ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

Last update: Nov 21, 2022

Overview

ZSL-KG

ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

Reference paper: Zero-shot Learning with Common Sense Knowledge graphs.

Performance

Performance ZSL-KG compared to other existing graph-based zero-shot learning frameworks.

Method	Ontonotes (Strict)	BBN (Strict)	SNIPS-NLU (Acc.)	AWA2 (H)	aPY (H)	ImageNet (All T-1)	Avg.
GCNZ	41.5	21.5	82.5	73.3	58.1	1.0	46.3
SGCN	42.6	24.9	50.3	73.7	56.8	1.5	41.6
DGP	41.1	24.0	64.4	75.1	55.7	1.4	43.6
ZSL-KG	45.2	26.7	89.0	74.6	61.6	1.7	49.8

ZSL-KG outperforms existing graph-based frameworks on five out of six benchmark datasets.

For more details on the experiments, refer to nayak-arxiv20-code.

Installation

The package requires python >= 3.7. To install the package, type the following command:

pip install .

Example Usage

In our framework, we use AutoGNN to easily create graph neural networks for zero-shot learning.

from zsl_kg.class_encoder.auto_gnn import AutoGNN
from zsl_kg.common.graph import NeighSampler

trgcn = {
    "input_dim": 300,
    "output_dim": 2049,
    "type": "trgcn",
    "gnn": [
        {
            "input_dim": 300,
            "output_dim": 2048,
            "activation": nn.LeakyReLU(0.2),
            "normalize": True,
            "sampler": NeighSampler(100, mode="topk"),
            "fh": 100,
        },
        {
            "input_dim": 2048,
            "output_dim": 2049,
            "activation": None,
            "normalize": True,
            "sampler": NeighSampler(50, mode="topk"),
        },
    ],
}

class_encoder = AutoGNN(trgcn)

Our framework supports the following graph neural networks: gcn, gat, rgcn, lstm, trgcn. You can change the type to any of the available to graph neural networks to instantly create a new graph neural network.

For more examples, refer to nayak-arxiv20-code.

Run Tests

To run the tests, please type the following command:

pytest

Citation

Please cite the following paper if you are using our framework.

@article{nayak:arxiv20,
  Author = {Nayak, Nihal V. and Bach, Stephen H.},
  Title = {Zero-Shot Learning with Common Sense Knowledge Graphs},
  Volume = {arXiv:2006.10713 [cs.LG]},
  Year = {2020}}

Comments

Graph usage query in object classification AWA2

Hi, I'm not entirely sure if this is where I should be asking this but I was wondering if you could help me clarify a question.

1 When performing zero shot object classification in AWA2, what is the graph that is used? (I assume a knowledge graph such as ConceptNet?)

2 If this is a knowledge graph, than what is the "input" to the trgcn? As in where are these "concepts" coming from, is it from the class attributes?

opened by shadowbourne 2
Quick note

Dear Team behind @zsl-kg,

This framework is really cool, however, I was quite disappointed to see you are re-writing everything from scratch when you could have used an awesome framework like PyTorch Geometric.

Best regards, Thomas Chaton

opened by tchaton 1

Releases(v0.0.1)

v0.0.1(Aug 27, 2021)

This is the initial release. Contains all the necessary code to run experiments in nayak-arxiv20-code.
Source code(tar.gz)
Source code(zip)

Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction".

GNN_PPI Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction". Lear

2 Dec 14, 2022

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Episodic Transformers (E.T.) Episodic Transformer for Vision-and-Language Navigation Alexander Pashevich, Cordelia Schmid, Chen Sun Episodic Transform

62 Dec 24, 2022

Neighborhood Contrastive Learning for Novel Class Discovery

Neighborhood Contrastive Learning for Novel Class Discovery This repository contains the official implementation of our paper: Neighborhood Contrastiv

56 Dec 9, 2022

Awesome Deep Graph Clustering is a collection of SOTA, novel deep graph clustering methods

ADGC: Awesome Deep Graph Clustering ADGC is a collection of state-of-the-art (SOTA), novel deep graph clustering methods (papers, codes and datasets).

297 Dec 27, 2022

The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Neural Deformation Graphs Project Page | Paper | Video Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction Aljaž Božič, Pablo P

134 Dec 16, 2022

Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph".

multilingual-mrc-isdg Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph". This r

5 Dec 7, 2022

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

PyTorch Implementation of Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers 1 Using Colab Please notic

489 Jan 7, 2023

sense-py-AnishaBaishya created by GitHub Classroom

Compute Statistics Here we compute statistics for a bunch of numbers. This project uses the unittest framework to test functionality. Pass the tests T

1 Oct 21, 2021

This repo is the code release of EMNLP 2021 conference paper "Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories".

Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories This repo is the code release of EMNLP 2021 con

12 Nov 22, 2022

ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

Related tags

Overview

ZSL-KG

Performance

Installation

Example Usage

Run Tests

Citation

You might also like...

Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction".

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Neighborhood Contrastive Learning for Novel Class Discovery

Awesome Deep Graph Clustering is a collection of SOTA, novel deep graph clustering methods

The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph".

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

sense-py-AnishaBaishya created by GitHub Classroom

This repo is the code release of EMNLP 2021 conference paper "Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories".

Comments

Graph usage query in object classification AWA2

Quick note

Releases(v0.0.1)

v0.0.1(Aug 27, 2021)

Owner

Bats Research

HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR. CVPR 2022

A complete speech segmentation system using Kaldi and x-vectors for voice activity detection (VAD) and speaker diarisation.

Everything you want about DP-Based Federated Learning, including Papers and Code. (Mechanism: Laplace or Gaussian, Dataset: femnist, shakespeare, mnist, cifar-10 and fashion-mnist. )

Grow Function: Generate 3D Stacked Bifurcating Double Deep Cellular Automata based organisms which differentiate using a Genetic Algorithm...

This is a model to classify Vietnamese sign language using Motion history image (MHI) algorithm and CNN.

PyTorch implementation of paper A Fast Knowledge Distillation Framework for Visual Recognition.

Selecting Parallel In-domain Sentences for Neural Machine Translation Using Monolingual Texts

This repository contains a pytorch implementation of "StereoPIFu: Depth Aware Clothed Human Digitization via Stereo Vision".

Identify the emotion of multiple speakers in an Audio Segment

An Implementation of Transformer in Transformer in TensorFlow for image classification, attention inside local patches

A simple baseline for 3d human pose estimation in tensorflow. Presented at ICCV 17.

网络协议2天集训

Equivariant layers for RC-complement symmetry in DNA sequence data

Code for the paper: On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations

Decorator for PyMC3

nanodet_plus,yolov5_v6.0

Acoustic mosquito detection code with Bayesian Neural Networks

A time series processing library

Graph Convolutional Networks for Temporal Action Localization (ICCV2019)

Code for the paper "On the Power of Edge Independent Graph Models"