ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

Last update: Nov 21, 2022

Overview

ZSL-KG

ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

Reference paper: Zero-shot Learning with Common Sense Knowledge graphs.

Performance

Performance ZSL-KG compared to other existing graph-based zero-shot learning frameworks.

Method	Ontonotes (Strict)	BBN (Strict)	SNIPS-NLU (Acc.)	AWA2 (H)	aPY (H)	ImageNet (All T-1)	Avg.
GCNZ	41.5	21.5	82.5	73.3	58.1	1.0	46.3
SGCN	42.6	24.9	50.3	73.7	56.8	1.5	41.6
DGP	41.1	24.0	64.4	75.1	55.7	1.4	43.6
ZSL-KG	45.2	26.7	89.0	74.6	61.6	1.7	49.8

ZSL-KG outperforms existing graph-based frameworks on five out of six benchmark datasets.

For more details on the experiments, refer to nayak-arxiv20-code.

Installation

The package requires python >= 3.7. To install the package, type the following command:

pip install .

Example Usage

In our framework, we use AutoGNN to easily create graph neural networks for zero-shot learning.

from zsl_kg.class_encoder.auto_gnn import AutoGNN
from zsl_kg.common.graph import NeighSampler

trgcn = {
    "input_dim": 300,
    "output_dim": 2049,
    "type": "trgcn",
    "gnn": [
        {
            "input_dim": 300,
            "output_dim": 2048,
            "activation": nn.LeakyReLU(0.2),
            "normalize": True,
            "sampler": NeighSampler(100, mode="topk"),
            "fh": 100,
        },
        {
            "input_dim": 2048,
            "output_dim": 2049,
            "activation": None,
            "normalize": True,
            "sampler": NeighSampler(50, mode="topk"),
        },
    ],
}

class_encoder = AutoGNN(trgcn)

Our framework supports the following graph neural networks: gcn, gat, rgcn, lstm, trgcn. You can change the type to any of the available to graph neural networks to instantly create a new graph neural network.

For more examples, refer to nayak-arxiv20-code.

Run Tests

To run the tests, please type the following command:

pytest

Citation

Please cite the following paper if you are using our framework.

@article{nayak:arxiv20,
  Author = {Nayak, Nihal V. and Bach, Stephen H.},
  Title = {Zero-Shot Learning with Common Sense Knowledge Graphs},
  Volume = {arXiv:2006.10713 [cs.LG]},
  Year = {2020}}

Comments

Graph usage query in object classification AWA2

Hi, I'm not entirely sure if this is where I should be asking this but I was wondering if you could help me clarify a question.

1 When performing zero shot object classification in AWA2, what is the graph that is used? (I assume a knowledge graph such as ConceptNet?)

2 If this is a knowledge graph, than what is the "input" to the trgcn? As in where are these "concepts" coming from, is it from the class attributes?

opened by shadowbourne 2
Quick note

Dear Team behind @zsl-kg,

This framework is really cool, however, I was quite disappointed to see you are re-writing everything from scratch when you could have used an awesome framework like PyTorch Geometric.

Best regards, Thomas Chaton

opened by tchaton 1

Releases(v0.0.1)

v0.0.1(Aug 27, 2021)

This is the initial release. Contains all the necessary code to run experiments in nayak-arxiv20-code.
Source code(tar.gz)
Source code(zip)

Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction".

GNN_PPI Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction". Lear

2 Dec 14, 2022

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Episodic Transformers (E.T.) Episodic Transformer for Vision-and-Language Navigation Alexander Pashevich, Cordelia Schmid, Chen Sun Episodic Transform

62 Dec 24, 2022

Neighborhood Contrastive Learning for Novel Class Discovery

Neighborhood Contrastive Learning for Novel Class Discovery This repository contains the official implementation of our paper: Neighborhood Contrastiv

56 Dec 9, 2022

Awesome Deep Graph Clustering is a collection of SOTA, novel deep graph clustering methods

ADGC: Awesome Deep Graph Clustering ADGC is a collection of state-of-the-art (SOTA), novel deep graph clustering methods (papers, codes and datasets).

297 Dec 27, 2022

The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Neural Deformation Graphs Project Page | Paper | Video Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction Aljaž Božič, Pablo P

134 Dec 16, 2022

Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph".

multilingual-mrc-isdg Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph". This r

5 Dec 7, 2022

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

PyTorch Implementation of Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers 1 Using Colab Please notic

489 Jan 7, 2023

sense-py-AnishaBaishya created by GitHub Classroom

Compute Statistics Here we compute statistics for a bunch of numbers. This project uses the unittest framework to test functionality. Pass the tests T

1 Oct 21, 2021

This repo is the code release of EMNLP 2021 conference paper "Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories".

Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories This repo is the code release of EMNLP 2021 con

12 Nov 22, 2022

ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

Related tags

Overview

ZSL-KG

Performance

Installation

Example Usage

Run Tests

Citation

You might also like...

Codes and models for the paper "Learning Unknown from Correlations: Graph Neural Network for Inter-novel-protein Interaction Prediction".

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Neighborhood Contrastive Learning for Novel Class Discovery

Awesome Deep Graph Clustering is a collection of SOTA, novel deep graph clustering methods

The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph".

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

sense-py-AnishaBaishya created by GitHub Classroom

This repo is the code release of EMNLP 2021 conference paper "Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories".

Comments

Graph usage query in object classification AWA2

Quick note

Releases(v0.0.1)

v0.0.1(Aug 27, 2021)

Owner

Bats Research

In this project, we develop a face recognize platform based on MTCNN object-detection netcwork and FaceNet self-supervised network.

Capsule endoscopy detection DACON challenge

Public Implementation of ChIRo from "Learning 3D Representations of Molecular Chirality with Invariance to Bond Rotations"

CR-Fill: Generative Image Inpainting with Auxiliary Contextual Reconstruction. ICCV 2021

Language models are open knowledge graphs ( non official implementation )

Public repository created to store my custom-made tools for Just Dance (UbiArt Engine)

An official reimplementation of the method described in the INTERSPEECH 2021 paper - Speech Resynthesis from Discrete Disentangled Self-Supervised Representations.

Composing methods for ML training efficiency

Deep Learning ❤️ OneFlow

This repo is to be freely used by ML devs to check the GAN performances without coding from scratch.

Collection of in-progress libraries for entity neural networks.

PyTorch implementation of "Dataset Knowledge Transfer for Class-Incremental Learning Without Memory" (WACV2022)

The codes and related files to reproduce the results for Image Similarity Challenge Track 2.

This is the repository of the NeurIPS 2021 paper "Curriculum Disentangled Recommendation withNoisy Multi-feedback"

Model-free Vehicle Tracking and State Estimation in Point Cloud Sequences

Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.

Benchmark VAE - Library for Variational Autoencoder benchmarking

Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies

Official implementation of Deep Convolutional Dictionary Learning for Image Denoising.

Alignment Attention Fusion framework for Few-Shot Object Detection