On Evaluation Metrics for Graph Generative Models

Overview

On Evaluation Metrics for Graph Generative Models

Authors: Rylee Thompson, Boris Knyazev, Elahe Ghalebi, Jungtaek Kim, Graham Taylor

This is the official repository for the paper On Evaluation Metrics for Graph Generative Models (hyperlink TBD). Our evaluation metrics enable the efficient computation of the distance between two sets of graphs regardless of domain. In addition, they are more expressive than previous metrics and easily incorporate continuous node and edge features in evaluation. If you're primarily interested in using our metrics in your work, please see evaluation/ for a more lightweight setup and installation and Evaluation_examples.ipynb for examples on how to utilize our code. The remainder of this README describes how to recreate our results which introduces additional dependencies.

Table of Contents

Requirements and installation

The main requirements are:

  • Python 3.7
  • PyTorch 1.8.1
  • DGL 0.6.1
pip install -r requirements.txt

Following that, install an appropriate version of DGL 0.6.1 for your system and download the proteins and ego datasets by running ./download_datasets.sh.

Reproducing main results

The arguments of our scripts are described in config.py.

Permutation experiments

Below, examples to run the scripts to run certain experiments are shown. In general, experiments can be run as:

python main.py --permutation_type={permutation type} --dataset={dataset}\
{feature_extractor} {feature_extractor_args}

For example, to run the mixing random graphs experiment on the proteins dataset using random-GNN-based metrics for a single random seed:

python main.py --permutation_type=mixing-random --dataset=proteins\
gnn

The hyperparameters of the GNN are set to our recommendations by default, however, they are easily changed by additional flags. To run the same experiment using the degree MMD metric:

python main.py --permutation_type=mixing-random --dataset=proteins\
mmd-structure --statistic=degree

Rank correlations are automatically computed and printed at the end of each experiment, and results are stored in experiment_results/. Recreating our results requires running variations of the above commands thousands of times. To generate these commands and store them in a bash script automatically, run python create_bash_script.py.

Pretraining GNNs

To pretrain a GNN for use in our permutation experiments, run python GIN_train.py, and see GIN_train.py for tweakable hyperparameters. Alternatively, the pretrained models used in our experiments can be downloaded by running ./download_pretrained_models.sh. Once you have a pretrained model, the permutation experiments can be ran using:

python main.py --permutation_type={permutation type} --dataset={dataset}\
gnn --use_pretrained {feature_extractor_args}

Generating graphs

Some of our experiments use graphs generated by GRAN. To find instructions on training and generating graphs using GRAN, please see the official GRAN repository. Alternatively, the graphs generated by GRAN used in our experiments can be downloaded by running ./download_gran_graphs.sh.

Visualization

All code for visualizing results and creating tables is found in data_visualization.ipynb.

License

We release our code under the MIT license.

Citation

@inproceedings{thompson2022evaluation,
  title={On Evaluation Metrics for Graph Generative Models},
  author={Thompson, Rylee, and Knyazev, Boris and Ghalebi, Elahe and Kim, Jungtaek, and Taylor, Graham W},
booktitle={International Conference on Learning Representations},
  year={2022}  
}
Kaggle | 9th place single model solution for TGS Salt Identification Challenge

UNet for segmenting salt deposits from seismic images with PyTorch. General We, tugstugi and xuyuan, have participated in the Kaggle competition TGS S

Erdene-Ochir Tuguldur 276 Dec 20, 2022
Codebase for Amodal Segmentation through Out-of-Task andOut-of-Distribution Generalization with a Bayesian Model

Codebase for Amodal Segmentation through Out-of-Task andOut-of-Distribution Generalization with a Bayesian Model

Yihong Sun 12 Nov 15, 2022
The ARCA23K baseline system

ARCA23K Baseline System This is the source code for the baseline system associated with the ARCA23K dataset. Details about ARCA23K and the baseline sy

4 Jul 02, 2022
Guided Internet-delivered Cognitive Behavioral Therapy Adherence Forecasting

Guided Internet-delivered Cognitive Behavioral Therapy Adherence Forecasting #Dataset The folder "Dataset" contains the dataset use in this work and m

0 Jan 08, 2022
PyTorch Implementation of Vector Quantized Variational AutoEncoders.

Pytorch implementation of VQVAE. This paper combines 2 tricks: Vector Quantization (check out this amazing blog for better understanding.) Straight-Th

Vrushank Changawala 2 Oct 06, 2021
Code and Datasets from the paper "Self-supervised contrastive learning for volcanic unrest detection from InSAR data"

Code and Datasets from the paper "Self-supervised contrastive learning for volcanic unrest detection from InSAR data" You can download the pretrained

Bountos Nikos 3 May 07, 2022
Sematic-Segmantation - Semantic Segmentation on MIT ADE20K dataset in PyTorch

Semantic Segmentation on MIT ADE20K dataset in PyTorch This is a PyTorch impleme

Berat Eren Terzioğlu 4 Mar 22, 2022
A high-level Python library for Quantum Natural Language Processing

lambeq About lambeq is a toolkit for quantum natural language processing (QNLP). Documentation: https://cqcl.github.io/lambeq/ Getting started Prerequ

Cambridge Quantum 315 Jan 01, 2023
Using LSTM write Tang poetry

本教程将通过一个示例对LSTM进行介绍。通过搭建训练LSTM网络,我们将训练一个模型来生成唐诗。本文将对该实现进行详尽的解释,并阐明此模型的工作方式和原因。并不需要过多专业知识,但是可能需要新手花一些时间来理解的模型训练的实际情况。为了节省时间,请尽量选择GPU进行训练。

56 Dec 15, 2022
Unbalanced Feature Transport for Exemplar-based Image Translation (CVPR 2021)

UNITE and UNITE+ Unbalanced Feature Transport for Exemplar-based Image Translation (CVPR 2021) Unbalanced Intrinsic Feature Transport for Exemplar-bas

Fangneng Zhan 183 Nov 09, 2022
Implementation of CVAE. Trained CVAE on faces from UTKFace Dataset to produce synthetic faces with a given degree of happiness/smileyness.

Conditional Smiles! (SmileCVAE) About Implementation of AE, VAE and CVAE. Trained CVAE on faces from UTKFace Dataset. Using an encoding of the Smile-s

Raúl Ortega 3 Jan 09, 2022
Unsupervised Feature Loss (UFLoss) for High Fidelity Deep learning (DL)-based reconstruction

Unsupervised Feature Loss (UFLoss) for High Fidelity Deep learning (DL)-based reconstruction Official github repository for the paper High Fidelity De

28 Dec 16, 2022
AOT (Associating Objects with Transformers) in PyTorch

An efficient modular implementation of Associating Objects with Transformers for Video Object Segmentation in PyTorch

162 Dec 14, 2022
QHack—the quantum machine learning hackathon

Official repo for QHack—the quantum machine learning hackathon

Xanadu 72 Dec 21, 2022
Pytorch implementation of One-Shot Affordance Detection

One-shot Affordance Detection PyTorch implementation of our one-shot affordance detection models. This repository contains PyTorch evaluation code, tr

46 Dec 12, 2022
OneFlow is a performance-centered and open-source deep learning framework.

OneFlow OneFlow is a performance-centered and open-source deep learning framework. Latest News Version 0.5.0 is out! First class support for eager exe

OneFlow 4.2k Jan 07, 2023
Fuzzing tool (TFuzz): a fuzzing tool based on program transformation

T-Fuzz T-Fuzz consists of 2 components: Fuzzing tool (TFuzz): a fuzzing tool based on program transformation Crash Analyzer (CrashAnalyzer): a tool th

HexHive 244 Nov 09, 2022
Starter code for the ICCV 2021 paper, 'Detecting Invisible People'

Detecting Invisible People [ICCV 2021 Paper] [Website] Tarasha Khurana, Achal Dave, Deva Ramanan Introduction This repository contains code for Detect

Tarasha Khurana 28 Sep 16, 2022
[CVPR 2021] A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts

Visual-Reasoning-eXplanation [CVPR 2021 A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts] Project Page | Vid

Andy_Ge 54 Dec 21, 2022