This repository contains the code used to quantitatively evaluate counterfactual examples in the associated paper.

Overview

On Quantitative Evaluations of Counterfactuals

Install

To install required packages with conda, run the following command:

> conda env create -f requirements.yml

Code

The code contains all the evaluation metrics used in the paper as well as the models and the data.

To evaluate methods, you need to choose a config from the configs directory and to choose which metric to apply. The code will then evaluate the chosen metrics on counterfactuals from all three methods (GB, GL, GEN) and store the results in an appropriate subdirectory in outputs. If you, e.g., want to run all metrics on the MNIST dataset, use the following command:

(cfeval) > python main.py --eval -c configs/mnist/mnist.ini -a

Afterwards you can enumerate the directory by

(cfeval) > python main.py --list

to get an output like the following:

> Listing dirs
000: ./output/celeba_makeup_[0]
001: ./output/fake_mnist_[0]
002: ./output/mnist_0_1_[0]
003: ./output/mnist_[0]

Now, results can be printed for the MNIST dataset (idx 3 above) by

(cfeval) > python main.py --print -c 3 

To get a result like

# # # # # # # # # # # # # # # # # # # # 
# MNIST
# # # # # # # # # # # # # # # # # # # # 
Method \ Metric    TargetClassValidity    ElasticNet    IM1          IM2             FID  Oracle
-----------------  ---------------------  ------------  -----------  -----------  ------  ------------
GB                 99.59 (0.13)           16.07 (0.18)  0.99 (0.00)  0.55 (0.01)   50.23  73.38 (0.87)
GL                 100.00 (0.00)          42.76 (0.31)  0.99 (0.00)  0.53 (0.00)  308.43  37.71 (0.95)
GEN                99.97 (0.03)           99.17 (0.58)  0.88 (0.00)  0.17 (0.00)   90.73  93.13 (0.50)

Directory overview:

File Description
ckpts Contains all the (Keras) models used by the various metrics.
data Contains the data used, both counterfactual examples from GB, GL, and GEN, and original input data.
configs Contains config files specifying experimental details like dataset, normalization, etc.
data Contains the data in numpy arrays.
dataset Code for loading data.
evaluate Implementations of all the metrics.
output Directory to hold computed results. Directory already contains results from paper.
config.py Reads config files from configs
constants.py Method and metric names.
listing.py Utility for indexing output dirs (see description below)
main.py Main file to run all code through.
print_results.py Utillity function for printing results from json files in the output directory.
Owner
Frederik Hvilshøj
PhD Student. Finishing PhD in Machine Learning Fall 2021.
Frederik Hvilshøj
Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

Self-Supervised Reward Regression (SSRR) Codebase for CoRL 2021 paper "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression "

19 Dec 12, 2022
MediaPipeのPythonパッケージのサンプルです。2020/12/11時点でPython実装のある4機能(Hands、Pose、Face Mesh、Holistic)について用意しています。

mediapipe-python-sample MediaPipeのPythonパッケージのサンプルです。 2020/12/11時点でPython実装のある以下4機能について用意しています。 Hands Pose Face Mesh Holistic Requirement mediapipe 0.

KazuhitoTakahashi 217 Dec 12, 2022
A simple, fast, and efficient object detector without FPN

You Only Look One-level Feature (YOLOF), CVPR2021 A simple, fast, and efficient object detector without FPN. This repo provides an implementation for

789 Jan 09, 2023
UAV-Networks-Routing is a Python simulator for experimenting routing algorithms and mac protocols on unmanned aerial vehicle networks.

UAV-Networks Simulator - Autonomous Networking - A.A. 20/21 UAV-Networks-Routing is a Python simulator for experimenting routing algorithms and mac pr

0 Nov 13, 2021
Code for the paper "Graph Attention Tracking". (CVPR2021)

SiamGAT 1. Environment setup This code has been tested on Ubuntu 16.04, Python 3.5, Pytorch 1.2.0, CUDA 9.0. Please install related libraries before r

122 Dec 24, 2022
给yolov5加个gui界面,使用pyqt5,yolov5是5.0版本

博文地址 https://xugaoxiang.com/2021/06/30/yolov5-pyqt5 代码执行 项目中使用YOLOv5的v5.0版本,界面文件是project.ui pip install -r requirements.txt python main.py 图片检测 视频检测

Xu GaoXiang 215 Dec 30, 2022
Implementation of Barlow Twins paper

barlowtwins PyTorch Implementation of Barlow Twins paper: Barlow Twins: Self-Supervised Learning via Redundancy Reduction This is currently a work in

IgorSusmelj 86 Dec 20, 2022
Byte-based multilingual transformer TTS for low-resource/few-shot language adaptation.

One model to speak them all 🌎 Audio Language Text ▷ Chinese 人人生而自由,在尊严和权利上一律平等。 ▷ English All human beings are born free and equal in dignity and rig

Mutian He 60 Nov 14, 2022
AWS documentation corpus for zero-shot open-book question answering.

aws-documentation We present the AWS documentation corpus, an open-book QA dataset, which contains 25,175 documents along with 100 matched questions a

Sia Gholami 2 Jul 07, 2022
PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.

PySlowFast PySlowFast is an open source video understanding codebase from FAIR that provides state-of-the-art video classification models with efficie

Meta Research 5.3k Jan 03, 2023
A Python library that provides a simplified alternative to DBAPI 2

A Python library that provides a simplified alternative to DBAPI 2. It provides a facade in front of DBAPI 2 drivers.

Tony Locke 44 Nov 17, 2021
Model Zoo of BDD100K Dataset

Model Zoo of BDD100K Dataset

ETH VIS Group 200 Dec 27, 2022
Bringing sanity to world of messed-up data

Sanitize sanitize is a Python module for making sure various things (e.g. HTML) are safe to use. It was originally written by Mark Pilgrim and is dist

Alireza Savand 63 Oct 26, 2021
Final project for Intro to CS class.

Financial Analysis Web App https://share.streamlit.io/mayurk1/fin-web-app-final-project/webApp.py 1. Project Description This project is a technical a

Mayur Khanna 1 Dec 10, 2021
Stratified Transformer for 3D Point Cloud Segmentation (CVPR 2022)

Stratified Transformer for 3D Point Cloud Segmentation Xin Lai*, Jianhui Liu*, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, Jiaya Jia

DV Lab 195 Jan 01, 2023
This is the official code for the paper "Tracker Meets Night: A Transformer Enhancer for UAV Tracking".

SCT This is the official code for the paper "Tracker Meets Night: A Transformer Enhancer for UAV Tracking" The spatial-channel Transformer (SCT) enhan

Intelligent Vision for Robotics in Complex Environment 27 Nov 23, 2022
🌎 The Modern Declarative Data Flow Framework for the AI Empowered Generation.

🌎 JSONClasses JSONClasses is a declarative data flow pipeline and data graph framework. Official Website: https://www.jsonclasses.com Official Docume

Fillmula Inc. 53 Dec 09, 2022
PyTorch implementation of convolutional neural networks-based text-to-speech synthesis models

Deepvoice3_pytorch PyTorch implementation of convolutional networks-based text-to-speech synthesis models: arXiv:1710.07654: Deep Voice 3: Scaling Tex

Ryuichi Yamamoto 1.8k Jan 08, 2023
PyTorch implementation for "Mining Latent Structures with Contrastive Modality Fusion for Multimedia Recommendation"

MIRCO PyTorch implementation for paper: Latent Structures Mining with Contrastive Modality Fusion for Multimedia Recommendation Dependencies Python 3.

Big Data and Multi-modal Computing Group, CRIPAC 9 Dec 08, 2022
Reinforcement learning for self-driving in a 3D simulation

SelfDrive_AI Reinforcement learning for self-driving in a 3D simulation (Created using UNITY-3D) 1. Requirements for the SelfDrive_AI Gym You need Pyt

Surajit Saikia 17 Dec 14, 2021