This repository contains the code used to quantitatively evaluate counterfactual examples in the associated paper.

Overview

On Quantitative Evaluations of Counterfactuals

Install

To install required packages with conda, run the following command:

> conda env create -f requirements.yml

Code

The code contains all the evaluation metrics used in the paper as well as the models and the data.

To evaluate methods, you need to choose a config from the configs directory and to choose which metric to apply. The code will then evaluate the chosen metrics on counterfactuals from all three methods (GB, GL, GEN) and store the results in an appropriate subdirectory in outputs. If you, e.g., want to run all metrics on the MNIST dataset, use the following command:

(cfeval) > python main.py --eval -c configs/mnist/mnist.ini -a

Afterwards you can enumerate the directory by

(cfeval) > python main.py --list

to get an output like the following:

> Listing dirs
000: ./output/celeba_makeup_[0]
001: ./output/fake_mnist_[0]
002: ./output/mnist_0_1_[0]
003: ./output/mnist_[0]

Now, results can be printed for the MNIST dataset (idx 3 above) by

(cfeval) > python main.py --print -c 3 

To get a result like

# # # # # # # # # # # # # # # # # # # # 
# MNIST
# # # # # # # # # # # # # # # # # # # # 
Method \ Metric    TargetClassValidity    ElasticNet    IM1          IM2             FID  Oracle
-----------------  ---------------------  ------------  -----------  -----------  ------  ------------
GB                 99.59 (0.13)           16.07 (0.18)  0.99 (0.00)  0.55 (0.01)   50.23  73.38 (0.87)
GL                 100.00 (0.00)          42.76 (0.31)  0.99 (0.00)  0.53 (0.00)  308.43  37.71 (0.95)
GEN                99.97 (0.03)           99.17 (0.58)  0.88 (0.00)  0.17 (0.00)   90.73  93.13 (0.50)

Directory overview:

File Description
ckpts Contains all the (Keras) models used by the various metrics.
data Contains the data used, both counterfactual examples from GB, GL, and GEN, and original input data.
configs Contains config files specifying experimental details like dataset, normalization, etc.
data Contains the data in numpy arrays.
dataset Code for loading data.
evaluate Implementations of all the metrics.
output Directory to hold computed results. Directory already contains results from paper.
config.py Reads config files from configs
constants.py Method and metric names.
listing.py Utility for indexing output dirs (see description below)
main.py Main file to run all code through.
print_results.py Utillity function for printing results from json files in the output directory.
Owner
Frederik Hvilshøj
PhD Student. Finishing PhD in Machine Learning Fall 2021.
Frederik Hvilshøj
(CVPR 2022) A minimalistic mapless end-to-end stack for joint perception, prediction, planning and control for self driving.

LAV Learning from All Vehicles Dian Chen, Philipp Krähenbühl CVPR 2022 (also arXiV 2203.11934) This repo contains code for paper Learning from all veh

Dian Chen 300 Dec 15, 2022
QR2Pass-project - A proof of concept for an alternative (passwordless) authentication system to a web server

QR2Pass This is a proof of concept for an alternative (passwordless) authenticat

4 Dec 09, 2022
PantheonRL is a package for training and testing multi-agent reinforcement learning environments.

PantheonRL is a package for training and testing multi-agent reinforcement learning environments. PantheonRL supports cross-play, fine-tuning, ad-hoc coordination, and more.

Stanford Intelligent and Interactive Autonomous Systems Group 57 Dec 28, 2022
Source code for Task-Aware Variational Adversarial Active Learning

Contrastive Coding for Active Learning under Class Distribution Mismatch Official PyTorch implementation of ["Contrastive Coding for Active Learning u

27 Nov 23, 2022
Code for "Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search"

Contextual Non-Local Alignment over Full-Scale Representation for Text-Based Person Search This is an implementation for our paper Contextual Non-Loca

Tencent YouTu Research 50 Dec 03, 2022
A simple configurable bot for sending arXiv article alert by mail

arXiv-newsletter A simple configurable bot for sending arXiv article alert by mail. Prerequisites PyYAML=5.3.1 arxiv=1.4.0 Configuration All config

SXKDZ 21 Nov 09, 2022
The code for 'Deep Residual Fourier Transformation for Single Image Deblurring'

Deep Residual Fourier Transformation for Single Image Deblurring Xintian Mao, Yiming Liu, Wei Shen, Qingli Li and Yan Wang News 2021.12.5 Release Deep

145 Jan 05, 2023
Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

30 Days Of Machine Learning Using Pytorch Objective of the repository is to learn and build machine learning models using Pytorch. List of Algorithms

Mayur 119 Nov 24, 2022
A system for quickly generating training data with weak supervision

Programmatically Build and Manage Training Data Announcement The Snorkel team is now focusing their efforts on Snorkel Flow, an end-to-end AI applicat

Snorkel Team 5.4k Jan 02, 2023
MoCap-Solver: A Neural Solver for Optical Motion Capture Data

MoCap-Solver is a data-driven-based robust marker denoising method, which takes raw mocap markers as input and outputs corresponding clean markers and skeleton motions.

55 Dec 28, 2022
PPO is a very popular Reinforcement Learning algorithm at present.

PPO is a very popular Reinforcement Learning algorithm at present. OpenAI takes PPO as the current baseline algorithm. We use the PPO algorithm to train a policy to give the best action in any situat

Rosefintech 11 Aug 23, 2021
A Factor Model for Persistence in Investment Manager Performance

Factor-Model-Manager-Performance A Factor Model for Persistence in Investment Manager Performance I apply methods and processes similar to those used

Omid Arhami 1 Dec 01, 2021
A list of all named GANs!

The GAN Zoo Every week, new GAN papers are coming out and it's hard to keep track of them all, not to mention the incredibly creative ways in which re

Avinash Hindupur 12.9k Jan 08, 2023
Transfer Learning library for Deep Neural Networks.

Transfer and meta-learning in Python Each folder in this repository corresponds to a method or tool for transfer/meta-learning. xfer-ml is a standalon

Amazon 245 Dec 08, 2022
Planner_backend - Academic planner application designed for students and counselors.

Planner (backend) Academic planner application designed for students and advisors.

2 Dec 31, 2021
Deploy tensorflow graphs for fast evaluation and export to tensorflow-less environments running numpy.

Deploy tensorflow graphs for fast evaluation and export to tensorflow-less environments running numpy. Now with tensorflow 1.0 support. Evaluation usa

Marcel R. 349 Aug 06, 2022
People log into different sites every day to get information and browse through these sites one by one

HyperLink People log into different sites every day to get information and browse through these sites one by one. And they are exposed to advertisemen

0 Feb 17, 2022
Torch implementation of various types of GAN (e.g. DCGAN, ALI, Context-encoder, DiscoGAN, CycleGAN, EBGAN, LSGAN)

gans-collection.torch Torch implementation of various types of GANs (e.g. DCGAN, ALI, Context-encoder, DiscoGAN, CycleGAN, EBGAN). Note that EBGAN and

Minchul Shin 53 Jan 22, 2022
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption, CVPR 2021 (Oral)

TAP: Text-Aware Pre-training TAP: Text-Aware Pre-training for Text-VQA and Text-Caption by Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Flo

Microsoft 61 Nov 14, 2022
Microscopy Image Cytometry Toolkit

Cytokit Cytokit is a collection of tools for quantifying and analyzing properties of individual cells in large fluorescent microscopy datasets with a

Hammer Lab 106 Jan 06, 2023