HIVE: Evaluating the Human Interpretability of Visual Explanations

Related tags

Deep LearningHIVE
Overview

HIVE: Evaluating the Human Interpretability of Visual Explanations

Project Page | Paper

This repo provides the code for HIVE, a human evaluation framework for interpretability methods in computer vision.

@article{kim2021hive,
  author = {Sunnie S. Y. Kim and Nicole Meister and Vikram V. Ramaswamy and Ruth Fong and Olga Russakovsky},
  title = {{HIVE}: Evaluating the Human Interpretability of Visual Explanations},
  journal = {CoRR},
  volume = {abs/2112.03184},
  year = {2021}
}

Our study UIs

Distinction task

  • combined_gradcam_nolabels.html
  • combined_bagnet_nolabels.html
  • combined_protopnet_distinction.html
  • combined_prototree_distinction.html

Agreement task

  • combined_protopnet_agreement.html
  • combined_prototree_agreement.html

Additional studies

  • combined_gradcam_labels.html
  • combined_bagnet_labels.html
  • combined_prototree_agreement_tree.html

Running human studies

We ran our studies through Human Intelligence Tasks (HITs) deployed on Amazon Mechanical Turk (AMT). We use simple-amt, a microframework for working with AMT. Here we describe which files correspond to which study UIs and provide brief instructions for running studies.

Brief instructions on how to run user studies on AMT

Please check out the original simple-amt repository for more information on how to run a HIT on AMT.

Launch HITs on AMT

python launch_hits.py \
--html_template=hit_templates/combined_prototree_distinction.html \
--hit_properties_file=hit_properties/properties.json \
--input_json_file=examples/input_prototree_distinction.txt \
--hit_ids_file=examples/hit_ids_prototree_distinction.txt --prod

Check HIT progress

python show_hit_progress.py \
--hit_ids_file=examples/hit_ids_prototree_distinction.txt --prod

Get results

python get_results.py \
  --hit_ids_file=examples/hit_ids_prototree_distinction.txt \
  --output_file=examples/results_prototree_distinction.txt \
  > examples/results_prototree_distinction.txt --prod

Approve work

python approve_hits.py \
--hit_ids_file=examples/hit_ids_prototree_distinction.txt --prod
Owner
Princeton Visual AI Lab
Princeton Visual AI Lab
Code for CVPR 2021 paper TransNAS-Bench-101: Improving Transferrability and Generalizability of Cross-Task Neural Architecture Search.

TransNAS-Bench-101 This repository contains the publishable code for CVPR 2021 paper TransNAS-Bench-101: Improving Transferrability and Generalizabili

Yawen Duan 17 Nov 20, 2022
Just Randoms Cats with python

Random-Cat Just Randoms Cats with python.

OriCode 2 Dec 21, 2021
Plato: A New Framework for Federated Learning Research

a new software framework to facilitate scalable federated learning research.

System <a href=[email protected] Lab"> 192 Jan 05, 2023
Malware Analysis Neural Network project.

MalanaNeuralNetwork Description Malware Analysis Neural Network project. Table of Contents Getting Started Requirements Installation Clone Set-Up VENV

2 Nov 13, 2021
Model serving at scale

Run inference at scale Cortex is an open source platform for large-scale machine learning inference workloads. Workloads Realtime APIs - respond to pr

Cortex Labs 7.9k Jan 06, 2023
Simple transformer model for CIFAR10

CIFAR-Transformer Simple transformer model for CIFAR10. Reference: https://www.tensorflow.org/text/tutorials/transformer https://github.com/huggingfac

9 Nov 07, 2022
A Neural Net Training Interface on TensorFlow, with focus on speed + flexibility

Tensorpack is a neural network training interface based on TensorFlow. Features: It's Yet Another TF high-level API, with speed, and flexibility built

Tensorpack 6.2k Jan 09, 2023
Repo for paper "Dynamic Placement of Rapidly Deployable Mobile Sensor Robots Using Machine Learning and Expected Value of Information"

Repo for paper "Dynamic Placement of Rapidly Deployable Mobile Sensor Robots Using Machine Learning and Expected Value of Information" Notes I probabl

Berkeley Expert System Technologies Lab 0 Jul 01, 2021
Code for "Learning Canonical Representations for Scene Graph to Image Generation", Herzig & Bar et al., ECCV2020

Learning Canonical Representations for Scene Graph to Image Generation (ECCV 2020) Roei Herzig*, Amir Bar*, Huijuan Xu, Gal Chechik, Trevor Darrell, A

roei_herzig 24 Jul 07, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

selfcontact This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] It includes the main function

Lea Müller 68 Dec 06, 2022
Attentional Focus Modulates Automatic Finger‑tapping Movements

"Attentional Focus Modulates Automatic Finger‑tapping Movements", in Scientific Reports

Xingxun Jiang 1 Dec 02, 2021
Active learning for Mask R-CNN in Detectron2

MaskAL - Active learning for Mask R-CNN in Detectron2 Summary MaskAL is an active learning framework that automatically selects the most-informative i

49 Dec 20, 2022
A nutritional label for food for thought.

Lexiscore As a first effort in tackling the theme of information overload in content consumption, I've been working on the lexiscore: a nutritional la

Paul Bricman 34 Nov 08, 2022
HyperPose is a library for building high-performance custom pose estimation applications.

HyperPose is a library for building high-performance custom pose estimation applications.

TensorLayer Community 1.2k Jan 04, 2023
(AAAI2022) Style Mixing and Patchwise Prototypical Matching for One-Shot Unsupervised Domain Adaptive Semantic Segmentation

SM-PPM This is a Pytorch implementation of our paper "Style Mixing and Patchwise Prototypical Matching for One-Shot Unsupervised Domain Adaptive Seman

W-zx-Y 10 Dec 07, 2022
Official implementation of VQ-Diffusion

Vector Quantized Diffusion Model for Text-to-Image Synthesis Overview This is the official repo for the paper: [Vector Quantized Diffusion Model for T

Microsoft 592 Jan 03, 2023
TAug :: Time Series Data Augmentation using Deep Generative Models

TAug :: Time Series Data Augmentation using Deep Generative Models Note!!! The package is under development so be careful for using in production! Fea

35 Dec 06, 2022
Template repository for managing machine learning research projects built with PyTorch-Lightning

Tutorial Repository with a minimal example for showing how to deploy training across various compute infrastructure.

Sidd Karamcheti 3 Feb 11, 2022
Fuzzy Overclustering (FOC)

Fuzzy Overclustering (FOC) In real-world datasets, we need consistent annotations between annotators to give a certain ground-truth label. However, in

2 Nov 08, 2022
Multi-agent reinforcement learning algorithm and environment

Multi-agent reinforcement learning algorithm and environment [en/cn] Pytorch implements multi-agent reinforcement learning algorithms including IQL, Q

万鲲鹏 7 Sep 20, 2022