Code for "The Box Size Confidence Bias Harms Your Object Detector"

Overview

The Box Size Confidence Bias Harms Your Object Detector - Code

Disclaimer: This repository is for research purposes only. It is designed to maintain reproducibility of the experiments described in "The Box Size Confidence Bias Harms Your Object Detector".

Setup

Download Annotations

Download COCO2017 annotations for train, val, and tes-dev from here and move them into the folder structure like this (alternatively change the config in config/all/paths/annotations/coco_2017.yaml to your local folder structure):

 .
 └── data
   └── coco
      └── annotations
        ├── instances_train2017.json
        ├── instances_val2017.json
        └── image_info_test-dev2017.json

Generate Detections

Generate detections on the train, val, and test-dev COCO2017 set, save them in the COCO file format as JSON files. Move detections to data/detections/MODEL_NAME, see config/all/detections/default_all.yaml for all the used detectors and to add other detectors.
The official implementations for the used detectors are:

Examples

CenterNet (Hourglass)

To generate the Detections for CenterNet with Hourglass backbone first follow the installation instructions. Then download ctdet_coco_hg.pth to /models from the official source Then generate the detections from the /src folder:

test_train.py python3 test_train.py ctdet --arch hourglass --exp_id Centernet_HG_train --dataset coco --load_model ../models/ctdet_coco_hg.pth ">
# On val
python3 test.py ctdet --arch hourglass --exp_id Centernet_HG_val --dataset coco --load_model ../models/ctdet_coco_hg.pth 
# On test-dev
python3 test.py ctdet --arch hourglass --exp_id Centernet_HG_test-dev --dataset coco --load_model ../models/ctdet_coco_hg.pth --trainval
# On train
sed '56s/.*/  split = "train"/' test.py > test_train.py
python3 test_train.py ctdet --arch hourglass --exp_id Centernet_HG_train --dataset coco --load_model ../models/ctdet_coco_hg.pth

The scaling for TTA is set via the "--test_scales LIST_SCALES" flag. So to generate only the 0.5x-scales: --test_scales 0.5

RetinaNet with MMDetection

To generate the de detection files using mmdet, first follow the installation instructions. Then download specific model weights, in this example retinanet_x101_64x4d_fpn_2x_coco_20200131-bca068ab.pth to PATH_TO_DOWNLOADED_WEIGHTS and execute the following commands:

python3 tools/test.py configs/retinanet/retinanet_x101_64x4d_fpn_2x_coco.py PATH_TO_DOWNLOADED_WEIGHTS/retinanet_x101_64x4d_fpn_2x_coco_20200131-bca068ab.pth  --eval bbox --eval-options jsonfile_prefix='PATH_TO_THIS_REPO/detections/retinanet_x101_64x4d_fpn_2x/train2017' --cfg-options data.test.img_prefix='PATH_TO_COCO_IMGS/train2017' data.test.ann_file='PATH_TO_COCO_ANNS/annotations/instances_train2017.json'
python3 tools/test.py configs/retinanet/retinanet_x101_64x4d_fpn_2x_coco.py PATH_TO_DOWNLOADED_WEIGHTS/retinanet_x101_64x4d_fpn_2x_coco_20200131-bca068ab.pth  --eval bbox --eval-options jsonfile_prefix='PATH_TO_THIS_REPO/detections/retinanet_x101_64x4d_fpn_2x/val2017' --cfg-options data.test.img_prefix='PATH_TO_COCO_IMGS/val2017' data.test.ann_file='PATH_TO_COCO_ANNS/annotations/instances_val2017.json'
python3 tools/test.py configs/retinanet/retinanet_x101_64x4d_fpn_2x_coco.py PATH_TO_DOWNLOADED_WEIGHTS/retinanet_x101_64x4d_fpn_2x_coco_20200131-bca068ab.pth  --eval bbox --eval-options jsonfile_prefix='PATH_TO_THIS_REPO/detections/retinanet_x101_64x4d_fpn_2x/test-dev2017' --cfg-options data.test.img_prefix='PATH_TO_COCO_IMGS/test2017' data.test.ann_file='PATH_TO_COCO_ANNS/annotations/image_info_test-dev2017.json'

Install Dependencies

pip3 install -r requirements.txt
Optional Dependencies
# Faster coco evaluation (used if available)
pip3 install fast_coco_eval
# Parallel multi-runs, if enough RAM is available (add "hydra/launcher=joblib" to every command with -m flag)
pip install hydra-joblib-launcher

Experiments

Most of the experiments are performed using the CenterNet(HG) detections to change the detector add detections=OTHER_DETECTOR, with the location of OTHER_DETECTORs detections specified in config/all/detections/default_all.yaml. The results of each experiment are saved to outputs/EXPERIMENT/DATE and multirun/EXPERIMENT/DATE in the case of a multirun (-m flag).

Figure 2: Calibration curve of histogram binning and modified version

# original histogram binning calibration curve
python3 create_plots.py -cn plot_org_hist_bin
# modified histogram binning calibration curve:
python3 create_plots.py -cn plot_mod_hist_bin

Table 1: Ablation of histogram binning modifications

python3 calibrate.py -cn ablate_modified_hist 

Table 2: Ablation of optimization metrics of calibration on validation split

python3 calibrate.py -cn ablate_metrics  "seed=range(4,14)" -m

Figure 3: Bounding box size bias on train and val data detections

Plot of calibration curve:

# on validation data
python3 create_plots.py -cn plot_miscal name="plot_miscal_val" split="val"
# on train data:
python3 create_plots.py -cn plot_miscal name="plot_miscal_train" split="train" calib.conf_bins=20

Table 3: Ablation of optimization metrics of calibration on training data

python3 calibrate.py -cn explore_train

Table 4: Effect of individual calibration on TTA

  1. Generate detections (on train and val split) for each scale-factor individually (CenterNet_HG_TTA_050, CenterNet_HG_TTA_075, CenterNet_HG_TTA_100, CenterNet_HG_TTA_125, CenterNet_HG_TTA_150) and for complete TTA (CenterNet_HG_TTA_ens)

  2. Generate individually calibrated detections..

    python3 calibrate.py -cn calibrate_train name="calibrate_train_tta" detector="CenterNet_HG_TTA_050","CenterNet_HG_TTA_075","CenterNet_HG_TTA_100","CenterNet_HG_TTA_125","CenterNet_HG_TTA_150","CenterNet_HG_TTA_ens" -m
  3. Copy calibrated detections from multirun/calibrate_train_tta/DATE/MODEL_NAME/quantile_spline_ontrain_opt_tradeoff_full/val/MODEL_NAME.json to data/calibrated/MODEL_NAME/val/results.json for MODEL_NAME in (CenterNet_HG_TTA_050, CenterNet_HG_TTA_075, CenterNet_HG_TTA_100, CenterNet_HG_TTA_125, CenterNet_HG_TTA_150).

  4. Generate TTA of calibrated detections

    python3 enseble.py -cn enseble

Figure 4: Ablation of IoU threshold

python3 calibrate.py -cn calibrate_train name="ablate_iou" "iou_threshold=range(0.5,0.96,0.05)" -m

Table 5: Calibration method on different model

python3 calibrate.py -cn calibrate_train name="calibrate_all_models" detector=LIST_ALL_MODELS -m

The test-dev predictions are found in multirun/calibrate_all_models/DATE/MODEL_NAME/quantile_spline_ontrain_opt_tradeoff_full/test/MODEL_NAME.json and can be evaluated using the official evaluation sever.

Supplementary Material

A.Figure 5 & 6: Performance Change for Extended Optimization Metrics

python3 calibrate.py -cn ablate_metrics_extended  "seed=range(4,14)" -m

A.Table 6: Influence of parameter search spaces on performance gain

# Results for B0, C0
python3 calibrate.py -cn calibrate_train
# Results for B0, C1
python3 calibrate.py -cn calibrate_train_larger_cbins
# Results for B0 union B1, C0
python3 calibrate.py -cn calibrate_train_larger_bbins
# Results for B0 union B1, C0 union C1
python3 calibrate.py -cn calibrate_train_larger_cbbins

A.Table 7: Influence of calibration method on different sized versions of EfficientDet

python3 calibrate.py -cn calibrate_train name="influence_modelsize" detector="Efficientdet_D0","Efficientdet_D1","Efficientdet_D2","Efficientdet_D3","Efficientdet_D4","Efficientdet_D5","Efficientdet_D6","Efficientdet_D7" -m
You might also like...
Code to use Augmented Shapiro Wilks Stopping, as well as code for the paper "Statistically Signifigant Stopping of Neural Network Training"

This codebase is being actively maintained, please create and issue if you have issues using it Basics All data files are included under losses and ea

Opinionated code formatter, just like Python's black code formatter but for Beancount

beancount-black Opinionated code formatter, just like Python's black code formatter but for Beancount Try it out online here Features MIT licensed - b

a delightful machine learning tool that allows you to train, test and use models without writing code
a delightful machine learning tool that allows you to train, test and use models without writing code

igel A delightful machine learning tool that allows you to train/fit, test and use models without writing code Note I'm also working on a GUI desktop

Pytorch Lightning code guideline for conferences

Deep learning project seed Use this seed to start new deep learning / ML projects. Built in setup.py Built in requirements Examples with MNIST Badges

Automatically Build Multiple ML Models with a Single Line of Code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.
Automatically Build Multiple ML Models with a Single Line of Code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.

Auto-ViML Automatically Build Variant Interpretable ML models fast! Auto_ViML is pronounced "auto vimal" (autovimal logo created by Sanket Ghanmare) N

Code samples for my book "Neural Networks and Deep Learning"

Code samples for "Neural Networks and Deep Learning" This repository contains code samples for my book on "Neural Networks and Deep Learning". The cod

Code for: https://berkeleyautomation.github.io/bags/

DeformableRavens Code for the paper Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks. Here is the

Code for our method RePRI for Few-Shot Segmentation. Paper at http://arxiv.org/abs/2012.06166
Code for our method RePRI for Few-Shot Segmentation. Paper at http://arxiv.org/abs/2012.06166

Region Proportion Regularized Inference (RePRI) for Few-Shot Segmentation In this repo, we provide the code for our paper : "Few-Shot Segmentation Wit

Applications using the GTN library and code to reproduce experiments in "Differentiable Weighted Finite-State Transducers"

gtn_applications An applications library using GTN. Current examples include: Offline handwriting recognition Automatic speech recognition Installing

Owner
Johannes G.
Johannes G.
text_recognition_toolbox: The reimplementation of a series of classical scene text recognition papers with Pytorch in a uniform way.

text recognition toolbox 1. 项目介绍 该项目是基于pytorch深度学习框架,以统一的改写方式实现了以下6篇经典的文字识别论文,论文的详情如下。该项目会持续进行更新,欢迎大家提出问题以及对代码进行贡献。 模型 论文标题 发表年份 模型方法划分 CRNN 《An End-t

168 Dec 24, 2022
Animal Sound Classification (Cats Vrs Dogs Audio Sentiment Classification)

this is a simple artificial neural network model using deep learning and torch-audio to classify cats and dog sounds.

crispengari 3 Dec 05, 2022
Aesara is a Python library that allows one to define, optimize, and efficiently evaluate mathematical expressions involving multi-dimensional arrays.

Aesara is a Python library that allows one to define, optimize, and efficiently evaluate mathematical expressions involving multi-dimensional arrays.

Aesara 898 Jan 07, 2023
A PyTorch implementation of "Semi-Supervised Graph Classification: A Hierarchical Graph Perspective" (WWW 2019)

SEAL ⠀⠀⠀ A PyTorch implementation of Semi-Supervised Graph Classification: A Hierarchical Graph Perspective (WWW 2019) Abstract Node classification an

Benedek Rozemberczki 202 Dec 27, 2022
Official repository for the paper "Self-Supervised Models are Continual Learners" (CVPR 2022)

Self-Supervised Models are Continual Learners This is the official repository for the paper: Self-Supervised Models are Continual Learners Enrico Fini

Enrico Fini 73 Dec 18, 2022
The Empirical Investigation of Representation Learning for Imitation (EIRLI)

The Empirical Investigation of Representation Learning for Imitation (EIRLI)

Center for Human-Compatible AI 31 Nov 06, 2022
Accurate identification of bacteriophages from metagenomic data using Transformer

PhaMer is a python library for identifying bacteriophages from metagenomic data. PhaMer is based on a Transorfer model and rely on protein-based vocab

Kenneth Shang 9 Nov 30, 2022
Pytorch implementation of VAEs for heterogeneous likelihoods.

Heterogeneous VAEs Beware: This repository is under construction 🛠️ Pytorch implementation of different VAE models to model heterogeneous data. Here,

Adrián Javaloy 35 Nov 29, 2022
HHP-Net: A light Heteroscedastic neural network for Head Pose estimation with uncertainty

HHP-Net: A light Heteroscedastic neural network for Head Pose estimation with uncertainty Giorgio Cantarini, Francesca Odone, Nicoletta Noceti, Federi

18 Aug 02, 2022
[CVPR 2021] Counterfactual VQA: A Cause-Effect Look at Language Bias

Counterfactual VQA (CF-VQA) This repository is the Pytorch implementation of our paper "Counterfactual VQA: A Cause-Effect Look at Language Bias" in C

Yulei Niu 94 Dec 03, 2022
Gesture Volume Control v.2

Gesture volume control v.2 In this project I am going to learn how to use Gesture Control to change the volume of a computer. I first look into hand t

Pavel Dat 23 Dec 26, 2022
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

[ICCV2021] TransReID: Transformer-based Object Re-Identification [pdf] The official repository for TransReID: Transformer-based Object Re-Identificati

DamoCV 569 Dec 30, 2022
A TensorFlow implementation of the Mnemonic Descent Method.

MDM A Tensorflow implementation of the Mnemonic Descent Method. Mnemonic Descent Method: A recurrent process applied for end-to-end face alignment G.

123 Oct 07, 2022
KIND: an Italian Multi-Domain Dataset for Named Entity Recognition

KIND (Kessler Italian Named-entities Dataset) KIND is an Italian dataset for Named-Entity Recognition. It contains more than one million tokens with t

Digital Humanities 5 Jun 21, 2022
PyTorch implementation of Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Simple PyTorch Implementation of "Grokking" Implementation of Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets Usage Running

Teddy Koker 15 Sep 29, 2022
Auto-Lama combines object detection and image inpainting to automate object removals

Auto-Lama Auto-Lama combines object detection and image inpainting to automate object removals. It is build on top of DE:TR from Facebook Research and

44 Dec 09, 2022
When are Iterative GPs Numerically Accurate?

When are Iterative GPs Numerically Accurate? This is a code repository for the paper "When are Iterative GPs Numerically Accurate?" by Wesley Maddox,

Wesley Maddox 1 Jan 06, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

SMPLify-XMC This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright Lic

Lea Müller 83 Dec 14, 2022
[NeurIPS 2021] Well-tuned Simple Nets Excel on Tabular Datasets

[NeurIPS 2021] Well-tuned Simple Nets Excel on Tabular Datasets Introduction This repo contains the source code accompanying the paper: Well-tuned Sim

52 Jan 04, 2023