Official Repository for the paper "Improving Baselines in the Wild".

Related tags

Deep Learningwilds
Overview

iWildCam and FMoW baselines (WILDS)

This repository was originally forked from the official repository of WILDS datasets (commit 7e103ed)

For general instructions, please refer to the original repositiory.

This repository contains code used to produce experimental results presented in:

Improving Baselines in the Wild

Apart from minor edits, the only main changes we introduce are:

  • --validate_every flag (default: 1000) to specify the frequency (number of training steps) of cross-validation/checkpoint tracking.
  • sub_val_metric option in the dataset (see examples/configs/datasets.py) to specify a secondary metric to be tracked during training. This activates additional cross-validation and checkpoint tracking for the specified metric.

Results

NB: To reproduce the numbers from the paper, the right PyTorch version must be used. All our experiments have been conducted using 1.9.0+cu102, except for + higher lr rows in Table 2/FMoW (which we ran for the camera-ready and for the public release) for which 1.10.0+cu102 was used.

The training scripts, logs, and model checkpoints for the best configurations from our experiments can be found here for iWildCam & FMoW.

iWildCam

CV based on "Valid F1"

Split / Metric mean (std) 3 runs
IID Valid Acc 82.5 (0.8) [0.817, 0.835, 0.822]
IID Valid F1 46.7 (1.0) [0.456, 0.481, 0.464]
IID Test Acc 76.2 (0.1) [0.762, 0.763, 0.761]
IID Test F1 47.9 (2.1) [0.505, 0.479, 0.453]
Valid Acc 64.1 (1.7) [0.644, 0.619, 0.661]
Valid F1 38.3 (0.9) [0.39, 0.371, 0.389]
Test Acc 69.0 (0.3) [0.69, 0.694, 0.687]
Test F1 32.1 (1.2) [0.338, 0.31, 0.314]

CV based on "Valid Acc"

Split / Metric mean (std) 3 runs
IID Valid Acc 82.6 (0.7) [0.836, 0.821, 0.822]
IID Valid F1 46.2 (0.9) [0.472, 0.45, 0.464]
IID Test Acc 75.8 (0.4) [0.76, 0.753, 0.761]
IID Test F1 44.9 (0.4) [0.444, 0.45, 0.453]
Valid Acc 66.6 (0.4) [0.666, 0.672, 0.661]
Valid F1 36.6 (2.1) [0.369, 0.339, 0.389]
Test Acc 68.6 (0.3) [0.688, 0.682, 0.687]
Test F1 28.7 (2.0) [0.279, 0.268, 0.314]

FMoW

CV based on "Valid Region"

Split / Metric mean (std) 3 runs
IID Valid Acc 63.9 (0.2) [0.64, 0.636, 0.641]
IID Valid Region 62.2 (0.5) [0.623, 0.616, 0.628]
IID Valid Year 49.8 (1.8) [0.52, 0.475, 0.5]
IID Test Acc 62.3 (0.2) [0.626, 0.621, 0.621]
IID Test Region 60.9 (0.6) [0.617, 0.603, 0.606]
IID Test Year 43.2 (1.1) [0.438, 0.417, 0.442]
Valid Acc 62.1 (0.0) [0.62, 0.621, 0.621]
Valid Region 52.5 (1.0) [0.538, 0.513, 0.524]
Valid Year 60.5 (0.2) [0.602, 0.605, 0.608]
Test Acc 55.6 (0.2) [0.555, 0.554, 0.558]
Test Region 34.8 (1.5) [0.369, 0.334, 0.34]
Test Year 50.2 (0.4) [0.499, 0.498, 0.508]

CV based on "Valid Acc"

Split / Metric mean (std) 3 runs
IID Valid Acc 64.0 (0.1) [0.641, 0.639, 0.641]
IID Valid Region 62.3 (0.4) [0.623, 0.617, 0.628]
IID Valid Year 50.8 (0.6) [0.514, 0.509, 0.5]
IID Test Acc 62.3 (0.4) [0.628, 0.62, 0.621]
IID Test Region 61.1 (0.6) [0.62, 0.608, 0.606]
IID Test Year 43.6 (1.4) [0.45, 0.417, 0.442]
Valid Acc 62.1 (0.0) [0.621, 0.621, 0.621]
Valid Region 51.4 (1.3) [0.522, 0.496, 0.524]
Valid Year 60.6 (0.3) [0.608, 0.601, 0.608]
Test Acc 55.6 (0.2) [0.556, 0.554, 0.558]
Test Region 34.2 (1.2) [0.357, 0.329, 0.34]
Test Year 50.2 (0.5) [0.496, 0.501, 0.508]

BibTex

@inproceedings{irie2021improving,
      title={Improving Baselines in the Wild}, 
      author={Kazuki Irie and Imanol Schlag and R\'obert Csord\'as and J\"urgen Schmidhuber},
      booktitle={Workshop on Distribution Shifts, NeurIPS},
      address={Virtual only},
      year={2021}
}
Owner
Kazuki Irie
Kazuki Irie
Official Repository for our ECCV2020 paper: Imbalanced Continual Learning with Partitioning Reservoir Sampling

Imbalanced Continual Learning with Partioning Reservoir Sampling This repository contains the official PyTorch implementation and the dataset for our

Chris Dongjoo Kim 40 Sep 18, 2022
Lingvo is a framework for building neural networks in Tensorflow, particularly sequence models.

Lingvo is a framework for building neural networks in Tensorflow, particularly sequence models.

2.7k Jan 05, 2023
This repository contains the code and models for the following paper.

DC-ShadowNet Introduction This is an implementation of the following paper DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised

AuAgCu 65 Dec 27, 2022
Vision-Language Transformer and Query Generation for Referring Segmentation (ICCV 2021)

Vision-Language Transformer and Query Generation for Referring Segmentation Please consider citing our paper in your publications if the project helps

Henghui Ding 143 Dec 23, 2022
Equivariant Imaging: Learning Beyond the Range Space

[Project] Equivariant Imaging: Learning Beyond the Range Space Project about the

Georges Le Bellier 3 Feb 06, 2022
CLOOB training (JAX) and inference (JAX and PyTorch)

cloob-training Pretrained models There are two pretrained CLOOB models in this repo at the moment, a 16 epoch and a 32 epoch ViT-B/16 checkpoint train

Katherine Crowson 64 Nov 27, 2022
YOLOv5🚀 reproduction by Guo Quanhao using PaddlePaddle

YOLOv5-Paddle YOLOv5 🚀 reproduction by Guo Quanhao using PaddlePaddle 支持AutoBatch 支持AutoAnchor 支持GPU Memory 快速开始 使用AIStudio高性能环境快速构建YOLOv5训练(PaddlePa

QuanHao Guo 20 Nov 14, 2022
Code to reproduce the results in "Visually Grounded Reasoning across Languages and Cultures", EMNLP 2021.

marvl-code [WIP] This is the implementation of the approaches described in the paper: Fangyu Liu*, Emanuele Bugliarello*, Edoardo M. Ponti, Siva Reddy

25 Nov 15, 2022
Implementations of the algorithms in the paper Approximative Algorithms for Multi-Marginal Optimal Transport and Free-Support Wasserstein Barycenters

Implementations of the algorithms in the paper Approximative Algorithms for Multi-Marginal Optimal Transport and Free-Support Wasserstein Barycenters

Johannes von Lindheim 3 Oct 29, 2022
The official PyTorch code implementation of "Personalized Trajectory Prediction via Distribution Discrimination" in ICCV 2021.

Personalized Trajectory Prediction via Distribution Discrimination (DisDis) The official PyTorch code implementation of "Personalized Trajectory Predi

25 Dec 20, 2022
Code implementation of Data Efficient Stagewise Knowledge Distillation paper.

Data Efficient Stagewise Knowledge Distillation Table of Contents Data Efficient Stagewise Knowledge Distillation Table of Contents Requirements Image

IvLabs 112 Dec 02, 2022
This is a repository for a No-Code object detection inference API using the OpenVINO. It's supported on both Windows and Linux Operating systems.

OpenVINO Inference API This is a repository for an object detection inference API using the OpenVINO. It's supported on both Windows and Linux Operati

BMW TechOffice MUNICH 68 Nov 24, 2022
torchbearer: A model fitting library for PyTorch

Note: We're moving to PyTorch Lightning! Read about the move here. From the end of February, torchbearer will no longer be actively maintained. We'll

631 Jan 04, 2023
Convert ONNX model graph to Keras model format.

Convert ONNX model graph to Keras model format.

Grigory Malivenko 175 Dec 28, 2022
Video Instance Segmentation with a Propose-Reduce Paradigm (ICCV 2021)

Propose-Reduce VIS This repo contains the official implementation for the paper: Video Instance Segmentation with a Propose-Reduce Paradigm Huaijia Li

DV Lab 39 Nov 23, 2022
Fast Differentiable Matrix Sqrt Root

Fast Differentiable Matrix Sqrt Root Geometric Interpretation of Matrix Square Root and Inverse Square Root This repository constains the official Pyt

YueSong 42 Dec 30, 2022
Syllabus del curso IIC2115 - Programación como Herramienta para la Ingeniería 2022/I

IIC2115 - Programación como Herramienta para la Ingeniería Videos y tutoriales Tutorial CMD Tutorial Instalación Python y Jupyter Tutorial de git-GitH

21 Nov 09, 2022
A Novel Incremental Learning Driven Instance Segmentation Framework to Recognize Highly Cluttered Instances of the Contraband Items

A Novel Incremental Learning Driven Instance Segmentation Framework to Recognize Highly Cluttered Instances of the Contraband Items This repository co

Taimur Hassan 3 Mar 16, 2022
Embeddinghub is a database built for machine learning embeddings.

Embeddinghub is a database built for machine learning embeddings.

Featureform 1.2k Jan 01, 2023
my graduation project is about live human face augmentation by projection mapping by using CNN

Live-human-face-expression-augmentation-by-projection my graduation project is about live human face augmentation by projection mapping by using CNN o

1 Mar 08, 2022