Official Repository for the paper "Improving Baselines in the Wild".

Related tags

Deep Learningwilds
Overview

iWildCam and FMoW baselines (WILDS)

This repository was originally forked from the official repository of WILDS datasets (commit 7e103ed)

For general instructions, please refer to the original repositiory.

This repository contains code used to produce experimental results presented in:

Improving Baselines in the Wild

Apart from minor edits, the only main changes we introduce are:

  • --validate_every flag (default: 1000) to specify the frequency (number of training steps) of cross-validation/checkpoint tracking.
  • sub_val_metric option in the dataset (see examples/configs/datasets.py) to specify a secondary metric to be tracked during training. This activates additional cross-validation and checkpoint tracking for the specified metric.

Results

NB: To reproduce the numbers from the paper, the right PyTorch version must be used. All our experiments have been conducted using 1.9.0+cu102, except for + higher lr rows in Table 2/FMoW (which we ran for the camera-ready and for the public release) for which 1.10.0+cu102 was used.

The training scripts, logs, and model checkpoints for the best configurations from our experiments can be found here for iWildCam & FMoW.

iWildCam

CV based on "Valid F1"

Split / Metric mean (std) 3 runs
IID Valid Acc 82.5 (0.8) [0.817, 0.835, 0.822]
IID Valid F1 46.7 (1.0) [0.456, 0.481, 0.464]
IID Test Acc 76.2 (0.1) [0.762, 0.763, 0.761]
IID Test F1 47.9 (2.1) [0.505, 0.479, 0.453]
Valid Acc 64.1 (1.7) [0.644, 0.619, 0.661]
Valid F1 38.3 (0.9) [0.39, 0.371, 0.389]
Test Acc 69.0 (0.3) [0.69, 0.694, 0.687]
Test F1 32.1 (1.2) [0.338, 0.31, 0.314]

CV based on "Valid Acc"

Split / Metric mean (std) 3 runs
IID Valid Acc 82.6 (0.7) [0.836, 0.821, 0.822]
IID Valid F1 46.2 (0.9) [0.472, 0.45, 0.464]
IID Test Acc 75.8 (0.4) [0.76, 0.753, 0.761]
IID Test F1 44.9 (0.4) [0.444, 0.45, 0.453]
Valid Acc 66.6 (0.4) [0.666, 0.672, 0.661]
Valid F1 36.6 (2.1) [0.369, 0.339, 0.389]
Test Acc 68.6 (0.3) [0.688, 0.682, 0.687]
Test F1 28.7 (2.0) [0.279, 0.268, 0.314]

FMoW

CV based on "Valid Region"

Split / Metric mean (std) 3 runs
IID Valid Acc 63.9 (0.2) [0.64, 0.636, 0.641]
IID Valid Region 62.2 (0.5) [0.623, 0.616, 0.628]
IID Valid Year 49.8 (1.8) [0.52, 0.475, 0.5]
IID Test Acc 62.3 (0.2) [0.626, 0.621, 0.621]
IID Test Region 60.9 (0.6) [0.617, 0.603, 0.606]
IID Test Year 43.2 (1.1) [0.438, 0.417, 0.442]
Valid Acc 62.1 (0.0) [0.62, 0.621, 0.621]
Valid Region 52.5 (1.0) [0.538, 0.513, 0.524]
Valid Year 60.5 (0.2) [0.602, 0.605, 0.608]
Test Acc 55.6 (0.2) [0.555, 0.554, 0.558]
Test Region 34.8 (1.5) [0.369, 0.334, 0.34]
Test Year 50.2 (0.4) [0.499, 0.498, 0.508]

CV based on "Valid Acc"

Split / Metric mean (std) 3 runs
IID Valid Acc 64.0 (0.1) [0.641, 0.639, 0.641]
IID Valid Region 62.3 (0.4) [0.623, 0.617, 0.628]
IID Valid Year 50.8 (0.6) [0.514, 0.509, 0.5]
IID Test Acc 62.3 (0.4) [0.628, 0.62, 0.621]
IID Test Region 61.1 (0.6) [0.62, 0.608, 0.606]
IID Test Year 43.6 (1.4) [0.45, 0.417, 0.442]
Valid Acc 62.1 (0.0) [0.621, 0.621, 0.621]
Valid Region 51.4 (1.3) [0.522, 0.496, 0.524]
Valid Year 60.6 (0.3) [0.608, 0.601, 0.608]
Test Acc 55.6 (0.2) [0.556, 0.554, 0.558]
Test Region 34.2 (1.2) [0.357, 0.329, 0.34]
Test Year 50.2 (0.5) [0.496, 0.501, 0.508]

BibTex

@inproceedings{irie2021improving,
      title={Improving Baselines in the Wild}, 
      author={Kazuki Irie and Imanol Schlag and R\'obert Csord\'as and J\"urgen Schmidhuber},
      booktitle={Workshop on Distribution Shifts, NeurIPS},
      address={Virtual only},
      year={2021}
}
Owner
Kazuki Irie
Kazuki Irie
Repository for benchmarking graph neural networks

Benchmarking Graph Neural Networks Updates Nov 2, 2020 Project based on DGL 0.4.2. See the relevant dependencies defined in the environment yml files

NTU Graph Deep Learning Lab 2k Jan 03, 2023
GANsformer: Generative Adversarial Transformers Drew A

GANformer: Generative Adversarial Transformers Drew A. Hudson* & C. Lawrence Zitnick Update: We released the new GANformer2 paper! *I wish to thank Ch

Drew Arad Hudson 1.2k Jan 02, 2023
Keras implementation of Real-Time Semantic Segmentation on High-Resolution Images

Keras-ICNet [paper] Keras implementation of Real-Time Semantic Segmentation on High-Resolution Images. Training in progress! Requisites Python 3.6.3 K

Aitor Ruano 87 Dec 16, 2022
A Tensorflow based library for Time Series Modelling with Gaussian Processes

Markovflow Documentation | Tutorials | API reference | Slack What does Markovflow do? Markovflow is a Python library for time-series analysis via prob

Secondmind Labs 24 Dec 12, 2022
Fuzzer for Linux Kernel Drivers

difuze: Fuzzer for Linux Kernel Drivers This repo contains all the sources (including setup scripts), you need to get difuze up and running. Tested on

seclab 344 Dec 27, 2022
The repo contains the code to train and evaluate a system which extracts relations and explanations from dialogue.

The repo contains the code to train and evaluate a system which extracts relations and explanations from dialogue. How do I cite D-REX? For now, cite

Alon Albalak 6 Mar 31, 2022
Implementation for HFGI: High-Fidelity GAN Inversion for Image Attribute Editing

HFGI: High-Fidelity GAN Inversion for Image Attribute Editing High-Fidelity GAN Inversion for Image Attribute Editing Update: We released the inferenc

Tengfei Wang 371 Dec 30, 2022
EqGAN - Improving GAN Equilibrium by Raising Spatial Awareness

EqGAN - Improving GAN Equilibrium by Raising Spatial Awareness Improving GAN Equilibrium by Raising Spatial Awareness Jianyuan Wang, Ceyuan Yang, Ying

GenForce: May Generative Force Be with You 149 Dec 19, 2022
Aggragrating Nested Transformer Official Jax Implementation

NesT is a simple method, which aggragrates nested local transformers on image blocks. The idea makes vision transformers attain better accuracy, data efficiency, and convergence on the ImageNet bench

Google Research 169 Dec 20, 2022
PyTorch implementation of Barlow Twins.

Barlow Twins: Self-Supervised Learning via Redundancy Reduction PyTorch implementation of Barlow Twins. @article{zbontar2021barlow, title={Barlow Tw

Facebook Research 839 Dec 29, 2022
[ICCV 2021] Relaxed Transformer Decoders for Direct Action Proposal Generation

RTD-Net (ICCV 2021) This repo holds the codes of paper: "Relaxed Transformer Decoders for Direct Action Proposal Generation", accepted in ICCV 2021. N

Multimedia Computing Group, Nanjing University 80 Nov 30, 2022
Fully Automatic Page Turning on Real Scores

Fully Automatic Page Turning on Real Scores This repository contains the corresponding code for our extended abstract Henkel F., Schwaiger S. and Widm

Florian Henkel 7 Jan 02, 2022
This repository contains the source codes for the paper AtlasNet V2 - Learning Elementary Structures.

AtlasNet V2 - Learning Elementary Structures This work was build upon Thibault Groueix's AtlasNet and 3D-CODED projects. (you might want to have a loo

Théo Deprelle 123 Nov 11, 2022
The codes and related files to reproduce the results for Image Similarity Challenge Track 1.

ISC-Track1-Submission The codes and related files to reproduce the results for Image Similarity Challenge Track 1. Required dependencies To begin with

Wenhao Wang 115 Jan 02, 2023
This project deals with the detection of skin lesions within the ISICs dataset using YOLOv3 Object Detection with Darknet.

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. Skin Lesion detection using YOLO This project deal

Lalith Veerabhadrappa Badiger 1 Nov 22, 2021
MPRNet-Cloud-removal: Progressive cloud removal

MPRNet-Cloud-removal Progressive cloud removal Requirements 1.Pytorch = 1.0 2.Python 3 3.NVIDIA GPU + CUDA 9.0 4.Tensorboard Installation 1.Clone the

Semi 95 Dec 18, 2022
Segmentation vgg16 fcn - cityscapes

VGGSegmentation Segmentation vgg16 fcn - cityscapes Priprema skupa skripta prepare_dataset_downsampled.py Iz slika cityscapesa izrezuje haubu automobi

6 Oct 24, 2020
A new framework, collaborative cascade prediction based on graph neural networks (CCasGNN) to jointly utilize the structural characteristics, sequence features, and user profiles.

CCasGNN A new framework, collaborative cascade prediction based on graph neural networks (CCasGNN) to jointly utilize the structural characteristics,

5 Apr 29, 2022
Code for layerwise detection of linguistic anomaly paper (ACL 2021)

Layerwise Anomaly This repository contains the source code and data for our ACL 2021 paper: "How is BERT surprised? Layerwise detection of linguistic

6 Dec 07, 2022
Registration Loss Learning for Deep Probabilistic Point Set Registration

RLLReg This repository contains a Pytorch implementation of the point set registration method RLLReg. Details about the method can be found in the 3DV

Felix Järemo Lawin 35 Nov 02, 2022