OODformer: Out-Of-Distribution Detection Transformer

This repo is the official the implementation of the OODformer: Out-Of-Distribution Detection Transformer in PyTorch using CIFAR as an illustrative example:
##Getting started

At first please install all the dependencies using : pip install -r requirement.txt ##Datasets Please download all the in-distribution (CIFAR-10,CIFAR-100,ImageNet-30) and out-of-distribution dataset(LSUN_resize, ImageNet_resize, Places-365, DTD, Stanford Dogs, Food-101, Caltech-256, CUB-200) to data folder under the root directory.

Training

For training Vision Transformer and its Data efficient variant please download their corresponding pre-train weight from ViT and DeiT repository.

To fine-tune vision transformer network on any in-distribution dataset on multi GPU settings:

srun --gres=gpu:4  python vit/src/train.py --exp-name name_of_the_experimet --tensorboard --model-arch b16 --checkpoint-path path/to/checkpoint --image-size 224 --data-dir data/ImageNet30 --dataset ImageNet --num-classes 30 --train-steps 4590 --lr 0.01 --wd 1e-5 --n-gpu 4 --num-workers 16 --batch-size 512 --method SupCE

• model-arch : specify the model of vit and deit variants (see vit/src/config.py )
• method : currently we support only supervised cross-entropy
• train_steps : cyclic lr has been used for lr scheduler, number of training epoch can be calculated using (#train steps* batch size)/#training samples
• checkpoint_path : for loading pre-trained weight of vision transformer based on their different model.

Training Support

OODformer can also be trained with various supervised and self-supervised loss like :

• Supervised Cross Entropy
• Self-Supervised Contrastive Loss[TBD]
• Supervised Contrastive Loss[TBD]
• Self-Supervised Vision Transformer[TBD]

Training Base ResNet model

To train resnet variants(e.g., resent-50,wide-resent) as base model on in-distribution dataset :

srun --gres=gpu:4  python main_ce.py --batch_size 512 --epochs 500 --model resent34 --learning_rate 0.8  --cosine --warm --dataset cifar10

Evaluation

To evaluate the similarity distance from the mean embedding of an in-distribution (e.g., CIFAR-10) class a list of distance metrics (e.g., Mahalanobis, Cosine, Euclidean, and Softmax) can be used with OODformer as stated below :

srun --gres=gpu:1 python OOD_Distance.py --ckpt checkpoint_path --model vit --model_arch b16 --distance Mahalanobis --dataset id_dataset --out_dataset ood_dataset

Visualization

Various embedding visualization can be viewed using generate_tsne.py

(1) UMAP of in-distribution embedding

(2) UMAP of combined in and out-of distribution embedding

Reference

@article{koner2021oodformer,
  title={OODformer: Out-Of-Distribution Detection Transformer},
  author={Koner, Rajat and Sinhamahapatra, Poulami and Roscher, Karsten and G{\"u}nnemann, Stephan and Tresp, Volker},
  journal={arXiv preprint arXiv:2107.08976},
  year={2021}
}

Acknowledgments

Part of this code is inspired by HobbitLong/SupContrast.