Code accompanying the paper "How Tight Can PAC-Bayes be in the Small Data Regime?"

Last update: Dec 21, 2021

Overview

How Tight Can PAC-Bayes be in the Small Data Regime?

This is the code to reproduce all experiments for the following paper:

@inproceedings{Foong:2021:How_Tight_Can_PAC-Bayes_Be,
    title = {How Tight Can {PAC}-{Bayes} Be in the Small Data Regime?},
    year = {2021},
    author = {Andrew Y. K. Foong and Wessel P. Bruinsma and David R. Burt and Richard E. Turner},
    booktitle = {Advances in Neural Information Processing Systems},
    volume = {35},
    eprint = {https://arxiv.org/abs/2106.03542},
}

Every experiment creates a folder in _experiments. The names of the files in those folders should be self-explanatory.

Installation

First, create and activate a virtual environment for Python 3.8.

virtualenv venv -p python3.8 
source venv/bin/activate

Then install an appropriate GPU-accelerated version of PyTorch.

Finally, install the requirements for the project.

pip install -e .

You should now be able to run the below commands.

Generating Datasets

In order to generate the synthetic 1D datasets used, run these commands from inside classification_1d:

python gen_data.py --class_scheme balanced --num_context 30 --name 30-context --num_train_batches 5000 --num_test_batches 64
python gen_data.py --class_scheme balanced --num_context 60 --name 60-context --num_train_batches 5000 --num_test_batches 64

The generated datasets will be in pacbayes/_data_caches

Theory Experiments

See Figure 2 in Section 3 and Appendix G.

python theory_experiments.py --setting det1-1
python theory_experiments.py --setting det1-2
python theory_experiments.py --setting det2-1
python theory_experiments.py --setting det2-1

python theory_experiments.py --setting stoch1
python theory_experiments.py --setting stoch2
python theory_experiments.py --setting stoch3

python theory_experiments.py --setting random --random-seed 1 --random-better-bound maurer
python theory_experiments.py --setting random --random-seed 6 --random-better-bound catoni

GNP Classification Experiments

See Figure 3 and 4 in Section 4 and Appendices I and J. The numbers from the graphs can be found in eval_metrics_no_post_opt.txt (without post optimisation) eval_metrics_post_opt.txt (with post optimisation).

MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh

MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh

MLP Classification Experiments

See Appendix J. The numbers from the graphs can be found in eval_metrics_no_post_opt.txt (without post optimisation) eval_metrics_post_opt.txt (with post optimisation).

MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=30 ./run_MLP.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=30 ./run_MLP.sh

MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=60 ./run_MLP.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=60 ./run_MLP.sh

Code accompanying the paper "How Tight Can PAC-Bayes be in the Small Data Regime?"

Related tags

Overview

How Tight Can PAC-Bayes be in the Small Data Regime?

Installation

Generating Datasets

Theory Experiments

GNP Classification Experiments

MLP Classification Experiments

Owner

Two-stage CenterNet

Continuous Time LiDAR odometry

A transformer model to predict pathogenic mutations

The easiest way to use deep metric learning in your application. Modular, flexible, and extensible. Written in PyTorch.

An official PyTorch implementation of the TKDE paper "Self-Supervised Graph Representation Learning via Topology Transformations".

Semantic Bottleneck Scene Generation

[ICCV2021] IICNet: A Generic Framework for Reversible Image Conversion

POPPY (Physical Optics Propagation in Python) is a Python package that simulates physical optical propagation including diffraction

EMNLP 2020 - Summarizing Text on Any Aspects

This repo uses a combination of logits and feature distillation method to teach the PSPNet model of ResNet18 backbone with the PSPNet model of ResNet50 backbone. All the models are trained and tested on the PASCAL-VOC2012 dataset.

A state-of-the-art semi-supervised method for image recognition

Simple API for UCI Machine Learning Dataset Repository (search, download, analyze)

DNA sequence classification by Deep Neural Network

Code for paper [ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shot] (ICCV 2021, oral))

Tensorflow implementation of "Learning Deconvolution Network for Semantic Segmentation"

Official repository for "Intriguing Properties of Vision Transformers" (2021)

PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network"

Code for Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid

An implementation of the [Hierarchical (Sig-Wasserstein) GAN] algorithm for large dimensional Time Series Generation

A Large-Scale Dataset for Spinal Vertebrae Segmentation in Computed Tomography