Code accompanying the paper "How Tight Can PAC-Bayes be in the Small Data Regime?"

Last update: Dec 21, 2021

Overview

How Tight Can PAC-Bayes be in the Small Data Regime?

This is the code to reproduce all experiments for the following paper:

@inproceedings{Foong:2021:How_Tight_Can_PAC-Bayes_Be,
    title = {How Tight Can {PAC}-{Bayes} Be in the Small Data Regime?},
    year = {2021},
    author = {Andrew Y. K. Foong and Wessel P. Bruinsma and David R. Burt and Richard E. Turner},
    booktitle = {Advances in Neural Information Processing Systems},
    volume = {35},
    eprint = {https://arxiv.org/abs/2106.03542},
}

Every experiment creates a folder in _experiments. The names of the files in those folders should be self-explanatory.

Installation

First, create and activate a virtual environment for Python 3.8.

virtualenv venv -p python3.8 
source venv/bin/activate

Then install an appropriate GPU-accelerated version of PyTorch.

Finally, install the requirements for the project.

pip install -e .

You should now be able to run the below commands.

Generating Datasets

In order to generate the synthetic 1D datasets used, run these commands from inside classification_1d:

python gen_data.py --class_scheme balanced --num_context 30 --name 30-context --num_train_batches 5000 --num_test_batches 64
python gen_data.py --class_scheme balanced --num_context 60 --name 60-context --num_train_batches 5000 --num_test_batches 64

The generated datasets will be in pacbayes/_data_caches

Theory Experiments

See Figure 2 in Section 3 and Appendix G.

python theory_experiments.py --setting det1-1
python theory_experiments.py --setting det1-2
python theory_experiments.py --setting det2-1
python theory_experiments.py --setting det2-1

python theory_experiments.py --setting stoch1
python theory_experiments.py --setting stoch2
python theory_experiments.py --setting stoch3

python theory_experiments.py --setting random --random-seed 1 --random-better-bound maurer
python theory_experiments.py --setting random --random-seed 6 --random-better-bound catoni

GNP Classification Experiments

See Figure 3 and 4 in Section 4 and Appendices I and J. The numbers from the graphs can be found in eval_metrics_no_post_opt.txt (without post optimisation) eval_metrics_post_opt.txt (with post optimisation).

MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh

MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh

MLP Classification Experiments

See Appendix J. The numbers from the graphs can be found in eval_metrics_no_post_opt.txt (without post optimisation) eval_metrics_post_opt.txt (with post optimisation).

MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=30 ./run_MLP.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=30 ./run_MLP.sh

MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=60 ./run_MLP.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=60 ./run_MLP.sh

Code accompanying the paper "How Tight Can PAC-Bayes be in the Small Data Regime?"

Related tags

Overview

How Tight Can PAC-Bayes be in the Small Data Regime?

Installation

Generating Datasets

Theory Experiments

GNP Classification Experiments

MLP Classification Experiments

Owner

Readings for "A Unified View of Relational Deep Learning for Polypharmacy Side Effect, Combination Therapy, and Drug-Drug Interaction Prediction."

Automatic caption evaluation metric based on typicality analysis.

The pytorch implementation of SOKD (BMVC2021).

Self-Adaptable Point Processes with Nonparametric Time Decays

This is the source code for generating the ASL-Skeleton3D and ASL-Phono datasets. Check out the README.md for more details.

PyTorch Implementation for AAAI'21 "Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selection"

Adaptive Dropblock Enhanced GenerativeAdversarial Networks for Hyperspectral Image Classification

Embeddinghub is a database built for machine learning embeddings.

Repository for the electrical and ICT benchmark model developed in the ERIGrid 2.0 project.

Pytorch Implementation for CVPR2018 Paper: Learning to Compare: Relation Network for Few-Shot Learning

This repository is the code of the paper Accelerating Deep Reinforcement Learning for Digital Twin Network Optimization with Evolutionary Strategies

A toolkit for Lagrangian-based constrained optimization in Pytorch

ruptures: change point detection in Python

Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather

A simple baseline for 3d human pose estimation in tensorflow. Presented at ICCV 17.

Code for Deterministic Neural Networks with Appropriate Inductive Biases Capture Epistemic and Aleatoric Uncertainty

ArcaneGAN by Alex Spirin

Iowa Project - My second project done at General Assembly, focused on feature engineering and understanding Linear Regression as a concept

PyTorch implementation of PSPNet segmentation network

Code for Deep Single-image Portrait Image Relighting