Fortuitous Forgetting in Connectionist Networks

Overview

Fortuitous Forgetting in Connectionist Networks

Introduction

This repository includes reference code for the paper Fortuitous Forgetting in Connectionist Networks (ICLR 2022).

@inproceedings{
  zhou2022fortuitous,
  title={Fortuitous Forgetting in Connectionist Networks},
  author={Hattie Zhou and Ankit Vani and Hugo Larochelle and Aaron Courville},
  booktitle={International Conference on Learning Representations},
  year={2022},
  url={https://openreview.net/forum?id=ei3SY1_zYsE}
}

Targeted Forgetting

This code implements the experiments on partial weight perturbations and their effects on easy or hard examples. Scripts are stored in /targeted_forgetting.

To run KE-style forgetting:

python mixed_group_training.py --seed 1 --train_perc 0.1 --random_perc 0.1 --keep_perc 0.5 --train_iters 50000 --fname new_rand_reinit_train0.1_mislabel0.1 --no_wandb

To run IMP-style forgetting:

python mixed_group_training.py --seed 1 --train_perc 1 --random_perc 0.0 --keep_perc 0.3 --train_iters 50000 --weight_mask --reset_to_zero --rewind_to_init --margin_groups --fname new_weight_rewind_zero_train1_margin0.1 --no_wandb

Later Layer Forgetting

This code builds upon the repository for Knowledge Evolution in Neural Networks. Scripts are stored in /llf_ke.

To run 10 generations of LLF on the Flower102 dataset:

python train_KE_cls.py --epochs 200 --num_generations 11 --name resetlayer4_flower_resnet18 --weight_decay 0.0001 --arch Split_ResNet18 --reset_layer_name layer4 --set Flower102 --data $DATA_DIR --no_wandb

To run 10 generations of KE:

python train_KE_cls.py --epochs 200 --num_generations 11 --name ke_kels_flower_resnet18 --weight_decay 0.0001 --arch Split_ResNet18 --split_rate 0.8 --split_mode kels --set Flower102 --data $DATA_DIR --no_wandb

To run 10 generations-equivalent of the long baseline on the Flower102 dataset:

python train_KE_cls.py --epochs 2200 --num_generations 1 --name resetlayer4_flower_resnet18_long2200 --weight_decay 0.0001 --arch Split_ResNet18 --reset_layer_name layer4 --set Flower102 --eval_intermediate_tst 200 --data $DATA_DIR --no_wandb

To run freeze later layers experiment:

python train_KE_cls.py --epochs 200 --num_generations 11 --name resetlayer4_flower_resnet18_freeze_reset_layers --weight_decay 0.0001 --arch Split_ResNet18 --reset_layer_name layer4 --data $DATA_DIR --set Flower102 --reverse_freeze --freeze_non_reset --optimizer sgd_TEMP --no_wandb

To run freeze early layers experiment:

python train_KE_cls.py --epochs 200 --num_generations 11 --name resetlayer4_flower_resnet18_freeze_nonreset_layers --weight_decay 0.0001 --arch Split_ResNet18 --reset_layer_name layer4 --data $DATA_DIR --set Flower102 --freeze_non_reset --optimizer sgd_TEMP --no_wandb

To run freeze later layers with fixed seed experiment:

python train_KE_cls.py --epochs 200 --num_generations 11 --name resetlayer4_flower_resnet18_freeze_reset_layers --weight_decay 0.0001 --arch Split_ResNet18 --reset_layer_name layer4 --data $DATA_DIR --set Flower102 --reverse_freeze --freeze_non_reset --optimizer sgd_TEMP --seed 0 --fix_seed --no_wandb

Ease-of-teaching

This code builds upon the repository for Ease-of-Teaching and Language Structure from Emergent Communication. Scripts are stored in /ease_of_teaching.

To run the no reset baseline:

python forget_train.py --fname baseline_no_reset --seed 0 --no_wandb

To run the reset receiver baseline:

python forget_train.py --resetNum 50 --fname baseline_reset_receiver --seed 0 --reset_receiver --no_wandb

To run partial balanced forgetting (PBF):

python forget_train.py --resetNum 100 --fname same_weight_reinit_sender10_receiver10_reset100 --seed 0 --forget_sender --sender_keep_perc 0.1 --forget_receiver --receiver_keep_perc 0.1 --weight_mask --same_mask --no_wandb

To run targeted forgettine experiments:

python mixed_language_forget_samebatch.py --group_vars same_mask weight_mask reset_to_zero keep_perc seed trainIters train_with_reset reset_every --seed 0 --keep_perc 0.5 --fname new_rand_reinit

python mixed_language_forget_samebatch.py --group_vars same_mask weight_mask reset_to_zero keep_perc seed trainIters train_with_reset reset_every --seed 0 --keep_perc 0.5 --fname same_weight_zero --same_mask --weight_mask --reset_to_zero

Owner
Hattie Zhou
Hattie Zhou
HuSpaCy: industrial-strength Hungarian natural language processing

HuSpaCy: Industrial-strength Hungarian NLP HuSpaCy is a spaCy model and a library providing industrial-strength Hungarian language processing faciliti

HuSpaCy 120 Dec 14, 2022
A fast, dataset-agnostic, deep visual search engine for digital art history

imgs.ai imgs.ai is a fast, dataset-agnostic, deep visual search engine for digital art history based on neural network embeddings. It utilizes modern

Fabian Offert 5 Dec 14, 2022
A containerized REST API around OpenAI's CLIP model.

OpenAI's CLIP — REST API This is a container wrapping OpenAI's CLIP model in a RESTful interface. Running the container locally First, build the conta

Santiago Valdarrama 48 Nov 06, 2022
Implementation of Basic Machine Learning Algorithms on small datasets using Scikit Learn.

Basic Machine Learning Algorithms All the basic Machine Learning Algorithms are implemented in Python using libraries Acknowledgements Machine Learnin

Piyal Banik 47 Oct 16, 2022
Official PyTorch code for CVPR 2020 paper "Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision"

Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision https://arxiv.org/abs/2003.00393 Abstract Active learning (AL) aims to min

Denis 29 Nov 21, 2022
Just playing with getting VQGAN+CLIP running locally, rather than having to use colab.

Just playing with getting VQGAN+CLIP running locally, rather than having to use colab.

Nerdy Rodent 2.3k Jan 04, 2023
Text and code for the forthcoming second edition of Think Bayes, by Allen Downey.

Think Bayes 2 by Allen B. Downey The HTML version of this book is here. Think Bayes is an introduction to Bayesian statistics using computational meth

Allen Downey 1.5k Jan 08, 2023
Sequential GCN for Active Learning

Sequential GCN for Active Learning Please cite if using the code: Link to paper. Requirements: python 3.6+ torch 1.0+ pip libraries: tqdm, sklearn, sc

45 Dec 26, 2022
Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning, NeurIPS 2021 (Spotlight)

Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning, NeurIPS 2021 (Spotlight) Abstract Due to the limited and even imbalanced dat

Hanzhe Hu 99 Dec 12, 2022
Active learning for Mask R-CNN in Detectron2

MaskAL - Active learning for Mask R-CNN in Detectron2 Summary MaskAL is an active learning framework that automatically selects the most-informative i

49 Dec 20, 2022
Evaluating Cross-lingual Sentence Representations

XNLI: The Cross-Lingual NLI Corpus XNLI is an evaluation corpus for language transfer and cross-lingual sentence classification in 15 languages. New:

Meta Research 395 Dec 19, 2022
Tool which allow you to detect and translate text.

Text detection and recognition This repository contains tool which allow to detect region with text and translate it one by one. Description Two pretr

Damian Panek 176 Nov 28, 2022
MILK: Machine Learning Toolkit

MILK: MACHINE LEARNING TOOLKIT Machine Learning in Python Milk is a machine learning toolkit in Python. Its focus is on supervised classification with

Luis Pedro Coelho 610 Dec 14, 2022
The official implementation of Autoregressive Image Generation using Residual Quantization (CVPR '22)

Autoregressive Image Generation using Residual Quantization (CVPR 2022) The official implementation of "Autoregressive Image Generation using Residual

Kakao Brain 529 Dec 30, 2022
Yolox-bytetrack-sample - Python sample of MOT (Multiple Object Tracking) using YOLOX and ByteTrack

yolox-bytetrack-sample YOLOXとByteTrackを用いたMOT(Multiple Object Tracking)のPythonサン

KazuhitoTakahashi 12 Nov 09, 2022
A PyTorch toolkit for 2D Human Pose Estimation.

PyTorch-Pose PyTorch-Pose is a PyTorch implementation of the general pipeline for 2D single human pose estimation. The aim is to provide the interface

Wei Yang 1.1k Dec 30, 2022
A hyperparameter optimization framework

Optuna: A hyperparameter optimization framework Website | Docs | Install Guide | Tutorial Optuna is an automatic hyperparameter optimization software

7.4k Jan 04, 2023
For encoding a text longer than 512 tokens, for example 800. Set max_pos to 800 during both preprocessing and training.

LongScientificFormer For encoding a text longer than 512 tokens, for example 800. Set max_pos to 800 during both preprocessing and training. Some code

Athar Sefid 6 Nov 02, 2022
traiNNer is an open source image and video restoration (super-resolution, denoising, deblurring and others) and image to image translation toolbox based on PyTorch.

traiNNer traiNNer is an open source image and video restoration (super-resolution, denoising, deblurring and others) and image to image translation to

202 Jan 04, 2023
A project to build an AI voice assistant using Python . The Voice assistant interacts with the humans to perform basic tasks.

AI_Personal_Voice_Assistant_Using_Python A project to build an AI voice assistant using Python . The Voice assistant interacts with the humans to perf

Chumui Tripura 1 Oct 30, 2021