Fortuitous Forgetting in Connectionist Networks

Overview

Fortuitous Forgetting in Connectionist Networks

Introduction

This repository includes reference code for the paper Fortuitous Forgetting in Connectionist Networks (ICLR 2022).

@inproceedings{
  zhou2022fortuitous,
  title={Fortuitous Forgetting in Connectionist Networks},
  author={Hattie Zhou and Ankit Vani and Hugo Larochelle and Aaron Courville},
  booktitle={International Conference on Learning Representations},
  year={2022},
  url={https://openreview.net/forum?id=ei3SY1_zYsE}
}

Targeted Forgetting

This code implements the experiments on partial weight perturbations and their effects on easy or hard examples. Scripts are stored in /targeted_forgetting.

To run KE-style forgetting:

python mixed_group_training.py --seed 1 --train_perc 0.1 --random_perc 0.1 --keep_perc 0.5 --train_iters 50000 --fname new_rand_reinit_train0.1_mislabel0.1 --no_wandb

To run IMP-style forgetting:

python mixed_group_training.py --seed 1 --train_perc 1 --random_perc 0.0 --keep_perc 0.3 --train_iters 50000 --weight_mask --reset_to_zero --rewind_to_init --margin_groups --fname new_weight_rewind_zero_train1_margin0.1 --no_wandb

Later Layer Forgetting

This code builds upon the repository for Knowledge Evolution in Neural Networks. Scripts are stored in /llf_ke.

To run 10 generations of LLF on the Flower102 dataset:

python train_KE_cls.py --epochs 200 --num_generations 11 --name resetlayer4_flower_resnet18 --weight_decay 0.0001 --arch Split_ResNet18 --reset_layer_name layer4 --set Flower102 --data $DATA_DIR --no_wandb

To run 10 generations of KE:

python train_KE_cls.py --epochs 200 --num_generations 11 --name ke_kels_flower_resnet18 --weight_decay 0.0001 --arch Split_ResNet18 --split_rate 0.8 --split_mode kels --set Flower102 --data $DATA_DIR --no_wandb

To run 10 generations-equivalent of the long baseline on the Flower102 dataset:

python train_KE_cls.py --epochs 2200 --num_generations 1 --name resetlayer4_flower_resnet18_long2200 --weight_decay 0.0001 --arch Split_ResNet18 --reset_layer_name layer4 --set Flower102 --eval_intermediate_tst 200 --data $DATA_DIR --no_wandb

To run freeze later layers experiment:

python train_KE_cls.py --epochs 200 --num_generations 11 --name resetlayer4_flower_resnet18_freeze_reset_layers --weight_decay 0.0001 --arch Split_ResNet18 --reset_layer_name layer4 --data $DATA_DIR --set Flower102 --reverse_freeze --freeze_non_reset --optimizer sgd_TEMP --no_wandb

To run freeze early layers experiment:

python train_KE_cls.py --epochs 200 --num_generations 11 --name resetlayer4_flower_resnet18_freeze_nonreset_layers --weight_decay 0.0001 --arch Split_ResNet18 --reset_layer_name layer4 --data $DATA_DIR --set Flower102 --freeze_non_reset --optimizer sgd_TEMP --no_wandb

To run freeze later layers with fixed seed experiment:

python train_KE_cls.py --epochs 200 --num_generations 11 --name resetlayer4_flower_resnet18_freeze_reset_layers --weight_decay 0.0001 --arch Split_ResNet18 --reset_layer_name layer4 --data $DATA_DIR --set Flower102 --reverse_freeze --freeze_non_reset --optimizer sgd_TEMP --seed 0 --fix_seed --no_wandb

Ease-of-teaching

This code builds upon the repository for Ease-of-Teaching and Language Structure from Emergent Communication. Scripts are stored in /ease_of_teaching.

To run the no reset baseline:

python forget_train.py --fname baseline_no_reset --seed 0 --no_wandb

To run the reset receiver baseline:

python forget_train.py --resetNum 50 --fname baseline_reset_receiver --seed 0 --reset_receiver --no_wandb

To run partial balanced forgetting (PBF):

python forget_train.py --resetNum 100 --fname same_weight_reinit_sender10_receiver10_reset100 --seed 0 --forget_sender --sender_keep_perc 0.1 --forget_receiver --receiver_keep_perc 0.1 --weight_mask --same_mask --no_wandb

To run targeted forgettine experiments:

python mixed_language_forget_samebatch.py --group_vars same_mask weight_mask reset_to_zero keep_perc seed trainIters train_with_reset reset_every --seed 0 --keep_perc 0.5 --fname new_rand_reinit

python mixed_language_forget_samebatch.py --group_vars same_mask weight_mask reset_to_zero keep_perc seed trainIters train_with_reset reset_every --seed 0 --keep_perc 0.5 --fname same_weight_zero --same_mask --weight_mask --reset_to_zero

Owner
Hattie Zhou
Hattie Zhou
Implementation of Graph Transformer in Pytorch, for potential use in replicating Alphafold2

Graph Transformer - Pytorch Implementation of Graph Transformer in Pytorch, for potential use in replicating Alphafold2. This was recently used by bot

Phil Wang 97 Dec 28, 2022
Awesome Remote Sensing Toolkit based on PaddlePaddle.

基于飞桨框架开发的高性能遥感图像处理开发套件,端到端地完成从训练到部署的全流程遥感深度学习应用。 最新动态 PaddleRS 即将发布alpha版本!欢迎大家试用 简介 PaddleRS是遥感科研院所、相关高校共同基于飞桨开发的遥感处理平台,支持遥感图像分类,目标检测,图像分割,以及变化检测等常用遥

146 Dec 11, 2022
Code for the paper "Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks"

ON-LSTM This repository contains the code used for word-level language model and unsupervised parsing experiments in Ordered Neurons: Integrating Tree

Yikang Shen 572 Nov 21, 2022
Ludwig Benchmarking Toolkit

Ludwig Benchmarking Toolkit The Ludwig Benchmarking Toolkit is a personalized benchmarking toolkit for running end-to-end benchmark studies across an

HazyResearch 17 Nov 18, 2022
VideoGPT: Video Generation using VQ-VAE and Transformers

VideoGPT: Video Generation using VQ-VAE and Transformers [Paper][Website][Colab][Gradio Demo] We present VideoGPT: a conceptually simple architecture

Wilson Yan 470 Dec 30, 2022
Deep-learning X-Ray Micro-CT image enhancement, pore-network modelling and continuum modelling

EDSR modelling A Github repository for deep-learning image enhancement, pore-network and continuum modelling from X-Ray Micro-CT images. The repositor

Samuel Jackson 7 Nov 03, 2022
This repository contains the segmentation user interface from the OpenSurfaces project, extracted as a lightweight tool

OpenSurfaces Segmentation UI This repository contains the segmentation user interface from the OpenSurfaces project, extracted as a lightweight tool.

Sean Bell 66 Jul 11, 2022
根据midi文件演奏“风物之诗琴”的脚本 "Windsong Lyre" auto play

Genshin-lyre-auto-play 简体中文 | English 简介 根据midi文件演奏“风物之诗琴”的脚本。由Python驱动,在此承诺, ⚠️ 项目内绝不含任何能够引起安全问题的代码。 前排提示:所有键盘在动但是原神没反应的都是因为没有管理员权限,双击run.bat或者以管理员模式

御坂17032号 386 Jan 01, 2023
RoBERTa Marathi Language model trained from scratch during huggingface 🤗 x flax community week

RoBERTa base model for Marathi Language (मराठी भाषा) Pretrained model on Marathi language using a masked language modeling (MLM) objective. RoBERTa wa

Nipun Sadvilkar 23 Oct 19, 2022
Code for SyncTwin: Treatment Effect Estimation with Longitudinal Outcomes (NeurIPS 2021)

SyncTwin: Treatment Effect Estimation with Longitudinal Outcomes (NeurIPS 2021) SyncTwin is a treatment effect estimation method tailored for observat

Zhaozhi Qian 3 Nov 03, 2022
Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals.

Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals This repo contains the Pytorch implementation of our paper: Unsupervised Seman

Wouter Van Gansbeke 335 Dec 28, 2022
Code for the paper Language as a Cognitive Tool to Imagine Goals in Curiosity Driven Exploration

IMAGINE: Language as a Cognitive Tool to Imagine Goals in Curiosity Driven Exploration This repo contains the code base of the paper Language as a Cog

Flowers Team 26 Dec 22, 2022
Official code for 'Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning' [ICCV 2021]

RTFM This repo contains the Pytorch implementation of our paper: Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Lear

Yu Tian 242 Jan 08, 2023
RIM: Reliable Influence-based Active Learning on Graphs.

RIM: Reliable Influence-based Active Learning on Graphs. This repository is the official implementation of RIM. Requirements To install requirements:

Wentao Zhang 4 Aug 29, 2022
Supplementary materials for ISMIR 2021 LBD paper "Evaluation of Latent Space Disentanglement in the Presence of Interdependent Attributes"

Evaluation of Latent Space Disentanglement in the Presence of Interdependent Attributes Supplementary materials for ISMIR 2021 LBD submission: K. N. W

Karn Watcharasupat 2 Oct 25, 2021
Generative Adversarial Text-to-Image Synthesis

###Generative Adversarial Text-to-Image Synthesis Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee This is the

Scott Ellison Reed 883 Dec 31, 2022
This repo will contain code to reproduce and build upon understanding transfer learning

What is being transferred in transfer learning? This repo contains the code for the following paper: Behnam Neyshabur*, Hanie Sedghi*, Chiyuan Zhang*.

4 Jun 16, 2021
Spatial Single-Cell Analysis Toolkit

Single-Cell Image Analysis Package Scimap is a scalable toolkit for analyzing spatial molecular data. The underlying framework is generalizable to spa

Laboratory of Systems Pharmacology @ Harvard 30 Nov 08, 2022
OpenMMLab Pose Estimation Toolbox and Benchmark.

Introduction English | 简体中文 MMPose is an open-source toolbox for pose estimation based on PyTorch. It is a part of the OpenMMLab project. The master b

OpenMMLab 2.8k Dec 31, 2022
Python implementation of MULTIseq barcode alignment using fuzzy string matching and GMM barcode assignment

Python implementation of MULTIseq barcode alignment using fuzzy string matching and GMM barcode assignment.

MT Schmitz 2 Feb 11, 2022