TensorFlow implementation of Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction)

Last update: Sep 14, 2022

Overview

Barlow-Twins-TF

This repository implements Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction) in TensorFlow and demonstrates it on the CIFAR10 dataset.

Summary:

With a ResNet20 as a trunk and a 3-layer MLP (each layer containing 2048 units) and 100 epochs of pre-training, this training notebook can give 62.61% accuracy on the CIFAR10 test set. The pre-training total takes ~23 minutes on a single Tesla V100. There are minor differences from the original implementation. However, the original loss function and the other minor details like having a big enough projection dimension have been maintained.

For details on Barlow Twins, I suggest reading the original paper, it's really well-written.

Loss progress during pre-training

Other notes

Pre-trained model is available here.
To follow the original implementation details as closely as possible, a WarmUpCosine learning rate schedule has been used during pre-training:
During linear evaluation, Cosine Decay has been used.

Acknowledgements

Thanks to Stéphane Deny (one of the authors of the paper) for helping me catch a pesky bug.

Releases(v1.0.0)

v1.0.0(May 10, 2021)

The archive contains a ResNet20 encoder (trunk + projection head) trained using Barlow Twins for 100 epochs with CIFAR10.
Source code(tar.gz)
Source code(zip)
barlow_twins.tar.gz(38.07 MB)

Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

Self-Supervised Reward Regression (SSRR) Codebase for CoRL 2021 paper "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression "

19 Dec 12, 2022

An official PyTorch implementation of the TKDE paper "Self-Supervised Graph Representation Learning via Topology Transformations".

Self-Supervised Graph Representation Learning via Topology Transformations This repository is the official PyTorch implementation of the following pap

2 Oct 31, 2022

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

Twins: Revisiting the Design of Spatial Attention in Vision Transformers Very recently, a variety of vision transformer architectures for dense predic

482 Dec 18, 2022

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular

18 Dec 17, 2022

PyTorch implementation of the paper Deep Networks from the Principle of Rate Reduction

A repository that shares tuning results of trained models generated by TensorFlow / Keras. Post-training quantization (Weight Quantization, Integer Quantization, Full Integer Quantization, Float16 Quantization), Quantization-aware training. TensorFlow Lite. OpenVINO. CoreML. TensorFlow.js. TF-TRT. MediaPipe. ONNX. [.tflite,.h5,.pb,saved_model,tfjs,tftrt,mlmodel,.xml/.bin, .onnx]

PINTO_model_zoo Please read the contents of the LICENSE file located directly under each folder before using the model. My model conversion scripts ar

2.4k Jan 5, 2023

TensorFlow implementation of Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction)

Related tags

Overview

Barlow-Twins-TF

Loss progress during pre-training

Other notes

Acknowledgements

You might also like...

Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

An official PyTorch implementation of the TKDE paper "Self-Supervised Graph Representation Learning via Topology Transformations".

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

PyTorch implementation of the paper Deep Networks from the Principle of Rate Reduction

Official NumPy Implementation of Deep Networks from the Principle of Rate Reduction (2021)

Model-based 3D Hand Reconstruction via Self-Supervised Learning, CVPR2021

TLDR: Twin Learning for Dimensionality Reduction

Releases(v1.0.0)

v1.0.0(May 10, 2021)

Owner

Sayak Paul

Official repository for Hierarchical Opacity Propagation for Image Matting

Multi-Task Deep Neural Networks for Natural Language Understanding

A simple baseline for 3d human pose estimation in tensorflow. Presented at ICCV 17.

Much faster than SORT(Simple Online and Realtime Tracking), a little worse than SORT

Data and analysis code for an MS on SK VOC genomes phenotyping/neutralisation assays

Repository for self-supervised landmark discovery

Angora is a mutation-based fuzzer. The main goal of Angora is to increase branch coverage by solving path constraints without symbolic execution.

PyTorch code of paper "LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering"

[WACV 2020] Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints

Reference implementation for Deep Unsupervised Learning using Nonequilibrium Thermodynamics

A reimplementation of DCGAN in PyTorch

(CVPR2021) Kaleido-BERT: Vision-Language Pre-training on Fashion Domain

Face Depixelizer based on "PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models" repository.

Codes for the paper Contrast and Mix: Temporal Contrastive Video Domain Adaptation with Background Mixing

Implementation of the paper Scalable Intervention Target Estimation in Linear Models (NeurIPS 2021), and the code to generate simulation results.

Keras implementation of "One pixel attack for fooling deep neural networks" using differential evolution on Cifar10 and ImageNet

Repo for flood prediction using LSTMs and HAND

When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset of 53,000+ Legal Holdings

The pytorch implementation of DG-Font: Deformable Generative Networks for Unsupervised Font Generation

sktime companion package for deep learning based on TensorFlow