The full training script for Enformer (Tensorflow Sonnet) on TPU clusters

Last update: Oct 19, 2022

Overview

Enformer TPU training script (wip)

The full training script for Enformer (Tensorflow Sonnet) on TPU clusters, in an effort to migrate the model to pytorch.

This was pieced together from the Deepmind Enformer repository, the colab training notebook, as well as Basenji sequence augmentation code

It accounts for:

distributed TPU training
distributed datasets
distributed validation
gradient clipping
cross replica batchnorms
dataset augmentation

Training takes about 3 days on v3-64

Todo

fix script for differences in sequence length in basenji training data, which is ~130k vs ~190k bp as in paper

Citations

@article {Avsec2021.04.07.438649,
    author  = {Avsec, {\v Z}iga and Agarwal, Vikram and Visentin, Daniel and Ledsam, Joseph R. and Grabska-Barwinska, Agnieszka and Taylor, Kyle R. and Assael, Yannis and Jumper, John and Kohli, Pushmeet and Kelley, David R.},
    title   = {Effective gene expression prediction from sequence by integrating long-range interactions},
    elocation-id = {2021.04.07.438649},
    year    = {2021},
    doi     = {10.1101/2021.04.07.438649},
    publisher = {Cold Spring Harbor Laboratory},
    URL     = {https://www.biorxiv.org/content/early/2021/04/08/2021.04.07.438649},
    eprint  = {https://www.biorxiv.org/content/early/2021/04/08/2021.04.07.438649.full.pdf},
    journal = {bioRxiv}
}

The full training script for Enformer (Tensorflow Sonnet) on TPU clusters

Related tags

Overview

Enformer TPU training script (wip)

Todo

Citations

Owner

Phil Wang

An introduction to bioimage analysis - http://bioimagebook.github.io

Multi-agent reinforcement learning algorithm and environment

Powerful unsupervised domain adaptation method for dense retrieval.

Look Who’s Talking: Active Speaker Detection in the Wild

Adversarial-Information-Bottleneck - Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck (NeurIPS21)

rliable is an open-source Python library for reliable evaluation, even with a handful of runs, on reinforcement learning and machine learnings benchmarks.

NLG evaluation via Statistical Measures of Similarity: BaryScore, DepthScore, InfoLM

Code for layerwise detection of linguistic anomaly paper (ACL 2021)

An end-to-end PyTorch framework for image and video classification

Get the partition that a file belongs and the percentage of space that consumes

Python inverse kinematics for your robot model based on Pinocchio.

a short visualisation script for pyvideo data

CVPR2022 (Oral) - Rethinking Semantic Segmentation: A Prototype View

AutoML library for deep learning

Contrastively Disentangled Sequential Variational Audoencoder

Software associated to AAAI paper "Planning with Biological Neurons and Synapses"

PyTorch Implementation of Daft-Exprt: Robust Prosody Transfer Across Speakers for Expressive Speech Synthesis

The datasets and code of ACL 2021 paper "Aspect-Category-Opinion-Sentiment Quadruple Extraction with Implicit Aspects and Opinions".

Label-Free Model Evaluation with Semi-Structured Dataset Representations

A pytorch reprelication of the model-based reinforcement learning algorithm MBPO