The full training script for Enformer (Tensorflow Sonnet) on TPU clusters

Last update: Oct 19, 2022

Overview

Enformer TPU training script (wip)

The full training script for Enformer (Tensorflow Sonnet) on TPU clusters, in an effort to migrate the model to pytorch.

This was pieced together from the Deepmind Enformer repository, the colab training notebook, as well as Basenji sequence augmentation code

It accounts for:

distributed TPU training
distributed datasets
distributed validation
gradient clipping
cross replica batchnorms
dataset augmentation

Training takes about 3 days on v3-64

Todo

fix script for differences in sequence length in basenji training data, which is ~130k vs ~190k bp as in paper

Citations

@article {Avsec2021.04.07.438649,
    author  = {Avsec, {\v Z}iga and Agarwal, Vikram and Visentin, Daniel and Ledsam, Joseph R. and Grabska-Barwinska, Agnieszka and Taylor, Kyle R. and Assael, Yannis and Jumper, John and Kohli, Pushmeet and Kelley, David R.},
    title   = {Effective gene expression prediction from sequence by integrating long-range interactions},
    elocation-id = {2021.04.07.438649},
    year    = {2021},
    doi     = {10.1101/2021.04.07.438649},
    publisher = {Cold Spring Harbor Laboratory},
    URL     = {https://www.biorxiv.org/content/early/2021/04/08/2021.04.07.438649},
    eprint  = {https://www.biorxiv.org/content/early/2021/04/08/2021.04.07.438649.full.pdf},
    journal = {bioRxiv}
}

The full training script for Enformer (Tensorflow Sonnet) on TPU clusters

Related tags

Overview

Enformer TPU training script (wip)

Todo

Citations

Owner

Phil Wang

Official PyTorch Implementation of Mask-aware IoU and maYOLACT Detector [BMVC2021]

Companion code for the paper "An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence" (NeurIPS 2021)

Multiview 3D object detection on MultiviewC dataset through moft3d.

Expand human face editing via Global Direction of StyleCLIP, especially to maintain similarity during editing.

Deep Reinforcement Learning based autonomous navigation for quadcopters using PPO algorithm.

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

TICC is a python solver for efficiently segmenting and clustering a multivariate time series

Bottom-up Human Pose Estimation

一个运行在 𝐞𝐥𝐞𝐜𝐕𝟐𝐏 或 𝐪𝐢𝐧𝐠𝐥𝐨𝐧𝐠 等定时面板的签到项目

Model-free Vehicle Tracking and State Estimation in Point Cloud Sequences

An end-to-end implementation of intent prediction with Metaflow and other cool tools

Image-to-image translation with conditional adversarial nets

PyTorch implementation for paper StARformer: Transformer with State-Action-Reward Representations.

ProMP: Proximal Meta-Policy Search

Patches desktop steam to look like the new steamdeck ui.

2D&3D human pose estimation

Proof-Of-Concept Piano-Drums Music AI Model/Implementation

A tool to analyze leveraged liquidity mining and find optimal option combination for hedging.

prior-based-losses-for-medical-image-segmentation

Semi-Supervised Graph Prototypical Networks for Hyperspectral Image Classification, IGARSS, 2021.