Reproduce results and replicate training fo T0 (Multitask Prompted Training Enables Zero-Shot Task Generalization)

Last update: Dec 27, 2022

Related tags

Overview

T-Zero

This repository serves primarily as codebase and instructions for training, evaluation and inference of T0.

T0 is the model developed in Multitask Prompted Training Enables Zero-Shot Task Generalization. In this paper, we demonstrate that massive multitask prompted fine-tuning is extremely effective to obtain task zero-shot generalization. T0 outperforms or matches GPT-3 while being 16x smaller.

While the codebase in this repository mainly reproduces and replicates the training and evaluation of T0, it will be useful for future research on massively multitask fine-tuning.

Training: reproducing (or replicating) the massively multitask fine-tuning
Evaluation: reproducing the main results reported in the paper
Inference: running inference with T0

Citation

If you find this resource useful, please cite the paper introducing T0:

@misc{sanh2021multitask,
      title={Multitask Prompted Training Enables Zero-Shot Task Generalization},
      author={Victor Sanh and Albert Webson and Colin Raffel and Stephen H. Bach and Lintang Sutawika and Zaid Alyafeai and Antoine Chaffin and Arnaud Stiegler and Teven Le Scao and Arun Raja and Manan Dey and M Saiful Bari and Canwen Xu and Urmish Thakker and Shanya Sharma Sharma and Eliza Szczechla and Taewoon Kim and Gunjan Chhablani and Nihal Nayak and Debajyoti Datta and Jonathan Chang and Mike Tian-Jian Jiang and Han Wang and Matteo Manica and Sheng Shen and Zheng Xin Yong and Harshit Pandey and Rachel Bawden and Thomas Wang and Trishala Neeraj and Jos Rozen and Abheesht Sharma and Andrea Santilli and Thibault Fevry and Jason Alan Fries and Ryan Teehan and Stella Biderman and Leo Gao and Tali Bers and Thomas Wolf and Alexander M. Rush},
      year={2021},
      eprint={2110.08207},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Reproduce results and replicate training fo T0 (Multitask Prompted Training Enables Zero-Shot Task Generalization)

Related tags

Overview

T-Zero

Contents

Citation

Owner

BigScience Workshop

An implementation for `Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction`

A lossless neural compression framework built on top of JAX.

Pytorch Implementation of "Desigining Network Design Spaces", Radosavovic et al. CVPR 2020.

Codes for the ICCV'21 paper "FREE: Feature Refinement for Generalized Zero-Shot Learning"

iBOT: Image BERT Pre-Training with Online Tokenizer

Semi-Supervised Learning with Ladder Networks in Keras. Get 98% test accuracy on MNIST with just 100 labeled examples !

Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pretrained models.

A simple rest api that classifies pneumonia infection weather it is Normal, Pneumonia Virus or Pneumonia Bacteria from a chest-x-ray image.

PyTorch implementation of Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Music source separation is a task to separate audio recordings into individual sources

(Preprint) Official PyTorch implementation of "How Do Vision Transformers Work?"

Implementation of the ICCV'21 paper Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases

Solving SMPL/MANO parameters from keypoint coordinates.

Official implementation of "Articulation Aware Canonical Surface Mapping"

AMTML-KD: Adaptive Multi-teacher Multi-level Knowledge Distillation

[CVPR 2022 Oral] Crafting Better Contrastive Views for Siamese Representation Learning

Scalable and Elastic Deep Reinforcement Learning Using PyTorch. Please star. 🔥

Code for DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning

NeROIC: Neural Object Capture and Rendering from Online Image Collections

This repository contains code, network definitions and pre-trained models for working on remote sensing images using deep learning