Sequence-to-Sequence learning using PyTorch

Last update: Nov 17, 2022

Overview

Seq2Seq in PyTorch

This is a complete suite for training sequence-to-sequence models in PyTorch. It consists of several models and code to both train and infer using them.

Using this code you can train:

Neural-machine-translation (NMT) models
Language models
Image to caption generation
Skip-thought sentence representations
And more...

Installation

git clone --recursive https://github.com/eladhoffer/seq2seq.pytorch
cd seq2seq.pytorch; python setup.py develop

Models

Models currently available:

Simple Seq2Seq recurrent model
Recurrent Seq2Seq with attentional decoder
Google neural machine translation (GNMT) recurrent model
Transformer - attention-only model from "Attention Is All You Need"

Datasets

Datasets currently available:

WMT16
WMT17
OpenSubtitles 2016
COCO image captions
Conceptual captions

All datasets can be tokenized using 3 available segmentation methods:

Character based segmentation
Word based segmentation
Byte-pair-encoding (BPE) as suggested by bpe with selectable number of tokens.

After choosing a tokenization method, a vocabulary will be generated and saved for future inference.

Training methods

The models can be trained using several methods:

Basic Seq2Seq - given encoded sequence, generate (decode) output sequence. Training is done with teacher-forcing.
Multi Seq2Seq - where several tasks (such as multiple languages) are trained simultaneously by using the data sequences as both input to the encoder and output for decoder.
Image2Seq - used to train image to caption generators.

Usage

Example training scripts are available in scripts folder. Inference examples are available in examples folder.

example for training a transformer on WMT16 according to original paper regime:

DATASET=${1:-"WMT16_de_en"}
DATASET_DIR=${2:-"./data/wmt16_de_en"}
OUTPUT_DIR=${3:-"./results"}

WARMUP="4000"
LR0="512**(-0.5)"

python main.py \
  --save transformer \
  --dataset ${DATASET} \
  --dataset-dir ${DATASET_DIR} \
  --results-dir ${OUTPUT_DIR} \
  --model Transformer \
  --model-config "{'num_layers': 6, 'hidden_size': 512, 'num_heads': 8, 'inner_linear': 2048}" \
  --data-config "{'moses_pretok': True, 'tokenization':'bpe', 'num_symbols':32000, 'shared_vocab':True}" \
  --b 128 \
  --max-length 100 \
  --device-ids 0 \
  --label-smoothing 0.1 \
  --trainer Seq2SeqTrainer \
  --optimization-config "[{'step_lambda':
                          \"lambda t: { \
                              'optimizer': 'Adam', \
                              'lr': ${LR0} * min(t ** -0.5, t * ${WARMUP} ** -1.5), \
                              'betas': (0.9, 0.98), 'eps':1e-9}\"
                          }]"

example for training attentional LSTM based model with 3 layers in both encoder and decoder:

python main.py \
  --save de_en_wmt17 \
  --dataset ${DATASET} \
  --dataset-dir ${DATASET_DIR} \
  --results-dir ${OUTPUT_DIR} \
  --model RecurrentAttentionSeq2Seq \
  --model-config "{'hidden_size': 512, 'dropout': 0.2, \
                   'tie_embedding': True, 'transfer_hidden': False, \
                   'encoder': {'num_layers': 3, 'bidirectional': True, 'num_bidirectional': 1, 'context_transform': 512}, \
                   'decoder': {'num_layers': 3, 'concat_attention': True,\
                               'attention': {'mode': 'dot_prod', 'dropout': 0, 'output_transform': True, 'output_nonlinearity': 'relu'}}}" \
  --data-config "{'moses_pretok': True, 'tokenization':'bpe', 'num_symbols':32000, 'shared_vocab':True}" \
  --b 128 \
  --max-length 80 \
  --device-ids 0 \
  --trainer Seq2SeqTrainer \
  --optimization-config "[{'epoch': 0, 'optimizer': 'Adam', 'lr': 1e-3},
                          {'epoch': 6, 'lr': 5e-4},
                          {'epoch': 8, 'lr':1e-4},
                          {'epoch': 10, 'lr': 5e-5},
                          {'epoch': 12, 'lr': 1e-5}]" \

Sequence-to-Sequence learning using PyTorch

Related tags

Overview

Seq2Seq in PyTorch

Installation

Models

Datasets

Training methods

Usage

Owner

Elad Hoffer

A library of extension and helper modules for Python's data analysis and machine learning libraries.

Easy way to add GoogleMaps to Flask applications. maintainer: @getcake

Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

The Pytorch code of "Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classification", CVPR 2022 (Oral).

Face Identity Disentanglement via Latent Space Mapping [SIGGRAPH ASIA 2020]

Information Gain Filtration (IGF) is a method for filtering domain-specific data during language model finetuning. IGF shows significant improvements over baseline fine-tuning without data filtration.

Jetson Nano-based smart camera system that measures crowd face mask usage in real-time.

HTSeq is a Python library to facilitate processing and analysis of data from high-throughput sequencing (HTS) experiments.

SimBERT升级版（SimBERTv2）！

RRxIO - Robust Radar Visual/Thermal Inertial Odometry: Robust and accurate state estimation even in challenging visual conditions.

Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning

Deep learning image registration library for PyTorch

A model to classify a piece of news as REAL or FAKE

The Adapter-Bot: All-In-One Controllable Conversational Model

ESL: Event-based Structured Light

Music source separation is a task to separate audio recordings into individual sources

Pytorch implementation for ACMMM2021 paper "I2V-GAN: Unpaired Infrared-to-Visible Video Translation".

Interacting Two-Hand 3D Pose and Shape Reconstruction from Single Color Image (ICCV 2021)

Based on Stockfish neural network(similar to LcZero)

NeuralCompression is a Python repository dedicated to research of neural networks that compress data