Implementation of Fast Transformer in Pytorch

Last update: Dec 27, 2022

Overview

Fast Transformer - Pytorch

Implementation of Fast Transformer in Pytorch. This only work as an encoder.

Install

$ pip install fast-transformer-pytorch

Usage

import torch
from fast_transformer_pytorch import FastTransformer

model = FastTransformer(
    num_tokens = 20000,
    dim = 512,
    depth = 2,
    max_seq_len = 4096,
    absolute_pos_emb = True   # default uses relative positional encoding, but if that isn't working, then turn on absolute positional embedding by setting this to True
)

x = torch.randint(0, 20000, (1, 4096))
mask = torch.ones(1, 4096).bool()

logits = model(x, mask = mask) # (1, 4096, 20000)

Citations

@misc{wu2021fastformer,
    title   = {Fastformer: Additive Attention is All You Need}, 
    author  = {Chuhan Wu and Fangzhao Wu and Tao Qi and Yongfeng Huang},
    year    = {2021},
    eprint  = {2108.09084},
    archivePrefix = {arXiv},
    primaryClass = {cs.CL}
}

You might also like...

Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time"

FLASH - Pytorch Implementation of the Transformer variant proposed in the paper Transformer Quality in Linear Time Install $ pip install FLASH-pytorch

209 Dec 28, 2022

FAST-RIR: FAST NEURAL DIFFUSE ROOM IMPULSE RESPONSE GENERATOR

This is the official implementation of our neural-network-based fast diffuse room impulse response generator (FAST-RIR) for generating room impulse responses (RIRs) for a given acoustic environment.

89 Dec 22, 2022

Fit Fast, Explain Fast

FastExplain Fit Fast, Explain Fast Installing pip install fast-explain About FastExplain FastExplain provides an out-of-the-box tool for analysts to

8 Dec 15, 2022

Official implement of Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer

Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer This repository contains the PyTorch code for Evo-ViT. This work proposes a slow-fas

53 Dec 5, 2022

GeoTransformer - Geometric Transformer for Fast and Robust Point Cloud Registration

Geometric Transformer for Fast and Robust Point Cloud Registration PyTorch imple

220 Jan 5, 2023

Official source code of Fast Point Transformer, CVPR 2022

Fast Point Transformer Project Page | Paper This repository contains the official source code and data for our paper: Fast Point Transformer Chunghyun

182 Dec 23, 2022

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Episodic Transformers (E.T.) Episodic Transformer for Vision-and-Language Navigation Alexander Pashevich, Cordelia Schmid, Chen Sun Episodic Transform

62 Dec 24, 2022

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped

CSWin-Transformer This repo is the official implementation of "CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows". Th

409 Jan 6, 2023

nnFormer: Interleaved Transformer for Volumetric Segmentation Code for paper "nnFormer: Interleaved Transformer for Volumetric Segmentation "

nnFormer: Interleaved Transformer for Volumetric Segmentation Code for paper "nnFormer: Interleaved Transformer for Volumetric Segmentation ". Please

610 Dec 28, 2022

Releases(0.0.4)

0.0.4(Aug 25, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(Aug 24, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(Aug 23, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(Aug 23, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need

GitHub Repository

Neural Logic Inductive Learning

Neural Logic Inductive Learning This is the implementation of the Neural Logic Inductive Learning model (NLIL) proposed in the ICLR 2020 paper: Learn

36 Nov 28, 2022

This repository contains the source code and data for reproducing results of Deep Continuous Clustering paper

Deep Continuous Clustering Introduction This is a Pytorch implementation of the DCC algorithms presented in the following paper (paper): Sohil Atul Sh

197 Nov 29, 2022

Codebase for Diffusion Models Beat GANS on Image Synthesis.

128 Dec 02, 2022

Implements MLP-Mixer: An all-MLP Architecture for Vision.

MLP-Mixer-CIFAR10 This repository implements MLP-Mixer as proposed in MLP-Mixer: An all-MLP Architecture for Vision. The paper introduces an all MLP (

51 Jan 04, 2023

Import Python modules from dicts and JSON formatted documents.

Paker Paker is module for importing Python packages/modules from dictionaries and JSON formatted documents. It was inspired by httpimporter. Important

1 Sep 07, 2022

OCRA (Object-Centric Recurrent Attention) source code

OCRA (Object-Centric Recurrent Attention) source code Hossein Adeli and Seoyoung Ahn Please cite this article if you find this repository useful: For

2 Jun 18, 2022

Code for layerwise detection of linguistic anomaly paper (ACL 2021)

Layerwise Anomaly This repository contains the source code and data for our ACL 2021 paper: "How is BERT surprised? Layerwise detection of linguistic

6 Dec 07, 2022

TensorFlow tutorials and best practices.

Effective TensorFlow 2 Table of Contents Part I: TensorFlow 2 Fundamentals TensorFlow 2 Basics Broadcasting the good and the ugly Take advantage of th

8.7k Dec 31, 2022

This is an unofficial implementation of the paper “Student-Teacher Feature Pyramid Matching for Unsupervised Anomaly Detection”.

32 Oct 26, 2022

Implementation of Fast Transformer in Pytorch

Related tags

Overview

Fast Transformer - Pytorch

Install

Usage

Citations

You might also like...

Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time"

FAST-RIR: FAST NEURAL DIFFUSE ROOM IMPULSE RESPONSE GENERATOR

Fit Fast, Explain Fast

Official implement of Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer

GeoTransformer - Geometric Transformer for Fast and Robust Point Cloud Registration

Official source code of Fast Point Transformer, CVPR 2022

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped

nnFormer: Interleaved Transformer for Volumetric Segmentation Code for paper "nnFormer: Interleaved Transformer for Volumetric Segmentation "

Releases(0.0.4)

0.0.4(Aug 25, 2021)

0.0.3(Aug 24, 2021)

0.0.2(Aug 23, 2021)

0.0.1(Aug 23, 2021)

Owner

Phil Wang

Neural Logic Inductive Learning

This repository contains the source code and data for reproducing results of Deep Continuous Clustering paper

Codebase for Diffusion Models Beat GANS on Image Synthesis.

Implements MLP-Mixer: An all-MLP Architecture for Vision.

Import Python modules from dicts and JSON formatted documents.

OCRA (Object-Centric Recurrent Attention) source code

Code for layerwise detection of linguistic anomaly paper (ACL 2021)

TensorFlow tutorials and best practices.

This is an unofficial implementation of the paper “Student-Teacher Feature Pyramid Matching for Unsupervised Anomaly Detection”.

Demo code for ICCV 2021 paper "Sensor-Guided Optical Flow"

Acute ischemic stroke dataset

Training data extraction on GPT-2

A scanpy extension to analyse single-cell TCR and BCR data.

Machine learning Bot detection technique, based on United States election dataset

Pytorch implementation of Feature Pyramid Network (FPN) for Object Detection

Learning the Beauty in Songs: Neural Singing Voice Beautifier; ACL 2022 (Main conference); Official code

Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter

Convert BART models to ONNX with quantization. 3X reduction in size, and upto 3X boost in inference speed

Self-attentive task GAN for space domain awareness data augmentation.

Neural Nano-Optics for High-quality Thin Lens Imaging