Fast Transformer - Pytorch
Implementation of Fast Transformer in Pytorch. This only work as an encoder.
Install
$ pip install fast-transformer-pytorch
Usage
import torch
from fast_transformer_pytorch import FastTransformer
model = FastTransformer(
num_tokens = 20000,
dim = 512,
depth = 2,
max_seq_len = 4096,
absolute_pos_emb = True # default uses relative positional encoding, but if that isn't working, then turn on absolute positional embedding by setting this to True
)
x = torch.randint(0, 20000, (1, 4096))
mask = torch.ones(1, 4096).bool()
logits = model(x, mask = mask) # (1, 4096, 20000)
Citations
@misc{wu2021fastformer,
title = {Fastformer: Additive Attention is All You Need},
author = {Chuhan Wu and Fangzhao Wu and Tao Qi and Yongfeng Huang},
year = {2021},
eprint = {2108.09084},
archivePrefix = {arXiv},
primaryClass = {cs.CL}
}
4 Jan 20, 2022
633 Dec 21, 2022
97 Nov 21, 2022
24.1k Jan 5, 2023
6.2k Dec 31, 2022
1.6k Dec 27, 2022
45 Nov 29, 2022
3k Jan 05, 2023
8 Oct 13, 2022
2 Oct 21, 2022
3.7k Jan 02, 2023
17 Oct 18, 2022
20 Jul 14, 2022
27 Dec 22, 2022
14 Jul 23, 2022
42 Jul 09, 2022
12 Nov 11, 2022
5.1k Jan 09, 2023
1 Nov 18, 2021
3 Jun 13, 2022
31 Oct 31, 2022
54 Dec 03, 2022
39 Aug 03, 2021
2 Jul 05, 2022
5 Dec 08, 2021
464 Jan 04, 2023