Fast Transformer 
This repo implements Fastformer: Additive Attention Can Be All You Need by Wu et al. in TensorFlow. Fast Transformer is a Transformer variant based on additive attention that can handle long sequences efficiently with linear complexity. Fastformer is much more efficient than many existing Transformer models and can meanwhile achieve comparable or even better long text modeling performance.
Installation
Run the following to install:
pip install fast-transformer
Developing fast-transformer
To install fast-transformer, along with tools you need to develop and test, run the following in your virtualenv:
git clone https://github.com/Rishit-dagli/Fast-Transformer.git
# or clone your own fork
cd fast-transformer
pip install -e .[dev]
Usage
import tensorflow as tf
from fast_transformer import FastTransformer
mask = tf.ones([1, 4096], dtype=tf.bool)
model = FastTransformer(
num_tokens = 20000,
dim = 512,
depth = 2,
max_seq_len = 4096,
absolute_pos_emb = True, # Absolute positional embeddings
mask = mask
)
x = tf.experimental.numpy.random.randint(0, 20000, (1, 4096))
logits = model(x) # (1, 4096, 20000)
Want to Contribute
🙋♂️
?
Awesome! If you want to contribute to this project, you're always welcome! See Contributing Guidelines. You can also take a look at open issues for getting more information about current or upcoming tasks.
Want to discuss?
💬
Have any questions, doubts or want to present your opinions, views? You're always welcome. You can start discussions.
Citation
@misc{wu2021fastformer,
title = {Fastformer: Additive Attention is All You Need},
author = {Chuhan Wu and Fangzhao Wu and Tao Qi and Yongfeng Huang},
year = {2021},
eprint = {2108.09084},
archivePrefix = {arXiv},
primaryClass = {cs.CL}
}
Yannic Kilcher's video was super helpful while building this.
License
Copyright 2020 Rishit Dagli
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
11 Mar 14, 2022
13 Dec 23, 2022
59 Sep 25, 2022
134 Dec 30, 2022
6.1k Jan 5, 2023
41 Jan 7, 2023
1 Nov 15, 2021
79 Dec 28, 2022
247 Dec 28, 2022
264 Dec 30, 2022
3 Nov 12, 2022
1 Jan 25, 2022
[email protected]">
66 Nov 23, 2022
5 Oct 20, 2022
13 May 11, 2022
27 Jul 20, 2022
43 Dec 14, 2022
16 Sep 12, 2022
55 Nov 29, 2022
12.2k Jan 03, 2023
2 Jan 07, 2022
29 Dec 14, 2022
157 Dec 26, 2022
2 Aug 20, 2022
22 Oct 13, 2022
24 Apr 15, 2021
15 Nov 28, 2022
2 Oct 20, 2022
6 Aug 24, 2022