Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

Last update: Dec 25, 2022

Overview

Hourglass Transformer - Pytorch (wip)

Implementation of Hourglass Transformer, in Pytorch. It will also contain some of my own ideas about how to make it work better.

Citations

@misc{nawrot2021hierarchical,
    title   = {Hierarchical Transformers Are More Efficient Language Models}, 
    author  = {Piotr Nawrot and Szymon Tworkowski and Michał Tyrolski and Łukasz Kaiser and Yuhuai Wu and Christian Szegedy and Henryk Michalewski},
    year    = {2021},
    eprint  = {2110.13711},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG}
}

You might also like...

A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Larger Google Sat2Map dataset This dataset extends the aerial ⟷ Maps dataset used in pix2pix (Isola et al., CVPR17). The provide script download_sat2m

34 Dec 28, 2022

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

GD-Thief Red Team tool for exfiltrating files from a target's Google Drive that you(the attacker) has access to, via the Google Drive API. This includ

39 Dec 27, 2022

Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

google-drive-to-sqlite Create a SQLite database containing metadata from Google

140 Dec 4, 2022

BasicRL: easy and fundamental codes for deep reinforcement learning。It is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up.

BasicRL: easy and fundamental codes for deep reinforcement learning BasicRL is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up. It is

12 Apr 28, 2022

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Swin Transformer for Object Detection This repo contains the supported code and configuration files to reproduce object detection results of Swin Tran

1.4k Dec 30, 2022

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

ImageProcessingTransformer Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

61 Jan 1, 2023

Transformer - Transformer in PyTorch

Transformer 完成进度 Embeddings and PositionalEncoding with example. MultiHeadAttent

1 Jan 6, 2022

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Deep Daze mist over green hills shattered plates on the grass cosmic love and attention a time traveler in the crowd life during the plague meditative

4.4k Jan 3, 2023

Plug-n-Play Reinforcement Learning in Python with OpenAI Gym and JAX

coax is built on top of JAX, but it doesn't have an explicit dependence on the jax python package. The reason is that your version of jaxlib will depend on your CUDA version.

128 Dec 27, 2022

Comments

No option for rotary positional embeddings

For the ImageNet64 generation experiments, the original authors used rotary positional embeddings. I think it would be worth adding this feature to the repository.

I can prepare a pull request that solves this~

opened by vvvm23 4
Issue with an example

@lucidrains Just Awesome to see this super fast launch !

The third example under "For Images...." throws the following error .,

Then , when I gave the inputs with num_tokens = 20000, max_seq_len = 1024, I got the following runtime error

opened by nsankar 1

Releases(0.0.6)

0.0.6(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.5(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.4(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need

GitHub Repository

Roadmap to becoming a machine learning engineer in 2020

Roadmap to becoming a machine learning engineer in 2020, inspired by web-developer-roadmap.

1.7k Dec 29, 2022

SporeAgent: Reinforced Scene-level Plausibility for Object Pose Refinement

SporeAgent: Reinforced Scene-level Plausibility for Object Pose Refinement This repository implements the approach described in SporeAgent: Reinforced

5 Jan 02, 2023

Code accompanying paper: Meta-Learning to Improve Pre-Training

Meta-Learning to Improve Pre-Training This folder contains code to run experiments in the paper Meta-Learning to Improve Pre-Training, NeurIPS 2021. P

28 Dec 31, 2022

An NVDA add-on to split screen reader and audio from other programs to different sound channels

An NVDA add-on to split screen reader and audio from other programs to different sound channels (add-on idea credit: Tony Malykh)

7 Dec 25, 2022

Count the MACs / FLOPs of your PyTorch model.

THOP: PyTorch-OpCounter How to install pip install thop (now continously intergrated on Github actions) OR pip install --upgrade git+https://github.co

3.9k Dec 29, 2022

A library for uncertainty quantification based on PyTorch

Torchuq [logo here] TorchUQ is an extensive library for uncertainty quantification (UQ) based on pytorch. TorchUQ currently supports 10 representation

96 Dec 12, 2022

Pose estimation with MoveNet Lightning

Pose Estimation With MoveNet Lightning MoveNet is the TensorFlow pre-trained model that identifies 17 different key points of the human body. It is th

2 Jan 04, 2022

Bachelor's Thesis in Computer Science: Privacy-Preserving Federated Learning Applied to Decentralized Data

federated is the source code for the Bachelor's Thesis Privacy-Preserving Federated Learning Applied to Decentralized Data (Spring 2021, NTNU) Federat

25 Nov 30, 2022

Hand tracking demo for DIY Smart Glasses with a remote computer doing the work

CameraStream This is a demonstration that streams the image from smartglasses to a pc, does the hand recognition on the remote pc and streams the proc

20 Oct 13, 2022

Official repository for "Restormer: Efficient Transformer for High-Resolution Image Restoration". SOTA results for single-image motion deblurring, image deraining, image denoising (synthetic and real data), and dual-pixel defocus deblurring.

Restormer: Efficient Transformer for High-Resolution Image Restoration Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan,

906 Dec 30, 2022

This repository contains the data and code for the paper "Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors" ([email protected])

GP-VAE This repository provides datasets and code for preprocessing, training and testing models for the paper: Diverse Text Generation via Variationa

18 Dec 29, 2022

Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

Related tags

Overview

Hourglass Transformer - Pytorch (wip)

Citations

You might also like...

A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

BasicRL: easy and fundamental codes for deep reinforcement learning。It is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up.

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

Transformer - Transformer in PyTorch

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Plug-n-Play Reinforcement Learning in Python with OpenAI Gym and JAX

Comments

No option for rotary positional embeddings

Issue with an example

Releases(0.0.6)

0.0.6(Nov 9, 2021)

0.0.5(Nov 9, 2021)

0.0.4(Nov 9, 2021)

0.0.3(Nov 9, 2021)

0.0.2(Nov 9, 2021)

0.0.1(Nov 9, 2021)

Owner

Phil Wang

Roadmap to becoming a machine learning engineer in 2020

SporeAgent: Reinforced Scene-level Plausibility for Object Pose Refinement

Code accompanying paper: Meta-Learning to Improve Pre-Training

An NVDA add-on to split screen reader and audio from other programs to different sound channels

Count the MACs / FLOPs of your PyTorch model.

A library for uncertainty quantification based on PyTorch

Pose estimation with MoveNet Lightning

Bachelor's Thesis in Computer Science: Privacy-Preserving Federated Learning Applied to Decentralized Data

Hand tracking demo for DIY Smart Glasses with a remote computer doing the work

Official repository for "Restormer: Efficient Transformer for High-Resolution Image Restoration". SOTA results for single-image motion deblurring, image deraining, image denoising (synthetic and real data), and dual-pixel defocus deblurring.

This repository contains the data and code for the paper "Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors" ([email protected])

A Survey on Deep Learning Technique for Video Segmentation

A small library of 3D related utilities used in my research.

CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

3D position tracking for soccer players with multi-camera videos

Retinal vessel segmentation based on GT-UNet

Beancount-mercury - Beancount importer for Mercury Startup Checking

HackBMU-5.0-Team-Ctrl-Alt-Elite - HackBMU 5.0 Team Ctrl Alt Elite

A forwarding MPI implementation that can use any other MPI implementation via an MPI ABI

Jiminy Cricket Environment (NeurIPS 2021)