Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Last update: Nov 22, 2022

Overview

HaloNet - Pytorch

Implementation of the Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones. This repository will only house the attention layer and not much more.

Install

$ pip install halonet-pytorch

Usage

import torch
from halonet_pytorch import HaloAttention

attn = HaloAttention(
    dim = 512,         # dimension of feature map
    block_size = 8,    # neighborhood block size (feature map must be divisible by this)
    halo_size = 4,     # halo size (block receptive field)
    dim_head = 64,     # dimension of each head
    heads = 4          # number of attention heads
).cuda()

fmap = torch.randn(1, 512, 32, 32).cuda()
attn(fmap) # (1, 512, 32, 32)

Citations

@misc{vaswani2021scaling,
    title   = {Scaling Local Self-Attention For Parameter Efficient Visual Backbones}, 
    author  = {Ashish Vaswani and Prajit Ramachandran and Aravind Srinivas and Niki Parmar and Blake Hechtman and Jonathon Shlens},
    year    = {2021},
    eprint  = {2103.12731},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

You might also like...

The official implementation of ELSA: Enhanced Local Self-Attention for Vision Transformer

ELSA: Enhanced Local Self-Attention for Vision Transformer By Jingkai Zhou, Pich

87 Dec 19, 2022

Unofficial PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution

PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution [arXiv 2021].

122 Dec 12, 2022

Official code for "Focal Self-attention for Local-Global Interactions in Vision Transformers"

Focal Transformer This is the official implementation of our Focal Transformer -- "Focal Self-attention for Local-Global Interactions in Vision Transf

486 Dec 20, 2022

A PyTorch implementation of " EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks."

EfficientNet A PyTorch implementation of EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. [arxiv] [Official TF Repo] Implemen

298 Dec 10, 2022

Official pytorch implementation of "Scaling-up Disentanglement for Image Translation", ICCV 2021.

41 Nov 29, 2022

Implementation of "Scaled-YOLOv4: Scaling Cross Stage Partial Network" using PyTorch framwork.

YOLOv4-large This is the implementation of "Scaled-YOLOv4: Scaling Cross Stage Partial Network" using PyTorch framwork. YOLOv4-CSP YOLOv4-tiny YOLOv4-

2k Jan 2, 2023

Official Pytorch Implementation of Relational Self-Attention: What's Missing in Attention for Video Understanding

Relational Self-Attention: What's Missing in Attention for Video Understanding This repository is the official implementation of "Relational Self-Atte

43 Dec 7, 2022

Image-Scaling Attacks and Defenses

Image-Scaling Attacks & Defenses This repository belongs to our publication: Erwin Quiring, David Klein, Daniel Arp, Martin Johns and Konrad Rieck. Ad

163 Nov 21, 2022

For auto aligning, cropping, and scaling HR and LR images for training image based neural networks

ImgAlign For auto aligning, cropping, and scaling HR and LR images for training image based neural networks Usage Make sure OpenCV is installed, 'pip

15 Dec 4, 2022

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Related tags

Overview

HaloNet - Pytorch

Install

Usage

Citations

You might also like...

The official implementation of ELSA: Enhanced Local Self-Attention for Vision Transformer

Unofficial PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution

Official code for "Focal Self-attention for Local-Global Interactions in Vision Transformers"

A PyTorch implementation of " EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks."

Official pytorch implementation of "Scaling-up Disentanglement for Image Translation", ICCV 2021.

Implementation of "Scaled-YOLOv4: Scaling Cross Stage Partial Network" using PyTorch framwork.

Official Pytorch Implementation of Relational Self-Attention: What's Missing in Attention for Video Understanding

Image-Scaling Attacks and Defenses

For auto aligning, cropping, and scaling HR and LR images for training image based neural networks

Releases(0.0.4)

0.0.4(Mar 24, 2021)

0.0.3(Mar 24, 2021)

0.0.2(Mar 24, 2021)

0.0.1(Mar 24, 2021)

Owner

Phil Wang

Code for EMNLP 2021 paper: "Learning Implicit Sentiment in Aspect-based Sentiment Analysis with Supervised Contrastive Pre-Training"

The FIRST GANs-based omics-to-omics translation framework

Container : Context Aggregation Network

Cobalt Strike teamserver detection.

Codes for TIM2021 paper "Anchor-Based Spatio-Temporal Attention 3-D Convolutional Networks for Dynamic 3-D Point Cloud Sequences"

NaturalCC is a sequence modeling toolkit that allows researchers and developers to train custom models

A tf.keras implementation of Facebook AI's MadGrad optimization algorithm

Deep Inside Convolutional Networks - This is a caffe implementation to visualize the learnt model

Numbering permanent and deciduous teeth via deep instance segmentation in panoramic X-rays

Python code to fuse multiple RGB-D images into a TSDF voxel volume.

Notepy is a full-featured Notepad Python app

Testability-Aware Low Power Controller Design with Evolutionary Learning, ITC2021

Computer vision - fun segmentation experience using classic and deep tools :)

Pyramid Pooling Transformer for Scene Understanding

Roach: End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

CLIPImageClassifier wraps clip image model from transformers

In this repo we reproduce and extend results of Learning in High Dimension Always Amounts to Extrapolation by Balestriero et al. 2021

DL & CV-based indicator toolset for the vehicle drivers via live dash-cam footage.

Final term project for Bayesian Machine Learning Lecture (XAI-623)

Creating predictive checklists from data using integer programming.