Long Expressive Memory (LEM)

Last update: Dec 17, 2022

Overview

Long Expressive Memory for Sequence Modeling

This repository contains the implementation to reproduce the numerical experiments of the paper Long Expressive Memory for Sequence Modeling

Requirements

To setup the environment, install the requirements using python 3.7.10 with

pip install -r requirements.txt

Experiments

This repository contains the codes to reproduce the results of the following experiments for the proposed LEM:

Adding Task
EigenWorms
FitzHugh-Nagumo
Google12 (V2)
Heart rate prediction
noisy CIFAR-10
Permuted Sequential MNIST
PennTree Bank (word- and char-level)
Sequential MNIST

All datasets get either synthetically generated or downloaded automatically, providing a full data pipline for each experiment.
Note that the PTB experiments are based on the open-source tensorflow project LAnguage Modelling Benchmarks (lamb) for tuning and testing Tensorflow language models. In order to reproduce these experiments, please follow the steps explained in the README of the PTB directory.

Results

The results of LEM for each of the experiments are:

Experiment	Result
EigenWorms	92.3% mean test accuracy
FitzHugh-Nagumo	0.002 test L^2 loss
Google12 (V2)	95.7% test accuracy
Heart rate prediction	0.85 test L^2 loss
noisy CIFAR-10	60.5% test accuracy
Permuted Sequential MNIST	96.6% test accuracy
PennTree Bank char-level (single layer)	1.25 test bits-per-character
PennTree Bank word-level (single layer)	72.8 test perplexity
Sequential MNIST	99.5% test accuracy

Long Expressive Memory (LEM)

Related tags

Overview

Long Expressive Memory for Sequence Modeling

Requirements

Experiments

Results

Owner

Konstantin Rusch

JAXDL: JAX (Flax) Deep Learning Library

PyTorch implementation of MSBG hearing loss model and MBSTOI intelligibility metric

Image based Human Fall Detection

Codes for TIM2021 paper "Anchor-Based Spatio-Temporal Attention 3-D Convolutional Networks for Dynamic 3-D Point Cloud Sequences"

Code for Learning to Segment The Tail (LST)

A PyTorch implementation of Sharpness-Aware Minimization for Efficiently Improving Generalization

Deep Q-learning for playing chrome dino game

Ultra-lightweight human body posture key point CNN model. ModelSize:2.3MB HUAWEI P40 NCNN benchmark: 6ms/img,

Code image classification of MNIST dataset using different architectures: simple linear NN, autoencoder, and highway network

Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving

classification task on dataset-CIFAR10,by using Tensorflow/keras

generate-2D-quadrilateral-mesh-with-neural-networks-and-tree-search

An Implementation of Transformer in Transformer in TensorFlow for image classification, attention inside local patches

Text-Based Ideal Points

Official implement of Paper：A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sening images

ML-Decoder: Scalable and Versatile Classification Head

Easily pull telemetry data and create beautiful visualizations for analysis.

History Aware Multimodal Transformer for Vision-and-Language Navigation

Collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning.

Official implementation of AAAI-21 paper "Label Confusion Learning to Enhance Text Classification Models"