A GPT, made only of MLPs, in Jax

Last update: Sep 27, 2022

Overview

MLP GPT - Jax (wip)

A GPT, made only of MLPs, in Jax. The specific MLP to be used are gMLPs with the Spatial Gating Units.

Install

$ pip install mlp-gpt-jax

Usage

from jax import random, numpy as np
from mlp_gpt_jax import MLPGpt

gpt = MLPGpt(
    num_tokens = 20000,
    dim = 512,
    depth = 6,
    seq_len = 512
)

key    = random.PRNGKey(0)
seq    = random.randint(key, (512,), 0, 20000)

params = gpt.init(key, seq)
logits = gpt.apply(params, seq) # (512, 20000)

Citations

@misc{liu2021pay,
    title   = {Pay Attention to MLPs}, 
    author  = {Hanxiao Liu and Zihang Dai and David R. So and Quoc V. Le},
    year    = {2021},
    eprint  = {2105.08050},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG}
}

Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

Storium GPT-2 Models This is the official repository for the GPT-2 models described in the EMNLP 2020 paper [STORIUM: A Dataset and Evaluation Platfor

27 Dec 20, 2022

Training data extraction on GPT-2

Training data extraction from GPT-2 This repository contains code for extracting training data from GPT-2, following the approach outlined in the foll

62 Dec 7, 2022

Simple Text-Generator with OpenAI gpt-2 Pytorch Implementation

GPT2-Pytorch with Text-Generator Better Language Models and Their Implications Our model, called GPT-2 (a successor to GPT), was trained simply to pre

775 Jan 8, 2023

ChatBot-Pytorch - A GPT-2 ChatBot implemented using Pytorch and Huggingface-transformers

ChatBot-Pytorch A GPT-2 ChatBot implemented using Pytorch and Huggingface-transf

42 Dec 9, 2022

AI-Bot - 一个基于watermelon改造的OpenAI-GPT-2的智能机器人

AI-Bot 一个基于watermelon改造的OpenAI-GPT-2的智能机器人在Binder上直接运行测试目前有两种实现方式 TF2的GPT-2 TF

9 Nov 16, 2022

Building Ellee — A GPT-3 and Computer Vision Powered Talking Robotic Teddy Bear With Human Level Conversation Intelligence

Using an object detection and facial recognition system built on MobileNetSSDV2 and Dlib and running on an NVIDIA Jetson Nano, a GPT-3 model, Google Speech Recognition, Amazon Polly and servo motors, I built Ellee - a robotic teddy bear who can move her head and converse naturally.

24 Oct 26, 2022

MAGMA - a GPT-style multimodal model that can understand any combination of images and language

MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning Authors repo (alphabetical) Constantin (CoEich), Mayukh (Mayukh

331 Jan 3, 2023

FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX.

FedJAX: Federated learning with JAX What is FedJAX? FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX. FedJAX priori

208 Dec 14, 2022

Flax is a neural network ecosystem for JAX that is designed for flexibility.

Flax: A neural network library and ecosystem for JAX designed for flexibility Overview | Quick install | What does Flax look like? | Documentation See

3.9k Jan 2, 2023

Comments

mistake in parameter initialization

floor division will always return 0 :(

https://github.com/lucidrains/mlp-gpt-jax/blob/c8a6d7738562e44d3c0b3018c83ae577f7931e78/mlp_gpt_jax/mlp_gpt_jax.py#L75

opened by guyd1995 1

Releases(0.0.19)

0.0.19(Jun 23, 2021)

Source code(tar.gz)
Source code(zip)
0.0.18(Jun 22, 2021)

Source code(tar.gz)
Source code(zip)
0.0.17(Jun 22, 2021)

Source code(tar.gz)
Source code(zip)
0.0.16(Jun 3, 2021)

Source code(tar.gz)
Source code(zip)
0.0.15(Jun 3, 2021)

Source code(tar.gz)
Source code(zip)
0.0.14(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.12(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.11(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.10(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.9(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.8(May 29, 2021)

Source code(tar.gz)
Source code(zip)
0.0.7(May 27, 2021)

Source code(tar.gz)
Source code(zip)
0.0.6(May 26, 2021)

Source code(tar.gz)
Source code(zip)
0.0.5(May 25, 2021)

Source code(tar.gz)
Source code(zip)
0.0.4(May 23, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(May 22, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(May 21, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(May 21, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention

GitHub Repository

A vanilla 3D face modeling on pose-invariant and multi-lightning image data

3D-Face-Modeling A vanilla 3D face modeling on pose-invariant and multi-lightning image data Table of Contents Background Install Usage Contributing B

1 Mar 12, 2022

Pyeventbus: a publish/subscribe event bus

pyeventbus pyeventbus is a publish/subscribe event bus for Python 2.7. simplifies the communication between python classes decouples event senders and

15 Apr 21, 2022

Veri Setinizi Yolov5 Formatına Dönüştürün

Veri Setinizi Yolov5 Formatına Dönüştürün! Bu Repo da Neler Var? Xml Formatındaki Veri Setini .Txt Formatına Çevirme Xml Formatındaki Dosyaları Silme

4 Aug 22, 2022

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Master Docs License Apache MXNet (incubating) is a deep learning framework designed for both efficiency an

29 Nov 16, 2022

A wrapper around SageMaker ML Lineage Tracking extending ML Lineage to end-to-end ML lifecycles, including additional capabilities around Feature Store groups, queries, and other relevant artifacts.

ML Lineage Helper This library is a wrapper around the SageMaker SDK to support ease of lineage tracking across the ML lifecycle. Lineage artifacts in

12 Nov 01, 2022

ML model to classify between cats and dogs

Cats-and-dogs-classifier This is my first ML model which can classify between cats and dogs. Here the accuracy is around 75%, however , the accuracy c

4 Aug 20, 2021

Object Tracking and Detection Using OpenCV

Object tracking is one such application of computer vision where an object is detected in a video, otherwise interpreted as a set of frames, and the object’s trajectory is estimated. For instance, yo

4 Aug 21, 2022

Time Series Forecasting with Temporal Fusion Transformer in Pytorch

Forecasting with the Temporal Fusion Transformer Multi-horizon forecasting often contains a complex mix of inputs – including static (i.e. time-invari

6 Jan 24, 2022

Build a small, 3 domain internet using Github pages and Wikipedia and construct a crawler to crawl, render, and index.

TechSEO Crawler Build a small, 3 domain internet using Github pages and Wikipedia and construct a crawler to crawl, render, and index. Play with the r

57 Nov 24, 2022

Pytorch implementation of Compressive Transformers, from Deepmind

Compressive Transformer in Pytorch Pytorch implementation of Compressive Transformers, a variant of Transformer-XL with compressed memory for long-ran

118 Dec 01, 2022

CCPD: a diverse and well-annotated dataset for license plate detection and recognition

CCPD (Chinese City Parking Dataset, ECCV) UPdate on 10/03/2019. CCPD Dataset is now updated. We are confident that images in subsets of CCPD is much m

1.8k Dec 30, 2022

Fully Convolutional Networks for Semantic Segmentation by Jonathan Long, Evan Shelhamer, and Trevor Darrell. CVPR 2015 and PAMI 2016.

Fully Convolutional Networks for Semantic Segmentation This is the reference implementation of the models and code for the fully convolutional network

3.2k Jan 08, 2023

A GPT, made only of MLPs, in Jax

Related tags

Overview

MLP GPT - Jax (wip)

Install

Usage

Citations

You might also like...

Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

Training data extraction on GPT-2

Simple Text-Generator with OpenAI gpt-2 Pytorch Implementation

ChatBot-Pytorch - A GPT-2 ChatBot implemented using Pytorch and Huggingface-transformers

AI-Bot - 一个基于watermelon改造的OpenAI-GPT-2的智能机器人

Building Ellee — A GPT-3 and Computer Vision Powered Talking Robotic Teddy Bear With Human Level Conversation Intelligence

MAGMA - a GPT-style multimodal model that can understand any combination of images and language

FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX.

Flax is a neural network ecosystem for JAX that is designed for flexibility.

Comments

mistake in parameter initialization

Releases(0.0.19)

0.0.19(Jun 23, 2021)

0.0.18(Jun 22, 2021)

0.0.17(Jun 22, 2021)

0.0.16(Jun 3, 2021)

0.0.15(Jun 3, 2021)

0.0.14(Jun 2, 2021)

0.0.12(Jun 2, 2021)

0.0.11(Jun 2, 2021)

0.0.10(Jun 2, 2021)

0.0.9(Jun 2, 2021)

0.0.8(May 29, 2021)

0.0.7(May 27, 2021)

0.0.6(May 26, 2021)

0.0.5(May 25, 2021)

0.0.4(May 23, 2021)

0.0.3(May 22, 2021)

0.0.2(May 21, 2021)

0.0.1(May 21, 2021)

Owner

Phil Wang

A vanilla 3D face modeling on pose-invariant and multi-lightning image data

Pyeventbus: a publish/subscribe event bus

Veri Setinizi Yolov5 Formatına Dönüştürün

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

A wrapper around SageMaker ML Lineage Tracking extending ML Lineage to end-to-end ML lifecycles, including additional capabilities around Feature Store groups, queries, and other relevant artifacts.

ML model to classify between cats and dogs

Object Tracking and Detection Using OpenCV

Time Series Forecasting with Temporal Fusion Transformer in Pytorch

Build a small, 3 domain internet using Github pages and Wikipedia and construct a crawler to crawl, render, and index.

Pytorch implementation of Compressive Transformers, from Deepmind

CCPD: a diverse and well-annotated dataset for license plate detection and recognition

Fully Convolutional Networks for Semantic Segmentation by Jonathan Long*, Evan Shelhamer*, and Trevor Darrell. CVPR 2015 and PAMI 2016.

3D-printable hand-strapped keyboard

Complementary Patch for Weakly Supervised Semantic Segmentation, ICCV21 (poster)

A simple python module to generate anchor (aka default/prior) boxes for object detection tasks.

Bottleneck Transformers for Visual Recognition

这是一个yolox-pytorch的源码，可以用于训练自己的模型。

Using pytorch to implement unet network for liver image segmentation.

A minimal solution to hand motion capture from a single color camera at over 100fps. Easy to use, plug to run.

ZeroVL - The official implementation of ZeroVL

Fully Convolutional Networks for Semantic Segmentation by Jonathan Long, Evan Shelhamer, and Trevor Darrell. CVPR 2015 and PAMI 2016.