Explore different way to mix speech model(wav2vec2, hubert) and nlp model(BART,T5,GPT) together

Last update: Nov 07, 2022

Related tags

Text Data & NLP SpeechMix

Overview

SpeechMix

Explore different way to mix speech model(wav2vec2, hubert) and nlp model(BART,T5,GPT) together.

Introduction

For the same input:

from datasets import load_dataset
import soundfile as sf


# define function to read in sound file
def map_to_array(batch):
    speech, _ = sf.read(batch["file"])
    batch["speech"] = speech
    return batch


# load dummy dataset and read soundfiles
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
ds = ds.map(map_to_array)

transcript = ds['text'][0]
speech = ds["speech"][0]

Speech encoder NLP decoder

model = SpeechMixED("facebook/wav2vec2-base-960h", "facebook/bart-large")

transcript_tensor = model.tokenizer(transcript, return_tensors="pt").input_ids
speech_tensor = model.processor(speech, return_tensors="pt").input_values

model(speech_tensor, transcript_tensor)

Speech encoder NLP decoder only fine-tune on cross attention/projection/decoder embedding

model = SpeechMixED("facebook/wav2vec2-base-960h", "facebook/bart-large", ftl=True)

transcript_tensor = model.tokenizer(transcript, return_tensors="pt").input_ids
speech_tensor = model.processor(speech, return_tensors="pt").input_values

model(speech_tensor, transcript_tensor)

Speech encoder NLP encoder decoder

model = SpeechMixEED("facebook/wav2vec2-base-960h", "facebook/bart-large")

transcript_tensor = model.tokenizer(transcript, return_tensors="pt").input_ids
speech_tensor = model.processor(speech, return_tensors="pt").input_values

model(speech_tensor, transcript_tensor)

Speech encoder NLP encoder decoder only fine-tune on layer norm and attention

model = SpeechMixEED("facebook/wav2vec2-base-960h", "facebook/bart-large", lna=True)

transcript_tensor = model.tokenizer(transcript, return_tensors="pt").input_ids
speech_tensor = model.processor(speech, return_tensors="pt").input_values

model(speech_tensor, transcript_tensor)

Speech encoder NLP encoder decoder only fine-tune on speech encoder

model = SpeechMixEED("facebook/wav2vec2-base-960h", "facebook/bart-large", fne=True)

transcript_tensor = model.tokenizer(transcript, return_tensors="pt").input_ids
speech_tensor = model.processor(speech, return_tensors="pt").input_values

model(speech_tensor, transcript_tensor)

Installation

pip install

pip install speechmix

Build from source

git clone and cd into this project.

pip install -e .

Explore different way to mix speech model(wav2vec2, hubert) and nlp model(BART,T5,GPT) together

Related tags

Overview

SpeechMix

Introduction

Speech encoder NLP decoder

Speech encoder NLP decoder only fine-tune on cross attention/projection/decoder embedding

Speech encoder NLP encoder decoder

Speech encoder NLP encoder decoder only fine-tune on layer norm and attention

Speech encoder NLP encoder decoder only fine-tune on speech encoder

Installation

pip install

Build from source

Owner

Eric Lam

Explore different way to mix speech model(wav2vec2, hubert) and nlp model(BART,T5,GPT) together

PyTorch code for EMNLP 2019 paper "LXMERT: Learning Cross-Modality Encoder Representations from Transformers".

ByT5: Towards a token-free future with pre-trained byte-to-byte models

A simple Flask site that allows users to create, update, and delete posts in a database, as well as perform basic NLP tasks on the posts.

Finally, some decent sample sentences

This repository contains the code for "Generating Datasets with Pretrained Language Models".

nlabel is a library for generating, storing and retrieving tagging information and embedding vectors from various nlp libraries through a unified interface.

DeepSpeech - Easy-to-use Speech Toolkit including SOTA ASR pipeline, influential TTS with text frontend and End-to-End Speech Simultaneous Translation.

Label data using HuggingFace's transformers and automatically get a prediction service

Original implementation of the pooling method introduced in "Speaker embeddings by modeling channel-wise correlations"

Incorporating KenLM language model with HuggingFace implementation of Wav2Vec2CTC Model using beam search decoding

Checking spelling of form elements

🐍 A hyper-fast Python module for reading/writing JSON data using Rust's serde-json.

Creating a chess engine using GPT-3

Exploration of BERT-based models on twitter sentiment classifications

Gold standard corpus annotated with verb-preverb connections for Hungarian.

A simple chatbot based on chatterbot that you can use for anything has basic features

The first online catalogue for Arabic NLP datasets.

Code for lyric-section-to-comment generation based on huggingface transformers.

Transformation spoken text to written text