Implementation of the paper "Fine-Tuning Transformers: Vocabulary Transfer"

Last update: Nov 30, 2022

Related tags

Overview

Transformer-vocabulary-transfer

Implementation of the paper "Fine-Tuning Transformers: Vocabulary Transfer"

Description

step 1 - Create SentencePiece vocabulary for dataset
step 2 - Train the first level model (BertForMaskedLM) on English Wikipedia from scratch
step 3 - Match vocabulary (first level model dataset & downstream task dataset)
step 4 - Transfer dictionary using mapping. Сreate folders and raw models for experiments.
step 5 - Train 1 epoch BertForMaskedLM on downstream task
step 6 - Train final (BertForSequenceClassification) downstream model

Citation

I. Samenko, A. Tikhonov, B. Kozlovsky, I. P. Yamshchikov. Fine-Tuning Transformers: Vocabulary Transfer.

Contact

*Igor Samenko: [email protected]

Owner

LEYA

LEYA is a Natural Language Processing research laboratory co-founded by Yandex and Higher School of Economics in Saint-Petersburg.

GitHub Repository

RRxIO - Robust Radar Visual/Thermal Inertial Odometry: Robust and accurate state estimation even in challenging visual conditions.

RRxIO - Robust Radar Visual/Thermal Inertial Odometry RRxIO offers robust and accurate state estimation even in challenging visual conditions. RRxIO c

64 Dec 29, 2022

FaceQgen: Semi-Supervised Deep Learning for Face Image Quality Assessment

FaceQgen FaceQgen: Semi-Supervised Deep Learning for Face Image Quality Assessment This repository is based on the paper: "FaceQgen: Semi-Supervised D

3 Aug 04, 2022

A PyTorch implementation: "LASAFT-Net-v2: Listen, Attend and Separate by Attentively aggregating Frequency Transformation"

LASAFT-Net-v2 Listen, Attend and Separate by Attentively aggregating Frequency Transformation Woosung Choi, Yeong-Seok Jeong, Jinsung Kim, Jaehwa Chun

29 Jun 04, 2022

Streaming Anomaly Detection Framework in Python (Outlier Detection for Streaming Data)

Python Streaming Anomaly Detection (PySAD) PySAD is an open-source python framework for anomaly detection on streaming multivariate data. Documentatio

181 Dec 18, 2022

Model Quantization Benchmark

Introduction MQBench is an open-source model quantization toolkit based on PyTorch fx. The envision of MQBench is to provide: SOTA Algorithms. With MQ

500 Jan 06, 2023

Playable Video Generation

Playable Video Generation Playable Video Generation Willi Menapace, Stéphane Lathuilière, Sergey Tulyakov, Aliaksandr Siarohin, Elisa Ricci Paper: ArX

136 Dec 31, 2022

Code for "Unsupervised Layered Image Decomposition into Object Prototypes" paper

DTI-Sprites Pytorch implementation of "Unsupervised Layered Image Decomposition into Object Prototypes" paper Check out our paper and webpage for deta

40 Dec 22, 2022

A novel method to tune language models. Codes and datasets for paper ``GPT understands, too''.

P-tuning A novel method to tune language models. Codes and datasets for paper ``GPT understands, too''. How to use our code We have released the code

562 Dec 27, 2022

Maximum Spatial Perturbation for Image-to-Image Translation (Official Implementation)

MSPC for I2I This repository is by Yanwu Xu and contains the PyTorch source code to reproduce the experiments in our CVPR2022 paper Maximum Spatial Pe

51 Dec 14, 2022

《Single Image Reflection Removal Beyond Linearity》(CVPR 2019)

Single-Image-Reflection-Removal-Beyond-Linearity Paper Single Image Reflection Removal Beyond Linearity. Qiang Wen, Yinjie Tan, Jing Qin, Wenxi Liu, G

51 Jun 24, 2022

MarcoPolo is a clustering-free approach to the exploration of bimodally expressed genes along with group information in single-cell RNA-seq data

MarcoPolo is a method to discover differentially expressed genes in single-cell RNA-seq data without depending on prior clustering Overview MarcoPolo

13 Dec 18, 2022

Implementation of Auto-Conditioned Recurrent Networks for Extended Complex Human Motion Synthesis

acLSTM_motion This folder contains an implementation of acRNN for the CMU motion database written in Pytorch. See the following links for more backgro

61 Sep 07, 2022

Everything about being a TA for ITP/AP course!

تی‌ای بودن! تی‌ای یا دستیار استاد از نقش‌های رایج بین دانشجویان مهندسی است، این ریپوزیتوری قرار است نکات مهم درمورد تی‌ای بودن و تی ای شدن را به ما نش

[email protected]"> 14 Sep 10, 2022

Paaster is a secure by default end-to-end encrypted pastebin built with the objective of simplicity.

Follow the development of our desktop client here Paaster Paaster is a secure by default end-to-end encrypted pastebin built with the objective of sim

211 Dec 25, 2022

Python library containing BART query generation and BERT-based Siamese models for neural retrieval.

Neural Retrieval Embedding-based Zero-shot Retrieval through Query Generation leverages query synthesis over large corpuses of unlabeled text (such as

35 Apr 14, 2022

[ICCV 2021] Focal Frequency Loss for Image Reconstruction and Synthesis

Focal Frequency Loss - Official PyTorch Implementation This repository provides the official PyTorch implementation for the following paper: Focal Fre

460 Jan 04, 2023

Image-based Navigation in Real-World Environments via Multiple Mid-level Representations: Fusion Models Benchmark and Efficient Evaluation

Image-based Navigation in Real-World Environments via Multiple Mid-level Representations: Fusion Models Benchmark and Efficient Evaluation This reposi

1 Aug 21, 2022

Implementation of the paper "Fine-Tuning Transformers: Vocabulary Transfer"

Related tags

Overview

Transformer-vocabulary-transfer

Description

Citation

Contact

Owner

LEYA

RRxIO - Robust Radar Visual/Thermal Inertial Odometry: Robust and accurate state estimation even in challenging visual conditions.

FaceQgen: Semi-Supervised Deep Learning for Face Image Quality Assessment

A PyTorch implementation: "LASAFT-Net-v2: Listen, Attend and Separate by Attentively aggregating Frequency Transformation"

Streaming Anomaly Detection Framework in Python (Outlier Detection for Streaming Data)

Model Quantization Benchmark

Playable Video Generation

Code for "Unsupervised Layered Image Decomposition into Object Prototypes" paper

A novel method to tune language models. Codes and datasets for paper ``GPT understands, too''.

Maximum Spatial Perturbation for Image-to-Image Translation (Official Implementation)

《Single Image Reflection Removal Beyond Linearity》(CVPR 2019)

MarcoPolo is a clustering-free approach to the exploration of bimodally expressed genes along with group information in single-cell RNA-seq data

Implementation of Auto-Conditioned Recurrent Networks for Extended Complex Human Motion Synthesis

Everything about being a TA for ITP/AP course!

Paaster is a secure by default end-to-end encrypted pastebin built with the objective of simplicity.

Python library containing BART query generation and BERT-based Siamese models for neural retrieval.

[ICCV 2021] Focal Frequency Loss for Image Reconstruction and Synthesis

Image-based Navigation in Real-World Environments via Multiple Mid-level Representations: Fusion Models Benchmark and Efficient Evaluation

This is the official implementation for "Do Transformers Really Perform Bad for Graph Representation?".

Adjusting for Autocorrelated Errors in Neural Networks for Time Series

Self-Supervised Contrastive Learning of Music Spectrograms