Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

Owner

Zirui Wang

Code accompanying our NeurIPS 2021 traffic4cast challenge

Official PyTorch implementation of the paper "TEMOS: Generating diverse human motions from textual descriptions"

Modeling Temporal Concept Receptive Field Dynamically for Untrimmed Video Analysis

chen2020iros: Learning an Overlap-based Observation Model for 3D LiDAR Localization.

Perturb-and-max-product: Sampling and learning in discrete energy-based models

Unoffical implementation about Image Super-Resolution via Iterative Refinement by Pytorch

The source code of the paper "Understanding Graph Neural Networks from Graph Signal Denoising Perspectives"

WORD: Revisiting Organs Segmentation in the Whole Abdominal Region

This implements the learning and inference/proposal algorithm described in "Learning to Propose Objects, Krähenbühl and Koltun"

Code accompanying our paper Feature Learning in Infinite-Width Neural Networks

RLMeta is a light-weight flexible framework for Distributed Reinforcement Learning Research.

Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking

Deep Learning tutorials in jupyter notebooks.

All the code and files related to the MI-Lab of UE19CS305 course in sem 5

Training BERT with Compute/Time (Academic) Budget

Diverse Object-Scene Compositions For Zero-Shot Action Recognition

Some methods for comparing network representations in deep learning and neuroscience.

50-days-of-Statistics-for-Data-Science - This repository consist of a 50-day program

An example project demonstrating how the Autonomous Learning Library can be used to build new reinforcement learning agents.

This repository provides a PyTorch implementation and model weights for HCSC (Hierarchical Contrastive Selective Coding)