Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

Owner

Zirui Wang

Point detection through multi-instance deep heatmap regression for sutures in endoscopy

Experiments with the Robust Binary Interval Search (RBIS) algorithm, a Query-Based prediction algorithm for the Online Search problem.

Exadel CompreFace is a free and open-source face recognition GitHub project

CBKH: The Cornell Biomedical Knowledge Hub

Semantic Image Synthesis with SPADE

This is an official implementation of CvT: Introducing Convolutions to Vision Transformers.

Natural Intelligence is still a pretty good idea.

Continuous Security Group Rule Change Detection & Response at scale

Model parallel transformers in Jax and Haiku

This is the official repository for our paper: ''Pruning Self-attentions into Convolutional Layers in Single Path''.

[ ICCV 2021 Oral ] Our method can estimate camera poses and neural radiance fields jointly when the cameras are initialized at random poses in complex scenarios (outside-in scenes, even with less texture or intense noise )

Official implementation of "CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding" (CVPR, 2022)

[CVPRW 2022] Attentions Help CNNs See Better: Attention-based Hybrid Image Quality Assessment Network

[CVPR 2021] Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion

Implementing Graph Convolutional Networks and Information Retrieval Mechanisms using pure Python and NumPy

StyleGAN2-ADA - Official PyTorch implementation

YoloV5 implemented by TensorFlow2 , with support for training, evaluation and inference.

📚 A collection of Jupyter notebooks for learning and experimenting with OpenVINO 👓

A Data Annotation Tool for Semantic Segmentation, Object Detection and Lane Line Detection.(In Development Stage)

Unofficial Tensorflow 2 implementation of the paper Implicit Neural Representations with Periodic Activation Functions