BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks

Python3 / PyTorch implementation of the following paper: Fine-grained Semantics-aware Representation Enhancement for Self-supervisedMonocular Depth Estimation. ICCV 2021 (oral)

Hand Gesture Volume Control is AIML based project which uses image processing to control the volume of your Computer.

My personal Home Assistant configuration.

[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views

Python scripts for performing lane detection using the LSTR model in ONNX

Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in ONNX

[ICML 2021] DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning | 斗地主AI

This repo contains the official code and pre-trained models for the Dynamic Vision Transformer (DVT).

[BMVC2021] The official implementation of "DomainMix: Learning Generalizable Person Re-Identification Without Human Annotations"

Demonstrational Session git repo for H SAF User Workshop (28/1)

Augmented Traffic Control: A tool to simulate network conditions

(ICCV 2021) ProHMR - Probabilistic Modeling for Human Mesh Recovery

Notification Triggers for Python

An efficient 3D semantic segmentation framework for Urban-scale point clouds like SensatUrban, Campus3D, etc.

COCO Style Dataset Generator GUI

Code for the Paper: Conditional Variational Capsule Network for Open Set Recognition

[3DV 2021] Channel-Wise Attention-Based Network for Self-Supervised Monocular Depth Estimation

A graph adversarial learning toolbox based on PyTorch and DGL.

VarCLR: Variable Semantic Representation Pre-training via Contrastive Learning