BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

SuperSDR: multiplatform KiwiSDR + CAT transceiver integrator

Source code related to the article submitted to the International Conference on Computational Science ICCS 2022 in London

Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers [CVPR 2021]

Tensorflow 2.x based implementation of EDSR, WDSR and SRGAN for single image super-resolution

Training Cifar-10 Classifier Using VGG16

Approximate Nearest Neighbors in C++/Python optimized for memory usage and loading/saving to disk

An intelligent, flexible grammar of machine learning.

(CVPR 2021) Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds

IAST: Instance Adaptive Self-training for Unsupervised Domain Adaptation (ECCV 2020)

Get the partition that a file belongs and the percentage of space that consumes

Object-aware Contrastive Learning for Debiased Scene Representation

Weakly- and Semi-Supervised Panoptic Segmentation (ECCV18)

Strongly local p-norm-cut algorithms for semi-supervised learning and local graph clustering

Code for How To Create A Fully Automated AI Based Trading System With Python

DockStream: A Docking Wrapper to Enhance De Novo Molecular Design

Learnable Multi-level Frequency Decomposition and Hierarchical Attention Mechanism for Generalized Face Presentation Attack Detection

Solution to the Weather4cast 2021 challenge

Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance

LERP : Label-dependent and event-guided interpretable disease risk prediction using EHRs

Python library for analysis of time series data including dimensionality reduction, clustering, and Markov model estimation