beyond-preserved-accuracy

Repo for EMNLP 2021 paper "Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression"

How to implement the metrics?

Probability Loyalty

We recommend the off-the-shelf implementation in scipy.

from scipy.spatial import distance
distance.jensenshannon([0.75, 0.2, 0.05], [0.8, 0.1, 0.1])  # softmax prediction of the teacher and student

Label Loyalty

We recommend the off-the-shelf implementation in sklearn.

from sklearn.metrics import accuracy_score
accuracy_score([0, 2, 1, 3], [0, 1, 2, 3]) # predicted labels of the teacher and student

Robustness

We use TextFooler, please follow the instructions there.

Citation

@inproceedings{beyond-preserved-accuracy,
    title = "Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression",
    author = "Canwen Xu and Wangchunshu Zhou and Tao Ge and Ke Xu and Julian McAuley and Furu Wei",
    booktitle = {{EMNLP}},
    year = "2021",
}