Behavioral Testing of Clinical NLP Models

This repository contains code for testing the behavior of clinical prediction models based on patient letters. For a detailed description of the testing framework see our paper What Do You See in this Patient? Behavioral Testing of Clinical NLP Models.

Usage

Install requirements: pip install -r requirements.txt

Run main.py, e.g. for diagnosis prediction test on gender, age and ethnicity:

python main.py 
    --test_set_path ./path_to_test_set
    --model_path bvanaken/CORe-clinical-diagnosis-prediction
    --task diagnosis
    --shift_keys gender,age,ethnicity
    --save_dir ./results
    --gpu False

Parameter	Description
test_set_path	Path to original test set file
model_path	Path to model or Huggingface model hub checkpoint
task	Current options: diagnosis, mortality
shift_keys	Which patient characteristics to test. Current options: age, gender, ethnicity, weight, intersectional (gender + ethnicity)
save_dir	Directory to save results, default: "./results"
gpu	Whether to use a gpu during inference or not, default: False

Using Non-Transformer models

The framework currently focuses on testing Transformer-based models. However, it is easy to extend it to any other prediction model. To do so, simply create a new class implementing the Predictor interface and add it to the TASK_MAP in main.py.

Cite

@inproceedings{vanAken2021,
  author    = {Betty van Aken and
               Sebastian Herrmann and
               Alexander Löser},
  title     = {What Do You See in this Patient? Behavioral Testing of Clinical NLP Models},
  booktitle = {Bridging the Gap: From Machine Learning Research to Clinical Practice, 
               Research2Clinics Workshop @ NeurIPS 2021},
  year      = {2021}
}

Behavioral Testing of Clinical NLP Models

Related tags

Overview

Behavioral Testing of Clinical NLP Models

Usage

Using Non-Transformer models

Cite

Owner

Betty van Aken

A repo for open resources & information for people to succeed in PhD in CS & career in AI / NLP

Extract rooms type, door, neibour rooms, rooms corners nad bounding boxes, and generate graph from rplan dataset

Behavioral Testing of Clinical NLP Models

GAP-text2SQL: Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

ChessCoach is a neural network-based chess engine capable of natural-language commentary.

Translation for Trilium Notes. Trilium Notes 中文版.

StarGAN - Official PyTorch Implementation

Repository for the paper "Optimal Subarchitecture Extraction for BERT"

Train and use generative text models in a few lines of code.

Shared, streaming Python dict

EasyTransfer is designed to make the development of transfer learning in NLP applications easier.

kochat

(ACL-IJCNLP 2021) Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models.

DeepSpeech - Easy-to-use Speech Toolkit including SOTA ASR pipeline, influential TTS with text frontend and End-to-End Speech Simultaneous Translation.

CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation

Japanese NLP Library

To create a deep learning model which can explain the content of an image in the form of speech through caption generation with attention mechanism on Flickr8K dataset.

Simple Speech to Text, Text to Speech

Command Line Text-To-Speech using Google TTS

This is a really simple text-to-speech app made with python and tkinter.