Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

Last update: Sep 20, 2022

Overview

Behavioral Testing of Clinical NLP Models

This repository contains code for testing the behavior of clinical prediction models based on patient letters. For a detailed description of the testing framework see our paper What Do You See in this Patient? Behavioral Testing of Clinical NLP Models.

Usage

Install requirements: pip install -r requirements.txt

Run main.py, e.g. for diagnosis prediction test on gender, age and ethnicity:

python main.py 
    --test_set_path ./path_to_test_set
    --model_path bvanaken/CORe-clinical-diagnosis-prediction
    --task diagnosis
    --shift_keys gender,age,ethnicity
    --save_dir ./results
    --gpu False

Parameter	Description
test_set_path	Path to original test set file
model_path	Path to model or Huggingface model hub checkpoint
task	Current options: diagnosis, mortality
shift_keys	Which patient characteristics to test. Current options: age, gender, ethnicity, weight, intersectional (gender + ethnicity)
save_dir	Directory to save results, default: "./results"
gpu	Whether to use a gpu during inference or not, default: False

Using Non-Transformer models

The framework currently focuses on testing Transformer-based models. However, it is easy to extend it to any other prediction model. To do so, simply create a new class implementing the Predictor interface and add it to the TASK_MAP in main.py.

Cite

@inproceedings{vanAken2021,
  author    = {Betty van Aken and
               Sebastian Herrmann and
               Alexander Löser},
  title     = {What Do You See in this Patient? Behavioral Testing of Clinical NLP Models},
  booktitle = {Bridging the Gap: From Machine Learning Research to Clinical Practice, 
               Research2Clinics Workshop @ NeurIPS 2021},
  year      = {2021}
}

Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

Related tags

Overview

Behavioral Testing of Clinical NLP Models

Usage

Using Non-Transformer models

Cite

Owner

Betty van Aken

Label-Free Model Evaluation with Semi-Structured Dataset Representations

Code for our paper "Multi-scale Guided Attention for Medical Image Segmentation"

adversarial_multi_armed_bandit_variable_plays

Keras udrl - Keras implementation of Upside Down Reinforcement Learning

A TensorFlow implementation of SOFA, the Simulator for OFfline LeArning and evaluation.

Learned Token Pruning for Transformers

Material for my PyConDE & PyData Berlin 2022 Talk "5 Steps to Speed Up Your Data-Analysis on a Single Core"

One line to host them all. Bootstrap your image search case in minutes.

Scalable Graph Neural Networks for Heterogeneous Graphs

Reproduction of Vision Transformer in Tensorflow2. Train from scratch and Finetune.

SlideGraph+: Whole Slide Image Level Graphs to Predict HER2 Status in Breast Cancer

Joint parameterization and fitting of stroke clusters

Cupytorch - A small framework mimics PyTorch using CuPy or NumPy

Learning Logic Rules for Document-Level Relation Extraction

PyTorch Implementation of Google Brain's WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis

An unopinionated replacement for PyTorch's Dataset and ImageFolder, that handles Tar archives

Using python and scikit-learn to make stock predictions

Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model

Re-implementation of the vector capsule with dynamic routing

Replication Code for "Self-Supervised Bug Detection and Repair" NeurIPS 2021