The official code of LM-Debugger, an interactive tool for inspection and intervention in transformer-based language models.

Overview

LM-Debugger is an open-source interactive tool for inspection and intervention in transformer-based language models. This repository includes the code and links for data files required for running LM-Debugger over GPT2 Large and GPT2 Medium. Adapting this tool to other models only requires changing the backend API (see details below). Contributions our welcome!

An online demo of LM-Debugger is available at:

For more details, please check our paper: "LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models".

⚙️ Requirements

LM-Debugger has two main views for (a) debugging and intervention in model predictions, and (b) exploration of information encoded in the model's feed-forward layers.

The tool runs in a React and python environment with Flask and Streamlit installed. In addition, the exploration view uses an Elasticsearch index. To set up the environment, please follow the steps below:

  1. Clone this repository:

    git clone https://github.com/mega002/lm-debugger
    cd lm-debugger
  2. Create a Python 3.8 environment, and install the following dependencies:

    pip install -r requirements.txt
  3. Install Yarn and NVM, and set up the React environment:

    cd ui
    nvm install
    yarn install
    cd ..
  4. Install Elasticsearch and make sure that the service is up.

🔎 Running LM-Debugger

Creating a Configuration File

LM-Debugger executes one model at a time, based on a given configuration file. The configuration includes IP addresses and port numbers for running the different services, as well as the following fields:

  • model_name: The current version of LM-Debugger supports GPT2 models from HuggingFace (e.g. gpt2-medium or gpt2-large).
  • server_files_dir: A path to store files with preprocessed model information, created by the script create_offline_files.py. The script creates 3 pickle files with (1) projections to the vocabulary of parameter vectors of the model's feed-forward layers, (2) two separate files with mappings between parameter vectors and clusters (and vice versa).
  • create_cluster_files: A boolean field (true/false) that indicates whether to run clustering or not. This is optional since clustering of the feed-forward parameter vectors can take several hours and might require extra computation resources (especially for large models).

Sample configuration files for the medium and large versions of GPT2 are provided in the config_files directory. The preprocessed data files for these models are available for download here.

Creating an Elasticsearch Index

The keyword search functionality in the exploration view is powered by an Elasticsearch index that stores the projections of feed-forward parameter vectors from the entire network. To create this index, run:

python es_index/index_value_projections_docs.py \
--config_path CONFIG_PATH

Executing LM-Debugger

To run LM-Debugger:

bash start.sh CONFIG_PATH

In case you are interested in running only one of the two views of LM-Debugger, this can be done as follows:

  1. To run the Flask server (needed for the prediction view):

    python flask_server/app.py --config_path CONFIG_PATH
  2. To run the prediction view:

    python ui/src/convert2runConfig.py --config_path CONFIG_PATH
    cd ui
    yarn start
  3. To run the exploration view:

    streamlit run streamlit/exploration.py -- --config_path CONFIG_PATH

Citation

Please cite as:

@article{geva2022lmdebugger,
  title={LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models},
  author={Geva, Mor and Caciularu, Avi and Dar, Guy and Roit, Paul and Sadde, Shoval and Shlain, Micah and Tamir, Bar and Goldberg, Yoav},
  journal={arXiv preprint arXiv:2204.12130},
  year={2022}
}
Owner
Mor Geva
Mor Geva
A drop-in replacement for Django's runserver.

About A drop in replacement for Django's built-in runserver command. Features include: An extendable interface for handling things such as real-time l

David Cramer 1.3k Dec 15, 2022
Automated bug/error reporting for napari

napari-error-monitor Want to help out napari? Install this plugin! This plugin will automatically send error reports to napari (via sentry.io) wheneve

Talley Lambert 2 Sep 15, 2022
pdb++, a drop-in replacement for pdb (the Python debugger)

pdb++, a drop-in replacement for pdb What is it? This module is an extension of the pdb module of the standard library. It is meant to be fully compat

1k Jan 02, 2023
A simple rubber duck debugger

Rubber Duck Debugger I found myself many times asking a question on StackOverflow or to one of my colleagues just for finding the solution simply by d

1 Nov 10, 2021
Hdbg - Historical Debugger

hdbg - Historical Debugger This is in no way a finished product. Do not use this

Fivreld 2 Jan 02, 2022
A powerful set of Python debugging tools, based on PySnooper

snoop snoop is a powerful set of Python debugging tools. It's primarily meant to be a more featureful and refined version of PySnooper. It also includ

Alex Hall 874 Jan 08, 2023
A web-based visualization and debugging platform for NuPIC

Cerebro 2 A web-based visualization and debugging platform for NuPIC. Usage Set up cerebro2.server to export your model state. Then, run: cd static py

Numenta 24 Oct 13, 2021
Trace all method entries and exits, the exit also prints the return value, if it is of basic type

Trace all method entries and exits, the exit also prints the return value, if it is of basic type. The apk must have set the android:debuggable="true" flag.

Kurt Nistelberger 7 Aug 10, 2022
VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.

VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.

2.8k Jan 08, 2023
Auto-detecting the n+1 queries problem in Python

nplusone nplusone is a library for detecting the n+1 queries problem in Python ORMs, including SQLAlchemy, Peewee, and the Django ORM. The Problem Man

Joshua Carp 837 Dec 29, 2022
Visual Interaction with Code - A portable visual debugger for python

VIC Visual Interaction with Code A simple tool for debugging and interacting with running python code. This tool is designed to make it easy to inspec

Nathan Blank 1 Nov 16, 2021
Silky smooth profiling for Django

Silk Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before prese

Jazzband 3.7k Jan 01, 2023
Hypothesis debugging with vscode

Hypothesis debugging with vscode

Oliver Mannion 0 Feb 09, 2022
Pyinstrument - a Python profiler. A profiler is a tool to help you optimize your code - make it faster.

Pyinstrument🚴 Call stack profiler for Python. Shows you why your code is slow!

Joe Rickerby 5k Jan 08, 2023
GDB plugin for streaming defmt messages over RTT from e.g. JLinkGDBServer

Defmt RTT plugin from GDB This small plugin runs defmt-print on the RTT stream produced by JLinkGDBServer, so that you can see the defmt logs in the G

Gaute Hope 1 Dec 30, 2021
A configurable set of panels that display various debug information about the current request/response.

Django Debug Toolbar The Django Debug Toolbar is a configurable set of panels that display various debug information about the current request/respons

Jazzband 7.3k Dec 31, 2022
Sweeter debugging and benchmarking Python programs.

Do you ever use print() or log() to debug your code? If so, ycecream, or y for short, will make printing debug information a lot sweeter. And on top o

42 Dec 12, 2022
Full-screen console debugger for Python

PuDB: a console-based visual debugger for Python Its goal is to provide all the niceties of modern GUI-based debuggers in a more lightweight and keybo

Andreas Klöckner 2.6k Jan 01, 2023
Never use print for debugging again

PySnooper - Never use print for debugging again PySnooper is a poor man's debugger. If you've used Bash, it's like set -x for Python, except it's fanc

Ram Rachum 15.5k Jan 01, 2023
A gdb-like Python3 Debugger in the Trepan family

Abstract Features More Exact location information Debugging Python bytecode (no source available) Source-code Syntax Colorization Command Completion T

R. Bernstein 126 Nov 24, 2022