The official code of LM-Debugger, an interactive tool for inspection and intervention in transformer-based language models.

Overview

LM-Debugger is an open-source interactive tool for inspection and intervention in transformer-based language models. This repository includes the code and links for data files required for running LM-Debugger over GPT2 Large and GPT2 Medium. Adapting this tool to other models only requires changing the backend API (see details below). Contributions our welcome!

An online demo of LM-Debugger is available at:

For more details, please check our paper: "LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models".

⚙️ Requirements

LM-Debugger has two main views for (a) debugging and intervention in model predictions, and (b) exploration of information encoded in the model's feed-forward layers.

The tool runs in a React and python environment with Flask and Streamlit installed. In addition, the exploration view uses an Elasticsearch index. To set up the environment, please follow the steps below:

  1. Clone this repository:

    git clone https://github.com/mega002/lm-debugger
    cd lm-debugger
  2. Create a Python 3.8 environment, and install the following dependencies:

    pip install -r requirements.txt
  3. Install Yarn and NVM, and set up the React environment:

    cd ui
    nvm install
    yarn install
    cd ..
  4. Install Elasticsearch and make sure that the service is up.

🔎 Running LM-Debugger

Creating a Configuration File

LM-Debugger executes one model at a time, based on a given configuration file. The configuration includes IP addresses and port numbers for running the different services, as well as the following fields:

  • model_name: The current version of LM-Debugger supports GPT2 models from HuggingFace (e.g. gpt2-medium or gpt2-large).
  • server_files_dir: A path to store files with preprocessed model information, created by the script create_offline_files.py. The script creates 3 pickle files with (1) projections to the vocabulary of parameter vectors of the model's feed-forward layers, (2) two separate files with mappings between parameter vectors and clusters (and vice versa).
  • create_cluster_files: A boolean field (true/false) that indicates whether to run clustering or not. This is optional since clustering of the feed-forward parameter vectors can take several hours and might require extra computation resources (especially for large models).

Sample configuration files for the medium and large versions of GPT2 are provided in the config_files directory. The preprocessed data files for these models are available for download here.

Creating an Elasticsearch Index

The keyword search functionality in the exploration view is powered by an Elasticsearch index that stores the projections of feed-forward parameter vectors from the entire network. To create this index, run:

python es_index/index_value_projections_docs.py \
--config_path CONFIG_PATH

Executing LM-Debugger

To run LM-Debugger:

bash start.sh CONFIG_PATH

In case you are interested in running only one of the two views of LM-Debugger, this can be done as follows:

  1. To run the Flask server (needed for the prediction view):

    python flask_server/app.py --config_path CONFIG_PATH
  2. To run the prediction view:

    python ui/src/convert2runConfig.py --config_path CONFIG_PATH
    cd ui
    yarn start
  3. To run the exploration view:

    streamlit run streamlit/exploration.py -- --config_path CONFIG_PATH

Citation

Please cite as:

@article{geva2022lmdebugger,
  title={LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models},
  author={Geva, Mor and Caciularu, Avi and Dar, Guy and Roit, Paul and Sadde, Shoval and Shlain, Micah and Tamir, Bar and Goldberg, Yoav},
  journal={arXiv preprint arXiv:2204.12130},
  year={2022}
}
Owner
Mor Geva
Mor Geva
VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.

VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.

2.8k Jan 08, 2023
Cyberbrain: Python debugging, redefined.

Cyberbrain1(电子脑) aims to free programmers from debugging.

laike9m 2.3k Jan 07, 2023
Sampling profiler for Python programs

py-spy: Sampling profiler for Python programs py-spy is a sampling profiler for Python programs. It lets you visualize what your Python program is spe

Ben Frederickson 9.5k Jan 08, 2023
Debugging manhole for python applications.

Overview docs tests package Manhole is in-process service that will accept unix domain socket connections and present the stacktraces for all threads

Ionel Cristian Mărieș 332 Dec 07, 2022
Never use print for debugging again

PySnooper - Never use print for debugging again PySnooper is a poor man's debugger. If you've used Bash, it's like set -x for Python, except it's fanc

Ram Rachum 15.5k Jan 01, 2023
Django package to log request values such as device, IP address, user CPU time, system CPU time, No of queries, SQL time, no of cache calls, missing, setting data cache calls for a particular URL with a basic UI.

django-web-profiler's documentation: Introduction: django-web-profiler is a django profiling tool which logs, stores debug toolbar statistics and also

MicroPyramid 77 Oct 29, 2022
AryaBota: An app to teach Python coding via gradual programming and visual output

AryaBota An app to teach Python coding, that gradually allows students to transition from using commands similar to natural language, to more Pythonic

5 Feb 08, 2022
A configurable set of panels that display various debug information about the current request/response.

Django Debug Toolbar The Django Debug Toolbar is a configurable set of panels that display various debug information about the current request/respons

Jazzband 7.3k Dec 29, 2022
Full featured multi arch/os debugger built on top of PyQt5 and frida

Full featured multi arch/os debugger built on top of PyQt5 and frida

iGio90 1.1k Dec 26, 2022
Pyinstrument - a Python profiler. A profiler is a tool to help you optimize your code - make it faster.

Pyinstrument🚴 Call stack profiler for Python. Shows you why your code is slow!

Joe Rickerby 5k Jan 08, 2023
A powerful set of Python debugging tools, based on PySnooper

snoop snoop is a powerful set of Python debugging tools. It's primarily meant to be a more featureful and refined version of PySnooper. It also includ

Alex Hall 874 Jan 08, 2023
Graphical Python debugger which lets you easily view the values of all evaluated expressions

birdseye birdseye is a Python debugger which records the values of expressions in a function call and lets you easily view them after the function exi

Alex Hall 1.5k Dec 24, 2022
Trashdbg - TrashDBG the world's worse debugger

The world's worse debugger Over the course of multiple OALABS Twitch streams we

OALabs 21 Jun 17, 2022
Trace all method entries and exits, the exit also prints the return value, if it is of basic type

Trace all method entries and exits, the exit also prints the return value, if it is of basic type. The apk must have set the android:debuggable="true" flag.

Kurt Nistelberger 7 Aug 10, 2022
The official code of LM-Debugger, an interactive tool for inspection and intervention in transformer-based language models.

LM-Debugger is an open-source interactive tool for inspection and intervention in transformer-based language models. This repository includes the code

Mor Geva 110 Dec 28, 2022
Hdbg - Historical Debugger

hdbg - Historical Debugger This is in no way a finished product. Do not use this

Fivreld 2 Jan 02, 2022
一个小脚本,用于trace so中native函数的调用。

trace_natives 一个IDA小脚本,获取SO代码段中所有函数的偏移地址,再使用frida-trace 批量trace so函数的调用。 使用方法 1.将traceNatives.py丢进IDA plugins目录中 2.IDA中,Edit-Plugins-traceNatives IDA输

296 Dec 28, 2022
Sweeter debugging and benchmarking Python programs.

Do you ever use print() or log() to debug your code? If so, ycecream, or y for short, will make printing debug information a lot sweeter. And on top o

42 Dec 12, 2022
Dahua Console, access internal debug console and/or other researched functions in Dahua devices.

Dahua Console, access internal debug console and/or other researched functions in Dahua devices.

bashis 156 Dec 28, 2022
Python's missing debug print command and other development tools.

python devtools Python's missing debug print command and other development tools. For more information, see documentation. Install Just pip install de

Samuel Colvin 637 Jan 02, 2023