This is the repo for Uncertainty Quantification 360 Toolkit.

Overview

UQ360

Build Status Documentation Status

The Uncertainty Quantification 360 (UQ360) toolkit is an open-source Python package that provides a diverse set of algorithms to quantify uncertainty, as well as capabilities to measure and improve UQ to streamline the development process. We provide a taxonomy and guidance for choosing these capabilities based on the user's needs. Further, UQ360 makes the communication method of UQ an integral part of development choices in an AI lifecycle. Developers can make a user-centered choice by following the psychology-based guidance on communicating UQ estimates, from concise descriptions to detailed visualizations.

The UQ360 interactive experience provides a gentle introduction to the concepts and capabilities by walking through an example use case. The tutorials and example notebooks offer a deeper, data scientist-oriented introduction. The complete API is also available.

We have developed the package with extensibility in mind. This library is still in development. We encourage the contribution of your uncertianty estimation algorithms, metrics and applications. To get started as a contributor, please join the #uq360-users or #uq360-developers channel of the AIF360 Community on Slack by requesting an invitation here.

Supported Uncertainty Evaluation Metrics

The toolbox provides several standard calibration metrics for classification and regression tasks. This includes Expected Calibration Error (Naeini et al., 2015), Brier Score (Murphy, 1973), etc for classification models. Regression metrics include Prediction Interval Coverage Probability (PICP) and Mean Prediction Interval Width (MPIW) among others. The toolbox also provides a novel operation-point agnostic approaches for the assessment of prediction uncertainty estimates called the Uncertainty Characteristic Curve (UCC). Several metrics and diagnosis tools such as reliability diagram (Niculescu-Mizil & Caruana, 2005) and risk-vs-rejection rate curves are provides which also support analysis by sub-groups in the population to study fairness implications of acting on given uncertainty estimates.

Supported Uncertainty Estimation Algorithms

UQ algorithms can be broadly classified as intrinsic or extrinsic depending on how the uncertainties are obtained from the AI models. Intrinsic methods encompass models that inherently provides an uncertainty estimate along with its predictions. The toolkit includes algorithms such as variational Bayesian neural networks (BNNs) (Graves, 2011), Gaussian processes (Rasmussen and Williams,2006), quantile regression (Koenker and Bassett, 1978) and hetero/homo-scedastic neuralnetworks (Kendall and Gal, 2017) which are models that fall in this category The toolkit also includes Horseshoe BNNs (Ghosh et al., 2019) that use sparsity promoting priors and can lead to better-calibrated uncertainties, especially in the small data regime. An Infinitesimal Jackknife (IJ) based algorithm (Ghosh et al., 2020)), provided in the toolkit, is a perturbation-based approach that perform uncertainty quantification by estimating model parameters under different perturbations of the original data. Crucially, here the estimation only requires the model to be trained once on the unperturbed dataset. For models that do not have an inherent notion of uncertainty built into them, extrinsic methods are employed to extract uncertainties post-hoc. The toolkit provides meta-models (Chen et al., 2019)that can be been used to successfully generate reliable confidence measures (in classification), prediction intervals (in regression), and to predict performance metrics such as accuracy on unseen and unlabeled data. For pre-trained models that captures uncertainties to some degree, the toolbox provides extrinsic algorithms that can improve the uncertainty estimation quality. This includes isotonic regression (Zadrozny and Elkan, 2001), Platt-scaling (Platt, 1999), auxiliary interval predictors (Thiagarajan et al., 2020), and UCC-Recalibration.

Setup

Supported Configurations:

OS Python version
macOS 3.7
Ubuntu 3.7
Windows 3.7

(Optional) Create a virtual environment

A virtual environment manager is strongly recommended to ensure dependencies may be installed safely. If you have trouble installing the toolkit, try this first.

Conda

Conda is recommended for all configurations though Virtualenv is generally interchangeable for our purposes. Miniconda is sufficient (see the difference between Anaconda and Miniconda if you are curious) and can be installed from here if you do not already have it.

Then, to create a new Python 3.7 environment, run:

conda create --name uq360 python=3.7
conda activate uq360

The shell should now look like (uq360) $. To deactivate the environment, run:

(uq360)$ conda deactivate

The prompt will return back to $ or (base)$.

Note: Older versions of conda may use source activate uq360 and source deactivate (activate uq360 and deactivate on Windows).

Installation

Clone the latest version of this repository:

(uq360)$ git clone https://github.ibm.com/UQ360/UQ360

If you'd like to run the examples and tutorial notebooks, download the datasets now and place them in their respective folders as described in uq360/datasets/data/README.md.

Then, navigate to the root directory of the project which contains setup.py file and run:

(uq360)$ pip install -e .

PIP Installation of Uncertainty Quantification 360

If you would like to quickly start using the UQ360 toolkit without cloning this repository, then you can install the uq360 pypi package as follows.

(your environment)$ pip install uq360

If you follow this approach, you may need to download the notebooks in the examples folder separately.

Using UQ360

The examples directory contains a diverse collection of jupyter notebooks that use UQ360 in various ways. Both examples and tutorial notebooks illustrate working code using the toolkit. Tutorials provide additional discussion that walks the user through the various steps of the notebook. See the details about tutorials and examples here.

Citing UQ360

A technical description of UQ360 is available in this paper. Below is the bibtex entry for this paper.

@misc{uq360-june-2021,
      title={Uncertainty Quantification 360: A Holistic Toolkit for Quantifying 
      and Communicating the Uncertainty of AI}, 
      author={Soumya Ghosh and Q. Vera Liao and Karthikeyan Natesan Ramamurthy 
      and Jiri Navratil and Prasanna Sattigeri 
      and Kush R. Varshney and Yunfeng Zhang},
      year={2021},
      eprint={2106.01410},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}

Acknowledgements

UQ360 is built with the help of several open source packages. All of these are listed in setup.py and some of these include:

License Information

Please view both the LICENSE file present in the root directory for license information.

Owner
International Business Machines
International Business Machines
:snake: Complete C99 parser in pure Python

pycparser v2.20 Contents 1 Introduction 1.1 What is pycparser? 1.2 What is it good for? 1.3 Which version of C does pycparser support? 1.4 What gramma

Eli Bendersky 2.8k Dec 29, 2022
Imitate Moulinette written in Python

Imitate Moulinette written in Python

Pumidol Leelerdsakulvong 2 Jul 26, 2022
A Python wrapper around Bacting

pybacting Python wrapper around bacting. Usage Based on the example from the bacting page, you can do: from pybacting import cdk print(cdk.fromSMILES

Charles Tapley Hoyt 5 Jan 03, 2022
Basit bir cc generator'ü.

Basit bir cc generator'ü. Setup What To Do; Python Installation We install python from CLICK Generator Board After installing the file and python, we

Lâving 7 Jan 09, 2022
Very Simple 2 Message Spammer!

Very Simple 2 Message Spammer!

Syntax. 4 Dec 06, 2022
Org agenda in the console

This Python script reads an org agenda file (i.e. a regular org file with some active dates) and displays an interactive and colored year calendar with detailed information for each day when the mous

Nicolas P. Rougier 113 Jan 03, 2023
SimilarWeb for Team ACT v.0.0.1

SimilarWeb for Team ACT v.0.0.1 This module has been built to provide a better environment specifically for Similarweb in Team ACT. This module itself

Sunkyeong Lee 0 Dec 29, 2021
Скрипт позволяет заводить задачи в Панель мониторинга YouTrack на основе парсинга сайта safe-surf.ru

Скрипт позволяет заводить задачи в Панель мониторинга YouTrack на основе парсинга сайта safe-surf.ru

Bad_karma 3 Feb 12, 2022
Spooky Castle Project

Spooky Castle Project Here is a repository where I have placed a few workflow scripts that could be used to automate the blender to godot sprite pipel

3 Jan 17, 2022
Standard mutable string (character array) implementation for Python.

chararray A standard mutable character array implementation for Python.

Tushar Sadhwani 3 Dec 18, 2021
This app is to use algorithms to find the root of the equation

In this repository, I made an amazing app with tkinter python language and other libraries the idea of this app is to use algorithms to find the root of the equation I used three methods from numeric

Mohammad Al Jadallah 3 Sep 16, 2022
A turtlebot auto controller allows robot to autonomously explore environment.

A turtlebot auto controller allows robot to autonomously explore environment.

Yuliang Zhong 1 Nov 10, 2021
A student information management system in Python

Student-information-management-system 本项目是一个学生信息管理系统,这个项目是用Python语言实现的,也实现了图形化界面的显示,同时也实现了管理员端,学生端两个登陆入口,同时底层使用的是Redis做的数据持久化。 This project is a stude

liuyunfei 7 Nov 15, 2022
Export transactions for an algorand wallet to a CSV file

algorand_txn_csv_exporter - (Algorand transaction CSV exporter) This script will export transactions for an algorand wallet to a CSV file. It is inten

TeneoPython01 5 Jun 19, 2022
This tool allows you to do goole dorking much easier

This tool allows you to do goole dorking much easier

Steven 8 Mar 06, 2022
My collection of mini-projects in various languages

Mini-Projects My collection of mini-projects in various languages About: This repository consists of a number of small projects. Most of these "mini-p

Siddhant Attavar 1 Jul 11, 2022
Tool to generate wrappers for Linux libraries allowing for dlopen()ing them without writing any boilerplate

Dynload wrapper This program will generate a wrapper to make it easy to dlopen() shared objects on Linux without writing a ton of boilerplate code. Th

Hein-Pieter van Braam 25 Oct 24, 2022
navigation_commander is a ROS package to command the robot to navigate autonomously to each table for food delivery inside a hotel.

navigation_commander navigation_commander is a ROS package to command the robot to navigate autonomously to each table for food delivery inside a hote

ALEENA LENTIN 9 Nov 08, 2021
Jarvis Python BOT acts like Google-assistance

Jarvis-Python-BOT Jarvis Python BOT acts like Google-assistance Setup Add Mail ID (Gmail) in the file at line no 82.

Ishan Jogalekar 1 Jan 08, 2022
Anki Addon idea by gbrl.sc to see previous ratings of a card in the reviewer

Card History At A Glance Stop having to press card browser and ctrl+i for every card and then WINCING to see it's history of reviews FEATURES Visualiz

Jerry Zhou 11 Dec 19, 2022