The open-source and free to use Python package miseval was developed to establish a standardized medical image segmentation evaluation procedure

Overview

miseval: a metric library for Medical Image Segmentation EVALuation

shield_python shield_build shield_pypi_version shield_pypi_downloads shield_license

The open-source and free to use Python package miseval was developed to establish a standardized medical image segmentation evaluation procedure. We hope that our this will help improve evaluation quality, reproducibility, and comparability in future studies in the field of medical image segmentation.

Guideline on Evaluation Metrics for Medical Image Segmentation

  1. Use DSC as main metric for validation and performance interpretation.
  2. Use AHD for interpretation on point position sensitivity (contour) if needed.
  3. Avoid any interpretations based on high pixel accuracy scores.
  4. Provide next to DSC also IoU, Sensitivity, and Specificity for method comparability.
  5. Provide sample visualizations, comparing the annotated and predicted segmentation, for visual evaluation as well as to avoid statistical bias.
  6. Avoid cherry-picking high-scoring samples.
  7. Provide histograms or box plots showing the scoring distribution across the dataset.
  8. For multi-class problems, provide metric computations for each class individually.
  9. Avoid confirmation bias through macro-averaging classes which is pushing scores via background class inclusion.
  10. Provide access to evaluation scripts and results with journal data services or third-party services like GitHub and Zenodo for easier reproducibility.

Implemented Metrics

Metric Index in miseval Function in miseval
Dice Similarity Index "DSC", "Dice", "DiceSimilarityCoefficient" miseval.calc_DSC()
Intersection-Over-Union "IoU", "Jaccard", "IntersectionOverUnion" miseval.calc_IoU()
Sensitivity "SENS", "Sensitivity", "Recall", "TPR", "TruePositiveRate" miseval.calc_Sensitivity()
Specificity "SPEC", "Specificity", "TNR", "TrueNegativeRate" miseval.calc_Specificity()
Precision "PREC", "Precision" miseval.calc_Precision()
Accuracy "ACC", "Accuracy", "RI", "RandIndex" miseval.calc_Accuracy()
Balanced Accuracy "BACC", "BalancedAccuracy" miseval.calc_BalancedAccuracy()
Adjusted Rand Index "ARI", "AdjustedRandIndex" miseval.calc_AdjustedRandIndex()
AUC "AUC", "AUC_trapezoid" miseval.calc_AUC()
Cohen's Kappa "KAP", "Kappa", "CohensKappa" miseval.calc_Kappa()
Hausdorff Distance "HD", "HausdorffDistance" miseval.calc_SimpleHausdorffDistance()
Average Hausdorff Distance "AHD", "AverageHausdorffDistance" miseval.calc_AverageHausdorffDistance()
Volumetric Similarity "VS", "VolumetricSimilarity" miseval.calc_VolumetricSimilarity()
True Positive "TP", "TruePositive" miseval.calc_TruePositive()
False Positive "FP", "FalsePositive" miseval.calc_FalsePositive()
True Negative "TN", "TrueNegative" miseval.calc_TrueNegative()
False Negative "FN", "FalseNegative" miseval.calc_FalseNegative()

How to Use

Example

# load libraries
import numpy as np
from miseval import evaluate

# Get some ground truth / annotated segmentations
np.random.seed(1)
real_bi = np.random.randint(2, size=(64,64))  # binary (2 classes)
real_mc = np.random.randint(5, size=(64,64))  # multi-class (5 classes)
# Get some predicted segmentations
np.random.seed(2)
pred_bi = np.random.randint(2, size=(64,64))  # binary (2 classes)
pred_mc = np.random.randint(5, size=(64,64))  # multi-class (5 classes)

# Run binary evaluation
dice = evaluate(real_bi, pred_bi, metric="DSC")    
  # returns single np.float64 e.g. 0.75

# Run multi-class evaluation
dice_list = evaluate(real_mc, pred_mc, metric="DSC", multi_class=True,
                     n_classes=5)   
  # returns array of np.float64 e.g. [0.9, 0.2, 0.6, 0.0, 0.4]
  # for each class, one score

Core function: Evaluate()

Every metric in miseval can be called via our core function evaluate().

The miseval eavluate function can be run with different metrics as backbone.
You can pass the following options to the metric parameter:

  • String naming one of the metric labels, for example "DSC"
  • Directly passing a metric function, for example calc_DSC_Sets (from dice.py)
  • Passing a custom metric function

List of metrics : See miseval/__init__.py under section "Access Functions to Metric Functions"

The classes in a segmentation mask must be ongoing starting from 0 (integers from 0 to n_classes-1).

A segmentation mask is allowed to have either no channel axis or just 1 (e.g. 512x512x1), which contains the annotation.

Binary mode. n_classes (Integer): Number of classes. By default 2 -> Binary Output: score (Float) or scores (List of Float) The multi_class parameter defines the output of this function. If n_classes > 2, multi_class is automatically True. If multi_class == False & n_classes == 2, only a single score (float) is returned. If multi_class == True, multiple scores as a list are returned (for each class one score). """ def evaluate(truth, pred, metric, multi_class=False, n_classes=2)">
"""
Arguments:
    truth (NumPy Matrix):            Ground Truth segmentation mask.
    pred (NumPy Matrix):             Prediction segmentation mask.
    metric (String or Function):     Metric function. Either a function directly or encoded as String from miseval or a custom function.
    multi_class (Boolean):           Boolean parameter, if segmentation is a binary or multi-class problem. By default False -> Binary mode.
    n_classes (Integer):             Number of classes. By default 2 -> Binary

Output:
    score (Float) or scores (List of Float)

    The multi_class parameter defines the output of this function.
    If n_classes > 2, multi_class is automatically True.
    If multi_class == False & n_classes == 2, only a single score (float) is returned.
    If multi_class == True, multiple scores as a list are returned (for each class one score).
"""
def evaluate(truth, pred, metric, multi_class=False, n_classes=2)

Installation

  • Install miseval from PyPI (recommended):
pip install miseval
  • Alternatively: install miseval from the GitHub source:

First, clone miseval using git:

git clone https://github.com/frankkramer-lab/miseval

Then, go into the miseval folder and run the install command:

cd miseval
python setup.py install

Author

Dominik Müller
Email: [email protected]
IT-Infrastructure for Translational Medical Research
University Augsburg
Bavaria, Germany

How to cite / More information

Dominik Müller, Dennis Hartmann, Philip Meyer, Florian Auer, Iñaki Soto-Rey, Frank Kramer. (2022)
MISeval: a Metric Library for Medical Image Segmentation Evaluation.
arXiv e-print: https://arxiv.org/abs/2201.09395

@inproceedings{misevalMUELLER2022,
  title={MISeval: a Metric Library for Medical Image Segmentation Evaluation},
  author={Dominik Müller, Dennis Hartmann, Philip Meyer, Florian Auer, Iñaki Soto-Rey, Frank Kramer},
  year={2022}
  eprint={2201.09395},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}

Thank you for citing our work.

License

This project is licensed under the GNU GENERAL PUBLIC LICENSE Version 3.
See the LICENSE.md file for license rights and limitations.

End-To-End Crowdsourcing

End-To-End Crowdsourcing Comparison of traditional crowdsourcing approaches to a state-of-the-art end-to-end crowdsourcing approach LTNet on sentiment

Andreas Koch 1 Mar 06, 2022
Vector Quantization, in Pytorch

Vector Quantization - Pytorch A vector quantization library originally transcribed from Deepmind's tensorflow implementation, made conveniently into a

Phil Wang 665 Jan 08, 2023
pixelNeRF: Neural Radiance Fields from One or Few Images

pixelNeRF: Neural Radiance Fields from One or Few Images Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa UC Berkeley arXiv: http://arxiv.org/abs/2

Alex Yu 1k Jan 04, 2023
Using pretrained language models for biomedical knowledge graph completion.

LMs for biomedical KG completion This repository contains code to run the experiments described in: Scientific Language Models for Biomedical Knowledg

Rahul Nadkarni 41 Nov 30, 2022
Image segmentation with private İstanbul Dataset

Image Segmentation This repo was created for academic research and test result. Repo will update after academic article online. This repo contains wei

İrem KÖMÜRCÜ 9 Dec 11, 2022
How to use TensorLayer

How to use TensorLayer While research in Deep Learning continues to improve the world, we use a bunch of tricks to implement algorithms with TensorLay

zhangrui 349 Dec 07, 2022
Code associated with the paper "Towards Understanding the Data Dependency of Mixup-style Training".

Mixup-Data-Dependency Code associated with the paper "Towards Understanding the Data Dependency of Mixup-style Training". Running Alternating Line Exp

Muthu Chidambaram 0 Nov 11, 2021
This project provides an unsupervised framework for mining and tagging quality phrases on text corpora with pretrained language models (KDD'21).

UCPhrase: Unsupervised Context-aware Quality Phrase Tagging To appear on KDD'21...[pdf] This project provides an unsupervised framework for mining and

Xiaotao Gu 146 Dec 22, 2022
Libraries, tools and tasks created and used at DeepMind Robotics.

Libraries, tools and tasks created and used at DeepMind Robotics.

DeepMind 270 Nov 30, 2022
Source code and dataset for ACL2021 paper: "ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning".

ERICA Source code and dataset for ACL2021 paper: "ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive L

THUNLP 75 Nov 02, 2022
CT Based COVID 19 Diagnose by Image Processing and Deep Learning

This project proposed the deep learning and image processing method to undertake the diagnosis on 2D CT image and 3D CT volume.

1 Feb 08, 2022
Official Matlab Implementation for "Tiny Obstacle Discovery by Occlusion-aware Multilayer Regression", TIP 2020

Tiny Obstacle Discovery by Occlusion-aware Multilayer Regression Official Matlab Implementation for "Tiny Obstacle Discovery by Occlusion-aware Multil

Xuefeng 5 Jan 15, 2022
A Review of Deep Learning Techniques for Markerless Human Motion on Synthetic Datasets

HOW TO USE THIS PROJECT A Review of Deep Learning Techniques for Markerless Human Motion on Synthetic Datasets Based on DeepLabCut toolbox, we run wit

1 Jan 10, 2022
Python package for covariance matrices manipulation and Biosignal classification with application in Brain Computer interface

pyRiemann pyRiemann is a python package for covariance matrices manipulation and classification through Riemannian geometry. The primary target is cla

447 Jan 05, 2023
Pneumonia Detection using machine learning - with PyTorch

Pneumonia Detection Pneumonia Detection using machine learning. Training was done in colab: DEMO: Result (Confusion Matrix): Data I uploaded my datase

Wilhelm Berghammer 12 Jul 07, 2022
GyroSPD: Vector-valued Distance and Gyrocalculus on the Space of Symmetric Positive Definite Matrices

GyroSPD Code for the paper "Vector-valued Distance and Gyrocalculus on the Space of Symmetric Positive Definite Matrices" accepted at NeurIPS 2021. Re

Federico Lopez 12 Dec 12, 2022
image scene graph generation benchmark

Scene Graph Benchmark in PyTorch 1.7 This project is based on maskrcnn-benchmark Highlights Upgrad to pytorch 1.7 Multi-GPU training and inference Bat

Microsoft 303 Dec 27, 2022
Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Improving Transferability of Representations via Augmentation-Aware Self-Supervision Accepted to NeurIPS 2021 TL;DR: Learning augmentation-aware infor

hankook 38 Sep 16, 2022