Code-free deep segmentation for computational pathology

Overview

NoCodeSeg: Deep segmentation made easy!

This is the official repository for the manuscript "Code-free development and deployment of deep segmentation models for digital pathology", submitted to Frontiers in Medicine.

The repository contains trained deep models for epithelium segmentation of HE and CD3 immunostained WSIs, as well as source code relevant for importing/exporting annotations/predictions in QuPath, from DeepMIB, and FastPathology. See here for how to download the 251 annotated WSIs.

Getting started

Watch the video

A video tutorial of the proposed pipeline was published on YouTube. It demonstrates the steps for:

  • Downloading and installing the softwares
  • QuPath
    • Create a project, then export annotations as patches with label files
    • Export patches from unannotated images for prediction in DeepMIB
    • (later) Import predictions for MIB and FastPathology as annotations
  • MIB
    • Use the annotated patches/labels exported from QuPath
    • Configuring and training deep segmentation models (i.e. U-Net/SegNet)
    • Use the trained U-net to predict unannotated patches exported from QuPath
    • Export trained models into the ONNX format for use in FastPathology
  • FastPathology
    • Importing and creating a configuration file for the DeepMIB exported ONNX model
    • Create a project and load WSIs into a project
    • Use the U-Net ONNX model to render predictions on top of the WSI in real time
    • Export full sized WSI tiffs for import into QuPath

Data

The 251 annotated WSIs are being processed before publishing on DataverseNO, where it will be made openly available for anyone

Reading annotations

The annotations are stored as tiled, pyramidal TIFFs, which makes it easy to generate patches from the data without the need for any preprocessing. Reading these files and working with them to generate training data, is already described in the tutorial video above.

TL;DR: Load TIFF as annotations in QuPath using provided groovy script and exporting these as labelled tiles.

Reading annotation in Python

However, if you wish to use Python, the annotations can be read exactly the same way as regular WSIs (for instance using OpenSlide):

import openslide

reader = ops.OpenSlide("path-to-annotation-image.tiff")
patch = reader.read_region(location=(x, y), level, size=(w, h))
reader.close()

Pixels here will be one-to-one with the original WSI. To generate patches for training, it is also possible to use pyFAST, which does the patching for you. For an example see here.

Citation

Please, consider citing our paper, if you find the work useful:

  @misc{pettersen2021codefree,
  title={Code-free development and deployment of deep segmentation models for digital pathology}, 
  author={Henrik Sahlin Pettersen and Ilya Belevich and Elin Synnøve Røyset and Erik Smistad and Eija Jokitalo and Ingerid Reinertsen and Ingunn Bakke and André Pedersen},
  year={2021},
  eprint={2111.08430},
  archivePrefix={arXiv},
  primaryClass={q-bio.QM}}
Comments
  • Create model multiple classes

    Create model multiple classes

    Hi @andreped

    Thanks for this great software and workflow. I would like to generate a model which classifies the epithelia in the colon from WSIs into villi (usually at the boundary) and crypts (circular structures). Essentially, there would be 3 classes,

    • background
    • villi and
    • crypts

    My plan is to use your models to generate annotations from my WSIs, import into QuPath, correct them, and then split annotations into different classes.

    How do I configure image export for 3 different classes from QuPath, train in MIB and then run on FastPathology?

    Is it possible to do this and is this a good workflow idea?

    Cheers Pradeep

    enhancement good first issue 
    opened by pr4deepr 19
  • Add instructions on how to run WSI-level predictions

    Add instructions on how to run WSI-level predictions

    As our TileImporter script does not support multi-class, the alternative method is to run predictions on WSI level.

    This requires you to do something slightly different when running predictions from what was done in the tutorial video.

    However, I don't see that there is any documentations for this. This should be added to assist users.

    enhancement 
    opened by andreped 10
  • Import tiles script QuPath

    Import tiles script QuPath

    Hi @andreped When I try the import pyramidal tiff script to improve annotations generated by FastPathology, the annotations are smaller by 4 times compared to the WSI. I can't enter a value to control for downsample in the new script anymore. It looks like the script has been updated compared to the on in the video I've reused the old version of the script to import annotations for now .

    Cheers Pradeep

    bug enhancement 
    opened by pr4deepr 7
  • Export Tiles from QuPath with multiple classes

    Export Tiles from QuPath with multiple classes

    First some context: I am attempting to obtain multi-class segmentation of cerebellum tissue with different cellular layers corresponding to separate classes. I have annotated four classes: 'EGL', 'Molecular layer', 'IGL', and 'WM' on QuPath. I tested export tiles with multiple classes from QuPath using two the two different scripts available. Here are the issues I am having with both:

    1. The script by @andreped is giving me an error:

    ERROR: MissingMethodException at line 61: No signature of method: qupath.lib.images.writers.TileExporter.labeledServer() is applicable for argument types: (qupath.lib.images.servers.LabeledImageServer$Builder) values: [[email protected]] Possible solutions: labeledServer(qupath.lib.images.servers.ImageServer)

    ERROR: org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:70) org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46) org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139) Script16.run(Script16.groovy:62) org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:317) org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:155) qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:982) qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:914) qupath.lib.gui.scripting.DefaultScriptEditor.executeScript(DefaultScriptEditor.java:829) qupath.lib.gui.scripting.DefaultScriptEditor$2.run(DefaultScriptEditor.java:1345) java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) java.base/java.util.concurrent.FutureTask.run(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.base/java.lang.Thread.run(Unknown Source)

    1. The script by @pr4deepr is yielding what I need (see below), however, many of the tiles are just background. Is there a way to incorporate the glassthreshold in the script by @andreped to avoid the tiles with too much background? Additionally, in order to get this script to work, I had to edit this line to .multichannelOutput(false).

    20211026_Javid_01Image_01 vsi - 20x  d=5 79044,x=10376,y=36318,w=2965,h=2964

    bug 
    opened by aaronsathya 6
  • generic multi-class support?

    generic multi-class support?

    The current importTiles script does not seem to support importing patches of multiple labels. This one is quite annoying to add multi-class support for, but is something we could try to do early next week? Let us schedule a session.

    There have been implemented multi-class variants for the other scripts (i.e., exportTiles and importPyramidalTIFF), but these have not been tested in various scenarios. Currently, these are split into separate scripts; one for single-class and one for multi-class, but I think the multi-class scripts might actually handle the single-class situation as well. Could you run some checks in your pipeline to see if the multi-class scripts are sufficient?

    enhancement 
    opened by andreped 4
  • Result from FastPathology fails to import in QuPath

    Result from FastPathology fails to import in QuPath

    This was observed and discussed in another Issue in the FastPathology repo: https://github.com/AICAN-Research/FAST-Pathology/issues/45

    Opening an Issue here to track the issue.

    This is likely due to the changed in how the predictions are stored on disk in the new FastPathology version v1.0.1.

    bug 
    opened by andreped 2
  • Download count is not updating

    Download count is not updating

    Currently, the current download count in the download shield is fixed.

    That is because there was no easy way to catch the current count through markdown.

    I attempted to create a JS and Python solution for doing that, which were to be run periodically through github actions, but that would result in lots of redundant commits, that are not relevant for the user.

    I'll have it fixed until I have time to come up with a better solution.

    enhancement help wanted 
    opened by andreped 1
  • multiclass_export_tiles

    multiclass_export_tiles

    The main change is multichannelOutput is set to true and there is a label export for each threshold.

    // Create an ImageServer where the pixels are derived from annotations
    //added option for multi-class export
    def labelServer = new LabeledImageServer.Builder(imageData)
        .backgroundLabel(0, ColorTools.WHITE) // Specify background label (usually 0 or 255)
        .downsample(downsample)    // Choose server resolution; this should match the resolution at which tiles are exported
        .addLabel('Epithelia', 1)      // Choose output labels (Define Threshold for each value for each label)
        .addLabel('Crypt', 2)
        .multichannelOutput(true)  // If true, each label is a different channel (required for multiclass probability)
        .build()
    
    

    Ideally, a loop to run through each label would be nice, so we don't have to define it manually

    opened by pr4deepr 0
  • Multi-class tile import?

    Multi-class tile import?

    The current tileImport script does not support multiclass tiles, that is produced prediction tiles generated from MIB by a trained multi-class model.

    It is possible to import predictions from MIB to QuPath by predicting on the full WSI in MIB which produces a stitched prediction image, which can be imported in QuPath using the importStitchedTIFfromMIBscript, similarly as done for FastPathology. And therefore, for the full pipeline a tile importer is not critical for using our pipeline.

    However, there are scenarios where having a tile importer is beneficial over the other alternative, for instance if one only wishes to run prediction on a smaller region of the WSI (based on segmentations in QuPath) - which can be especially helpful if the image is extremely large. This can make inference a lot faster.

    It is possible to do such a workflow in FastPathology, given that you provide the region of interest to run inference on. However, no such method has been made easily accessible. It would also require annotations or a model that can produce these ROI segmentation to use as a preprocessing step for the next patch-wise model.

    Therefore, having multi-class support for the tile importer could be a valuable feature.

    enhancement 
    opened by andreped 1
Releases(download-badge)
  • download-badge(Jun 21, 2022)

  • v1.0.0(Jun 13, 2022)

    Released code relevant for Frontiers article.

    Code is freezed to a release to keep supporting the old versions of FastPathology and QuPath used in the tutorial video.

    Future source code will be made compatible with more recent versions of both softwares.

    Full Changelog: https://github.com/andreped/NoCodeSeg/commits/v1.0.0

    Source code(tar.gz)
    Source code(zip)
Owner
André Pedersen
PhD Candidate in Medical Technology at NTNU | Master of Science at SINTEF Health Research
André Pedersen
PyTorch implementation of Neural Combinatorial Optimization with Reinforcement Learning.

neural-combinatorial-rl-pytorch PyTorch implementation of Neural Combinatorial Optimization with Reinforcement Learning. I have implemented the basic

Patrick E. 454 Jan 06, 2023
Toolkit for collecting and applying prompts

PromptSource Promptsource is a toolkit for collecting and applying prompts to NLP datasets. Promptsource uses a simple templating language to programa

BigScience Workshop 998 Jan 03, 2023
Collection of machine learning related notebooks to share.

ML_Notebooks Collection of machine learning related notebooks to share. Notebooks GAN_distributed_training.ipynb In this Notebook, TensorFlow's tutori

Sascha Kirch 14 Dec 22, 2022
Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter

ACE Please find the preliminary version published at BMVC 2020 in the folder BMVC_version, and its extended journal version in Journal_version. Datase

28 Dec 25, 2022
DaReCzech is a dataset for text relevance ranking in Czech

Dataset DaReCzech is a dataset for text relevance ranking in Czech. The dataset consists of more than 1.6M annotated query-documents pairs,

Seznam.cz a.s. 8 Jul 26, 2022
The code for our paper "AutoSF: Searching Scoring Functions for Knowledge Graph Embedding"

AutoSF The code for our paper "AutoSF: Searching Scoring Functions for Knowledge Graph Embedding" and this paper has been accepted by ICDE2020. News:

AutoML Research 64 Dec 17, 2022
DataCLUE: 国内首个以数据为中心的AI测评(含模型分析报告)

DataCLUE: A Benchmark Suite for Data-centric NLP You can get the english version of README. 以数据为中心的AI测评(DataCLUE) 内容导引 章节 描述 简介 介绍以数据为中心的AI测评(DataCLUE

CLUE benchmark 135 Dec 22, 2022
Learning Multiresolution Matrix Factorization and its Wavelet Networks on Graphs

Project Learning Multiresolution Matrix Factorization and its Wavelet Networks on Graphs, https://arxiv.org/pdf/2111.01940.pdf. Authors Truong Son Hy

5 Jun 28, 2022
DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021)

Evaluation, Training, Demo, and Inference of DeFMO DeFMO: Deblurring and Shape Recovery of Fast Moving Objects (CVPR 2021) Denys Rozumnyi, Martin R. O

Denys Rozumnyi 139 Dec 26, 2022
SegNet including indices pooling for Semantic Segmentation with tensorflow and keras

SegNet SegNet is a model of semantic segmentation based on Fully Comvolutional Network. This repository contains the implementation of learning and te

Yuta Kamikawa 172 Dec 23, 2022
MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts (ICLR 2022)

MetaShift: A Dataset of Datasets for Evaluating Distribution Shifts and Training Conflicts This repo provides the PyTorch source code of our paper: Me

88 Jan 04, 2023
A dataset for online Arabic calligraphy

Calliar Calliar is a dataset for Arabic calligraphy. The dataset consists of 2500 json files that contain strokes manually annotated for Arabic callig

ARBML 114 Dec 28, 2022
Autotype on websites that have copy-paste disabled like Moodle, HackerEarth contest etc.

Autotype A quick and small python script that helps you autotype on websites that have copy paste disabled like Moodle, HackerEarth contests etc as it

Tushar 32 Nov 03, 2022
Pixray is an image generation system

Pixray is an image generation system

pixray 883 Jan 07, 2023
wmctrl ported to Python Ctypes

work in progress wmctrl is a command that can be used to interact with an X Window manager that is compatible with the EWMH/NetWM specification. wmctr

Iyad Ahmed 22 Dec 31, 2022
A task-agnostic vision-language architecture as a step towards General Purpose Vision

Towards General Purpose Vision Systems By Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem Overview Welcome to the official code base f

AI2 79 Dec 23, 2022
Restricted Boltzmann Machines in Python.

How to Use First, initialize an RBM with the desired number of visible and hidden units. rbm = RBM(num_visible = 6, num_hidden = 2) Next, train the m

Edwin Chen 928 Dec 30, 2022
torchbearer: A model fitting library for PyTorch

Note: We're moving to PyTorch Lightning! Read about the move here. From the end of February, torchbearer will no longer be actively maintained. We'll

631 Jan 04, 2023
Official Implementation for "StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery" (ICCV 2021 Oral)

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (ICCV 2021 Oral) Run this model on Replicate Optimization: Global directions: Mapper: Check ou

3.3k Jan 05, 2023
Project of 'TBEFN: A Two-branch Exposure-fusion Network for Low-light Image Enhancement '

TBEFN: A Two-branch Exposure-fusion Network for Low-light Image Enhancement Codes for TMM20 paper "TBEFN: A Two-branch Exposure-fusion Network for Low

KUN LU 31 Nov 06, 2022