E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation

Related tags

Deep Learninge2ec
Overview

E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation

city

E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation
Tao Zhang, Shiqing Wei, Shunping Ji
CVPR 2022

Any questions or discussions are welcomed!

Installation

Please see INSTALL.md.

Performances

We re-tested the speed on a single RTX3090.

Dtataset AP Image size FPS
SBD val 59.2 512×512 59.60
COCO test-dev 33.8 original size 35.25
KINS val 34.0 768×2496 12.39
Cityscapes val 34.0 1216×2432 8.58

The accuracy and inference speed of the contours at different stages on SBD val set. We also re-tested the speed on a single RTX3090.

stage init coarse final final-dml
AP 51.4 55.9 58.8 59.2
FPS 101.73 91.35 67.48 59.6

The accuracy and inference speed of the contours at different stages on coco val set.

stage init coarse final final-dml
AP 27.8 31.6 33.5 33.6
FPS 80.97 72.81 42.55 35.25

Testing

Testing on COCO

  1. Download the pretrained model here or Baiduyun(password is e2ec).

  2. Prepared the COCO dataset according to the INSTALL.md.

  3. Test:

    # testing segmentation accuracy on coco val set
    python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True
    # testing detection accuracy on coco val set
    python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --eval bbox
    # testing the speed
    python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --type speed
    # testing the contours of specified stage(init/coarse/final/final-dml)
    python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --stage coarse
    # testing on coco test-dev set, run and submit data/result/results.json
    python test.py coco --checkpoint /path/to/model_coco.pth --with_nms True --dataset coco_test
    

Testing on SBD

  1. Download the pretrained model here or Baiduyun(password is e2ec).

  2. Prepared the SBD dataset according to the INSTALL.md.

  3. Test:

    # testing segmentation accuracy on SBD
    python test.py sbd --checkpoint /path/to/model_sbd.pth
    # testing detection accuracy on SBD
    python test.py sbd --checkpoint /path/to/model_sbd.pth --eval bbox
    # testing the speed
    python test.py sbd --checkpoint /path/to/model_sbd.pth --type speed
    # testing the contours of specified stage(init/coarse/final/final-dml)
    python test.py sbd --checkpoint /path/to/model_sbd.pth --stage coarse
    

Testing on KINS

  1. Download the pretrained model here or Baiduyun(password is e2ec).

  2. Prepared the KINS dataset according to the INSTALL.md.

  3. Test:

    Maybe you will find some troules, such as object of type <class 'numpy.float64'> cannot be safely interpreted as an integer. Please modify the /path/to/site-packages/pycocotools/cooceval.py. Replace np.round((0.95 - .5) / .05) in lines 506 and 507 with int(np.round((0.95 - .5) / .05)).

    # testing segmentation accuracy on KINS
    python test.py kitti --checkpoint /path/to/model_kitti.pth
    # testing detection accuracy on KINS
    python test.py kitti --checkpoint /path/to/model_kitti.pth --eval bbox
    # testing the speed
    python test.py kitti --checkpoint /path/to/model_kitti.pth --type speed
    # testing the contours of specified stage(init/coarse/final/final-dml)
    python test.py kitti --checkpoint /path/to/model_kitti.pth --stage coarse
    

Testing on Cityscapes

  1. Download the pretrained model here or Baiduyun(password is e2ec).

  2. Prepared the KINS dataset according to the INSTALL.md.

  3. Test:

    We will soon release the code for e2ec with multi component detection. Currently only supported for testing e2ec performance on cityscapes dataset.

    # testing segmentation accuracy on Cityscapes with coco evaluator
    python test.py cityscapesCoco --checkpoint /path/to/model_cityscapes.pth
    # with cityscapes official evaluator
    python test.py cityscapes --checkpoint /path/to/model_cityscapes.pth
    # testing the detection accuracy
    python test.py cityscapesCoco \
    --checkpoint /path/to/model_cityscapes.pth --eval bbox
    # testing the speed
    python test.py cityscapesCoco \
    --checkpoint /path/to/model_cityscapes.pth --type speed
    # testing the contours of specified stage(init/coarse/final/final-dml)
    python test.py cityscapesCoco \
    --checkpoint /path/to/model_cityscapes.pth --stage coarse
    # testing on test set, run and submit the result file
    python test.py cityscapes --checkpoint /path/to/model_cityscapes.pth \
    --dataset cityscapes_test
    

Evaluate boundary AP

  1. Install the Boundary IOU API according boundary iou.

  2. Testing segmentation accuracy with coco evaluator.

  3. Using offline evaluation pipeline.

    python /path/to/boundary_iou_api/tools/coco_instance_evaluation.py \
        --gt-json-file /path/to/annotation_file.json \
        --dt-json-file data/result/result.json \
        --iou-type boundary
    

Visualization

  1. Download the pretrained model.

  2. Visualize:

    # inference and visualize the images with coco pretrained model
    python visualize.py coco /path/to/images \
    --checkpoint /path/to/model_coco.pth --with_nms True
    # you can using other pretrained model, such as cityscapes 
    python visualize.py cityscapesCoco /path/to/images \
    --checkpoint /path/to/model_cityscapes.pth
    # if you want to save the visualisation, please specify --output_dir
    python visualize.py coco /path/to/images \
    --checkpoint /path/to/model_coco.pth --with_nms True \
    --output_dir /path/to/output_dir
    # visualize the results at different stage
    python visualize.py coco /path/to/images \
    --checkpoint /path/to/model_coco.pth --with_nms True --stage coarse
    # you can reset the score threshold, default is 0.3
    python visualize.py coco /path/to/images \
    --checkpoint /path/to/model_coco.pth --with_nms True --ct_score 0.1
    # if you want to filter some of the jaggedness caused by dml 
    # please using post_process
    python visualize.py coco /path/to/images \
    --checkpoint /path/to/model_coco.pth --with_nms True \
    --with_post_process True
    

Training

We have only released the code for single GPU training, multi GPU training with ddp will be released soon.

Training on SBD

python train_net.py sbd --bs $batch_size
# if you do not want to use dinamic matching loss (significantly improves 
# contour detail but introduces jaggedness), please set --dml as False
python train_net.py sbd --bs $batch_size --dml False

Training on KINS

python train_net.py kitti --bs $batch_size

Training on Cityscapes

python train_net.py cityscapesCoco --bs $batch_size

Training on COCO

In fact it is possible to achieve the same accuracy without training so many epochs.

# first to train with adam
python train_net.py coco --bs $batch_size
# then finetune with sgd
python train_net.py coco_finetune --bs $batch_size \
--type finetune --checkpoint data/model/139.pth

Training on the other dataset

If the annotations is in coco style:

  1. Add dataset information to dataset/info.py.

  2. Modify the configs/coco.py, reset the train.dataset , model.heads['ct_hm'] and test.dataset. Maybe you also need to change the train.epochs, train.optimizer['milestones'] and so on.

  3. Train the network.

    python train_net.py coco --bs $batch_size
    

If the annotations is not in coco style:

  1. Prepare dataset/train/your_dataset.py and dataset/test/your_dataset.py by referring to dataset/train/base.py and dataset/test/base.py.

  2. Prepare evaluator/your_dataset/snake.py by referring to evaluator/coco/snake.py.

  3. Prepare configs/your_dataset.py and by referring to configs/base.py.

  4. Train the network.

    python train_net.py your_dataset --bs $batch_size
    

Citation

If you find this project helpful for your research, please consider citing using BibTeX below:

@article{zhang2022e2ec,
  title={E2EC: An End-to-End Contour-based Method for High-Quality High-Speed Instance Segmentation},
  author={Zhang, Tao and Wei, Shiqing and Ji, Shunping},
  journal={arXiv preprint arXiv:2203.04074},
  year={2022}
}

Acknowledgement

Code is largely based on Deep Snake. Thanks for their wonderful works.

Owner
zhangtao
zhangtao
We present a regularized self-labeling approach to improve the generalization and robustness properties of fine-tuning.

Overview This repository provides the implementation for the paper "Improved Regularization and Robustness for Fine-tuning in Neural Networks", which

NEU-StatsML-Research 21 Sep 08, 2022
Code for our CVPR 2021 Paper "Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes".

Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes (CVPR 2021) Project page | Paper | Colab | Colab for Drawing App Rethinking Style

CompVis Heidelberg 153 Jan 04, 2023
High performance Cross-platform Inference-engine, you could run Anakin on x86-cpu,arm, nv-gpu, amd-gpu,bitmain and cambricon devices.

Anakin2.0 Welcome to the Anakin GitHub. Anakin is a cross-platform, high-performance inference engine, which is originally developed by Baidu engineer

514 Dec 28, 2022
Progressive Growing of GANs for Improved Quality, Stability, and Variation

Progressive Growing of GANs for Improved Quality, Stability, and Variation — Official TensorFlow implementation of the ICLR 2018 paper Tero Karras (NV

Tero Karras 5.9k Jan 05, 2023
An open source python library for automated feature engineering

"One of the holy grails of machine learning is to automate more and more of the feature engineering process." ― Pedro Domingos, A Few Useful Things to

alteryx 6.4k Jan 03, 2023
LibMTL: A PyTorch Library for Multi-Task Learning

LibMTL LibMTL is an open-source library built on PyTorch for Multi-Task Learning (MTL). See the latest documentation for detailed introductions and AP

765 Jan 06, 2023
Tutorial page of the Climate Hack, the greatest hackathon ever

Tutorial page of the Climate Hack, the greatest hackathon ever

UCL Artificial Intelligence Society 12 Jul 02, 2022
[제 13회 투빅스 컨퍼런스] OK Mugle! - 장르부터 멜로디까지, Content-based Music Recommendation

Ok Mugle! 🎵 장르부터 멜로디까지, Content-based Music Recommendation 'Ok Mugle!'은 제13회 투빅스 컨퍼런스(2022.01.15)에서 진행한 음악 추천 프로젝트입니다. Description 📖 본 프로젝트에서는 Kakao

SeongBeomLEE 5 Oct 09, 2022
Interpretable and Generalizable Person Re-Identification with Query-Adaptive Convolution and Temporal Lifting

QAConv Interpretable and Generalizable Person Re-Identification with Query-Adaptive Convolution and Temporal Lifting This PyTorch code is proposed in

Shengcai Liao 166 Dec 28, 2022
A python script to lookup Passport Index Dataset

visa-cli A python script to lookup Passport Index Dataset Installation pip install visa-cli Usage usage: visa-cli [-h] [-d DESTINATION_COUNTRY] [-f]

rand-net 16 Oct 18, 2022
A visualisation tool for Deep Reinforcement Learning

DRLVIS - Visualising Deep Reinforcement Learning Created by Marios Sirtmatsis with the support of Alex Bäuerle. DRLVis is an application used for visu

Marios Sirtmatsis 1 Nov 04, 2021
Official implementation of "CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding" (CVPR, 2022)

CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding (CVPR'22) Paper Link | Project Page Abstract : Manual an

Mohamed Afham 152 Dec 23, 2022
StarGAN - Official PyTorch Implementation (CVPR 2018)

StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation

Yunjey Choi 5.1k Dec 30, 2022
Efficient Sparse Attacks on Videos using Reinforcement Learning

EARL This repository provides a simple implementation of the work "Efficient Sparse Attacks on Videos using Reinforcement Learning" Example: Demo: Her

12 Dec 05, 2021
Job Assignment System by Real-time Emotion Detection

Emotion-Detection Job Assignment System by Real-time Emotion Detection Emotion is the essential role of facial expression and it could provide a lot o

1 Feb 08, 2022
Utilities and information for the signals.numer.ai tournament

dsignals Utilities and information for the signals.numer.ai tournament using eodhistoricaldata.com eodhistoricaldata.com provides excellent historical

Degerhan Usluel 23 Dec 18, 2022
Code for Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks Under construction. Description Code for Phase diagram of S

Rodrigo Veiga 3 Nov 24, 2022
Convenient tool for speeding up the intern/officer review process.

icpc-app-screen Convenient tool for speeding up the intern/officer applicant review process. Eliminates the pain from reading application responses of

1 Oct 30, 2021
Facial Image Inpainting with Semantic Control

Facial Image Inpainting with Semantic Control In this repo, we provide a model for the controllable facial image inpainting task. This model enables u

Ren Yurui 8 Nov 22, 2021
RetinaFace: Deep Face Detection Library in TensorFlow for Python

RetinaFace is a deep learning based cutting-edge facial detector for Python coming with facial landmarks.

Sefik Ilkin Serengil 512 Dec 29, 2022