MiVOS (CVPR 2021) - Scribble To Mask
Ho Kei Cheng, Yu-Wing Tai, Chi-Keung Tang
[arXiv] [Paper PDF] [Project Page]
A simplistic network that turns scribbles to mask. It supports multi-object segmentation using soft-aggregation. Don't expect SOTA results from this model!
Overall structure and capabilities
| MiVOS | Mask-Propagation | Scribble-to-Mask | |
|---|---|---|---|
| DAVIS/YouTube semi-supervised evaluation |
|
|
|
| DAVIS interactive evaluation |
|
|
|
| User interaction GUI tool |
|
|
|
| Dense Correspondences |
|
|
|
| Train propagation module |
|
|
|
| Train S2M (interaction) module |
|
|
|
| Train fusion module |
|
|
|
| Generate more synthetic data |
|
|
|
Requirements
The package versions shown here are the ones that I used. You might not need the exact versions.
- PyTorch
1.6.0 - torchvision
0.7.0 - opencv-contrib
4.2.0 - davis-interactive (https://github.com/albertomontesg/davis-interactive)
- gitpython for training
- gdown for downloading pretrained models
Refer to the official PyTorch guide for installing PyTorch/torchvision. The rest can be installed by:
pip install opencv-contrib-python gitpython gdown
Pretrained model
Download and put the model in ./saves/. Alternatively use the provided download_model.py.
Interactive GUI
python interactive.py --image <image>
Controls:
Mouse Left - Draw scribbles
Mouse middle key - Switch positive/negative
Key f - Commit changes, clear scribbles
Key r - Clear everything
Key d - Switch between overlay/mask view
Key s - Save masks into a temporary output folder (./output/)
Known issues
The model almost always needs to focus on at least one object. It is very difficult to erase all existing masks from an image using scribbles.
Training
Datasets
- Download and extract LVIS training set.
- Download and extract a set of static image segmentation datasets. These are already downloaded for you if you used the
download_datasets.pyin Mask-Propagation.
├── lvis
│ ├── lvis_v1_train.json
│ └── train2017
├── Scribble-to-Mask
└── static
├── BIG_small
└── ...
Commands
Use the deeplabv3plus_resnet50 pretrained model provided here.
CUDA_VISIBLE_DEVICES=0,1 OMP_NUM_THREADS=4 python -m torch.distributed.launch --master_port 9842 --nproc_per_node=2 train.py --id s2m --load_deeplab <path_to_deeplab.pth>
Credit
Deeplab implementation and pretrained model: https://github.com/VainF/DeepLabV3Plus-Pytorch.
Citation
Please cite our paper if you find this repo useful!
@inproceedings{MiVOS_2021,
title={Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion},
author={Cheng, Ho Kei and Tai, Yu-Wing and Tang, Chi-Keung},
booktitle={CVPR},
year={2021}
}
Contact: [email protected]
I put the static folder outside this repository as you mentioned. It is confusing that I can use the same datasets for the pretraining propagation module, the train.py in Mask-Propagation works fine.
I used command : 
454 Jan 03, 2023
6 Dec 07, 2022
11 Sep 28, 2022
501 Dec 28, 2022
54 Dec 06, 2022
49 Dec 27, 2022
4.3k Jan 07, 2023
17 Nov 21, 2022
76 Sep 05, 2022
529 Dec 30, 2022
16 Dec 25, 2022
9 Nov 04, 2022
9 Nov 08, 2022
24 May 20, 2022
10 May 18, 2022
36 Dec 08, 2022
34 Nov 22, 2022
39 Jul 21, 2022
1.6k Dec 31, 2022
14 Oct 31, 2022