Panoptic SegFormer is accepted by CVPR'22 and we update our latest paper on arXiv
results on COCO val
| Backbone | Method | Lr Schd | PQ | Config | Download |
|---|---|---|---|---|---|
| R-50 | Panoptic-SegFormer | 1x | 48.0 | config | model |
| R-50 | Panoptic-SegFormer | 2x | 49.6 | config | model |
| R-101 | Panoptic-SegFormer | 2x | 50.6 | config | model |
| PVTv2-B5 (much lighter) | Panoptic-SegFormer | 2x | 55.6 | config | model |
| Swin-L (window size 7) | Panoptic-SegFormer | 2x | 55.8 | config | model |
- Linux
- Python 3.6+
- PyTorch 1.5+
- torchvision
- CUDA 9.2+ (If you build PyTorch from source, CUDA 9.0 is also compatible)
- GCC 5+
- mmcv-full==1.3.4
- mmdet==2.12.0 # higher version may not work
- timm==0.4.5
- einops==0.3.0
- Pillow==8.0.1
- opencv-python==4.5.2
note: PyTorch1.8 has a bug in its adamw.py and it is solved in PyTorch1.9(see), you can easily solve it by comparing the difference.
python setup.py install
When I began this project, mmdet dose not support panoptic segmentation officially. I convert the dataset from panoptic segmentation format to instance segmentation format for convenience.
cd Panoptic-SegFormer
mkdir datasets
cd datasets
ln -s path_to_coco coco
mkdir annotations/
cd annotations
wget http://images.cocodataset.org/annotations/panoptic_annotations_trainval2017.zip
unzip panoptic_annotations_trainval2017.zip
Then the directory structure should be the following:
Panoptic-SegFormer
├── datasets
│ ├── annotations/
│ │ ├── panoptic_train2017/
│ │ ├── panoptic_train2017.json
│ │ ├── panoptic_val2017/
│ │ └── panoptic_val2017.json
│ └── coco/
│
├── config
├── checkpoints
├── easymd
...
cd Panoptic-SegFormer
./tools/convert_panoptic_coco.sh coco
Then the directory structure should be the following:
Panoptic-SegFormer
├── datasets
│ ├── annotations/
│ │ ├── panoptic_train2017/
│ │ ├── panoptic_train2017_detection_format.json
│ │ ├── panoptic_train2017.json
│ │ ├── panoptic_val2017/
│ │ ├── panoptic_val2017_detection_format.json
│ │ └── panoptic_val2017.json
│ └── coco/
│
├── config
├── checkpoints
├── easymd
...
single-machine with 8 gpus.
./tools/dist_train.sh ./configs/panformer/panformer_r50_24e_coco_panoptic.py 8
./tools/dist_test.sh ./configs/panformer/panformer_r50_24e_coco_panoptic.py path/to/model.pth 8
If you use Panoptic SegFormer in your research, please use the following BibTeX entry.
@misc{li2021panoptic,
title={Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers},
author={Zhiqi Li and Wenhai Wang and Enze Xie and Zhiding Yu and Anima Anandkumar and Jose M. Alvarez and Tong Lu and Ping Luo},
year={2021},
eprint={2109.03814},
archivePrefix={arXiv},
primaryClass={cs.CV}
}Mainly based on Defromable DETR from MMdet.
Thanks very much for other open source works: timm, Panoptic FCN, MaskFomer, QueryInst