A Survey on Deep Learning Technique for Video Segmentation

A Survey on Deep Learning Technique for Video Segmentation
Wenguan Wang, Tianfei Zhou, Fatih Porikli, David Crandall, and Luc Van Gool.

Contributing

Please feel free to create issues or pull requests to add papers.

Welcome any discussions on video segmentation at

1. Introduction

Video segmentation, i.e., partitioning video frames into multiple segments or objects, plays a critical role in a broad range of practical applications, from enhancing visual effects in movie, to understanding scenes in autonomous driving, to virtual background creation in video conferencing. In this survey, we comprehensively review two basic lines of research — video object segmentation and video semantic segmentation — by introducing their respective task settings, background concepts, perceived need, development history, and main challenges. In particular, we review eight sub-fields as given in the following figure:

2. Deep Learning-based Video Object Segmentation

• 2.1 Automatic Video Object Segmentation (AVOS)
  • 2.1.1 Deep Learning Module based
  • 2.1.2 Pixel Instance Embedding based
  • 2.1.3 Short-term Information Encoding
  • 2.1.4 Long-term Context Encoding
  • 2.1.5 Un-/Weakly-supervised based
  • 2.1.6 Instance-level AVOS
• 2.2 Semi-automatic Video Object Segmentation (SVOS)
  • 2.2.1 Online Fine-tuning based
  • 2.2.2 Propagation based
  • 2.2.3 Matching based
  • 2.2.4 Box-initialization based
  • 2.2.5 Un-/Weakly-supervised based
  • 2.2.6 Other Specific Methods
• 2.3 Interactive Video Object Segmentation (IVOS)
  • 2.3.1 Interaction-propagation based
  • 2.3.2 Other Methods
• 2.4 Language-guided Video Object Segmentation (LVOS)
  • 2.4.1 Dynamic Convolution based
  • 2.4.2 Capsule Routing based
  • 2.4.3 Attention based

3. Deep Learning-based Video Semantic Segmentation

• 3.1 (Instance-agnostic) Video Semantic Segmentation (VSS)
  • 3.1.1 Efforts towards More Accurate Segmentation
  • 3.1.2 Efforts towards Faster Segmentation
  • 3.1.3 Semi-/Weakly-supervised based
• 3.2 Video Instance Segmentation (VIS)
• 3.3 Video Panoptic Segmentation (VPS)

4. Datasets

• Popular Datasets in VOS and VSS

Citation

If you find our survey and repository useful for your research, please consider citing our paper:

@article{wang2021survey,
  title={A survey on deep learning technique for video segmentation},
  author={Wang, Wenguan and Zhou, Tianfei and Porikli, Fatih and Crandall, David and Van Gool, Luc},
  journal={arXiv preprint arXiv:2107.01153},
  year={2021}
}