The difference between classification, object detection, semantic segmentation, and instance segmentation

There are many tasks in computer vision, including image classification, object detection, semantic segmentation, instance segmentation and panorama segmentation, etc. What is the difference between them?

1, Image Classification

Image classification (left in the picture below) is to determine the classification of the image, for example, in the learning classification, the data set is a person (person), a sheep (sheep),There are four kinds of dog (dog) and cat (cat). Image classification requires which categories are contained in a given image output image. For example, the example in the figure below contains three types of person, sheep and dog.

2, Object detection

target detection (above right) is simply what's in the picture?Where are they?(frame them in rectangles)

Currently commonly used target detection algorithms include Faster R-CNN and YOLO-based target detection algorithms

3, semantic segmentation

Target segmentation in the usual sense refers to semantic segmentation

Semantic segmentation (left in the picture below) is to distinguish every pixel in the picture, not just a rectangular frame.But different instances of the same object do not need to be segmented separately.On the left side of the picture below, label it as people, sheep, dogs, and grass.Instead of Sheep 1, Sheep 2, Sheep 3, Sheep 4, Sheep 5, etc.

4, Instance segmentation

Instance segmentation (right above) is actually a combination of **target detection and semantic segmentation**.Relative to the bounding box of target detection, instance segmentation can be accurate to the edge of the object; relative to semantic segmentation, instance segmentation needs to label different individuals of the same object on the map (sheep 1, sheep 2, sheep 3...)

The most commonly used instance segmentation algorithm is Mask R-CNN.

Mask R-CNN performs pixel-level segmentation by adding a branch to Faster R-CNN that outputs a binary mask indicating whether a given pixel is part of the target object: this branch is based on convolutionalFully Convolutional Network for Neural Network Feature Mapping.Taking a given convolutional neural network feature map as input, the output is a matrix where all positions where a pixel belongs to the object is represented by 1 and other positions are represented by 0, which is the binary mask.

Once these masks are generated, Mask R-CNN combines RoIAlign with classification and bounding boxes from Faster R-CNN for accurate segmentation:

5, Panoramic segmentation

Panoramic segmentationis a combination of semantic segmentation and instance segmentation.Different from instance segmentation: instance segmentation only detects objects in the image and segments the detected objects, while panorama segmentation is to detect and segment all objects in the image, including the background.