Thesis unscramble TransFG: A Transformer Architecture for Fine - grained Recognition

This article is the application of transformers in fine-grained fields.

Problem: Transformer has not been used in the field of image segmentation

Contribution points: 1. The input of the vision transformer divides the image into patches, but there is no overlap. The article is changed to split patches and use overlap (this can only be counted as a trick)

2.Part Selection Module

In layman's terms, the input of the last layer is different from the vision transformer, that is, the weights of all layers before the last layer (shown in the red box) are multiplied, and then the tokens with great weight are filtered and spliced ​​together as the L-th layer.enter.

First of all, the output of the L-1 layer is originally like this:

The weight of a previous layer is as follows:

The value range of the subscript l is (1,2,...,L-1)

Assuming there are K self-attention heads, the weight in each head is:

The value range of the superscript i is (0,1,...,K)

The weights are multiplied to all layers before the last layer:

Then select the A_k tokens with the largest weight as the input of the last layer.

So after processing, its input can be expressed as:

From the perspective of the model architecture, it can be found that the token with the arrow in the red box is selected, and it is also the token with a large weight after the weight is multiplied. The blue box on the right represents the patch corresponding to the selected token.

3.Contrastive loss

The author said that the features between different categories in the fine-grained field are very similar, so it is not enough to use the cross-entropy loss to learn the features. After the cross-entropy loss, a new Contrastive loss is added, which introduces the cosine similarity.(Used to estimate the similarity of two vectors), the more similar the vectors, the greater the cosine similarity.

The purpose of the author's proposal of this loss function is to reduce the similarity of "classification tokens" of different categories, and maximize the similarity of the same "classification tokens".The formula of Contrastive loss is as follows:

Where a is an artificially set constant.

So the overall function is:

Experiment:

Comparison with CNN and ViT on several datasets for sub-classification, SOTA