Two-stage Knowledge Learning (Collation-Examination) is designed on the basis of knowledge ledgedistillation and the method of contrastive learning is introduced.

Method

Collaborative Knowledge Transfer (CKT)

Progressive Feature Projector (PFP)

Project the features of the teacher and student networks into a common feature space, and calculate the L1 loss of the two projected features: Q represents the number of coding layers, Ti represents the ith teacher network, and S represents the student network
Projection is achieved by a small network consisting of several convolutional blocks with stride 1 and a ReLU activation function

Bidirectional Feature Matching

Project the projected features of the teacher network back to the original input space, and calculate the L1 loss of the two features to ensure the validity of the projection

Multi-contrastive Regularization (Two-stage Knowledge Learning)

Contrastive Regularization: v, v+, v- represent predicted samples, positive samples (target samples), and negative samples respectively; Ψ represents the features extracted by VGG-19; R is the number of negative samples.

Soft Contrastive Regularization. (Knowledge Collation (KC).)

Soft: Use teacher's predictions as positive samples to avoid the challenges of early learning

Hard Contrastive Regularization.（Knowledge Examination (KE).)

Hard: Use groundtruth as a positive sample to learn directly after training for a period of time