Cvpr2022 𞓜 collaborative dual stream visual language pre training model for cross modal retrieval

Thesis link ：https://arxiv.org/abs/2204.07441

Large scale single tower pre training model , It has achieved amazing retrieval results in cross modal Retrieval . Unfortunately , Because most of them use time-consuming real parameters and cross modal interaction , The retrieval efficiency is very low . lately , image CLIP and ALIGN In this way, the two tower model with high reasoning efficiency also shows good results , However , They only consider instance level alignment between modes （ Therefore, there is still room for improvement ）. To overcome these limitations , We propose a novel collaborative two tower visual language pre training model , Referred to as COTS. in general , What we proposed COTS It is to improve the image quality by strengthening the interaction between modes - The effect of text retrieval . In addition to instance level alignment through momentum contrast learning , We also propose two additional cross modal interactions .(1)Token Level interaction — Without using the argument interaction model , We designed a masking visual language modeling （MVLM） Learning objectives of , The variational self encoder is used for visual coding , Visual information can be generated for each image token Level markers .(2) Task level interaction — An algorithm is designed between text to image and image to text retrieval tasks KL- Align learning objectives , The probability distribution of each task is calculated by using the negative sample queue in momentum contrast learning . In a fair comparison , What we proposed COTS The best result of all the two tower methods , Compared with the latest single tower method ,COTS Show considerable ability （ But reasoning is fast 10,800 times ）. meanwhile , What we proposed COTS It is also applicable to the retrieval from text to video , In widely used MSR-VTT The best results so far have been obtained on the data set .