1. Overview of the scene

Retrieval system exists in many products we use everyday , For example, commodity search system 、 Academic literature retrieval department, etc , This scheme provides a complete implementation of the retrieval system . Limit the scene It's the user through Enter the search term Query, Quickly find similar documents in massive data .

Semantic retrieval （ Also known as vector based retrieval ）, It means that the retrieval system is no longer confined to users Query Literal itself , It can accurately capture users Query The real intention behind it and use it to search , So as to more accurately return the most consistent results to the user . Find the vector representation of text by using the most advanced semantic indexing model , Index them in high-dimensional vector space , And measure the similarity between the query vector and the index document , Thus, the defects brought by keyword index are solved .

For example, the following Two sets of text Pair, If the similarity is calculated based on keywords , The similarity between the two groups is the same . From the practical semantic point of view , The similarity of the first group is higher than that of the second group .

 How to place the license plate on the front of the car      How to install the front license plate 
 How to place the license plate on the front of the car      How to install the rear license plate 

The key of semantic retrieval system is , Recall using semantics rather than keywords , Achieve more accuracy 、 The purpose of a broader recall of similar results .

2. Product function introduction

Usually, the data of retrieval business is relatively large , Will be divided into Recall （ Indexes ）、 Sort Two links . The recall phase is mainly from at least In the ten million level candidate collection , Filter out relevant documents , such The number of candidate sets will be greatly reduced , In the later sorting stage, you can use some complex models to do fine or personalized sorting . It is generally used Multiple recall strategy （ for example Key words recall 、 Hot recall 、 Semantic recall Combination, etc ）, After aggregation of multiple recall results , After unified scoring, the best TopK Result .

2.1 System features

• Low threshold

  • Build up... Hand in hand Search system
  • The retrieval system can be built without labeling data
  • Provide Training 、 forecast 、ANN Engine one-stop capability

• The effect is good

  • Professional solutions for a variety of data scenarios
    • Only unsupervised data : SimCSE
    • There are only surveillance data : InBatchNegative
    • With unsupervised data and There are monitoring data ： Fusion model
  • Further optimize the scheme : Domain oriented pre training Domain-adaptive Pretraining

• Fast performance

  • be based on Paddle Inference Fast extraction of vectors
  • be based on Milvus Fast query and high-performance database building

2.2 Functional architecture

There are two types of indexing methods ： Literal based keyword index ; Semantic index . Semantic index Can better Representing semantic information , Solve situations that are not literally similar but semantically similar . This system gives a semantic indexing scheme , Other schemes can be used in actual business . The architecture and functions of the whole scheme are described in detail below .