Detailed explanation of IK tokenizer

What is IK tokenizer

Word segmentation: that is, to divide a paragraph of Chinese or other words into keywords, we will segment our own information when searching, segment the data in the database or the index database, and then perform a matchOperation, the default Chinese word segmentation is to treat each character as a word, such as "I love Wang Yaotian" will be divided into: "I", "love", "王", "Yao", "天", which is obviouslyDoes not meet the requirements, so we need to install the Chinese tokenizer ik to solve this problem.

Ik provides two word segmentation algorithms: ik_smart and ik_max_word, where ik_smart is the least segmentation and ik_max_word is the most fine-grained segmentation!We will test in a while

Install

1. GitHub - medcl/elasticsearch-analysis-ik: The IK Analysis plugin integrates Lucene IK analyzer into elasticsearch, support customized dictionary.

2. After downloading, put it into our elasticsearch plugin!

3. Restart and observe ES, you can see that the ik tokenizer is loaded

4. The elasticsearch-plugin can use this command to view the loaded plugins

5. Use kibana to test!

View different word segmentation effects

ik_smart is the least segmentation

ik_max_word is the most fine-grained division!Possibility of exhausting thesaurus Dictionaries

We input super like Wang Yaotian

Problem found: Wang Yaotian was disassembled

This kind of word you need needs to be added to the dictionary of our tokenizer by yourself

ik tokenizer adds its own configuration!

Restart ES to see details

Test Wang Yaotian again to see the effect

In the future, we need to configure it by ourselves, and the word segmentation can be configured in the dic file defined by ourselves