当前位置：网站首页>bpe Chinese tokens

bpe Chinese tokens

2022-08-06 07:53:00 【Kun Li】

BPE Algorithm of Algorithms to Understand in One Minute - KnowingFrom Deep Learning Natural Language ProcessingThe author of the WeChat public account zenRRan found one of the scripts of a machine translation when summarizing the analysis of the experimental data yesterday. The algorithm used in it is the BPE algorithm. It felt very high at first, because I always heard things with algorithm hats.Just think 666...https://zhuanlan.zhihu.com/p/38130825

bpe is very detailed as mentioned above.

The Chinese word code, currently mainly vocab.txt, is the 21128 version, which is the same version in Chinese_clip or Chinese_bert.

bpe English word encoding, currently the bpe_simple_vocab_16e6 version, 262145 version, used by both dalle and clip.

原网站

版权声明
本文为[Kun Li]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/218/202208060747060019.html

边栏推荐

猜你喜欢

随机推荐