使用 Tokenizers 的 tokenizers

PreTrainedTokenizerFast 依赖于 Tokenizers 库.从 Tokenizers library acquiredtokenizerscan be loaded very simplyTransformers.

在详细讨论之前,Let's first create a dummy with a few lines of codetokenizer:

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])

tokenizer.pre_tokenizer = Whitespace()
files = [...]
tokenizer.train(files, trainer)

We now have a trainer for the file we defined.We can continue to use it at runtime,or save it to JSON file for future reuse.

直接从 tokenizer 对象加载

让我们看看如何在TransformersUse this in the library tokenizer 对象.By accepting instantiated tokenizer 对象作为参数,PreTrainedTokenizerFast Classes allow easy instantiation:

from transformers import PreTrainedTokenizerFast

fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)

This object is now shared by all methodsTransformerstokenizer！更多信息请访问 tokenizer 页面.

从 JSON 文件加载

为了从 JSON 文件中加载 tokenizer,Let's save first tokenizer:

tokenizer.save("tokenizer.json")

The path where we saved this file can be used tokenizer_file 参数传递给 PreTrainedTokenizerFast 初始化方法:

from transformers import PreTrainedTokenizerFast

fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")

This object is now shared by all methodsTransformerstokenizer！更多信息请访问 tokenizer 页面.

本文是抱抱脸（Hugging Face）教程中文翻译,仅学习使用

原文链接