最終更新:2024-11-25 (月) 14:44:19 (58d)
Tokenizers
Top / Tokenizers
Tokenizers provides an implementation of today’s most used tokenizers, with a focus on performance and versatility. These tokenizers are also used in Transformers.
https://github.com/huggingface/tokenizers
https://huggingface.co/docs/tokenizers/index
