最終更新:2025-02-14 (金) 10:46:10 (33d)  

Hugging Face/モデル/tokenizer.json
Top / Hugging Face / モデル / tokenizer.json

https://huggingface.co/docs/transformers/fast_tokenizers

保存

  • tokenizer.save( "tokenizer.json" )
↑

ロード

  • transformers.PreTrainedTokenizerFast(tokenizer_file= "tokenizer.json" )
    • A path to a local JSON file representing a previously serialized tokenizers. Tokenizer? object from 🤗 tokenizers.



検索：