最終更新:2025-02-14 (金) 10:15:01 (33d)
transformers.LlamaTokenizer
Top / transformers.LlamaTokenizer
https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaTokenizer
メモ
- The LLaMA tokenizer is a BPE model based on SentencePiece.
- One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. “Banana”), the tokenizer does not prepend the prefix space to the string.
関連
- transformers.LlamaTokenizerFast?
