Skip to content

huggingface

huggingface

Compatibility helpers for HuggingFace tokenizer loading.

load_huggingface_tokenizer 

load_huggingface_tokenizer(tokenizer_model, trust_remote_code=False)

Load a HuggingFace tokenizer with local compatibility patches applied.

Parameters:

Name Type Description Default
tokenizer_model str | Path

HuggingFace model identifier or local path.

 required
trust_remote_code bool

Whether to allow tokenizers that ship custom code.

 False
Source code in src/versatil/data/tokenization/huggingface.py 
38
39
40
41
42
43
44
45
46
47
48
49
50
51
def load_huggingface_tokenizer(
    tokenizer_model: str | Path,
    trust_remote_code: bool = False,
) -> PreTrainedTokenizerBase:
    """Load a HuggingFace tokenizer with local compatibility patches applied.

    Args:
        tokenizer_model: HuggingFace model identifier or local path.
        trust_remote_code: Whether to allow tokenizers that ship custom code.
    """
    _patch_roberta_processing_cls_alias()
    return AutoTokenizer.from_pretrained(
        tokenizer_model, trust_remote_code=trust_remote_code
    )