Skip to content

fast

fast

FAST action processor loading compatible with transformers>=5.0.

Transformers 5.0 changed ProcessorMixin._get_arguments_from_pretrained to load any attribute containing "tokenizer" from a subfolder. This breaks custom processors like physical-intelligence/fast's UniversalActionProcessor, whose bpe_tokenizer attribute has no corresponding subfolder on the hub. This module bypasses AutoProcessor.from_pretrained by loading the config and tokenizer separately, then constructing the processor directly.

load_fast_processor 

load_fast_processor(model_path)

Load a FAST action processor from a HuggingFace hub model or local directory.

Parameters:

Name Type Description Default
model_path str

HuggingFace model ID or local directory path.

 required

Returns:

Type Description
ProcessorMixin

Instantiated FAST action processor.
Source code in src/versatil/data/tokenization/fast.py 
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
def load_fast_processor(model_path: str) -> ProcessorMixin:  # UniversalActionProcessor
    """Load a FAST action processor from a HuggingFace hub model or local directory.

    Args:
        model_path: HuggingFace model ID or local directory path.

    Returns:
        Instantiated FAST action processor.
    """
    local_path = Path(model_path)
    is_local = local_path.is_dir()
    config = _load_processor_config(
        model_path=model_path, local_path=local_path, is_local=is_local
    )
    bpe_tokenizer = _load_bpe_tokenizer(
        model_path=model_path, local_path=local_path, is_local=is_local
    )
    processor_class = _resolve_processor_class(config=config, model_path=model_path)
    return processor_class(
        bpe_tokenizer=bpe_tokenizer,
        scale=config.get("scale", 10),
        vocab_size=config.get("vocab_size", 2048),
        min_token=config.get("min_token", 0),
        action_dim=config.get("action_dim"),
        time_horizon=config.get("time_horizon"),
    )