tokenizer

tokenizer

ObservationTokenizationConfig dataclass

ObservationTokenizationConfig(tokenizer_model=value, observation_keys=list(), bin_continuous_data=True, num_bins=256, max_token_len=256, raw_text=False, prompt_template=None, padding_strategy=value, trust_remote_code=False)

Configuration for converting observations into text token IDs.

Attributes:

Name	Type	Description
tokenizer_model	str	Language tokenizer model name.
observation_keys	list[str]	Observation keys to include in prompt (order preserved in prompt construction) Example: ["language", "proprio_robot_frame", "proprio_camera_frame"].
bin_continuous_data	bool	Whether to bin continuous observations into quantiles before string conversion.
num_bins	int	Number of discretization bins for continuous observations.
max_token_len	int	Maximum token length for the prompt.
raw_text	bool	Pass language text through unformatted (no "Task:" prefix, no lowercasing). Use for VLM policies (SmolVLA, Pi0) that expect raw text.
prompt_template	str | None	Optional template wrapped around the language instruction in raw-text mode, with an "{instruction}" placeholder. The instruction is lowercased and stripped before insertion (OpenVLA convention).
padding_strategy	str	Padding strategy: "max_length" pads all sequences to max_token_len, "longest" pads to the longest sequence in the batch.
trust_remote_code	bool	Allow tokenizers that ship custom HuggingFace code.

ActionDiscretizerConfig dataclass

ActionDiscretizerConfig(type=value, use_pretrained=True, tokenizer_model='physical-intelligence/fast', num_bins=256, binning_strategy=value, min_value=-1.0, max_value=1.0)

Configuration for discretizing continuous action chunks.

Attributes:

Name	Type	Description
type	str	Strategy that turns continuous action chunks into local discrete action IDs.
use_pretrained	bool	FAST-specific options.
tokenizer_model	str	FAST tokenizer model identifier.
num_bins	int	Binned discretizer options. Uniform binning places equal-width bins over [min_value, max_value]; quantile binning adapts edges to the action distribution and ignores the range bounds.
binning_strategy	str	Strategy mapping continuous values to bins.
min_value	float	Lower bound of the discretized value range.
max_value	float	Upper bound of the discretized value range.

ActionTokenIdMappingConfig dataclass

ActionTokenIdMappingConfig(type=value, language_tokenizer_model=None, num_special_tokens_to_skip=128)

Configuration for mapping action-local IDs into model token IDs.

Attributes:

Name	Type	Description
type	str	Mapping from local action IDs into the model token-id space.
language_tokenizer_model	str | None	Language-tokenizer mapping options.
num_special_tokens_to_skip	int	Vocabulary offset reserved for special tokens.

ActionTokenizationConfig dataclass

ActionTokenizationConfig(action_discretizer=ActionDiscretizerConfig(), token_id_mapping=ActionTokenIdMappingConfig(), max_token_len=128)

Configuration for action tokenization.

Attributes:

Name	Type	Description
action_discretizer	ActionDiscretizerConfig	Discretizer turning continuous actions into bins.
token_id_mapping	ActionTokenIdMappingConfig	Mapping between action bins and vocabulary token ids.
max_token_len	int	Maximum action token sequence length.

TokenizationConfig dataclass

TokenizationConfig(tokenize_observations=False, observation_tokenizer=None, tokenize_actions=False, action_tokenizer=None)

Top-level observation/action tokenization configuration.

Attributes:

Name	Type	Description
tokenize_observations	bool	Whether observations are tokenized into prompts.
observation_tokenizer	ObservationTokenizationConfig | None	Observation tokenizer settings.
tokenize_actions	bool	Whether actions are tokenized.
action_tokenizer	ActionTokenizationConfig | None	Action tokenizer settings.