smolvlm

smolvlm

SmolVLM/Idefics3 component for VLA decoders.

SmolVLM

SmolVLM(input_keys, pretrained, frozen, model_name=value, attention_type=value, model_dtype=None, max_text_length=None, lora_config=None)

Bases: HuggingFaceGenerativeVLM

SmolVLM/Idefics3 component with native multi-image support.

Camera images are stacked along the num_images dimension and processed through SigLIP + connector in a single call, then concatenated with language embeddings before the SmolLM pass.

Initialize the SmolVLM component.

Parameters:

Name	Type	Description	Default
input_keys	str | list[str]	Input keys for cameras and tokenized text.	required
pretrained	bool	Whether to load pretrained HuggingFace weights.	required
frozen	bool	Whether to freeze all model weights.	required
model_name	str	HuggingFace model identifier for SmolVLM.	value
attention_type	str	Attention implementation (e.g. SDPA, eager).	value
model_dtype	str | None	Precision string from experiment config (e.g. "bf16-mixed").	None
max_text_length	int | None	Maximum text sequence length. Defaults to model's max_position_embeddings if None.	None
lora_config	LoRAAdaptation | None	Optional LoRA adapter configuration.	None

Source code in src/versatil/models/decoding/generative_language_models/vision_language/smolvlm.py

def __init__(
    self,
    input_keys: str | list[str],
    pretrained: bool,
    frozen: bool,
    model_name: str = SmolVLMModelType.SMOLVLM_256M.value,
    attention_type: str = AttentionImplementation.SDPA.value,
    model_dtype: str | None = None,
    max_text_length: int | None = None,
    lora_config: LoRAAdaptation | None = None,
):
    """Initialize the SmolVLM component.

    Args:
        input_keys: Input keys for cameras and tokenized text.
        pretrained: Whether to load pretrained HuggingFace weights.
        frozen: Whether to freeze all model weights.
        model_name: HuggingFace model identifier for SmolVLM.
        attention_type: Attention implementation (e.g. SDPA, eager).
        model_dtype: Precision string from experiment config (e.g. ``"bf16-mixed"``).
        max_text_length: Maximum text sequence length. Defaults to model's
            max_position_embeddings if None.
        lora_config: Optional LoRA adapter configuration.
    """
    super().__init__(
        input_keys=input_keys,
        pretrained=pretrained,
        frozen=frozen,
        model_name=model_name,
        attention_type=attention_type,
        model_dtype=model_dtype,
        max_text_length=max_text_length,
        lora_config=lora_config,
    )

get_explainability_targets

get_explainability_targets()

Return the connector output used by SmolVLA as visual context.

SmolVLM stacks cameras before image encoding, and HuggingFace flattens the stack to B * num_cameras while producing one token sequence per camera. The explainability package splits this stacked batch back into camera-specific heatmaps through is_stacked_camera_batch.

Returns:

Type	Description
list[VisionExplanationTarget]	One token-sequence target over SmolVLM image tokens.

Source code in src/versatil/models/decoding/generative_language_models/vision_language/smolvlm.py

def get_explainability_targets(self) -> list[VisionExplanationTarget]:
    """Return the connector output used by SmolVLA as visual context.

    SmolVLM stacks cameras before image encoding, and HuggingFace flattens
    the stack to ``B * num_cameras`` while producing one token sequence per
    camera. The explainability package splits this stacked batch back into
    camera-specific heatmaps through ``is_stacked_camera_batch``.

    Returns:
        One token-sequence target over SmolVLM image tokens.
    """
    return [
        VisionExplanationTarget(
            layer=self.vlm.model.connector,
            target_kind=ExplanationTargetKind.TOKEN_SEQUENCE.value,
            activation_layout=ActivationLayout.NLC.value,
            patch_grid=self._get_image_token_grid(),
        )
    ]