paligemma

paligemma

PaliGemma VLM component for VLA decoders.

PaliGemmaVLM

PaliGemmaVLM(input_keys, pretrained, frozen, model_name=value, attention_type=value, model_dtype=None, max_text_length=None, lora_config=None)

Bases: HuggingFaceGenerativeVLM

PaliGemma VLM with per-camera sequential image encoding.

Each camera image is encoded through SigLIP + multi-modal projector separately, then concatenated with language embeddings before the Gemma language-model pass. Scaling follows the HF reference: text embeddings are scaled by sqrt(hidden_dimension) inside Gemma's embedding module, image tokens enter unscaled.

Initialize the PaliGemma VLM component.

Parameters:

Name	Type	Description	Default
input_keys	str | list[str]	Input keys for cameras and tokenized text.	required
pretrained	bool	Whether to load pretrained HuggingFace weights.	required
frozen	bool	Whether to freeze all model weights.	required
model_name	str	HuggingFace model identifier for PaliGemma.	value
attention_type	str	Attention implementation (e.g. SDPA, eager).	value
model_dtype	str | None	Precision string from experiment config (e.g. "bf16-mixed").	None
max_text_length	int | None	Maximum text sequence length. Defaults to model's max_position_embeddings if None.	None
lora_config	LoRAAdaptation | None	Optional LoRA adapter configuration.	None

Source code in src/versatil/models/decoding/generative_language_models/vision_language/paligemma.py

def __init__(
    self,
    input_keys: str | list[str],
    pretrained: bool,
    frozen: bool,
    model_name: str = PaliGemmaModelType.PALIGEMMA2_3B_224.value,
    attention_type: str = AttentionImplementation.SDPA.value,
    model_dtype: str | None = None,
    max_text_length: int | None = None,
    lora_config: LoRAAdaptation | None = None,
):
    """Initialize the PaliGemma VLM component.

    Args:
        input_keys: Input keys for cameras and tokenized text.
        pretrained: Whether to load pretrained HuggingFace weights.
        frozen: Whether to freeze all model weights.
        model_name: HuggingFace model identifier for PaliGemma.
        attention_type: Attention implementation (e.g. SDPA, eager).
        model_dtype: Precision string from experiment config (e.g. ``"bf16-mixed"``).
        max_text_length: Maximum text sequence length. Defaults to model's
            max_position_embeddings if None.
        lora_config: Optional LoRA adapter configuration.
    """
    super().__init__(
        input_keys=input_keys,
        pretrained=pretrained,
        frozen=frozen,
        model_name=model_name,
        attention_type=attention_type,
        model_dtype=model_dtype,
        max_text_length=max_text_length,
        lora_config=lora_config,
    )

forward_language_model

forward_language_model(input_ids=None, inputs_embeds=None, attention_mask=None, past_key_values=None, use_cache=False, cache_position=None, position_ids=None, output_hidden_states=True)

Run the Gemma language tower with PaliGemma 1-indexed positions.

Source code in src/versatil/models/decoding/generative_language_models/vision_language/paligemma.py

def forward_language_model(
    self,
    input_ids: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    past_key_values: Cache | tuple[tuple[torch.Tensor, ...], ...] | None = None,
    use_cache: bool = False,
    cache_position: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    output_hidden_states: bool = True,
) -> CausalLanguageModelOutput:
    """Run the Gemma language tower with PaliGemma 1-indexed positions."""
    paligemma_position_ids = None
    if position_ids is not None:
        paligemma_position_ids = position_ids + 1
    return super().forward_language_model(
        input_ids=input_ids,
        inputs_embeds=inputs_embeds,
        attention_mask=attention_mask,
        past_key_values=past_key_values,
        use_cache=use_cache,
        cache_position=cache_position,
        position_ids=paligemma_position_ids,
        output_hidden_states=output_hidden_states,
    )

get_explainability_targets

get_explainability_targets()

Return the projector output used as PaliGemma visual context.

PaliGemma encodes each camera independently with SigLIP and projects patch tokens into the Gemma hidden size through multi_modal_projector. For multi-camera inputs this layer is invoked once per camera, and the base VLM is_multi_camera flag lets the explainability runner select the requested camera invocation.

Returns:

Type	Description
list[VisionExplanationTarget]	One token-sequence target over PaliGemma image tokens.

Source code in src/versatil/models/decoding/generative_language_models/vision_language/paligemma.py

def get_explainability_targets(self) -> list[VisionExplanationTarget]:
    """Return the projector output used as PaliGemma visual context.

    PaliGemma encodes each camera independently with SigLIP and projects
    patch tokens into the Gemma hidden size through ``multi_modal_projector``.
    For multi-camera inputs this layer is invoked once per camera, and the
    base VLM ``is_multi_camera`` flag lets the explainability runner select
    the requested camera invocation.

    Returns:
        One token-sequence target over PaliGemma image tokens.
    """
    return [
        VisionExplanationTarget(
            layer=self._get_paligemma_model().multi_modal_projector,
            target_kind=ExplanationTargetKind.TOKEN_SEQUENCE.value,
            activation_layout=ActivationLayout.NLC.value,
            patch_grid=self._get_image_token_grid(),
        )
    ]