image_mixin

image_mixin

Mixin for image encoders with multi-camera support.

ImageEncoderMixin

Bases: ABC

Shared logic for encoders that process camera images.

Provides camera key storage, multi-camera detection, vision feature naming, and multi-camera encode dispatch. Camera modality checks are performed by experiment validation because they require observation-space metadata.

set_camera_metadata

set_camera_metadata(camera_metadata)

Store observation-space camera metadata for runtime camera routing.

Parameters:

Name	Type	Description	Default
camera_metadata	dict[str, CameraMetadata]	Observation-space camera metadata keyed by observation key.	required

Source code in src/versatil/models/encoding/encoders/image_mixin.py

def set_camera_metadata(self, camera_metadata: dict[str, CameraMetadata]) -> None:
    """Store observation-space camera metadata for runtime camera routing.

    Args:
        camera_metadata: Observation-space camera metadata keyed by
            observation key.
    """
    declared_camera_keys = [
        key
        for key in self.input_specification.keys
        if key not in _NON_CAMERA_INPUT_KEYS
    ]
    unknown_keys = [
        key for key in declared_camera_keys if key not in camera_metadata
    ]
    if unknown_keys:
        raise ValueError(
            f"{type(self).__name__} declares camera keys {unknown_keys} "
            "that are not part of the observation-space cameras "
            f"{sorted(camera_metadata)}."
        )
    self.camera_keys = declared_camera_keys
    self.camera_metadata = camera_metadata
    self.is_multi_camera = len(self.camera_keys) > 1

RGBEncoderMixin

Bases: ImageEncoderMixin

Mixin for encoders processing RGB camera images.

DepthEncoderMixin

Bases: ImageEncoderMixin

Mixin for encoders processing single-channel depth camera images.

RGBDEncoderMixin

Bases: ImageEncoderMixin

Mixin for encoders processing both RGB and depth camera images.

validate_input_metadata

validate_input_metadata(key, metadata)

Validate that RGBD inputs use camera metadata.

Parameters:

Name	Type	Description	Default
key	str	Observation key being validated.	required
metadata	BaseMetadata	Metadata from the observation space for this key.	required

Returns:

Type	Description
str | None	Error message if incompatible, None if valid.

Source code in src/versatil/models/encoding/encoders/image_mixin.py

def validate_input_metadata(self, key: str, metadata: BaseMetadata) -> str | None:
    """Validate that RGBD inputs use camera metadata.

    Args:
        key: Observation key being validated.
        metadata: Metadata from the observation space for this key.

    Returns:
        Error message if incompatible, None if valid.
    """
    if not isinstance(metadata, CameraMetadata):
        return f"Expected CameraMetadata for '{key}', got {type(metadata).__name__}"
    return None

resize_to_target_size

resize_to_target_size(images, target_height, target_width)

Resize images preserving aspect ratio and zero-pad to target size.

Scales down so the largest dimension fits the target, then zero-pads the shorter dimension. No-op if images already match the target.

Parameters:

Name	Type	Description	Default
images	Tensor	Image tensor of shape (B*T, C, H, W); forward() flattens the temporal axis into the batch before dispatching here.	required
target_height	int	Target height in pixels.	required
target_width	int	Target width in pixels.	required

Returns:

Type	Description
Tensor	Resized and padded tensor of shape (B, C, target_height, target_width).

Source code in src/versatil/models/encoding/encoders/image_mixin.py

def resize_to_target_size(
    images: torch.Tensor,
    target_height: int,
    target_width: int,
) -> torch.Tensor:
    """Resize images preserving aspect ratio and zero-pad to target size.

    Scales down so the largest dimension fits the target, then zero-pads
    the shorter dimension. No-op if images already match the target.

    Args:
        images: Image tensor of shape (B*T, C, H, W); ``forward()`` flattens
                the temporal axis into the batch before dispatching here.
        target_height: Target height in pixels.
        target_width: Target width in pixels.

    Returns:
        Resized and padded tensor of shape (B, C, target_height, target_width).
    """
    current_height, current_width = images.shape[2], images.shape[3]
    if current_height == target_height and current_width == target_width:
        return images
    ratio = max(current_height / target_height, current_width / target_width)
    resized_height = int(current_height / ratio)
    resized_width = int(current_width / ratio)
    resized = torch.nn.functional.interpolate(
        images,
        size=(resized_height, resized_width),
        mode="bilinear",
        align_corners=False,
    )
    pad_height = target_height - resized_height
    pad_width = target_width - resized_width
    if pad_height > 0 or pad_width > 0:
        resized = torch.nn.functional.pad(
            resized, (0, pad_width, 0, pad_height), value=0.0
        )
    return resized