encoder

encoder

Configuration classes for observation encoders of different data modalities.

input_keys exposes only user-facing observation keys (camera names, proprioceptive keys). Language is excluded: the tokenizer rewrites it to SampleKey.TOKENIZED_OBSERVATIONS during preprocessing, and language/VLM encoders bind to that internal key automatically. VLM encoder configs therefore list only their vision keys; the tokenized text is routed to the language tower without user config.

LanguageEncoderConfig does not inherit from EncoderConfig because LanguageEncoder's constructor does not accept input_keys — its input specification is tied to the tokenized-observations key.

EncoderConfig dataclass

EncoderConfig(_target_=MISSING, input_keys=MISSING, pretrained=False, frozen=False, model_dtype='${experiment.precision}')

Base encoder configuration.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
input_keys	list[str]	Observation keys consumed as inputs.
pretrained	bool	Whether to use pretrained weights.
frozen	bool	Whether to freeze encoder weights.
model_dtype	str | None	Precision string from experiment config (e.g. "bf16-mixed").

SpatialDepthEncoderConfig dataclass

SpatialDepthEncoderConfig(_target_='versatil.models.encoding.encoders.depth.spatial.SpatialDepthEncoder', input_keys=MISSING, pretrained=False, frozen=False, model_dtype='${experiment.precision}', backbone=MISSING, batch_norm_handling=value, pooling_method=value, intermediate_layer_index=None, lora_config=None)

Bases: EncoderConfig

Spatial depth encoder configuration for backbones producing (B, C, H, W) feature maps.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
backbone	str	timm backbone name producing spatial feature maps.
batch_norm_handling	str	BatchNorm strategy: keep, freeze, or replace.
pooling_method	str	Spatial pooling applied to the feature map, or null to keep it.
intermediate_layer_index	int | None	Backbone stage the features are taken from, or null for the last.
lora_config	LoRAAdaptationConfig | None	LoRA adaptation settings, or null to fine-tune directly.

DFormerEncoderConfig dataclass

DFormerEncoderConfig(_target_='versatil.models.encoding.encoders.cross_modal.rgbd.dformerv2.DFormerEncoder', input_keys=(lambda: [LEFT.value, DEPTH.value])(), pretrained=False, frozen=False, model_dtype='${experiment.precision}', variant=value, pretrained_weights=value, pooling_method=value, lora_config=None)

Bases: EncoderConfig

DFormer RGB+Depth encoder configuration.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
input_keys	list[str]	Input keys for RGB and depth.
variant	str	Model variant (S/B/L).
pretrained_weights	str	Which checkpoint family to download from https://huggingface.co/bbynku/DFormerv2 when pretrained is set: the ImageNet backbone or the NYU/SUNRGBD finetuned models.
pooling_method	str	Feature pooling method (spatial_softmax or global_average).
lora_config	LoRAAdaptationConfig | None	Optional LoRA adapter configuration applied to the stage linears.

GeometricRGBDEncoderConfig dataclass

GeometricRGBDEncoderConfig(_target_='versatil.models.encoding.encoders.cross_modal.rgbd.geometric_rgbd.GeometricRGBDEncoder', input_keys=(lambda: [LEFT.value, DEPTH.value])(), pretrained=False, frozen=False, model_dtype='${experiment.precision}', embedding_dimension=512, number_of_heads=8, ffn_dimension=2048, patch_size=16, pooling_method=value)

Bases: EncoderConfig

Geometric RGB+Depth encoder configuration.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
input_keys	list[str]	Input keys for RGB and depth observations.
embedding_dimension	int	Dimension of patch embeddings and attention.
number_of_heads	int	Number of attention heads.
ffn_dimension	int	Hidden dimension of the feed-forward network.
patch_size	int	Size of image patches for the patch embedding.
pooling_method	str	Feature pooling method applied after attention.

ProprioEncoderConfig dataclass

ProprioEncoderConfig(_target_='versatil.models.encoding.encoders.proprioceptive.base.ProprioceptiveEncoder', input_keys=MISSING, pretrained=False, frozen=False, model_dtype='${experiment.precision}', output_dim=128, hidden_dimensions=None, activation=value, dropout=0.1)

Bases: EncoderConfig

State encoder configuration for proprioceptive data.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
output_dim	int	Output feature dimension.
hidden_dimensions	list[int] | None	Hidden layer dimensions. If None or [], creates simple linear layer. If [128], creates one hidden layer. If [256, 128], creates two hidden layers.
activation	str	Activation function from ActivationFunction enum.
dropout	float	Dropout rate between layers.

VLMEncoderConfig dataclass

VLMEncoderConfig(_target_='versatil.models.encoding.encoders.cross_modal.vision_language.vlm_encoder.VLMEncoder', input_keys=MISSING, pretrained=False, frozen=False, model_dtype='${experiment.precision}', model_name=MISSING, pooling_method=value, lora_config=None)

Bases: EncoderConfig

VLM encoder configuration for image-text embedding models.

its input_keys should only include vision keys; the tokenized text is routed to the language

tower automatically via the fixed key SampleKey.TOKENIZED_OBSERVATIONS, so it doesn't

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
model_name	str	HuggingFace model identifier for the VLM.
pooling_method	str	Feature pooling strategy for vision and language outputs.
lora_config	LoRAAdaptationConfig | None	Optional LoRA adapter configuration.

LanguageEncoderConfig dataclass

LanguageEncoderConfig(_target_='versatil.models.encoding.encoders.language.language.LanguageEncoder', model_name=value, pooling_method=value, pretrained=False, frozen=False, lora_config=None, max_token_len=128, use_embeddings_only=False, model_dtype='${experiment.precision}', trust_remote_code=False)

Language encoder configuration.

Note

It doesn't inherit from EncoderConfig because its input key is fixed, i.e. SampleKey.TOKENIZED_OBSERVATIONS

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
model_name	str	Model identifier from LanguageEncoderType.
pooling_method	str	How to extract features from transformer output.
pretrained	bool	Whether to use pretrained weights.
frozen	bool	Whether to freeze backbone weights.
lora_config	LoRAAdaptationConfig | None	Optional LoRA adapter configuration.
max_token_len	int	Maximum token sequence length for the encoder.
use_embeddings_only	bool	If True, use only the pretrained token embedding layer.
model_dtype	str | None	Precision string from experiment config (e.g. "bf16-mixed").
trust_remote_code	bool	Whether to allow HuggingFace models that ship custom modeling code (e.g. nvidia/llama-nemotron-embed).

ImageEncoderConfig dataclass

ImageEncoderConfig(_target_=MISSING, input_keys=MISSING, pretrained=False, frozen=False, model_dtype='${experiment.precision}', backbone=MISSING)

Bases: EncoderConfig

Abstract base config for image encoders.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
backbone	str	Backbone name.

SpatialRGBEncoderConfig dataclass

SpatialRGBEncoderConfig(_target_='versatil.models.encoding.encoders.rgb.spatial.SpatialRGBEncoder', input_keys=MISSING, pretrained=False, frozen=False, model_dtype='${experiment.precision}', backbone=MISSING, pooling_method=value, batch_norm_handling=value, intermediate_layer_index=None, lora_config=None)

Bases: ImageEncoderConfig

Spatial RGB encoder configuration for backbones producing (B, C, H, W) feature maps.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
pooling_method	str	Spatial pooling applied to the feature map, or null to keep it.
batch_norm_handling	str	BatchNorm strategy: keep, freeze, or replace.
intermediate_layer_index	int | None	Backbone stage the features are taken from, or null for the last.
lora_config	LoRAAdaptationConfig | None	LoRA adaptation settings, or null to fine-tune directly.

ConditionalCNNEncoderConfig dataclass

ConditionalCNNEncoderConfig(_target_='versatil.models.encoding.encoders.rgb.conditional_cnn.ConditionalCNNEncoder', input_keys=MISSING, pretrained=False, frozen=False, model_dtype='${experiment.precision}', backbone=MISSING, condition_key=MISSING, conditioning_dimension=MISSING, pooling_method=value, batch_norm_handling=value, lora_config=None)

Bases: ImageEncoderConfig

Feature-conditioned CNN encoder configuration.

this vision encoder receives as conditioning an encoded feature from

another unconditional encoder in the pipeline. Conditional encoders are always run after conditional encoders, and their condition_key must be the output key of the desired unconditional encoder's feature.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
condition_key	str	Key for the conditioning feature tensor.
conditioning_dimension	int	Dimensionality of the conditioning feature.
pooling_method	str	Feature pooling strategy.
batch_norm_handling	str	How to handle batch normalization layers.
lora_config	LoRAAdaptationConfig | None	Optional PEFT LoRA adapter configuration.

FlatRGBEncoderConfig dataclass

FlatRGBEncoderConfig(_target_='versatil.models.encoding.encoders.rgb.flat.FlatRGBEncoder', input_keys=MISSING, pretrained=False, frozen=False, model_dtype='${experiment.precision}', backbone=MISSING, pooling_method=value, image_size=None, intermediate_layer_index=None, lora_config=None)

Bases: ImageEncoderConfig

Flat RGB encoder configuration for backbones producing (B, S, D) token sequences.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
pooling_method	str	Feature pooling strategy for patch tokens. Defaults to CLS token selection.
image_size	int | None	Optional image size passed to timm during backbone construction.
intermediate_layer_index	int | None	Optional intermediate layer index for feature extraction. Negative values index from the end.
lora_config	LoRAAdaptationConfig | None	Optional PEFT LoRA adapter configuration.

DinoV2SigLIPRGBEncoderConfig dataclass

DinoV2SigLIPRGBEncoderConfig(_target_='versatil.models.encoding.encoders.rgb.dinov2_siglip.DinoV2SigLIPRGBEncoder', input_keys=MISSING, pretrained=False, frozen=False, model_dtype='${experiment.precision}', backbone=value, lora_config=None)

Bases: ImageEncoderConfig

DINOv2+SigLIP RGB encoder configuration for fused patch-token sequences.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
backbone	str	DINOv2+SigLIP paired backbone identifier.
lora_config	LoRAAdaptationConfig | None	Optional LoRA adapter configuration for the timm towers.