act

act

Action Chunking Transformer (ACT) architecture for action decoding.

Reference: https://arxiv.org/abs/2304.13705

ACT

ACT(input_keys, action_space, action_heads, observation_space, observation_horizon, prediction_horizon, device, embedding_dimension=256, number_of_heads=8, feedforward_dimension=512, number_of_encoder_layers=6, number_of_decoder_layers=6, activation=value, dropout_rate=0.1, normalize_before=False)

Bases: BaseParallelTransformerDecoder

Action Chunking Transformer network for action decoding.

This architecture: - Encodes multi-camera images into spatial features - Optionally accepts a latent embedding from the algorithm layer (e.g., from VAE) - Convert flat and spatial features into a sequence of token embeddings with shared embedding dimension - Decodes actions in parallel using a DETR-style non-autoregressive transformer with learnable queries - Supports multiple action heads: position, orientation, gripper

Note: Latent action encoding is handled at the Algorithm level, not within this decoder. The decoder expects latent embeddings to be passed via the features dictionary with key LatentKey.POSTERIOR_LATENT.

Initialize ACT-style decoder.

Parameters:

Name	Type	Description	Default
input_keys	list[str]	List of feature keys expected from encoder pipeline	required
action_space	ActionSpace	Action space configuration	required
observation_space	ObservationSpace	Observation space configuration	required
observation_horizon	int	Number of observation timesteps	required
prediction_horizon	int	Number of actions to predict	required
device	str	Device to run the model on	required
embedding_dimension	int	Transformer hidden dimension	256
number_of_heads	int	Number of attention heads	8
feedforward_dimension	int	Feedforward network dimension	512
number_of_encoder_layers	int	Number of transformer encoder layers	6
number_of_decoder_layers	int	Number of transformer decoder layers	6
activation	str	Activation function name	value
dropout_rate	float	Dropout probability	0.1
normalize_before	bool	Use pre-normalization	False

Source code in src/versatil/models/decoding/decoders/factory/act.py

def __init__(
    self,
    input_keys: list[str],
    action_space: ActionSpace,
    action_heads: dict[str, ActionHead],
    observation_space: ObservationSpace,
    observation_horizon: int,
    prediction_horizon: int,
    device: str,
    embedding_dimension: int = 256,
    number_of_heads: int = 8,
    feedforward_dimension: int = 512,
    number_of_encoder_layers: int = 6,
    number_of_decoder_layers: int = 6,
    activation: str = ActivationFunction.RELU.value,
    dropout_rate: float = 0.1,
    normalize_before: bool = False,
) -> None:
    """Initialize ACT-style decoder.

    Args:
        input_keys: List of feature keys expected from encoder pipeline
        action_space: Action space configuration
        observation_space: Observation space configuration
        observation_horizon: Number of observation timesteps
        prediction_horizon: Number of actions to predict
        device: Device to run the model on
        embedding_dimension: Transformer hidden dimension
        number_of_heads: Number of attention heads
        feedforward_dimension: Feedforward network dimension
        number_of_encoder_layers: Number of transformer encoder layers
        number_of_decoder_layers: Number of transformer decoder layers
        activation: Activation function name
        dropout_rate: Dropout probability
        normalize_before: Use pre-normalization

    """
    decoder_input = DecoderInput(
        keys=input_keys,
        required_types=[FeatureType.SPATIAL.value],
        requires_actions=False,
    )
    super().__init__(
        decoder_input=decoder_input,
        action_space=action_space,
        action_heads=action_heads,
        observation_space=observation_space,
        prediction_horizon=prediction_horizon,
        observation_horizon=observation_horizon,
        device=device,
        embedding_dimension=embedding_dimension,
    )
    self.number_of_heads = number_of_heads
    self.feedforward_dimension = feedforward_dimension
    self.number_of_encoder_layers = number_of_encoder_layers
    self.number_of_decoder_layers = number_of_decoder_layers
    self.activation = activation
    self.dropout_rate = dropout_rate
    self.normalize_before = normalize_before
    self._build_transformer_components()
    self.to(self.device)

forward

forward(features, actions=None)

Forward pass of ACT architecture.

Parameters:

Name	Type	Description	Default
features	dict[str, Tensor]	Dictionary of encoded features from EncodingPipeline	required
actions	dict[str, Tensor] | None	Not used, present for API compatibility.	None

Returns:

Type	Description
dict[str, Tensor]	Dictionary containing action head predictions (e.g. position, orientation, gripper)

Note

If LatentKey.POSTERIOR_LATENT is present in features, it will be used as an extra token embedding for the transformer cross-attention.

Source code in src/versatil/models/decoding/decoders/factory/act.py

def forward(
    self,
    features: dict[str, torch.Tensor],
    actions: dict[str, torch.Tensor] | None = None,
) -> dict[str, torch.Tensor]:
    """Forward pass of ACT architecture.

    Args:
        features: Dictionary of encoded features from EncodingPipeline
        actions: Not used, present for API compatibility.

    Returns:
        Dictionary containing action head predictions (e.g. position, orientation, gripper)

    Note:
        If LatentKey.POSTERIOR_LATENT is present in features, it will be used as an extra token embedding for the transformer cross-attention.
    """
    # This creates a sequence of input tokens and positional encodings in the format ACT expects
    input_tokens, pos_encodings, padding_mask = self.input_sequence_builder(
        features
    )  # (B, observation_token_count, embedding_dimension)
    action_embeddings = self._decode_actions(
        input_tokens=input_tokens,
        positional_encodings=pos_encodings,
        padding_mask=padding_mask,
    )  # (B, prediction_horizon, embedding_dimension)
    predictions = self._apply_action_heads(action_embeddings)
    return predictions