decoder

decoder

Configuration classes for different action decoder architectures.

DecodingNetworkConfig dataclass

DecodingNetworkConfig(_target_=MISSING, action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}')

Base architecture configuration.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
action_heads	dict[str, Any] | None	Action head config per predicted action key.
input_keys	list[str]	Observation keys consumed as inputs.
observation_space	ObservationSpaceConfig	Observation space of the task, wired via interpolation.
action_space	ActionSpaceConfig	Action space of the task, wired via interpolation.
observation_horizon	int	Number of past observation frames consumed.
prediction_horizon	int	Number of future actions predicted per chunk.
device	str	Torch device for the module.

ACTConfig dataclass

ACTConfig(_target_='versatil.models.decoding.decoders.factory.act.ACT', action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', embedding_dimension=512, number_of_heads=8, feedforward_dimension=3200, number_of_encoder_layers=4, number_of_decoder_layers=7, activation=value, dropout_rate=0.1, normalize_before=False)

Bases: DecodingNetworkConfig

Action Chunking Transformer with encoder-decoder architecture and parallel generation.

Note

Ref. https://arxiv.org/abs/2304.13705

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
embedding_dimension	int	Transformer hidden dimension.
number_of_heads	int	Number of attention heads.
feedforward_dimension	int	Feedforward network dimension.
number_of_encoder_layers	int	Number of transformer encoder layers.
number_of_decoder_layers	int	Number of transformer decoder layers.
activation	str	Activation function name.
dropout_rate	float	Dropout probability.
normalize_before	bool	Use pre-normalization.

PhaseACTConfig dataclass

PhaseACTConfig(_target_='versatil.models.decoding.decoders.factory.phase_act.PhaseACT', action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', embedding_dimension=512, number_of_heads=8, feedforward_dimension=3200, number_of_encoder_layers=4, number_of_decoder_layers=7, activation=value, dropout_rate=0.1, normalize_before=False, phase_routing_key=MISSING)

Bases: ACTConfig

Phase-conditioned ACT decoder with MoE routing.

Note

Cf. https://arxiv.org/abs/2601.21971 Extends the base ACT architecture to support phase-based expert routing. The phase classifier head produces routing logits that are used to route action predictions through phase-specific expert networks.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
phase_routing_key	str	Key for the phase classifier head that provides routing weights.

GPTActionTransformerConfig dataclass

GPTActionTransformerConfig(_target_='versatil.models.decoding.decoders.factory.gpt_action_transformer.GPTActionTransformer', action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', max_seq_len=512, embedding_dimension=256, number_of_heads=8, number_of_key_value_heads=None, feedforward_dimension=None, number_of_layers=6, activation=value, normalization_type=value, attention_type=value, dropout_rate=0.1, attention_dropout=0.0, positional_encoding_type=value, temperature=1.0, learnable_temperature=False, deterministic=True)

Bases: DecodingNetworkConfig

Autoregressive transformer that models a categorical distribution over discrete action tokens.

Pure GPT-style autoregressive action decoder: - Concatenates visual/proprioceptive features as prefix tokens - Supports variable-length action token sequences (e.g. for FAST tokenization) - Teacher forcing during training - Autoregressive generation with KV caching during inference

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
max_seq_len	int	Maximum sequence length for GPT (features + action tokens).
embedding_dimension	int	Common embedding dimension to bring input tokens to, also Transformer hidden size.
number_of_heads	int	Number of query attention heads.
number_of_key_value_heads	int | None	Number of K/V heads for GQA (None = same as heads = MHA).
feedforward_dimension	int | None	FFN hidden dimension (default: 4 * embedding_dimension).
number_of_layers	int	Number of transformer layers.
activation	str	Activation function (swiglu, gelu, relu, silu).
normalization_type	str	Normalization type (rmsnorm, layernorm).
attention_type	str	Attention type (gqa, mha).
dropout_rate	float	Dropout probability.
attention_dropout	float	Attention dropout probability.
positional_encoding_type	str | None	Type of positional encoding (sinusoidal, rope, None).
temperature	float	Initial temperature for sampling (not used in greedy decoding).
learnable_temperature	bool	If True, make temperature a learnable parameter.
deterministic	bool	If True, use greedy decoding during inference.

AutoregressiveVLAConfig dataclass

AutoregressiveVLAConfig(_target_='versatil.models.decoding.decoders.factory.autoregressive_vla.AutoregressiveVLADecoder', action_heads=dict(), input_keys=list(), observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', vlm_backbone=MISSING, max_seq_len=512, temperature=1.0, learnable_temperature=False, deterministic=True, causal_prefix=False)

Bases: DecodingNetworkConfig

Autoregressive VLA action-token decoder backed by a VLM.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
action_heads	dict[str, Any]	Must be empty. This decoder predicts action tokens with the VLM language vocabulary head.
input_keys	list[str]	Must be empty. Raw observation keys are declared by vlm_backbone.input_specification.
vlm_backbone	Any	Generative VLM that builds image-language prefix embeddings and exposes the causal language model vocabulary.
max_seq_len	int	Maximum prefix plus generated action-token length.
temperature	float	Softmax temperature for stochastic inference.
learnable_temperature	bool	Whether temperature is optimized as a model parameter.
deterministic	bool	Whether inference uses greedy token selection.
causal_prefix	bool	Whether to use a standard causal padding mask (OpenVLA) for the whole sequence instead of bidirectional prefix attention (Pi0-FAST).

OpenVLAOFTConfig dataclass

OpenVLAOFTConfig(_target_='versatil.models.decoding.decoders.factory.openvla_oft.OpenVLAOFTDecoder', action_heads=None, input_keys=list(), observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', vlm_backbone=MISSING, slots_per_action_dimension=True, causal_action_slots=True, min_period=0.004, max_period=4.0)

Bases: DecodingNetworkConfig

OpenVLA-OFT-style continuous action chunk decoder backed by a VLM.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
input_keys	list[str]	Must be empty. Raw observation keys are declared by vlm_backbone.input_specification.
vlm_backbone	Any	Generative VLM that builds image-language prefix embeddings and exposes the language tower.
slots_per_action_dimension	bool	When True, each action scalar owns one VLM hidden-state slot before the joint action projection. When False, each timestep owns one slot.
causal_action_slots	bool	Whether action slots use causal self-attention.
min_period	float	Minimum period for sinusoidal timestep embeddings used by denoising algorithms.
max_period	float	Maximum period for sinusoidal timestep embeddings used by denoising algorithms.

ActionTransformerConfig dataclass

ActionTransformerConfig(_target_='versatil.models.decoding.decoders.factory.action_transformer.ActionTransformer', action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', embedding_dimension=256, number_of_heads=8, number_of_key_value_heads=None, feedforward_dimension=None, number_of_layers=6, activation=value, normalization_type=value, attention_type=value, dropout_rate=0.1, attention_dropout=0.0, positional_encoding_type=value)

Bases: DecodingNetworkConfig

Action Decoder-only Transformer architecture with cross-attention to encoded features.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
embedding_dimension	int	Embedding dimension of the model tokens.
number_of_heads	int	Number of query attention heads.
number_of_key_value_heads	int | None	Key/value head count for grouped-query attention.
feedforward_dimension	int | None	Feedforward layer width.
number_of_layers	int	Transformer layer count.
activation	str	Activation function.
normalization_type	str	Normalization type.
attention_type	str	Attention type.
dropout_rate	float	Dropout probability.
attention_dropout	float	Dropout probability inside attention.
positional_encoding_type	str | None	Self-attention positional encoding: rope, sinusoidal, learned, or null.

MixtureOfDensitiesActionTransformerConfig dataclass

MixtureOfDensitiesActionTransformerConfig(_target_='versatil.models.decoding.decoders.factory.mode_act.MixtureOfDensitiesActionTransformer', action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', embedding_dimension=256, number_of_heads=8, number_of_key_value_heads=None, feedforward_dimension=None, number_of_layers=6, activation=value, normalization_type=value, attention_type=value, dropout_rate=0.1, attention_dropout=0.0, positional_encoding_type=value, num_mixture_components=8, gating_hidden_dims=(lambda: [256, 128])(), gating_activation=value, gating_dropout=0.1, gating_normalization=True, temperature=1.0, learnable_temperature=False, gating_feature_key=None, gmm_init_strategy=value, inference_sampling_mode=value)

Bases: DecodingNetworkConfig

Mixture of Densities Action Transformer (MODE-ACT) configuration.

MODE-ACT extends ActionTransformer with mixture density network capabilities for multi-modal action prediction. It uses K copies of each action head and a gating network to predict mixture weights.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
embedding_dimension	int	Transformer embedding dimension.
number_of_heads	int	Number of attention heads.
number_of_key_value_heads	int | None	Number of key-value heads for GQA.
feedforward_dimension	int | None	FFN hidden dimension.
number_of_layers	int	Number of decoder layers.
activation	str	Activation function.
normalization_type	str	Normalization type.
attention_type	str	Attention type.
dropout_rate	float	Dropout rate.
attention_dropout	float	Attention dropout rate.
positional_encoding_type	str | None	Positional encoding type.
num_mixture_components	int	Number of mixture components (K).
gating_hidden_dims	list[int]	Hidden dimensions for gating MLP.
gating_activation	str	Activation for gating MLP.
gating_dropout	float	Dropout rate in gating MLP.
gating_normalization	bool	Whether to normalize gating input.
temperature	float	Temperature for softmax scaling.
learnable_temperature	bool	Whether temperature is learnable.
gating_feature_key	str | None	If set, use this feature for gating instead of mode embedding.
gmm_init_strategy	str	Strategy for initializing GMM component means.
inference_sampling_mode	str	How to sample from the mixture at inference. DETERMINISTIC: argmax component, return mean. STOCHASTIC_MEAN: multinomial component, return mean (no noise). STOCHASTIC_SAMPLE: multinomial component, add Gaussian noise.

LACTConfig dataclass

LACTConfig(_target_='versatil.models.decoding.decoders.factory.lact.LACT', action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', latent_dimension=MISSING, embedding_dimension=256, number_of_heads=8, number_of_key_value_heads=None, feedforward_dimension=None, number_of_layers=6, activation=value, normalization_type=value, attention_type=value, positional_encoding_type=value, dropout_rate=0.1, attention_dropout=0.0, use_gating=True)

Bases: DecodingNetworkConfig

Latent Action Transformer (LACT) architecture configuration.

LACT extends a standard Action Transformer with latent-conditioned decoding. It uses a Pix-Art style DiT with AdaLN-Zero modulation on the latent token and cross-attention to encoder features.

Must be used with a variational algorithm (e.g., VariationalAlgorithm) that provides a latent embedding indexed by LatentKey.POSTERIOR_LATENT in the features dictionary.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
latent_dimension	int	Dimension of latent conditioning vector.
embedding_dimension	int	Transformer hidden dimension.
number_of_heads	int	Number of attention heads.
number_of_key_value_heads	int | None	Number of K/V heads for GQA (None for MHA).
feedforward_dimension	int | None	FFN hidden dimension (default: 4 * embedding_dimension).
number_of_layers	int	Number of conditional transformer decoder layers.
activation	str	Activation function name.
normalization_type	str	Type of adaptive normalization layer.
attention_type	str	Type of attention mechanism (multi-head, grouped query, etc.).
positional_encoding_type	str | None	Type of positional encoding.
dropout_rate	float	Dropout probability for residual connections.
attention_dropout	float	Dropout probability for attention weights.
use_gating	bool	Whether to use AdaLN-Zero gating on residual connections.

MixtureOfExpertsDecoderConfig dataclass

MixtureOfExpertsDecoderConfig(_target_='versatil.models.decoding.decoders.moe.MoEDecoder', base_expert=MISSING, num_experts=MISSING, gating_feature_key=MISSING, inference_gating_key=None, gating_input_dim=None, gating_hidden_dims=list(), gating_activation=value, routing_type=value, top_k=2, temperature=1.0, learnable_temperature=False, gating_dropout=0.1, gating_normalization=True)

Mixture of Experts (MoE) decoder configuration.

The wrapper takes every generic decoder setting (action heads, input keys, spaces, horizons, device) from base_expert, so it intentionally does not extend DecodingNetworkConfig.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
base_expert	Any	Decoder config replicated per expert.
num_experts	int	Number of expert decoders.
gating_feature_key	str	Feature key routed to the gating network during training.
inference_gating_key	str | None	Feature key routed to the gating network at inference when it differs from gating_feature_key. None uses gating_feature_key. The key must exist in the decoder features at inference; variational algorithms expose their sampled latent under the posterior key ('latent') during both training and inference.
gating_input_dim	int | None	Gating network input dimension; null derives routing from the latent.
gating_hidden_dims	list[int]	Gating network hidden layer widths.
gating_activation	str	Activation function inside the gating network.
routing_type	str	Expert routing: soft or top_k.
top_k	int	Number of expert outputs mixed per sample with top_k routing. Every expert still runs a forward pass; routing selects and renormalizes the top-k outputs rather than skipping computation.
temperature	float	Routing softmax temperature.
learnable_temperature	bool	Whether the routing temperature is learned.
gating_dropout	float	Dropout probability inside the gating network.
gating_normalization	bool	Normalization layer name inside the gating network.

DiTBlockActionTransformerConfig dataclass

DiTBlockActionTransformerConfig(_target_='versatil.models.decoding.decoders.factory.dit_block_action_transformer.DiTBlockActionTransformer', action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', max_sequence_length=1024, embedding_dimension=512, timestep_embedding_dimension=256, number_of_heads=8, number_of_key_value_heads=None, number_of_encoder_layers=6, number_of_decoder_layers=6, feedforward_dimension=2048, activation=value, normalization_type=value, attention_type=value, dropout_rate=0.1, attention_dropout=0.0, positional_encoding_type=value, use_gating=True)

Bases: DecodingNetworkConfig

DiTBlock action transformer with pooled conditioning.

Note

Ref. https://arxiv.org/abs/2410.10088v1 It uses an encoder-decoder architecture that pools encoder output to a single conditioning vector. Must be used with a denoising algorithm that provides timesteps and noisy actions.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
max_sequence_length	int	Maximum sequence length for input tokens.
embedding_dimension	int	Transformer hidden dimension.
timestep_embedding_dimension	int	Diffusion timestep embedding dimension.
number_of_heads	int	Number of attention heads.
number_of_key_value_heads	int | None	Number of K/V heads for GQA.
number_of_encoder_layers	int	Number of transformer encoder layers.
number_of_decoder_layers	int	Number of transformer decoder layers.
feedforward_dimension	int	Feedforward network dimension.
activation	str	Activation function name.
normalization_type	str	Normalization type name.
attention_type	str	Attention type name (gqa, mha).
dropout_rate	float	Dropout probability for residual connections.
attention_dropout	float	Dropout probability for attention weights.
positional_encoding_type	str | None	Type of positional encoding.
use_gating	bool	Whether to use gating in AdaLN-Zero layers.

DiffusionActionTransformerConfig dataclass

DiffusionActionTransformerConfig(_target_='versatil.models.decoding.decoders.factory.diffusion_action_transformer.DiffusionActionTransformer', action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', diffusion_transformer_type=value, max_sequence_length=1024, embedding_dimension=512, timestep_embedding_dimension=256, number_of_heads=8, number_of_key_value_heads=None, number_of_layers=6, feedforward_dimension=2048, activation=value, normalization_type=value, attention_type=value, dropout_rate=0.1, attention_dropout=0.0, positional_encoding_type=value, use_gating=True)

Bases: DecodingNetworkConfig

Diffusion action transformer for CrossAttentionDiT and MultiModal DiT.

Note

Must be used with a denoising algorithm that provides timesteps and noisy actions.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
diffusion_transformer_type	str	Type of Diffusion Transformer architecture.
max_sequence_length	int	Maximum sequence length for input tokens.
embedding_dimension	int	Transformer hidden dimension.
timestep_embedding_dimension	int	Diffusion timestep embedding dimension.
number_of_heads	int	Number of attention heads.
number_of_key_value_heads	int | None	Number of K/V heads for GQA.
number_of_layers	int	Number of transformer layers.
feedforward_dimension	int	Feedforward network dimension.
activation	str	Activation function name.
normalization_type	str	Normalization type name.
attention_type	str	Attention type name (gqa, mha).
dropout_rate	float	Dropout probability for residual connections.
attention_dropout	float	Dropout probability for attention weights.
positional_encoding_type	str | None	Type of positional encoding.
use_gating	bool	Whether to use gating in AdaLN-Zero layers.

ConditionalActionUNetConfig dataclass

ConditionalActionUNetConfig(_target_='versatil.models.decoding.decoders.factory.conditional_action_unet.ConditionalActionUNet', action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', embedding_dimension=256, down_dimensions=(lambda: [256, 512, 1024])(), kernel_size=5, num_groups=8, condition_predict_scale=False)

Bases: DecodingNetworkConfig

Conditional U-Net action decoder configuration with FiLM conditioning. Note: Ref. https://diffusion-policy.cs.columbia.edu. Must be used with a denoising algorithm that provides timesteps and noisy actions.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
embedding_dimension	int	Diffusion timestep embedding dimension.
down_dimensions	list[int]	List of channel dimensions for downsampling layers.
kernel_size	int	Kernel size for convolutions in residual blocks.
num_groups	int	Number of groups for group normalization.
condition_predict_scale	bool	If True, conditions predict scaling factors in FiLM.

SmolVLADecoderConfig dataclass

SmolVLADecoderConfig(_target_='versatil.models.decoding.decoders.factory.smolvla.SmolVLADecoder', action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', vlm_backbone=MISSING, expert_width_multiplier=0.75, num_expert_layers=-1, num_vlm_layers=16, self_attention_every_n_layers=2, proprioceptive_feature_key=None, min_period=0.004, max_period=4.0, freeze_vlm=True, normalization_type=value, activation=value, dropout=0.1)

Bases: DecodingNetworkConfig

SmolVLA Vision Language Action model with interleaved VLM + expert cross-attention.

Note

Ref. https://arxiv.org/abs/2506.01844 Must be used with a denoising algorithm that provides timesteps and noisy actions.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
vlm_backbone	Any	Generative VLM backbone that builds the raw observation prefix with shape (B, P, D_vlm).
expert_width_multiplier	float	Expert hidden size as fraction of VLM hidden size.
num_expert_layers	int	Number of expert layers. -1 uses the same count as VLM.
num_vlm_layers	int	Number of VLM layers to use (truncates if fewer than available).
self_attention_every_n_layers	int	Period for joint self-attention layers. 0 disables joint self-attention (all cross- attention).
proprioceptive_feature_key	str | None	Feature key for proprioceptive state from the encoding pipeline. When set, the feature is prepended to the VLM prefix before interleaved processing. None disables state prepend.
min_period	float	Minimum period for sinusoidal timestep embedding.
max_period	float	Maximum period for sinusoidal timestep embedding.
freeze_vlm	bool	Whether to freeze VLM layer parameters (disable gradients).
normalization_type	str	Normalization layer type.
activation	str	Activation function for expert feedforward layers.
dropout	float	Dropout rate.

Pi0DecoderConfig dataclass

Pi0DecoderConfig(_target_='versatil.models.decoding.decoders.factory.pi0.Pi0Decoder', action_heads=None, input_keys=MISSING, observation_space='${policy.observation_space}', action_space='${policy.action_space}', observation_horizon='${policy.observation_horizon}', prediction_horizon='${policy.prediction_horizon}', device='${policy.device}', vlm_backbone=MISSING, expert_hidden_size=1024, expert_intermediate_size=4096, expert_number_of_heads=8, expert_number_of_key_value_heads=1, expert_number_of_layers=18, expert_head_dimension=256, time_conditioning=value, min_period=0.004, max_period=4.0, proprioceptive_feature_key=None, normalization_type=value, activation=value, dropout=0.0)

Bases: DecodingNetworkConfig

Pi0/Pi0.5 Vision Language Action model with interleaved VLM + expert joint attention.

Note

Ref. https://arxiv.org/abs/2410.24164, https://arxiv.org/abs/2504.16054 Must be used with a denoising algorithm that provides timesteps and noisy actions.

Attributes:

Name	Type	Description
_target_	str	Import path instantiated by Hydra.
vlm_backbone	Any	Generative VLM backbone that builds the raw observation prefix with shape (B, P, D_vlm).
expert_hidden_size	int	Expert network hidden dimension.
expert_intermediate_size	int	Expert feedforward intermediate dimension.
expert_number_of_heads	int	Number of attention heads in expert layers.
expert_number_of_key_value_heads	int	Number of K/V heads in expert layers.
expert_number_of_layers	int	Number of expert layers (must match VLM layers).
expert_head_dimension	int	Per-head dimension in expert layers.
time_conditioning	str	Timestep conditioning mode (use TimeConditioning enum values).
min_period	float	Minimum period for sinusoidal timestep embedding.
max_period	float	Maximum period for sinusoidal timestep embedding.
proprioceptive_feature_key	str | None	Feature key for proprioceptive state. When set, the feature is prepended to the VLM prefix.
normalization_type	str	Normalization layer type.
activation	str	Activation function for expert feedforward layers.
dropout	float	Dropout rate.