Skip to content

base

base

Fusion modules for combining multi-modal features.

FusionInput dataclass 

FusionInput(input_features)

Structured input specification for fusion modules.

FusionModule 

FusionModule(input_specification, output_name)

Bases: Module

Base fusion module with validation.

Source code in src/versatil/models/encoding/fusion/base.py 
22
23
24
25
26
27
28
29
30
def __init__(
    self,
    input_specification: FusionInput,
    output_name: str,
):
    super().__init__()
    self.input_specification = input_specification
    self.output_name = output_name
    self._initialized = False

input_features property writable 

input_features

Get list of input feature names.

get_output_specification abstractmethod 

get_output_specification()

Get structured output specification.

Source code in src/versatil/models/encoding/fusion/base.py 
42
43
44
45
@abc.abstractmethod
def get_output_specification(self) -> FeatureMetadata:
    """Get structured output specification."""
    raise NotImplementedError

setup 

setup(feature_registry)

Setup layers once feature metadata is known.

Note

Called once by the encoding pipeline after feature metadata is available. Allows fusion modules to be created without knowing input dimensions ahead of time.

Parameters:

Name Type Description Default
feature_registry dict[str, FeatureMetadata]

Dict mapping available feature names to their metadata.

 required
Source code in src/versatil/models/encoding/fusion/base.py 
47
48
49
50
51
52
53
54
55
56
57
58
59
60
def setup(self, feature_registry: dict[str, FeatureMetadata]):
    """Setup layers once feature metadata is known.

    Note:
        Called once by the encoding pipeline after feature metadata is available.
        Allows fusion modules to be created without knowing input dimensions ahead of time.

    Args:
        feature_registry: Dict mapping available feature names to their metadata.
    """
    if self._initialized:
        return
    self._setup_layers(feature_registry)
    self._initialized = True

forward abstractmethod 

forward(features)

Fuse a list of feature tensors into one tensor.

Source code in src/versatil/models/encoding/fusion/base.py 
67
68
69
70
@abc.abstractmethod
def forward(self, features: list[torch.Tensor]) -> torch.Tensor:
    """Fuse a list of feature tensors into one tensor."""
    raise NotImplementedError

SequentialFusion 

SequentialFusion(input_features, output_name, hidden_dimension)

Bases: FusionModule, ABC

Base class for fusion modules that project features to a shared dimension.

Parameters:

Name Type Description Default
input_features list[str]

List of feature names to fuse.

 required
output_name str

Name of the output fused feature.

 required
hidden_dimension int

Dimension to project each input feature to before fusion.

 required
Source code in src/versatil/models/encoding/fusion/base.py 
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
def __init__(
    self,
    input_features: list[str],
    output_name: str,
    hidden_dimension: int,
):
    """
    Args:
        input_features: List of feature names to fuse.
        output_name: Name of the output fused feature.
        hidden_dimension: Dimension to project each input feature to before fusion.
    """
    input_specification = FusionInput(input_features=input_features)
    super().__init__(
        input_specification=input_specification, output_name=output_name
    )
    self.projections: nn.ModuleList | None = None
    self.hidden_dimension = hidden_dimension
    self._output_feature_type: str | None = None
    self._output_sequence_length: int | None = None