Skip to content

dual_stream_base

dual_stream_base

Base class for dual-stream attention blocks.

DualStreamBlock 

DualStreamBlock(attention_normalization_primary, dropout=0.1)

Bases: Module

Shared components for dual-stream attention blocks.

Both full and secondary-stream-precomputed variants have primary-stream normalization and dropout. This base holds those modules.

Source code in src/versatil/models/layers/transformer/block/dual_stream_base.py 
16
17
18
19
20
21
def __init__(
    self, attention_normalization_primary: BlockNormalization, dropout: float = 0.1
):
    super().__init__()
    self.attention_normalization_primary = attention_normalization_primary
    self.attention_dropout_primary = nn.Dropout(dropout)