Skip to content

activation

activation

ActivationFunction

Bases: StrEnum

Available activation functions.

is_gated property 

is_gated

Whether this activation uses a gated linear unit (two projections).

to_torch_activation 

to_torch_activation()

Convert to corresponding PyTorch activation module.

Gated activations (SwiGLU, GeGLU) require input_dimension and hidden_dimension constructor args. Standard activations are zero-arg.

Source code in src/versatil/models/layers/activation.py 
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
def to_torch_activation(self) -> type[nn.Module]:
    """Convert to corresponding PyTorch activation module.

    Gated activations (SwiGLU, GeGLU) require ``input_dimension`` and
    ``hidden_dimension`` constructor args. Standard activations are zero-arg.
    """
    mapping = {
        ActivationFunction.RELU.value: nn.ReLU,
        ActivationFunction.GELU.value: nn.GELU,
        ActivationFunction.SILU.value: nn.SiLU,
        ActivationFunction.SWIGLU.value: SwiGLU,
        ActivationFunction.GEGLU.value: GeGLU,
        ActivationFunction.SIGMOID.value: nn.Sigmoid,
        ActivationFunction.TANH.value: nn.Tanh,
        ActivationFunction.LEAKY_RELU.value: nn.LeakyReLU,
        ActivationFunction.LINEAR.value: nn.Identity,
        ActivationFunction.MISH.value: nn.Mish,
    }
    if self.value not in mapping:
        raise ValueError(
            f"Unsupported activation: {self.value}. "
            f"Choose from {[e.value for e in ActivationFunction]}"
        )
    return mapping[self.value]