azula.nn.normalization¶

Normalization layers.

Classes¶

Creates a layer that standardizes features along a dimension.

Descriptions¶

class azula.nn.normalization.LayerNorm(dim, eps=1e-05)¶

Creates a layer that standardizes features along a dimension.

\[y = \frac{x - \mathbb{E}[x]}{\sqrt{\mathbb{V}[x] + \epsilon}}\]

References

Layer Normalization (Lei Ba et al., 2016)

https://arxiv.org/abs/1607.06450

Parameters:

dim (int | Sequence[int]) – The dimension(s) to standardize.
eps (float) – A numerical stability term.

forward(x)¶

Parameters:: x (Tensor) – The input tensor \(x\), with shape :math:(*).
Returns:: The standardized tensor \(y\), with shape \((*)\).
Return type:: Tensor