azula.nn.embedding

Embedding and encoding modules.

Note

The terms embedding and encoding are often used interchangeably in the literature. We adopt the following nomenclature: an embedding is a learned function while an encoding is a static function.

Classes

SineEncoding

Creates a sinusoidal positional encoding.

Functions

sine_encoding

Descriptions

class azula.nn.embedding.SineEncoding(features, omega=10000.0)

Creates a sinusoidal positional encoding.

\[\begin{split}e_{2i} & = \sin \left( x \times \omega^\frac{-2i}{D} \right) \\ e_{2i+1} & = \cos \left( x \times \omega^\frac{-2i}{D} \right)\end{split}\]

References

Attention Is All You Need (Vaswani et al., 2017)

https://arxiv.org/abs/1706.03762

Parameters:

• features (int) – The number of embedding features \(D\). Must be even.

• omega (float) – The maximum frequency \(\omega\).

forward(x)

Parameters:

x (Tensor) – The position \(x\), with shape \((*)\).

Returns:

The embedding vector \(e\), with shape \((*, D)\).

Return type:

Tensor