syna.layers.normalization module

Normalization layers

class syna.layers.normalization.LayerNorm(dim: int, eps: float = 1e-05, dtype=<class 'numpy.float32'>)

Bases: Layer

Layer Normalization (Jimmy Lei Ba et al. 2016).

paper: https://arxiv.org/abs/1607.06450

\[y = \gamma \odot \frac{x - \mathbb{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} + \beta\]

Args:

dim: size of the last (normalized) dimension eps: small constant added to variance for numerical stability

forward(x)

Apply layer normalization to input tensor.