syna.optim.optimizer module

Optimization algorithms and helper utilities.

Includes common optimizers (SGD, Adam, AdaGrad, etc.) and small utilities that are used as hooks (weight decay, gradient clipping, parameter freezing).

class syna.optim.optimizer.ClipGrad(max_norm: float)

Bases: object

Clip gradients by global norm.

max_norm: maximum allowed norm for concatenated gradients.

class syna.optim.optimizer.FreezeParam(*layers)

Bases: object

Freeze specified parameters or layers (set their grads to None).

class syna.optim.optimizer.Optimizer

Bases: object

Base optimizer.

Subclasses should implement update_one(param). The optimizer keeps a reference to the target (model) via setup(target) and supports hooks that run before updates.

add_hook(f: Callable[[Iterable], None]) → None

Add a hook function called with the list of parameters before updates.

setup(target)

Attach optimizer to a target (model) which must provide params().

update() → None

Run hooks and update all parameters with non-None gradients.

update_one(param) → None

Update a single parameter. Must be implemented by subclasses.

class syna.optim.optimizer.WeightDecay(rate: float)

Bases: object

L2 weight decay applied to gradients.

rate: multiplier applied to parameter data and added to gradient.