easycv.core.optimizer package

Submodules

easycv.core.optimizer.lars module

class easycv.core.optimizer.lars.LARS(params, lr=<required parameter>, momentum=0, dampening=0, weight_decay=0, eta=0.001, nesterov=False)

Bases: torch.optim.optimizer.Optimizer

Implements layer-wise adaptive rate scaling for SGD.

Parameters

• params (iterable) – iterable of parameters to optimize or dicts defining parameter groups

• lr (float) – base learning rate (gamma_0)

• momentum (float, optional) – momentum factor (default: 0) (“m”)

• weight_decay (float, optional) – weight decay (L2 penalty) (default: 0) (“beta”)

• dampening (float, optional) – dampening for momentum (default: 0)

• eta (float, optional) – LARS coefficient

• nesterov (bool, optional) – enables Nesterov momentum (default: False)

Based on Algorithm 1 of the following paper by You, Gitman, and Ginsburg. Large Batch Training of Convolutional Networks:

https://arxiv.org/abs/1708.03888

Example

>>> optimizer = LARS(model.parameters(), lr=0.1, momentum=0.9,
>>>                  weight_decay=1e-4, eta=1e-3)
>>> optimizer.zero_grad()
>>> loss_fn(model(input), target).backward()
>>> optimizer.step()

__init__(params, lr=<required parameter>, momentum=0, dampening=0, weight_decay=0, eta=0.001, nesterov=False)

Initialize self. See help(type(self)) for accurate signature.

step(closure=None)

Performs a single optimization step.

Parameters

closure (callable, optional) – A closure that reevaluates the model and returns the loss.

easycv.core.optimizer.ranger module

easycv.core.optimizer.ranger.centralized_gradient(x, use_gc=True, gc_conv_only=False)

credit - https://github.com/Yonghongwei/Gradient-Centralization

class easycv.core.optimizer.ranger.Ranger(params, lr=0.001, alpha=0.5, k=6, N_sma_threshhold=5, betas=(0.95, 0.999), eps=1e-05, weight_decay=0, use_gc=True, gc_conv_only=False, gc_loc=True)

Bases: torch.optim.optimizer.Optimizer

Adam+LookAhead: refer to https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer

__init__(params, lr=0.001, alpha=0.5, k=6, N_sma_threshhold=5, betas=(0.95, 0.999), eps=1e-05, weight_decay=0, use_gc=True, gc_conv_only=False, gc_loc=True)

Initialize self. See help(type(self)) for accurate signature.

step(closure=None)

Performs a single optimization step (parameter update).

Parameters

closure (Callable) – A closure that reevaluates the model and returns the loss. Optional for most optimizers.

Note

Unless otherwise specified, this function should not modify the .grad field of the parameters.