neupy.algorithms.polynomial_decay

neupy.algorithms.polynomial_decay(initial_value, decay_iter, minstep=0.001, power=1.0, cycle=False, start_iter=0, name='step')[source]

Applies polynomial decay to the learning rate. This function is a wrapper for the tensorflow’s polynomial_decay function.

iteration = min(current_iteration, decay_iter)
step = minstep + (
    (initial_value - minstep) *
    (1 - iteration / decay_iter) ^ power
)

If cycle is True then a multiple of decay_iter is used, the first one that is bigger than current_iterations.

decay_iter = decay_iter * ceil(current_iteration / decay_iter)
step = minstep + (
    (initial_value - minstep) *
    (1 - current_iteration / decay_iter) ^ power
)
Parameters:
initial_value : float

Initial value for the learning rate.

decay_iter : int

When cycle=False parameter identifies number of iterations when minstep will be reached. When cycle=True than the decay_iter value will be increased. See code above.

minstep : float

Step will never be lower than that minimum possible step, specified by this parameter. Defaults to 0.001.

power : float

The power of the polynomial. Defaults to 1.

cycle : bool

When value equal to True than step will be further reduced when current_iteration > decay_iter. Defaults to False.

start_iter : int

Start iteration. At has to be equal to 0 when network just started the training. Defaults to 0.

name : str

Learning rate’s variable name. Defaults to step.

Notes

Step will be reduced faster when you have smaller training batches.

Examples

>>> from neupy import algorithms
>>> from neupy.layers import *
>>>
>>> optimizer = algorithms.Momentum(
...     Input(5) >> Relu(10) >> Sigmoid(1),
...     step=algorithms.polynomial_decay(
...         initial_value=0.1,
...         decay_iter=1000,
...         minstep=0.01,
...     )
... )