# neupy.algorithms.exponential_decay

neupy.algorithms.exponential_decay(initial_value, reduction_freq, reduction_rate, staircase=False, start_iter=0, name='step')[source]

Applies exponential decay to the learning rate. This function is a wrapper for the tensorflow’s exponential_decay function.

$\alpha_{t + 1} = \alpha_{0} \cdot d^{\frac{t}{r}}$

where $$\alpha$$ is a step, $$t$$ is an iteration number, $$d$$ is a reduction_freq and $$r$$ is a reduction_rate.

step = initial_value * reduction_rate ^ (
current_iteration / reduction_freq)


When staircase=True the $$\frac{t}{r}$$ value will be rounded.

step = initial_value * reduction_rate ^ floor(
current_iteration / reduction_freq)

Parameters: initial_value : float Initial value for the learning rate. reduction_freq : int Parameter controls step reduction frequency. The larger the value the slower step parameter decreases. reduction_rate : float Parameter controls step reduction rate. The larger the value the slower step parameter decreases. staircase : bool If True decay the learning rate at discrete intervals. Defaults to False. start_iter : int Start iteration. At has to be equal to 0 when network just started the training. Defaults to 0. name : str Learning rate’s variable name. Defaults to step.

Notes

Step will be reduced faster when you have smaller training batches.

Examples

>>> from neupy import algorithms
>>> from neupy.layers import *
>>>
>>> optimizer = algorithms.Momentum(
...     Input(5) >> Relu(10) >> Sigmoid(1),
...     step=algorithms.exponential_decay(
...         initial_value=0.1,
...         reduction_freq=1000,
...         reduction_rate=0.95,
...     )
... )