neupy.layers.normalization module

class neupy.layers.normalization.BatchNorm[source]

Batch-normalization layer.

Parameters:

axes : int, tuple with int or None

The axis or axes along which normalization is applied. None means that normalization will be applied over all axes except the first one. In case of 4D tensor it will be equal to (0, 2, 3). Defaults to None.

epsilon : float

Epsilon is a positive constant that adds to the standard deviation to prevent the division by zero. Defaults to 1e-5.

alpha : float

Coefficient for the exponential moving average of batch-wise means and standard deviations computed during training; the closer to one, the more it will depend on the last batches seen. Value needs to be between 0 and 1. Defaults to 0.1.

gamma : array-like, Theano variable, scalar or Initializer

Default initialization methods you can find here. Defaults to Constant(value=1).

beta : array-like, Theano variable, scalar or Initializer

Default initialization methods you can find here. Defaults to Constant(value=0).

running_mean : array-like, Theano variable, scalar or Initializer

Default initialization methods you can find here. Defaults to Constant(value=0).

running_inv_std : array-like, Theano variable, scalar or Initializer

Default initialization methods you can find here. Defaults to Constant(value=1).

name : str or None

Layer’s identifier. If name is equal to None than name will be generated automatically. Defaults to None.

References

[R3]Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, http://arxiv.org/pdf/1502.03167v3.pdf

Attributes

input_shape (tuple) Layer’s input shape.
output_shape (tuple) Layer’s output shape.
training_state (bool) Defines whether layer in training state or not.
parameters (dict) Trainable parameters.
graph (LayerGraph instance) Graphs that stores all relations between layers.

Methods

disable_training_state() Swith off trainig state.
initialize() Set up important configurations related to the layer.
alpha = None[source]
axes = None[source]
beta = None[source]
epsilon = None[source]
gamma = None[source]
initialize()[source]

Initialize connection

options = {'name': Option(class_name='BaseLayer', value=Property(name="name")), 'epsilon': Option(class_name='BatchNorm', value=NumberProperty(name="epsilon")), 'axes': Option(class_name='BatchNorm', value=AxesProperty(name="axes")), 'running_inv_std': Option(class_name='BatchNorm', value=ParameterProperty(name="running_inv_std")), 'beta': Option(class_name='BatchNorm', value=ParameterProperty(name="beta")), 'running_mean': Option(class_name='BatchNorm', value=ParameterProperty(name="running_mean")), 'alpha': Option(class_name='BatchNorm', value=ProperFractionProperty(name="alpha")), 'gamma': Option(class_name='BatchNorm', value=ParameterProperty(name="gamma"))}[source]
output(input_value)[source]

Return output base on the input value.

Parameters:input_value
running_inv_std = None[source]
running_mean = None[source]
class neupy.layers.normalization.LocalResponseNorm[source]

Local Response Normalization Layer.

Aggregation is purely across channels, not within channels, and performed “pixelwise”.

If the value of the \(i\) th channel is \(x_i\), the output is

\[x_i = \frac{x_i}{ (k + ( \alpha \sum_j x_j^2 ))^\beta }\]

where the summation is performed over this position on \(n\) neighboring channels.

Parameters:

alpha : float

coefficient, see equation above

beta : float

offset, see equation above

k : float

exponent, see equation above

n : int

Number of adjacent channels to normalize over, must be odd

name : str or None

Layer’s identifier. If name is equal to None than name will be generated automatically. Defaults to None.

Attributes

input_shape (tuple) Layer’s input shape.
output_shape (tuple) Layer’s output shape.
training_state (bool) Defines whether layer in training state or not.
parameters (dict) Trainable parameters.
graph (LayerGraph instance) Graphs that stores all relations between layers.

Methods

disable_training_state() Swith off trainig state.
initialize() Set up important configurations related to the layer.
alpha = None[source]
beta = None[source]
k = None[source]
n = None[source]
options = {'beta': Option(class_name='LocalResponseNorm', value=NumberProperty(name="beta")), 'alpha': Option(class_name='LocalResponseNorm', value=NumberProperty(name="alpha")), 'k': Option(class_name='LocalResponseNorm', value=NumberProperty(name="k")), 'name': Option(class_name='BaseLayer', value=Property(name="name")), 'n': Option(class_name='LocalResponseNorm', value=IntProperty(name="n"))}[source]
output(input_value)[source]

Return output base on the input value.

Parameters:input_value
validate(input_shape)[source]

Validate input shape value before assigning it.

Parameters:input_shape : tuple with int