neupy.layers.GroupNorm

class neupy.layers.GroupNorm[source]

Group Normalization layer. This layer is a simple alternative to the Batch Normalization layer for cases when batch size is small.

Parameters:
n_groups : int

During normalization all the channels will be break down into separate groups and mean and variance will be estimated per group. This parameter controls number of groups.

gamma : array-like, Tensorfow variable, scalar or Initializer

Scale. Default initialization methods you can find here. Defaults to Constant(value=1).

beta : array-like, Tensorfow variable, scalar or Initializer

Offset. Default initialization methods you can find here. Defaults to Constant(value=0).

epsilon : float

Epsilon ensures that input rescaling procedure that uses estimated variance will never cause division by zero. Defaults to 1e-5.

name : str or None

Layer’s name. Can be used as a reference to specific layer. Name Can be specified as:

  • String: Specified name will be used as a direct reference to the layer. For example, name=”fc”
  • Format string: Name pattern could be defined as a format string and specified field will be replaced with an index. For example, name=”fc{}” will be replaced with fc1, fc2 and so on. A bit more complex formatting methods are acceptable, for example, name=”fc-{:<03d}” will be converted to fc-001, fc-002, fc-003 and so on.
  • None: When value specified as None than name will be generated from the class name.

Defaults to None.

References

[1]Group Normalization, Yuxin Wu, Kaiming He, https://arxiv.org/pdf/1803.08494.pdf

Examples

Convolutional Neural Networks (CNN)

>>> from neupy.layers import *
>>> network = join(
...     Input((28, 28, 1)),
...     Convolution((3, 3, 16)) >> GroupNorm(4) >> Relu(),
...     Convolution((3, 3, 16)) >> GroupNorm(4) >> Relu(),
...     Reshape(),
...     Softmax(10),
... )
Attributes:
variables : dict

Variable names and their values. Dictionary can be empty in case if variables hasn’t been created yet.

Methods

variable(value, name, shape=None, trainable=True) Initializes variable with specified values.
get_output_shape(input_shape) Computes expected output shape from the layer based on the specified input shape.
output(*inputs, **kwargs) Propagates input through the layer. The kwargs variable might contain additional information that propagates through the network.
beta = None[source]
create_variables(input_shape)[source]
epsilon = None[source]
gamma = None[source]
get_output_shape(input_shape)[source]
n_groups = None[source]
options = {'beta': Option(class_name='GroupNorm', value=ParameterProperty(name="beta")), 'epsilon': Option(class_name='GroupNorm', value=NumberProperty(name="epsilon")), 'gamma': Option(class_name='GroupNorm', value=ParameterProperty(name="gamma")), 'n_groups': Option(class_name='GroupNorm', value=IntProperty(name="n_groups")), 'name': Option(class_name='BaseLayer', value=Property(name="name"))}[source]
output(input)[source]