Replicate layers and networks

Copy layers/networks

Any layer or network could be copied. For example, we can create layer and copy it as many times as we want.

>>> from neupy.layers import *
>>> layer = layers.Relu(10)
>>> layer
Relu(10, alpha=0, weight=HeNormal(gain=2), bias=Constant(0), name='relu-1')
>>>
>>> import copy
>>> copy.copy(layer)
Relu(10, alpha=0, weight=HeNormal(gain=2), bias=Constant(0), name='relu-2')
>>>
>>> copy.copy(layer)
Relu(10, alpha=0, weight=HeNormal(gain=2), bias=Constant(0), name='relu-3')

Each new copy gets the same set of parameters. Also, notice that each new copy get new name. It works because we didn’t specify exact name of the layer.

>>> layer = Relu(10, name='relu-layer')
>>> layer
Relu(10, alpha=0, weight=HeNormal(gain=2), bias=Constant(0), name='relu-layer')
>>>
>>> copy.copy(layer)
Relu(10, alpha=0, weight=HeNormal(gain=2), bias=Constant(0), name='relu-layer')

In order to create layers with custom name, we can specify name as a string that could be formatted with a unique index value.

>>> layer = Relu(10, name='relu-layer-{}')
>>> layer
Relu(10, alpha=0, weight=HeNormal(gain=2), bias=Constant(0), name='relu-layer-1')
>>>
>>> copy.copy(layer)
Relu(10, alpha=0, weight=HeNormal(gain=2), bias=Constant(0), name='relu-layer-2')

All the examples above also work for the networks

>>> block = Conv((3, 3, 32)) >> Relu() >> BN()
>>> block
<unknown> -> [... 3 layers ...] -> (?, ?, ?, 32)
>>>
>>> copy.copy(block)
<unknown> -> [... 3 layers ...] -> (?, ?, ?, 32)

Repeat the same layer/network multiple times

Certain neural network architectures might have single layer or group of layers repeated multiple times in a sequence. For example:

>>> from neupy.layers import *
>>> network = Input(20) >> Relu(10) >> Relu(10) >> Relu(10) >> Relu(10)
>>> network
(?, 20) -> [... 5 layers ...] -> (?, 10)

In order to avoid redundant repetitions, NeuPy introduced function called repeat. This function copies layer multiple times and connects original and all copied layers into a sequence. We can rewrite previous example, using repeat function in order to get the exactly the same neural network architecture.

>>> network = Input(20) >> repeat(Relu(10), n=4)
>>> network
(?, 20) -> [... 5 layers ...] -> (?, 10)

And the same function will work if applied to the network.

>>> block = Conv((3, 3, 32)) >> Relu() >> BN()
>>> block
<unknown> -> [... 3 layers ...] -> (?, ?, ?, 32)
>>>
>>> repeat(block, n=5)
<unknown> -> [... 15 layers ...] -> (?, ?, ?, 32)

It’s important to remember that input shape of the layer/network should be compatible with it’s output shape. Otherwise exception will be triggered.

Caveats

Copying and repetition make more sense when layer hasn’t been initialized yet. Let’s check the following example:

>>> from neupy.layers import *
>>> layer = layers.Relu(10)
>>> layer
Relu(10, alpha=0, weight=HeNormal(gain=2), bias=Constant(0), name='relu-1')

We can see that weight and bias hasn’t been generated yet. We can add layer to the network and create variables for it.

>>> network = Input(20) >> layer
>>> network.create_variables()
>>> layer
Relu(10, alpha=0, weight=<Variable shape=(20, 10)>, bias=<Variable shape=(10,)>, name='relu-1')

We can see that now each parameter of the layer has it’s own variable. If we try to copy layer with initialized variables that’s what we will get.

>>> copy.copy(layer)
Relu(10, alpha=0, weight=<Array shape=(20, 10)>, bias=<Array shape=(10,)>, name='relu-2')

Now each parameter has it’s value specified as an array, which is just a copy of the value stored in the original variable. For this layer, variables hasn’t been created yet, since it’s not a part of any network.