neupy.architectures.mixture_of_experts module

neupy.architectures.mixture_of_experts.mixture_of_experts(networks, gating_layer=None)[source]

Generates mixture of experts architecture from the set of networks that has the same input and output shapes.

Mixture of experts learns to how to mix results from different networks in order to get better performances. It adds gating layer that using input data tries to figure out which of the networks will make better contribution to the final result. The final result mixes from all networks using different weights. The higher the weight the larger contribution from the individual layer.

Parameters:

networks : list of connections or networks

These networks will be combine into mixture of experts. Every network should have single 1D input layer and single output layer. Another restriction is that all networks should expect the same input and output layers.

gating_layer : None or layer

In case if value equal to None that the following layer will be created.

gating_layer = layers.Softmax(len(networks))

Output from the gating layer should be 1D and equal to the number of networks.

Returns:

connection

Mixture of experts network that combine all networks into single one and adds gating layer to it.

Raises:

ValueError

In case if there is some problem with input networks or custom gating layer.

Examples

>>> from neupy import layers, algorithms, architectures
>>>
>>> network = architectures.mixture_of_experts([
...     layers.join(
...         layers.Input(10),
...         layers.Relu(5),
...     ),
...     layers.join(
...         layers.Input(10),
...         layers.Relu(33),
...         layers.Relu(5),
...     ),
...     layers.join(
...         layers.Input(10),
...         layers.Relu(12),
...         layers.Relu(25),
...         layers.Relu(5),
...     ),
... ])
>>> network
10 -> [... 12 layers ...] -> 5
>>>
>>> gdnet = algorithms.Momentum(network, step=0.1)