neupy.layers.recurrent module

class neupy.layers.recurrent.LSTM[source]

Long Short Term Memory (LSTM) Layer.

Parameters:

size : int

Number of hidden units in the layer.

input_weights : Initializer, ndarray

Weight parameters for input connection. Defaults to HeNormal().

hidden_weights : Initializer, ndarray

Weight parameters for hidden connection. Defaults to HeNormal().

cell_weights : Initializer, ndarray

Weight parameters for cell connection. Require only when peepholes=True otherwise it will be ignored. Defaults to HeNormal().

bias : Initializer, ndarray

Bias parameters for all gates. Defaults to Constant(0).

activation_functions : dict, callable

Activation functions for different gates. Defaults to:

# import tensorflow as tf
dict(
    ingate=tf.nn.sigmoid,
    forgetgate=tf.nn.sigmoid,
    outgate=tf.nn.sigmoid,
    cell=tf.tanh,
)

If application requires modification to only one parameter then it’s better to specify the one that you need to modify and ignore other parameters

dict(ingate=tf.tanh)

Other parameters like forgetgate or outgate will be equal to their default values.

learn_init : bool

If True, make cell_init and hidden_init trainable variables. Defaults to False.

cell_init : array-like, Tensorfow variable, scalar or Initializer

Initializer for initial cell state (\(c_0\)). Defaults to Constant(0).

hidden_init : array-like, Tensorfow variable, scalar or Initializer

Initializer for initial hidden state (\(h_0\)). Defaults to Constant(0).

backwards : bool

If True, process the sequence backwards and then reverse the output again such that the output from the layer is always from \(x_1\) to \(x_n\). Defaults to False

only_return_final : bool

If True, only return the final sequential output (e.g. for tasks where a single target value for the entire sequence is desired). In this case, Tensorfow makes an optimization which saves memory. Defaults to True.

peepholes : bool

If True, the LSTM uses peephole connections. When False, cell parameters are ignored. Defaults to False.

unroll_scan : bool

If True the recursion is unrolled instead of using scan. For some graphs this gives a significant speed up but it might also consume more memory. When unroll_scan=True, backpropagation always includes the full sequence, so n_gradient_steps must be set to -1 and the input sequence length must be known at compile time (i.e., cannot be given as None). Defaults to False.

gradient_clipping : float or int

If nonzero, the gradient messages are clipped to the given value during the backward pass. Defaults to 0.

name : str or None

Layer’s identifier. If name is equal to None than name will be generated automatically. Defaults to None.

Notes

Code was adapted from the Lasagne library.

Examples

Sequence classification

from neupy import layers, algorithms

n_time_steps = 40
n_categories = 20
embedded_size = 10

network = algorithms.RMSProp(
    [
        layers.Input(n_time_steps),
        layers.Embedding(n_categories, embedded_size),
        layers.LSTM(20),
        layers.Sigmoid(1),
    ]
)
activation_functions = None[source]
backwards = None[source]
biases = None[source]
cell_init = None[source]
cell_weights = None[source]
gradient_clipping = None[source]
hidden_init = None[source]
hidden_weights = None[source]
initialize()[source]

Initialize connection

input_weights = None[source]
learn_init = None[source]
options = {'name': Option(class_name='BaseLayer', value=Property(name="name")), 'size': Option(class_name='BaseRNNLayer', value=IntProperty(name="size")), 'only_return_final': Option(class_name='BaseRNNLayer', value=Property(name="only_return_final")), 'input_weights': Option(class_name='LSTM', value=ParameterProperty(name="input_weights")), 'hidden_weights': Option(class_name='LSTM', value=ParameterProperty(name="hidden_weights")), 'cell_weights': Option(class_name='LSTM', value=ParameterProperty(name="cell_weights")), 'biases': Option(class_name='LSTM', value=ParameterProperty(name="biases")), 'activation_functions': Option(class_name='LSTM', value=MultiCallableProperty(name="activation_functions")), 'learn_init': Option(class_name='LSTM', value=Property(name="learn_init")), 'cell_init': Option(class_name='LSTM', value=ParameterProperty(name="cell_init")), 'hidden_init': Option(class_name='LSTM', value=ParameterProperty(name="hidden_init")), 'unroll_scan': Option(class_name='LSTM', value=Property(name="unroll_scan")), 'backwards': Option(class_name='LSTM', value=Property(name="backwards")), 'peepholes': Option(class_name='LSTM', value=Property(name="peepholes")), 'gradient_clipping': Option(class_name='LSTM', value=NumberProperty(name="gradient_clipping"))}[source]
output(input_value)[source]

Return output base on the input value.

Parameters:input_value
peepholes = None[source]
unroll_scan = None[source]
class neupy.layers.recurrent.GRU[source]

Gated Recurrent Unit (GRU) Layer.

Parameters:

size : int

Number of hidden units in the layer.

input_weights : Initializer, ndarray

Weight parameters for input connection. Defaults to HeNormal().

hidden_weights : Initializer, ndarray

Weight parameters for hidden connection. Defaults to HeNormal().

bias : Initializer, ndarray

Bias parameters for all gates. Defaults to Constant(0).

activation_functions : dict, callable

Activation functions for different gates. Defaults to:

# import tensorflow as tf
dict(
    resetgate=tf.nn.sigmoid,
    updategate=tf.nn.sigmoid,
    hidden_update=tf.tanh,
)

If application requires modification to only one parameter then it’s better to specify the one that you need to modify and ignore other parameters

dict(resetgate=tf.tanh)

Other parameters like updategate or hidden_update will be equal to their default values.

learn_init : bool

If True, make hidden_init trainable variable. Defaults to False.

hidden_init : array-like, Tensorfow variable, scalar or Initializer

Initializer for initial hidden state (\(h_0\)). Defaults to Constant(0).

only_return_final : bool

If True, only return the final sequential output (e.g. for tasks where a single target value for the entire sequence is desired). In this case, Tensorfow makes an optimization which saves memory. Defaults to True.

backwards : bool

If True, process the sequence backwards and then reverse the output again such that the output from the layer is always from \(x_1\) to \(x_n\). Defaults to False.

unroll_scan : bool

If True the recursion is unrolled instead of using scan. For some graphs this gives a significant speed up but it might also consume more memory. When unroll_scan=True, backpropagation always includes the full sequence, so n_gradient_steps must be set to -1 and the input sequence length must be known at compile time (i.e., cannot be given as None). Defaults to False.

name : str or None

Layer’s identifier. If name is equal to None than name will be generated automatically. Defaults to None.

Notes

Code was adapted from the Lasagne library.

Examples

Sequence classification

from neupy import layers, algorithms

n_time_steps = 40
n_categories = 20
embedded_size = 10

network = algorithms.RMSProp(
    [
        layers.Input(n_time_steps),
        layers.Embedding(n_categories, embedded_size),
        layers.GRU(20),
        layers.Sigmoid(1),
    ]
)
activation_functions = None[source]
backwards = None[source]
biases = None[source]
gradient_clipping = None[source]
hidden_init = None[source]
hidden_weights = None[source]
initialize()[source]

Initialize connection

input_weights = None[source]
learn_init = None[source]
options = {'name': Option(class_name='BaseLayer', value=Property(name="name")), 'size': Option(class_name='BaseRNNLayer', value=IntProperty(name="size")), 'only_return_final': Option(class_name='BaseRNNLayer', value=Property(name="only_return_final")), 'input_weights': Option(class_name='GRU', value=ParameterProperty(name="input_weights")), 'hidden_weights': Option(class_name='GRU', value=ParameterProperty(name="hidden_weights")), 'biases': Option(class_name='GRU', value=ParameterProperty(name="biases")), 'activation_functions': Option(class_name='GRU', value=MultiCallableProperty(name="activation_functions")), 'learn_init': Option(class_name='GRU', value=Property(name="learn_init")), 'hidden_init': Option(class_name='GRU', value=ParameterProperty(name="hidden_init")), 'backwards': Option(class_name='GRU', value=Property(name="backwards")), 'unroll_scan': Option(class_name='GRU', value=Property(name="unroll_scan")), 'gradient_clipping': Option(class_name='GRU', value=NumberProperty(name="gradient_clipping"))}[source]
output(input_value)[source]

Return output base on the input value.

Parameters:input_value
unroll_scan = None[source]