neupy.layers.LSTM

class neupy.layers.LSTM[source]

Long Short Term Memory (LSTM) Layer.

Parameters:
n_units : int

Number of hidden units in the layer.

only_return_final : bool

If True, only return the final sequential output (e.g. for tasks where a single target value for the entire sequence is desired). In this case, Tensorfow makes an optimization which saves memory. Defaults to True.

input_weights : Initializer, ndarray

Weight parameters for input connection. Defaults to HeNormal().

hidden_weights : Initializer, ndarray

Weight parameters for hidden connection. Defaults to HeNormal().

cell_weights : Initializer, ndarray

Weight parameters for cell connection. Require only when peepholes=True otherwise it will be ignored. Defaults to HeNormal().

bias : Initializer, ndarray

Bias parameters for all gates. Defaults to Constant(0).

ingate : function

Activation function for the input gate. Defaults to tf.nn.sigmoid.

forgetgate : function

Activation function for the forget gate. Defaults to tf.nn.sigmoid.

outgate : function

Activation function for the output gate. Defaults to tf.nn.sigmoid.

cell : function

Activation function for the cell. Defaults to tf.tanh.

learn_init : bool

If True, make cell_init and hidden_init trainable variables. Defaults to False.

cell_init : array-like, Tensorfow variable, scalar or Initializer

Initializer for initial cell state (\(c_0\)). Defaults to Constant(0).

hidden_init : array-like, Tensorfow variable, scalar or Initializer

Initializer for initial hidden state (\(h_0\)). Defaults to Constant(0).

backwards : bool

If True, process the sequence backwards and then reverse the output again such that the output from the layer is always from \(x_1\) to \(x_n\). Defaults to False

peepholes : bool

If True, the LSTM uses peephole connections. When False, cell parameters are ignored. Defaults to False.

unroll_scan : bool

If True the recursion is unrolled instead of using scan. For some graphs this gives a significant speed up but it might also consume more memory. When unroll_scan=True, backpropagation always includes the full sequence, so n_gradient_steps must be set to -1 and the input sequence length must be known at compile time (i.e., cannot be given as None). Defaults to False.

gradient_clipping : float or int

If nonzero, the gradient messages are clipped to the given value during the backward pass. Defaults to 0.

name : str or None

Layer’s name. Can be used as a reference to specific layer. Name Can be specified as:

  • String: Specified name will be used as a direct reference to the layer. For example, name=”fc”
  • Format string: Name pattern could be defined as a format string and specified field will be replaced with an index. For example, name=”fc{}” will be replaced with fc1, fc2 and so on. A bit more complex formatting methods are acceptable, for example, name=”fc-{:<03d}” will be converted to fc-001, fc-002, fc-003 and so on.
  • None: When value specified as None than name will be generated from the class name.

Defaults to None.

Notes

Code was adapted from the Lasagne library.

Examples

Sequence classification

>>> from neupy.layers import *
>>>
>>> n_time_steps = 40
>>> n_categories = 20
>>> embedded_size = 10
>>>
>>> network = join(
...     Input(n_time_steps),
...     Embedding(n_categories, embedded_size),
...     LSTM(20),
...     Sigmoid(1),
... )
>>> network
(?, 40) -> [... 4 layers ...] -> (?, 1)
backwards = None[source]
biases = None[source]
cell = None[source]
cell_init = None[source]
cell_weights = None[source]
create_variables(input_shape)[source]
forgetgate = None[source]
gradient_clipping = None[source]
hidden_init = None[source]
hidden_weights = None[source]
ingate = None[source]
input_weights = None[source]
learn_init = None[source]
options = {'backwards': Option(class_name='LSTM', value=Property(name="backwards")), 'biases': Option(class_name='LSTM', value=ParameterProperty(name="biases")), 'cell': Option(class_name='LSTM', value=Property(name="cell")), 'cell_init': Option(class_name='LSTM', value=ParameterProperty(name="cell_init")), 'cell_weights': Option(class_name='LSTM', value=ParameterProperty(name="cell_weights")), 'forgetgate': Option(class_name='LSTM', value=Property(name="forgetgate")), 'gradient_clipping': Option(class_name='LSTM', value=NumberProperty(name="gradient_clipping")), 'hidden_init': Option(class_name='LSTM', value=ParameterProperty(name="hidden_init")), 'hidden_weights': Option(class_name='LSTM', value=ParameterProperty(name="hidden_weights")), 'ingate': Option(class_name='LSTM', value=Property(name="ingate")), 'input_weights': Option(class_name='LSTM', value=ParameterProperty(name="input_weights")), 'learn_init': Option(class_name='LSTM', value=Property(name="learn_init")), 'n_units': Option(class_name='BaseRNNLayer', value=IntProperty(name="n_units")), 'name': Option(class_name='BaseLayer', value=Property(name="name")), 'only_return_final': Option(class_name='BaseRNNLayer', value=Property(name="only_return_final")), 'outgate': Option(class_name='LSTM', value=Property(name="outgate")), 'peepholes': Option(class_name='LSTM', value=Property(name="peepholes")), 'unroll_scan': Option(class_name='LSTM', value=Property(name="unroll_scan"))}[source]
outgate = None[source]
output(input, **kwargs)[source]
peepholes = None[source]
unroll_scan = None[source]