neupy.algorithms.gd.lev_marq module

class neupy.algorithms.gd.lev_marq.LevenbergMarquardt[source]

Levenberg-Marquardt algorithm is a variation of the Newton’s method. It minimizes MSE error. The algorithm approximates Hessian matrix using dot product between two jacobian matrices.

Parameters:
network : list, tuple or LayerConnection instance

Network’s architecture. There are a few ways to define it.

  • List of layers. For instance, [Input(2), Tanh(4), Relu(1)].
  • Constructed layers. For instance, Input(2) >> Tanh(4) >> Relu(1).
mu : float

Control inversion for J.T * J matrix, defaults to 0.1.

mu_update_factor : float

Factor to decrease the mu if error was reduced after last update, otherwise increase mu by the same factor. Defaults to 1.2

error : {mse}

Levenberg-Marquardt works only for quadratic functions. Defaults to mse.

show_epoch : int

This property controls how often the network will display information about training. It has to be defined as positive integer. For instance, number 100 mean that network shows summary at 1st, 100th, 200th, 300th … and last epochs.

Defaults to 1.

shuffle_data : bool

If it’s True than training data will be shuffled before the training. Defaults to True.

signals : dict, list or function

Function that will be triggered after certain events during the training.

verbose : bool

Property controls verbose output in terminal. The True value enables informative output in the terminal and False - disable it. Defaults to False.

See also

BaseOptimizer
BaseOptimizer algorithm.

Notes

  • Method requires all training data during propagation, which means it cannot be trained with mini-batches.
  • Network minimizes only Mean Squared Error (MSE) loss function.
  • Efficient for small training datasets, because it computes gradient per each sample separately.
  • Efficient for small-sized networks.

Examples

>>> import numpy as np
>>> from neupy import algorithms
>>> from neupy.layers import *
>>>
>>> x_train = np.array([[1, 2], [3, 4]])
>>> y_train = np.array([[1], [0]])
>>>
>>> network = Input(2) >> Sigmoid(3) >> Sigmoid(1)
>>> optimizer = algorithms.LevenbergMarquardt(network)
>>> optimizer.train(x_train, y_train)
Attributes:
errors : list

Information about errors. It has two main attributes, namely train and valid. These attributes provide access to the training and validation errors respectively.

last_epoch : int

Value equals to the last trained epoch. After initialization it is equal to 0.

n_updates_made : int

Number of training updates applied to the network.

Methods

predict(X) Predicts output for the specified input.
train(X_train, y_train, X_test=None, y_test=None, epochs=100) Train network. You can control network’s training procedure with epochs parameter. The X_test and y_test should be presented both in case network’s validation required after each training epoch.
fit(*args, **kwargs) Alias to the train method.
init_functions()[source]
init_train_updates()[source]
loss = None[source]
mu = None[source]
mu_update_factor = None[source]
one_training_update(X_train, y_train)[source]

Function would be trigger before run all training procedure related to the current epoch.

Parameters:
epoch : int

Current epoch number.
options = {'loss': Option(class_name='LevenbergMarquardt', value=ChoiceProperty(name="loss")), 'mu': Option(class_name='LevenbergMarquardt', value=BoundedProperty(name="mu")), 'mu_update_factor': Option(class_name='LevenbergMarquardt', value=BoundedProperty(name="mu_update_factor")), 'show_epoch': Option(class_name='BaseNetwork', value=IntProperty(name="show_epoch")), 'shuffle_data': Option(class_name='BaseNetwork', value=Property(name="shuffle_data")), 'signals': Option(class_name='BaseNetwork', value=Property(name="signals")), 'target': Option(class_name='BaseOptimizer', value=Property(name="target")), 'verbose': Option(class_name='Verbose', value=VerboseProperty(name="verbose"))}[source]
regularizer = None[source]
step = None[source]