# Cheat sheet

## Algorithms

### Deep Learning

#### Optimizers

neupy.algorithms.Momentum | Momentum algorithm. |

neupy.algorithms.GradientDescent | Mini-batch Gradient Descent algorithm. |

neupy.algorithms.Adam | Adam algorithm. |

neupy.algorithms.Adamax | AdaMax algorithm. |

neupy.algorithms.RMSProp | RMSProp algorithm. |

neupy.algorithms.Adadelta | Adadelta algorithm. |

neupy.algorithms.Adagrad | Adagrad algorithm. |

neupy.algorithms.ConjugateGradient | Conjugate Gradient algorithm. |

neupy.algorithms.QuasiNewton | Quasi-Newton algorithm. |

neupy.algorithms.LevenbergMarquardt | Levenberg-Marquardt algorithm is a variation of the Newton’s method. |

neupy.algorithms.Hessian | Hessian gradient decent optimization, also known as Newton’s method. |

neupy.algorithms.HessianDiagonal | Hissian diagonal is a Hessian algorithm approximation which require only computation of hessian matrix diagonal elements and makes it invertion much easier and faster. |

neupy.algorithms.RPROP | Resilient backpropagation (RPROP) is an optimization algorithm for supervised learning. |

neupy.algorithms.IRPROPPlus | iRPROP+ is an optimization algorithm for supervised learning. |

#### Regularizers

```
from neupy import algorithms
from neupy.layers import *
optimizer = algorithms.Momentum(
Input(5) >> Relu(10) >> Sigmoid(1),
step=algorithms.l2(decay_rate=0.1)
)
```

neupy.algorithms.l1 | Applies l1 regularization to the trainable parameters in the network. |

neupy.algorithms.l2 | Applies l2 regularization to the trainable parameters in the network. |

neupy.algorithms.maxnorm | Applies max-norm regularization to the trainable parameters in the network. |

#### Learning rate update rules

```
from neupy import algorithms
from neupy.layers import *
optimizer = algorithms.Momentum(
Input(5) >> Relu(10) >> Sigmoid(1),
step=algorithms.step_decay(
initial_value=0.1,
reduction_freq=100,
)
)
```

neupy.algorithms.step_decay | Algorithm minimizes learning step monotonically after each iteration. |

neupy.algorithms.exponential_decay | Applies exponential decay to the learning rate. |

neupy.algorithms.polynomial_decay | Applies polynomial decay to the learning rate. |

### Neural Networks with Radial Basis Functions (RBFN)

neupy.algorithms.GRNN | Generalized Regression Neural Network (GRNN). |

neupy.algorithms.PNN | Probabilistic Neural Network (PNN). |

### Autoasociative Memory

neupy.algorithms.DiscreteBAM | Discrete BAM Network with associations. |

neupy.algorithms.CMAC | Cerebellar Model Articulation Controller (CMAC) Network based on memory. |

neupy.algorithms.DiscreteHopfieldNetwork | Discrete Hopfield Network. |

### Competitive Networks

neupy.algorithms.ART1 | Adaptive Resonance Theory (ART1) Network for binary data clustering. |

neupy.algorithms.GrowingNeuralGas | Growing Neural Gas (GNG) algorithm. |

neupy.algorithms.SOFM | Self-Organizing Feature Map (SOFM or SOM). |

neupy.algorithms.LVQ | Learning Vector Quantization (LVQ) algorithm. |

neupy.algorithms.LVQ2 | Learning Vector Quantization 2 (LVQ2) algorithm. |

neupy.algorithms.LVQ21 | Learning Vector Quantization 2.1 (LVQ2.1) algorithm. |

neupy.algorithms.LVQ3 | Learning Vector Quantization 3 (LVQ3) algorithm. |

### Associative

neupy.algorithms.Oja | Oja is an unsupervised technique used for the dimensionality reduction tasks. |

neupy.algorithms.Kohonen | Kohonen Neural Network used for unsupervised learning. |

neupy.algorithms.Instar | Instar is a simple unsupervised Neural Network algortihm which detects associations. |

neupy.algorithms.HebbRule | Neural Network with Hebbian Learning. |

### Boltzmann Machine

neupy.algorithms.RBM | Boolean/Bernoulli Restricted Boltzmann Machine (RBM). |

## Layers

```
from neupy.layers import *
network = Input(32) >> Relu(16) >> Softmax(10)
```

### Layers with activation function

neupy.layers.Linear | Layer with linear activation function. |

neupy.layers.Sigmoid | Layer with the sigmoid used as an activation function. |

neupy.layers.HardSigmoid | Layer with the hard sigmoid used as an activation function. |

neupy.layers.Tanh | Layer with the hyperbolic tangent used as an activation function. |

neupy.layers.Relu | Layer with the rectifier (ReLu) used as an activation function. |

neupy.layers.LeakyRelu | Layer with the leaky rectifier (Leaky ReLu) used as an activation function. |

neupy.layers.Elu | Layer with the exponensial linear unit (ELU) used as an activation function. |

neupy.layers.PRelu | Layer with the parametrized ReLu used as an activation function. |

neupy.layers.Softplus | Layer with the softplus used as an activation function. |

neupy.layers.Softmax | Layer with the softmax activation function. |

### Convolutional layers

neupy.layers.Convolution | Convolutional layer. |

neupy.layers.Deconvolution | Deconvolution layer (also known as Transposed Convolution.). |

### Recurrent layers

neupy.layers.LSTM | Long Short Term Memory (LSTM) Layer. |

neupy.layers.GRU | Gated Recurrent Unit (GRU) Layer. |

### Pooling layers

neupy.layers.MaxPooling | Maximum pooling layer. |

neupy.layers.AveragePooling | Average pooling layer. |

neupy.layers.Upscale | Upscales input over two axis (height and width). |

neupy.layers.GlobalPooling | Global pooling layer. |

### Normalization layers

neupy.layers.BatchNorm | Batch normalization layer. |

neupy.layers.GroupNorm | Group Normalization layer. |

neupy.layers.LocalResponseNorm | Local Response Normalization Layer. |

### Stochastic layers

neupy.layers.Dropout | Dropout layer. |

neupy.layers.GaussianNoise | Add gaussian noise to the input value. |

### Merge layers

neupy.layers.Elementwise | Layer merges multiple input with elementwise function and generate single output. |

neupy.layers.Concatenate | Concatenate multiple inputs into one. |

neupy.layers.GatedAverage | Layer uses applies weighted elementwise addition to multiple outputs. |

### Other layers

neupy.layers.Input | Layer defines network’s input. |

neupy.layers.Identity | Passes input through the layer without changes. |

neupy.layers.Reshape | Layer reshapes input tensor. |

neupy.layers.Transpose | Layer transposes input tensor. |

neupy.layers.Embedding | Embedding layer accepts indeces as an input and returns rows from the weight matrix associated with these indeces. |

### Operations

Additional operations that can be performed on the layers or graphs

neupy.layers.join(*networks) | Sequentialy combines layers and networks into single network. |

neupy.layers.parallel(*networks) | Merges all networks/layers into single network without joining input and output layers together. |

## Architectures

```
>>> from neupy import architectures
>>> resnet = architectures.resnet50()
>>> resnet
(?, 224, 224, 3) -> [... 187 layers ...] -> (?, 1000)
```

neupy.architectures.vgg16 | VGG16 network architecture with random parameters. |

neupy.architectures.vgg19 | VGG19 network architecture with random parameters. |

neupy.architectures.squeezenet | SqueezeNet network architecture with random parameters. |

neupy.architectures.resnet50 | ResNet50 network architecture with random parameters. |

neupy.architectures.mixture_of_experts | Generates mixture of experts architecture from the set of networks that has the same input and output shapes. |

## Parameter initialization

```
from neupy.init import *
from neupy.layers import *
from neupy import algorithms
gdnet = algorithms.GradientDescent([
Input(784),
Relu(100, weight=HeNormal(), bias=Constant(0)),
Softmax(10, weight=Uniform(-0.01, 0.01)),
])
```

neupy.init.Constant | Initialize parameter that has constant values. |

neupy.init.Normal | Initialize parameter sampling from the normal distribution. |

neupy.init.Uniform | Initialize parameter sampling from the uniformal distribution. |

neupy.init.Orthogonal | Initialize matrix with orthogonal basis. |

neupy.init.HeNormal | Kaiming He parameter initialization method based on the normal distribution. |

neupy.init.HeUniform | Kaiming He parameter initialization method based on the uniformal distribution. |

neupy.init.XavierNormal | Xavier Glorot parameter initialization method based on normal distribution. |

neupy.init.XavierUniform | Xavier Glorot parameter initialization method based on uniform distribution. |

## Datasets

neupy.datasets.load_digits | Returns dataset that contains discrete digits. |

neupy.datasets.make_digits | Returns discrete digits dataset. |

neupy.datasets.make_reber | Generate list of words valid by Reber grammar. |

neupy.datasets.make_reber_classification | Generate random dataset for Reber grammar classification. |