Modeling biological neurons
Neural nets are modeled upon real biological neurons, which have the following characteristics:
- All or none: the input from each entry in the neural net is either O or 1, the output is also 0 or 1
- Cumulative influence: the influence of various input neurons accumulates to produce the final output result, if a certain threshold is reached
- Synaptic weight: each input neuron is given a weight depending on its importance
Characteristics that are not modeled
Several characteristics of biological neurons are not modeled in neural nets, as their purpose is not clearly understood in brain information processing.
- Real neurons also have a refractory period when they do not respond to stimuli after firing.
- In a real neuron, the resulting output can go to either one (of a few) axonal bifurcations.
- The timing of the various inputs in the dendritic tree is not well understood in the resulting pulse in the axon.
Neural net principles
In a neural net, the resulting vector z, is a function of the different inputs x[n], the weights w[n] and the thresholds t[n]. A neural net is therefore a function approximator.
By comparing a known desired results vector (such as the content of a picture) with the actual output vector, a performance function can be determined to know how well the neural net is performing.
To simplify the performance function, thresholds are eliminated by adding an extra weight w[0] that nullify the threshold, and the step function resulting in {0, 1} values is smoothed to a sigmoid function resulting in the [0, 1] interval.
Backpropagation
Backpropagation is the name of the algorithm generally used to train a neural net.
Varying the weights little by little and with a certain randomization allows the performance function to measure if progress is being made or not, and to improve the weighing accordingly to progress towards an optimal performance.
The amount of computation of the performance function is linearly increased by the depth and squared by the width of the net.