Chapter 1: Introduction to Neural Networks
Perceptron, MLP, and Activation Functions
Perceptron
Perceptron (Rosenblatt, 1958)
The simplest form of a neural network (a single-layer model). It thresholds a weighted sum of inputs:
$$y = \begin{cases} 1 & \text{if } \mathbf{w}^\top \mathbf{x} + b > 0 \\ 0 & \text{otherwise} \end{cases}$$Limitations of the Perceptron
A single-layer perceptron splits the input space with a single line (a hyperplane in higher dimensions), so it can only solve linearly separable problems. Patterns such as XOR, which cannot be separated by a straight line, are unreachable.
In 1969, Minsky and Papert rigorously showed this limitation of the single-layer perceptron in their book Perceptrons, which is regarded as one of the causes of the subsequent slowdown in neural network research (the AI winter). A multilayer network can represent XOR, but it was in 1986 (Rumelhart et al.) that backpropagation spread as a way to train such networks, making the training of multilayer networks practical. This broader history of booms and AI winters is surveyed in A Brief History of Machine Learning.
Multilayer Perceptron (MLP)
Multilayer Perceptron
A neural network built by stacking multiple layers. Composing nonlinear functions lets it represent complex functions.
Forward Propagation
The output of the $l$-th layer:
$$\mathbf{h}^{(l)} = \sigma(\mathbf{W}^{(l)} \mathbf{h}^{(l-1)} + \mathbf{b}^{(l)})$$- $\mathbf{W}^{(l)}$: weight matrix of the $l$-th layer
- $\mathbf{b}^{(l)}$: bias vector of the $l$-th layer
- $\sigma$: activation function (the output layer's function is chosen according to the task; see "Output Layer and Loss Function" below)
Universal Approximation Theorem
An MLP with a single hidden layer, using a non-polynomial activation function such as the sigmoid, can approximate any continuous function on a bounded closed (compact) set to arbitrary precision (uniformly), given enough hidden units (Cybenko, 1989; Hornik, 1991).
Practical caveat: being "approximable" does not imply being "learnable", and it is known that the number of hidden units required can grow exponentially. One reason modern deep learning stacks many layers is that the same function can often be represented far more efficiently — with far fewer parameters — by multiple layers.
Activation Functions
Role of the Activation Function
It introduces nonlinearity. Without it, no matter how many layers you stack the network would just compute a composition of linear transformations (which is again linear; with biases these are affine transformations, and a composition of affine transformations is again affine).
Comparison of Activation Functions
| Function | Output Range | Characteristics |
|---|---|---|
| Sigmoid | (0, 1) | Convenient for probabilistic output. Prone to vanishing gradients. |
| tanh | (-1, 1) | Zero-centered. Larger gradients than Sigmoid near the origin (max 1 vs. 0.25). |
| ReLU | [0, ∞) | Fast to compute. Suffers from the Dead ReLU problem. |
| Leaky ReLU | (-∞, ∞) | Mitigates the Dead ReLU problem. |
Why ReLU Is Popular
- Fast to compute: only max(0, z).
- Resistant to vanishing gradients: the gradient is 1 in the positive region.
- Sparsity: depending on the input distribution and initialization, many units output zero (efficient).
What Is the Dead ReLU Problem?
ReLU outputs 0 for any negative input and its gradient is also 0 there. If a large update during training pushes a neuron into a regime where its preactivation $z$ is negative for almost all training data, the neuron's output and gradient become 0, no error flows back through it, and it is hard for it to recover. This failure mode is called the Dead ReLU problem.
Common countermeasures include Leaky ReLU ($\max(\alpha z, z)$, where $\alpha$ is a small positive hyperparameter, commonly $\alpha = 0.01$ by default), PReLU (where $\alpha$ is learned), and ELU, all of which keep a small gradient on the negative side. Lowering the learning rate and using He initialization (designed for ReLU) are also effective preventive measures.
Output Layer and Loss Function
Output Layer by Task
| Task | Output Layer | Loss Function |
|---|---|---|
| Regression | Linear (identity) | MSE |
| Binary Classification | Sigmoid | Binary Cross-Entropy |
| Multiclass Classification (mutually exclusive classes) | Softmax | Cross-Entropy |
Summary
- Perceptron: a single neuron (single layer), only linearly separable problems
- MLP: multiple layers, can represent nonlinear functions
- Activation function: introduces nonlinearity; ReLU is the mainstream choice
- Universal approximation theorem: a single hidden layer can approximate any continuous function on a bounded closed set (under conditions on the activation function)
Frequently Asked Questions
What is the difference between a perceptron and a multilayer perceptron (MLP)?
A perceptron is the simplest classifier: a single neuron that thresholds a weighted sum of inputs, and as a single layer it can only solve linearly separable problems (it cannot solve XOR). An MLP is a network of stacked layers; by inserting an activation function (a nonlinear function) at each layer, it can express complex nonlinear functions, and with enough hidden units it can approximate any continuous function on a bounded closed set (the universal approximation theorem of Cybenko and others, under conditions on the activation function).
Why do activation functions need to be nonlinear?
The composition of linear transformations (affine transformations, once biases are included) is again linear (affine), so if the activation is linear (e.g., the identity function) then no matter how many layers you stack you only get the expressive power of a single layer. Inserting a nonlinear function such as Sigmoid, tanh, or ReLU is what gives stacking layers any meaning and lets the network represent complex functions.
When should I use Sigmoid versus ReLU?
ReLU is the mainstream choice in modern deep learning. It is fast to compute (just max(0, z)), it has a gradient of 1 in the positive region so it suffers less from vanishing gradients, and depending on the input distribution and initialization many units output zero, which gives a useful sparsity. Sigmoid is prone to vanishing gradients and so is avoided in hidden layers, but it is still used in the output layer for binary classification (mapping to a probability in (0, 1)) and in the gating mechanisms of LSTMs.
How do I choose the output layer and loss function?
The choice is typically determined by the task. For regression, use a linear (identity) output layer with MSE loss. For binary classification, use a Sigmoid output with Binary Cross-Entropy loss. For multiclass classification with mutually exclusive classes, use a Softmax output with Cross-Entropy loss (for multi-label classification, where one input can have several labels, use a Sigmoid with Binary Cross-Entropy for each class). These pairings can be derived as the negative log-likelihood of a probabilistic model (normal distribution, Bernoulli distribution, or categorical distribution, respectively).