Normal Distribution

Normal Distribution (Gaussian Distribution)

Beginner

Overview

The normal distribution is the continuous probability distribution that appears most often in nature and in social phenomena, and it sits at the center of statistics. It is also called the Gaussian distribution after Carl Friedrich Gauss (1777–1855).

The normal distribution describes many random phenomena: heights and weights, measurement errors, exam scores, fluctuations in stock prices, and more. By the central limit theorem, sums of independent random variables approach a normal distribution, which makes it extremely important in practice.

Definition

Definition: Normal distribution

When a random variable $X$ follows a normal distribution with mean $\mu$ and variance $\sigma^2$, we write $X \sim N(\mu, \sigma^2)$. Its probability density function (PDF) is

$$f(x) = \dfrac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\dfrac{(x - \mu)^2}{2\sigma^2}\right)$$

where:

$\mu \in \mathbb{R}$: the expected value (mean)
$\sigma^2 > 0$: the variance
$\sigma$: the standard deviation

Shape of the graph

The probability density function of the normal distribution is a bell-shaped curve centered at $x = \mu$. The larger $\mu$ is, the further to the right it sits; the larger $\sigma$ is, the wider it spreads.

Properties

Mean and variance

Mean: $E[X] = \mu$
Variance: $\text{Var}(X) = \sigma^2$
Standard deviation: $\text{SD}(X) = \sigma$

Proof: that the mean is $\mu$ and the variance is $\sigma^2$

We show this directly from the definition of the probability density function. Set the deviation $t = x-\mu$ (so $dx = dt$); then

$$g(t) := f(t+\mu) = \dfrac{1}{\sqrt{2\pi}\,\sigma}\,\exp\!\left(-\dfrac{t^2}{2\sigma^2}\right)$$

is an even function of $t$. Also, since $f$ is a probability density, its integral over the whole line equals total probability 1. This is called normalization, and the substitution $t=x-\mu$ carries it over to $g$ unchanged:

$$\int_{-\infty}^{\infty} g(t)\,dt = \int_{-\infty}^{\infty} f(x)\,dx = 1.$$

The fact that this equals $1$ is itself equivalent to the normalizing constant $\dfrac{1}{\sqrt{2\pi}\,\sigma}$ exactly cancelling the Gaussian integral $\displaystyle\int_{-\infty}^{\infty} e^{-t^2/(2\sigma^2)}\,dt = \sqrt{2\pi}\,\sigma$. That Gaussian integral itself is obtained by squaring it and switching to polar coordinates:

$$\left(\int_{-\infty}^{\infty} e^{-u^2/2}\,du\right)^{\!2} = \iint_{\mathbb{R}^2} e^{-(u^2+v^2)/2}\,du\,dv = 2\pi.$$

Mean $E[X]=\mu$:

$$E[X] = \int_{-\infty}^{\infty} x\,f(x)\,dx = \int_{-\infty}^{\infty} (t+\mu)\,g(t)\,dt = \underbrace{\int_{-\infty}^{\infty} t\,g(t)\,dt}_{=\,0} \;+\; \mu\underbrace{\int_{-\infty}^{\infty} g(t)\,dt}_{=\,1} = \mu.$$

The first term vanishes because the integrand $t\,g(t)$ is an odd function (an even function $g$ times $t$), and the integral of an odd function over the symmetric interval $(-\infty,\infty)$ is $0$.

Variance $\text{Var}(X)=\sigma^2$: rewrite the expected value of the squared deviation with the same substitution.

$$\text{Var}(X) = E[(X-\mu)^2] = \int_{-\infty}^{\infty}(x-\mu)^2 f(x)\,dx = \int_{-\infty}^{\infty} t^2\,g(t)\,dt.$$

Differentiating the exponent gives $\dfrac{d}{dt}e^{-t^2/(2\sigma^2)} = -\dfrac{t}{\sigma^2}\,e^{-t^2/(2\sigma^2)}$, hence $t\,e^{-t^2/(2\sigma^2)} = -\sigma^2\,\dfrac{d}{dt}e^{-t^2/(2\sigma^2)}$. Using this and integrating by parts,

$$\int_{-\infty}^{\infty} t^2 g(t)\,dt = \dfrac{-\sigma^2}{\sqrt{2\pi}\,\sigma}\int_{-\infty}^{\infty} t\,\dfrac{d}{dt}e^{-t^2/(2\sigma^2)}\,dt = \dfrac{-\sigma^2}{\sqrt{2\pi}\,\sigma}\left(\Bigl[\,t\,e^{-t^2/(2\sigma^2)}\Bigr]_{-\infty}^{\infty} - \int_{-\infty}^{\infty} e^{-t^2/(2\sigma^2)}\,dt\right).$$

The boundary term vanishes because $t\,e^{-t^2/(2\sigma^2)}\to 0$ as $|t|\to\infty$. For the remainder, use normalization $\displaystyle\dfrac{1}{\sqrt{2\pi}\,\sigma}\int_{-\infty}^{\infty} e^{-t^2/(2\sigma^2)}\,dt = \int_{-\infty}^{\infty} g(t)\,dt = 1$:

$$\int_{-\infty}^{\infty} t^2 g(t)\,dt = \dfrac{\sigma^2}{\sqrt{2\pi}\,\sigma}\int_{-\infty}^{\infty} e^{-t^2/(2\sigma^2)}\,dt = \sigma^2 \cdot 1 = \sigma^2.$$

Therefore $\text{Var}(X) = \sigma^2$. $\blacksquare$

Symmetry

The normal distribution is symmetric about the mean $\mu$: $f(\mu + x) = f(\mu - x)$.

Figure 1: The normal distribution is symmetric about the axis through the mean μ. Two points equidistant from μ have equal heights (f(μ−a) = f(μ+a)).

Cumulative distribution function (CDF)

The cumulative distribution function $F(x) = P(X \leq x)$ is $$F(x) = \displaystyle\int_{-\infty}^{x} f(t) \, dt = \Phi\left(\dfrac{x - \mu}{\sigma}\right)$$ where $\Phi$ is the cumulative distribution function of the standard normal distribution. Since $\Phi$ cannot be expressed with elementary functions, numerical computation or statistical tables are used.

Figure 2: The cumulative distribution function F(x) = P(X ≤ x). A monotonically increasing S-shaped curve from 0 to 1; at x=μ, F(μ)=1/2 (the inflection point of the curve).

The 68-95-99.7 rule

Empirical rule

When $X \sim N(\mu, \sigma^2)$:

$P(\mu - \sigma \leq X \leq \mu + \sigma) \approx 0.683$ (about 68%)
$P(\mu - 2\sigma \leq X \leq \mu + 2\sigma) \approx 0.954$ (about 95%)
$P(\mu - 3\sigma \leq X \leq \mu + 3\sigma) \approx 0.997$ (about 99.7%)

In other words, almost all of the data falls within three standard deviations of the mean.

Standard normal distribution

The normal distribution with mean 0 and variance 1 is called the standard normal distribution, written $Z \sim N(0, 1)$. Its probability density function is $$\varphi(z) = \dfrac{1}{\sqrt{2\pi}} e^{-z^2/2}$$ and its cumulative distribution function is $$\Phi(z) = \displaystyle\int_{-\infty}^{z} \varphi(t) \, dt$$

Figure 3: The probability density function φ(z) of the standard normal distribution N(0, 1). A bell shape with mean 0 and variance 1; the range ±1σ (z = −1 to 1) contains about 68.3% of the total probability.

Standardization

When $X \sim N(\mu, \sigma^2)$, the change of variable $$Z = \dfrac{X - \mu}{\sigma}$$ gives $Z \sim N(0, 1)$. This operation is called standardization.

This lets any probability calculation for a normal distribution be carried out using a table for the standard normal distribution: $$P(a \leq X \leq b) = \Phi\left(\dfrac{b - \mu}{\sigma}\right) - \Phi\left(\dfrac{a - \mu}{\sigma}\right)$$

Examples

Example 1: Distribution of heights

Suppose the heights of adult men in some population follow a normal distribution with mean $\mu = 170$ cm and standard deviation $\sigma = 6$ cm.

Question: What fraction of people are at least 176 cm tall?

Solution: From $Z = (176 - 170) / 6 = 1$, $$P(X \geq 176) = 1 - \Phi(1) \approx 1 - 0.8413 = 0.1587$$ so about 15.9% of people are at least 176 cm tall.

Example 2: Exam scores

If exam scores follow $N(70, 10^2)$, what score is needed to be in the top 10%?

Solution: From $\Phi^{-1}(0.9) \approx 1.28$, $$x = \mu + \sigma \cdot 1.28 = 70 + 10 \times 1.28 = 82.8$$ so a score of about 83 or higher puts you in the top 10%.

Central limit theorem

When $X_1, X_2, \ldots, X_n$ are independent and identically distributed (mean $\mu$, variance $\sigma^2$), the sample mean $$\bar{X}_n = \dfrac{1}{n}\displaystyle\sum_{i=1}^{n} X_i$$ approaches the normal distribution $N(\mu, \sigma^2/n)$ as $n \to \infty$: $$\dfrac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0, 1)$$

By this theorem, whatever the original distribution is, the sample mean can be approximated by a normal distribution. This is the foundation of statistical inference.

For the proof of the central limit theorem (via characteristic functions and Lévy's continuity theorem), see the advanced section "Proof of the Central Limit Theorem."

Sums and products of normal distributions

Reproductive property of sums

Sum of normal distributions

If $X \sim N(\mu_1, \sigma_1^2)$ and $Y \sim N(\mu_2, \sigma_2^2)$ are independent, then $$X + Y \sim N(\mu_1 + \mu_2, \sigma_1^2 + \sigma_2^2)$$ (the reproductive property of the normal distribution).

Proof: that the sum of independent normal variables is again normal

Since $X, Y$ are independent, the density of the sum $Z = X+Y$ is given by the convolution:

$$f_Z(z) = \int_{-\infty}^{\infty} f_X(x)\,f_Y(z-x)\,dx.$$

Substituting the two normal densities, the content of the exponent (apart from the factor $-\tfrac{1}{2}$) becomes $\dfrac{(x-\mu_1)^2}{\sigma_1^2} + \dfrac{(z-x-\mu_2)^2}{\sigma_2^2}$. We complete the square in $x$. Setting $\sigma^2 := \sigma_1^2 + \sigma_2^2$, the following identity holds (verify it by expanding the right-hand side):

$$\dfrac{(x-\mu_1)^2}{\sigma_1^2} + \dfrac{(z-x-\mu_2)^2}{\sigma_2^2} = \dfrac{\sigma^2}{\sigma_1^2\sigma_2^2}\,(x-x_0)^2 + \dfrac{\bigl(z-(\mu_1+\mu_2)\bigr)^2}{\sigma^2},\quad x_0 = \dfrac{\sigma_2^2\mu_1 + \sigma_1^2(z-\mu_2)}{\sigma^2}.$$

The second term on the right does not contain $x$. So the integral over $x$ is a constant Gaussian integral, and all dependence on $z$ collects into the second term:

$$f_Z(z) = \underbrace{\dfrac{1}{2\pi\sigma_1\sigma_2}\int_{-\infty}^{\infty} \exp\!\left(-\dfrac{\sigma^2}{2\sigma_1^2\sigma_2^2}(x-x_0)^2\right)dx}_{=\;1/\sqrt{2\pi\sigma^2}}\;\exp\!\left(-\dfrac{\bigl(z-(\mu_1+\mu_2)\bigr)^2}{2\sigma^2}\right).$$

For the underbraced part, using $\displaystyle\int_{-\infty}^{\infty} e^{-(x-x_0)^2/(2\tau^2)}\,dx = \sqrt{2\pi}\,\tau$ (with $\tau^2 = \sigma_1^2\sigma_2^2/\sigma^2$) gives $\dfrac{1}{2\pi\sigma_1\sigma_2}\cdot\sqrt{2\pi}\,\tau = \dfrac{1}{\sqrt{2\pi\sigma^2}}$. Therefore

$$f_Z(z) = \dfrac{1}{\sqrt{2\pi(\sigma_1^2+\sigma_2^2)}}\,\exp\!\left(-\dfrac{\bigl(z-(\mu_1+\mu_2)\bigr)^2}{2(\sigma_1^2+\sigma_2^2)}\right),$$

which is exactly the probability density function of $N(\mu_1+\mu_2,\ \sigma_1^2+\sigma_2^2)$. Hence $X+Y \sim N(\mu_1+\mu_2,\ \sigma_1^2+\sigma_2^2)$. $\blacksquare$

Linear transformation

For $X \sim N(\mu, \sigma^2)$, setting $Y = aX + b$ gives $$Y \sim N(a\mu + b, a^2\sigma^2)$$

Proof: that a linear transformation stays normal ($a \ne 0$)

Assume $a \ne 0$ (if $a = 0$, then $Y = b$ is a constant and the distribution degenerates). $Y = aX+b$ is a monotone transformation with inverse $X = \dfrac{Y-b}{a}$ and $\left|\dfrac{dx}{dy}\right| = \dfrac{1}{|a|}$. By the change-of-variables formula, the density of $Y$ is

$$f_Y(y) = f_X\!\left(\dfrac{y-b}{a}\right)\cdot\dfrac{1}{|a|}.$$

Substitute the normal density. Focusing on the content of the exponent,

$$\dfrac{y-b}{a} - \mu = \dfrac{y - (a\mu+b)}{a} \quad\Longrightarrow\quad \left(\dfrac{y-b}{a}-\mu\right)^2 = \dfrac{\bigl(y-(a\mu+b)\bigr)^2}{a^2}$$

so, noting $|a|\,\sigma = \sqrt{a^2\sigma^2}$,

$$f_Y(y) = \dfrac{1}{|a|}\cdot\dfrac{1}{\sqrt{2\pi}\,\sigma}\exp\!\left(-\dfrac{\bigl(y-(a\mu+b)\bigr)^2}{2a^2\sigma^2}\right) = \dfrac{1}{\sqrt{2\pi\,a^2\sigma^2}}\exp\!\left(-\dfrac{\bigl(y-(a\mu+b)\bigr)^2}{2a^2\sigma^2}\right).$$

This is exactly the probability density function of $N(a\mu+b,\ a^2\sigma^2)$. Hence $Y = aX+b \sim N(a\mu+b,\ a^2\sigma^2)$. $\blacksquare$

Applications

Statistical inference

It is used as the foundation of confidence intervals and hypothesis tests ($t$-tests, $z$-tests). By the central limit theorem, the distribution of the sample mean is approximated by a normal distribution.

Quality control

Measurements such as product dimensions and weights often follow a normal distribution. Process control is carried out with control charts.

Machine learning

The normal distribution plays a central role in Gaussian processes, Gaussian mixture models (GMM), and Bayesian inference.

Financial engineering

Logarithmic returns of stock prices are approximated by a normal distribution (in reality they have fat tails). It is used in models such as Black-Scholes.

Signal processing

Thermal noise, shot noise, and the like follow a normal distribution (by the central limit theorem). It is important in filter design and noise removal.

Natural science

Measurement errors are assumed to follow a normal distribution (the basis of least squares). It is widely used in physics, chemistry, and biology.

Multivariate normal distribution

When an $n$-dimensional vector $\mathbf{X} = (X_1, \ldots, X_n)^T$ follows a multivariate normal distribution with mean vector $\boldsymbol{\mu}$ and covariance matrix $\Sigma$, $$f(\mathbf{x}) = \dfrac{1}{(2\pi)^{n/2}|\Sigma|^{1/2}} \exp\left(-\dfrac{1}{2}(\mathbf{x} - \boldsymbol{\mu})^T \Sigma^{-1} (\mathbf{x} - \boldsymbol{\mu})\right)$$

The multivariate normal distribution is fundamental to multivariate statistical analysis, principal component analysis, and machine learning (discriminant analysis, Gaussian processes).

References

Ross, S. M. (2014). Introduction to Probability Models (11th ed.). Academic Press.
Casella, G., & Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury.
Feller, W. (1968). An Introduction to Probability Theory and Its Applications, Vol. 1 (3rd ed.). Wiley.
Weisstein, E. W. "Normal Distribution." From MathWorld--A Wolfram Web Resource.

Frequently asked questions

Why is the normal distribution so important?

The importance of the normal (Gaussian) distribution $N(\mu,\sigma^2)$ comes from the central limit theorem: the sample mean of independent, identically distributed random variables converges to a normal distribution as $n\to\infty$. In addition, many natural phenomena (heights, measurement errors, and so on) are well approximated by it, and it is characterized by the maximum-entropy principle (it is the least-informative distribution once the mean and variance are fixed).

How do you standardize to the standard normal distribution?

If $X\sim N(\mu,\sigma^2)$, then $Z=(X-\mu)/\sigma\sim N(0,1)$ (the standard normal distribution). Standardization lets you compute probabilities as $P(a\leq X\leq b)=\Phi((b-\mu)/\sigma)-\Phi((a-\mu)/\sigma)$, where $\Phi$ is the standard normal CDF. The 68-95-99.7 rule, for example $P(\mu-\sigma<X<\mu+\sigma)\approx 68\%$, is easy to remember.

What is the shape of the normal probability density function?

The probability density function of the normal distribution $N(\mu,\sigma^2)$ is $f(x)=\frac{1}{\sqrt{2\pi}\sigma}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$. It is a symmetric bell shape centered at $\mu$, with wider tails for larger $\sigma$. The distribution function $\Phi(x)=\int_{-\infty}^x f(t)\,dt$ cannot be written in closed form, so numerical tables (z-tables) are used.