advanced

Wiener-Khinchin Theorem

Q: To what kinds of signals does the Wiener-Khinchin theorem apply?

The Wiener-Khinchin theorem holds for wide-sense stationary processes. A wide-sense stationary process is a stochastic process whose mean is constant and whose autocorrelation function depends only on the time lag. It also applies to deterministic signals provided their autocorrelation function is absolutely integrable.

Q: What is the relationship between the periodogram and the power spectral density?

The periodogram is one method for estimating the power spectral density from a finite-length signal. It is the squared magnitude of the finite-interval Fourier transform of the signal divided by the data length, and it converges to the true power spectral density as the data length tends to infinity. However, the periodogram is an unbiased but not a consistent estimator, because its variance does not decrease.

Q: How is the Wiener-Khinchin theorem used in optics?

In optics the theorem is used to analyze the temporal coherence of a light source. The visibility of the interference fringes measured with a Michelson interferometer corresponds to the autocorrelation function of the optical field, and its Fourier transform gives the spectral distribution of the source. Fourier-transform infrared spectroscopy (FTIR) is based on this principle.

The autocorrelation function and power spectral density as a Fourier transform pair

1. Overview

The Wiener-Khinchin theorem states that, for a wide-sense stationary process, the autocorrelation function and the power spectral density (PSD) form a Fourier transform pair.

The theorem is a fundamental bridge between the "correlation structure in the time domain" and the "energy distribution in the frequency domain" of a signal, and it is used across an extremely wide range of fields including signal processing, communication engineering, statistical physics, and optics.

The theorem is named after Norbert Wiener (1930) and Aleksandr Khinchin (1934), who proved it independently.

2. Statement of the theorem

2.1 Preliminaries: wide-sense stationary processes

A stochastic process $\{X(t)\}$ is a wide-sense stationary process (WSS) when it satisfies the following two conditions:

The mean is constant: $E[X(t)] = \mu$ (independent of $t$).
The autocorrelation function depends only on the time lag: $R_{XX}(t_1, t_2) = R_{XX}(t_2 - t_1)$.

Here the autocorrelation function is defined as follows:

$$R(\tau) = E[X(t+\tau)\overline{X(t)}]$$

2.2 The statement

Theorem (Wiener-Khinchin)

For a wide-sense stationary process $\{X(t)\}$, the autocorrelation function $R(\tau)$ and the power spectral density $S(\omega)$ form a Fourier transform pair:

$$\boxed{S(\omega) = \displaystyle\int_{-\infty}^{\infty} R(\tau)\, e^{-i\omega\tau}\, d\tau}$$ $$\boxed{R(\tau) = \dfrac{1}{2\pi}\displaystyle\int_{-\infty}^{\infty} S(\omega)\, e^{i\omega\tau}\, d\omega}$$

Here $S(\omega) \geq 0$ (the power spectral density is non-negative), and when $R(\tau)$ is real-valued and even, $S(\omega)$ is also a real-valued even function.

In particular, substituting $\tau = 0$ gives the relation between the average power of the signal and its spectrum:

$$R(0) = E[|X(t)|^2] = \dfrac{1}{2\pi}\displaystyle\int_{-\infty}^{\infty} S(\omega)\, d\omega$$

This is the stochastic-process counterpart of Parseval's theorem.

Graph showing the Fourier transform pair of the autocorrelation function R(tau) and the power spectral density S(omega). The top panel shows an oscillatory decaying autocorrelation function; the bottom panel shows the corresponding power spectral density with Lorentzian peaks.

Figure 1: The autocorrelation function $R(\tau)$ and the power spectral density $S(\omega)$ as a Fourier transform pair. The spectrum corresponding to $R(\tau) = 2\cos(2\pi f_1 \tau)e^{-|\tau|/2} + \cos(2\pi f_2 \tau)e^{-|\tau|/3}$ shows Lorentzian peaks at $f_1, f_2$.

3. Outline of the proof

Proof (derivation for a wide-sense stationary process)

Consider the Fourier transform of the signal truncated to the finite interval $[-T, T]$:

$$X_T(\omega) = \displaystyle\int_{-T}^{T} X(t)\, e^{-i\omega t}\, dt$$

Step 1: Compute the expectation of $|X_T(\omega)|^2$.

\begin{align} E[|X_T(\omega)|^2] &= E\left[\displaystyle\int_{-T}^{T}\displaystyle\int_{-T}^{T} X(t)\overline{X(s)}\, e^{-i\omega(t-s)}\, dt\, ds\right] \\ &= \displaystyle\int_{-T}^{T}\displaystyle\int_{-T}^{T} E[X(t)\overline{X(s)}]\, e^{-i\omega(t-s)}\, dt\, ds \\ &= \displaystyle\int_{-T}^{T}\displaystyle\int_{-T}^{T} R(t-s)\, e^{-i\omega(t-s)}\, dt\, ds \end{align}

Step 2: Apply the change of variables $\tau = t - s$.

$$E[|X_T(\omega)|^2] = \displaystyle\int_{-2T}^{2T} R(\tau)\, e^{-i\omega\tau}\left(2T - |\tau|\right) d\tau$$

Step 3: Divide both sides by $2T$ and take the limit $T \to \infty$.

$$\lim_{T\to\infty} \dfrac{E[|X_T(\omega)|^2]}{2T} = \displaystyle\int_{-\infty}^{\infty} R(\tau)\, e^{-i\omega\tau}\, d\tau$$

The left-hand side is exactly the definition of the power spectral density $S(\omega)$:

$$S(\omega) = \lim_{T\to\infty} \dfrac{E[|X_T(\omega)|^2]}{2T}$$

Therefore $S(\omega) = \mathcal{F}\{R(\tau)\}$ has been shown. The inverse transform is obtained directly from the Fourier inversion formula. $\blacksquare$

$R(\tau)$ $S(\omega)$

Figure 2: Conceptual diagram of the Wiener-Khinchin theorem. The time-domain autocorrelation function $R(\tau)$ and the frequency-domain power spectral density $S(\omega)$ are linked by the Fourier transform and its inverse.

4. Discrete version

The same relationship holds for a discrete-time wide-sense stationary process $\{X[n]\}$.

4.1 Discrete-time autocorrelation function

$$R[m] = E[X[n+m]\overline{X[n]}]$$

4.2 Discrete Wiener-Khinchin theorem

Theorem (discrete version)

The power spectral density of a discrete-time wide-sense stationary process is given by the discrete-time Fourier transform (DTFT) of the autocorrelation sequence:

$$S(e^{j\omega}) = \displaystyle\sum_{m=-\infty}^{\infty} R[m]\, e^{-j\omega m}$$

The inverse transform is:

$$R[m] = \dfrac{1}{2\pi}\displaystyle\int_{-\pi}^{\pi} S(e^{j\omega})\, e^{j\omega m}\, d\omega$$

Here $S(e^{j\omega})$ is a function of period $2\pi$, and $S(e^{j\omega}) \geq 0$ holds. Setting $m = 0$ gives:

$$R[0] = E[|X[n]|^2] = \dfrac{1}{2\pi}\displaystyle\int_{-\pi}^{\pi} S(e^{j\omega})\, d\omega$$

This means that the average power of the discrete signal equals the integral of the spectral density over the whole band.

5. Relationship to the periodogram

A naive method for estimating the power spectral density from finite-length data $X[0], X[1], \ldots, X[N-1]$ is the periodogram (periodogram).

5.1 Definition

$$\hat{S}_N(\omega) = \dfrac{1}{N}\left|\displaystyle\sum_{n=0}^{N-1} X[n]\, e^{-j\omega n}\right|^2$$

5.2 Relationship to the Wiener-Khinchin theorem

The periodogram can be expressed as the DTFT of the sample autocorrelation function:

$$\hat{S}_N(\omega) = \displaystyle\sum_{m=-(N-1)}^{N-1} \hat{R}[m]\, e^{-j\omega m}$$

Here $\hat{R}[m] = \dfrac{1}{N}\displaystyle\sum_{n=0}^{N-1-|m|} X[n+|m|]\overline{X[n]}$ is the biased sample autocorrelation.

5.3 Properties and issues

Bias: $E[\hat{S}_N(\omega)]$ equals $S(\omega)$ convolved with the Fejer kernel, so it is not strictly unbiased but asymptotically unbiased ($E[\hat{S}_N(\omega)] \to S(\omega)$ as $N \to \infty$).
Inconsistency: $\text{Var}[\hat{S}_N(\omega)]$ does not converge to 0 even as $N \to \infty$. The periodogram is not a consistent estimator.
Improved methods: The variance can be reduced by Bartlett's method (segment averaging), Welch's method (overlapping segments with a window function), and the Blackman-Tukey method (applying a window function to the autocorrelation).

6. Applications

6.1 Spectral estimation

The Wiener-Khinchin theorem is the theoretical foundation of frequency analysis of time-series data. The method of estimating the autocorrelation function from real data and taking its Fourier transform to obtain the spectrum (the Blackman-Tukey method) is based directly on this theorem. Parametric spectral estimation using AR models and ARMA models also rests on the correspondence between autocorrelation and spectrum.

6.2 Signal processing

The Wiener filter is designed on the premise of this theorem. When the power spectral densities of the signal and the noise are known, the transfer function of the optimal linear filter that minimizes the mean squared error is expressed in the frequency domain as a ratio of spectral densities. In communication engineering it is also used in the design of matched filters and in the computation of the signal-to-noise ratio.

6.3 Optics (coherence)

In optics, the temporal coherence of a light source is described by the Wiener-Khinchin theorem. The Fourier transform of the autocorrelation function $\Gamma(\tau) = \langle E(t+\tau)E^*(t)\rangle$ of the optical field $E(t)$ gives the spectral distribution $I(\omega)$ of the source:

$$I(\omega) = \displaystyle\int_{-\infty}^{\infty} \Gamma(\tau)\, e^{-i\omega\tau}\, d\tau$$

Fourier-transform infrared spectroscopy (FTIR) measures the interference fringes (the interferogram) with a Michelson interferometer and recovers the spectrum from their Fourier transform, which is precisely a direct application of the Wiener-Khinchin theorem.

FAQ

Q1. To what kinds of signals does the Wiener-Khinchin theorem apply?

It holds for wide-sense stationary processes. The target is a stochastic process whose mean is constant and whose autocorrelation function depends only on the time lag. The same correspondence also holds for deterministic signals provided their autocorrelation function is absolutely integrable (in that case a time average is used instead of the stochastic expectation).

Q2. What is the relationship between the periodogram and the power spectral density?

The periodogram is a method for estimating the power spectral density from finite-length data, and it equals the DTFT of the sample autocorrelation function. As $N \to \infty$ its expectation converges to the true $S(\omega)$, but its variance does not decrease, so it is not a consistent estimator. In practice, improved methods such as Welch's method and the Blackman-Tukey method are needed.

Q3. How is the Wiener-Khinchin theorem used in optics?

It is used to analyze the temporal coherence of a light source. The interferogram (the intensity pattern of the interference fringes) measured with a Michelson interferometer corresponds to the autocorrelation function of the optical field, and its Fourier transform gives the spectral distribution of the source. Fourier-transform infrared spectroscopy (FTIR) is a measurement technique based directly on this principle.

References

A. Papoulis, S. U. Pillai, Probability, Random Variables and Stochastic Processes, 4th ed., McGraw-Hill, 2002. — Bibliographic information (Google Books)
S. Haykin, Adaptive Filter Theory, 5th ed., Pearson, 2014. — Publisher page
Wikipedia, "Wiener-Khinchin theorem"