Wiener Filter

Frequency-Domain and Time-Domain Approaches with Wirtinger Derivative

What Is the Wiener Filter?

Among linear filters, the Wiener filter optimally estimates the original signal from noisy observations in the minimum mean square error (MMSE) sense (when the signal and noise are Gaussian, it is optimal even among nonlinear estimators). Developed by Norbert Wiener in the 1940s, it is widely used in signal processing, image processing, and communications.

X(ω) Original signal H(ω) Degradation filter + N(ω) Y(ω) Degraded signal G(ω) Wiener filter
Figure 1: Signal degradation model and estimation by the Wiener filter

Solution of the Wiener Filter

Frequency domain:

$$G(\omega) = \frac{H^*(\omega) P_S(\omega)}{|H(\omega)|^2 P_S(\omega) + P_N(\omega)}$$

Time domain (solution of the Wiener-Hopf equation $\boldsymbol{R}\boldsymbol{w}=\boldsymbol{p}$, when $\boldsymbol{R}$ is nonsingular):

$$\boldsymbol{w}_{\rm opt} = \boldsymbol{R}^{-1}\boldsymbol{p}$$

Here $P_S(\omega)$ and $P_N(\omega)$ are the power spectra of the original signal $X$ and the noise $N$, $\boldsymbol{w}$ is the vector of FIR filter coefficients, $\boldsymbol{R}=E[\boldsymbol{x}\boldsymbol{x}^H]$ is the autocorrelation matrix of the observation vector $\boldsymbol{x}$, and $\boldsymbol{p}=E[\boldsymbol{x}d^*]$ is the cross-correlation vector between the observation and the desired signal $d$.

Both assume that the signal and the noise are uncorrelated and wide-sense stationary, with known power spectra (correlations). $G(\omega)$ is the solution without a causality constraint (two-sided); the solution under a causality constraint is treated in Chapter 7.

Table of Contents

This series derives the Wiener filter through multiple approaches. Parts 1 and 2 arrive at the same result by different methods under the same assumptions, which deepens understanding of the underlying theory; Parts 3 and 4 cover extensions that change the assumptions, such as causality, prediction, and multiple channels.

Part 1: Fundamental Derivations

  1. Chapter 1 Frequency-Domain Derivation

    Three approaches: orthogonality principle, completing the square, and Wirtinger derivative

  2. Chapter 2 Time-Domain Derivation

    Derivation of the Wiener-Hopf equation via the orthogonality principle and Wirtinger derivative

  3. Chapter 3 Relationship Between Frequency and Time Domains

    Correspondence between the two domains and their essential equivalence

Part 2: Advanced Derivation Methods

  1. Chapter 4 Advanced Derivation Methods

    Real/imaginary decomposition, variational method, projection theorem, MAP estimation under Gaussian assumptions, eigenvalue decomposition (spectral decomposition), and LMS convergence point

  1. Chapter 5 Practical Considerations

    Non-causality, regularization, numerical stability, Levinson-Durbin algorithm, and spectral estimation

  2. Chapter 6 Related Theory

    Relationship with the Kalman filter, connections to machine learning, and comparison of derivation methods

Part 4: Advanced Topics

  1. Chapter 7 Causal Wiener Filter

    Spectral factorization, Wiener-Hopf decomposition, and real-time processing

  2. Chapter 8 Smoothing, Filtering, and Prediction

    Wiener's three classical problems and performance comparison

  3. Chapter 9 Multichannel Wiener Filter

    MIMO extension, microphone arrays, and beamforming

  4. Chapter 10 Applications

    Echo cancellation, channel equalization, speech enhancement, image restoration, and Python implementation

Appendix

  1. Appendix Supplementary Materials

    Wirtinger derivative formulas, optimality for Gaussian signals, references, and trivia

Why Study Multiple Derivation Approaches

  1. Multifaceted understanding: Arriving at the same conclusion through different approaches provides deeper insight into the essence of the theory
  2. Improved applicability: Enables choosing the most suitable derivation method depending on the nature of the problem
  3. Mathematical breadth: Offers exposure to various mathematical techniques including functional analysis, probability theory, and optimization theory
  4. Historical perspective: Provides understanding of how derivation methods have evolved alongside the development of the field

The main approaches are as follows:

  1. Frequency-domain derivation: completing the square, orthogonality principle, Wirtinger derivative
  2. Time-domain derivation: orthogonality principle, Wirtinger derivative
  3. Other derivation methods: real/imaginary decomposition, variational method, projection theorem, MAP estimation under Gaussian assumptions, eigenvalue decomposition (spectral decomposition), LMS convergence point

These are different representations of the same theory and, under their respective assumptions (Gaussianity for MAP estimation, a sufficiently small step size for LMS), ultimately yield the equivalent result (the Wiener-Hopf equation).

Frequently Asked Questions

Q1. What is a Wiener filter?

Among linear filters, the Wiener filter is the one that minimizes the mean square error in estimating the original signal from noisy observations (optimal in the MMSE sense among linear estimators). When the signal and noise are uncorrelated and wide-sense stationary with known power spectra, it is given in the frequency domain by $G(\omega) = \dfrac{H^*(\omega)P_S(\omega)}{|H(\omega)|^2P_S(\omega)+P_N(\omega)}$ (the solution without a causality constraint) and in the time domain by the solution $\boldsymbol{w}_{\rm opt}=\boldsymbol{R}^{-1}\boldsymbol{p}$ of the Wiener-Hopf equation $\boldsymbol{R}\boldsymbol{w}=\boldsymbol{p}$ (when $\boldsymbol{R}$ is nonsingular). When the signal and noise are Gaussian, it is optimal even among nonlinear estimators.

Q2. What is the relationship between Wirtinger derivatives and the Wiener filter?

Wirtinger derivatives allow differentiation of functions of complex variables without splitting them into real and imaginary parts, making them convenient for MMSE optimization of complex signals. For the mean square error $J(\boldsymbol{w}) = E[|d - \boldsymbol{w}^H\boldsymbol{x}|^2]$ (where $d$ is the desired signal and $\boldsymbol{x}$ the observation vector), simply setting $\partial J/\partial\boldsymbol{w}^* = \boldsymbol{R}\boldsymbol{w}-\boldsymbol{p} = 0$ yields the Wiener-Hopf equation, giving the same result as the orthogonality principle or completing the square, but more concisely.