Probability — Intermediate

Measure-Theoretic Probability (Third- to Fourth-Year University Level)

Overview of the Intermediate level

The Intermediate level covers modern probability theory based on measure theory. It rebuilds the "discrete" probability of the earlier levels within a more general framework.

Learning goals

  • Understand the axiomatic foundations of probability
  • Master the concepts of σ-algebras and measures
  • Handle general random variables and their expectations
  • Understand continuous distributions and density functions
  • Distinguish the modes of convergence of random variables
  • Be able to prove the law of large numbers
  • Understand the principle and asymptotic properties of maximum likelihood estimation

Contents

  1. Chapter 1 Axiomatization of Probability

    Kolmogorov's axioms and the rigorous definition of a probability space $(\Omega, \mathcal{F}, P)$

  2. Chapter 2 σ-Algebras and Measures

    Definition of a σ-algebra, Borel sets $\mathcal{B}(\mathbb{R})$, extension of measures

  3. Chapter 3 Generalizing Random Variables

    Measurable functions, distribution functions, transformations of random variables

  4. Chapter 4 Continuous Distributions

    Density functions, the normal distribution $N(\mu, \sigma^2)$, exponential distribution, gamma distribution

  5. Chapter 5 Redefining Expectation

    Expectation via the Lebesgue integral, the monotone convergence theorem, the dominated convergence theorem

  6. Chapter 6 Modes of Convergence

    Almost-sure convergence, convergence in probability, $L^p$ convergence, convergence in distribution and their relationships

  7. Chapter 7 Law of Large Numbers

    The weak law, the strong law, the Borel-Cantelli lemma

  8. Chapter 8 Statistical Inference

    Likelihood function, maximum likelihood estimation, consistency, asymptotic normality

By topic

  • Probability Distribution

    Definition and properties of discrete and continuous probability distributions

  • Cross-Entropy

    Definition of cross-entropy, relation to the KL divergence, loss functions in machine learning

  • Shannon Entropy

    Definition and basic properties of entropy, mutual information, KL divergence, the source coding theorem

  • Monte Carlo Method

    Basic principle of Monte Carlo methods, variance reduction, MCMC (Metropolis-Hastings, Gibbs sampling), convergence diagnostics

  • Random Walk

    Definition and basic properties of random walks, martingale property, the ruin problem, Pólya's recurrence theorem, relation to Brownian motion

  • Cauchy Distribution

    Definition and properties of the Cauchy distribution, non-existence of the mean and variance, characteristic function, stability, failure of the law of large numbers, the Lorentzian

  • Law of Large Numbers (Proof and Applications)

    The difference between the weak and strong laws, proof via Chebyshev's inequality, applications to Monte Carlo methods, insurance, and opinion polls

  • Cumulants

    Cumulant generating function, relation to moments, additivity, characterization of the normal distribution, Edgeworth expansion, application to ICA

  • Moment Generating Function

    Definition of the MGF $M_X(t)=E[e^{tX}]$, generating moments, the uniqueness theorem, a table of MGFs for major distributions, application to proving the central limit theorem

  • Mutual Information

    Definition of mutual information, relation to entropy, relation to the KL divergence, basic properties, conditional mutual information and the chain rule, applications

  • Multivariate Normal Distribution

    Definition (PDF), covariance and correlation matrices, marginal and conditional distributions, Mahalanobis distance, closure under affine transformations, applications (PCA, Gaussian processes, portfolio theory)

  • Order Statistics

    The distribution of statistics obtained by sorting a sample. Order statistics of the uniform distribution and the beta distribution.

  • Total Variation Distance

    A distance between probability measures. Coupling representation and Pinsker's inequality.

  • Copula

    Separation of marginal distributions and dependence structure via Sklar's theorem. Tail dependence and risk management.

  • Pinsker's Inequality

    The inequality linking total variation distance and the KL divergence, and its best constant.

  • Probability Generating Function — Definition, radius of convergence, moment extraction, and independence.
  • Hellinger Distance — Definition, inequality relations with the total variation distance and KL divergence, and statistical hypothesis testing.
  • Wasserstein Distance — Definition, relation to optimal transport, and Kantorovich-Rubinstein duality.
  • Kolmogorov's Zero-One Law — The statement, definition of tail events, proof, and independence.
  • Chernoff Bound — Definition, derivation via the exponential-moment method, application to the binomial distribution, and algorithms.
  • Hoeffding's Inequality — Definition, Hoeffding's lemma, proof, and relation to Azuma's inequality.
  • Branching Process — Definition, analysis via generating functions, extinction probability, mean growth and criticality, Galton-Watson process.
  • Renewal Process — Definition, renewal equation, renewal (limit) theorems, age and residual life.
  • Little's Law — Definition of Little's Law $L=\lambda W$, proof idea, and the relation between arrival rate, mean number in system, and mean sojourn time.
  • Cycle Lemma — The cycle lemma and the Dvoretzky-Motzkin theorem, proof of the ballot problem, relation to Catalan numbers, and lattice paths.
  • Reflection Principle — Definition, application to random walks, and the maximum distribution of Brownian motion.
  • Detailed Balance — Definition of the detailed balance condition, reversible Markov chains, finding stationary distributions, and MCMC.
  • Coupling Method — Definition, construction of basic couplings, relation to the total variation distance, and mixing time of Markov chains.

Distribution Reference

  1. Beta Distribution

    Definition, relation to the beta function, mean, variance, mode, and special cases.

  2. Gamma Distribution

    Definition, meaning of the shape and rate parameters, mean, and variance.

  3. Log-Normal Distribution

    Definition, probability density function, mean and variance, and the geometric mean.

  4. Weibull Distribution

    Definition, shape and scale parameters, mean and variance, and reliability applications.

  5. Pareto Distribution

    Definition, probability density function, mean and variance, and the Pareto principle (80/20).

  6. Laplace Distribution

    Definition, probability density function, mean and variance, and the moment generating function.

  7. Rayleigh Distribution

    Probability density and cumulative distribution functions, mean and variance, and the two-dimensional Gaussian magnitude.

  8. Gumbel Distribution

    The Type I extreme value distribution: definition, PDF and CDF, mean, and applications.

  9. Erlang Distribution

    A continuous distribution expressed as the sum of $k$ exponential distributions: definition, PDF, and applications.

  10. Half-Normal Distribution

    Definition, probability density function, mean and variance, and relation to the normal distribution.

  11. Inverse Gaussian Distribution

    Definition, PDF, mean, variance, and applications.

  12. Noncentral Chi-Squared Distribution

    Definition, PDF, mean, variance, and applications.

  13. Logistic Distribution

    PDF and CDF (the logistic function), mean, and variance.

  14. Dirichlet Distribution

    Definition, probability density function, mean and variance, and conjugacy with the multinomial distribution.

  15. Von Mises Distribution

    A probability distribution on the circle: definition, PDF, and mean.

  16. Maxwell Distribution

    Probability density function, mean, mode, root-mean-square, and relation to the chi distribution.

  17. Benford's Law

    The leading digit of natural data is not uniform but follows $\log_{10}(1+1/d)$.

  18. Zipf's Law

    A power law in which word frequency is inversely proportional to rank: definition, PMF, mean, and applications.

  19. Fréchet Distribution

    Definition, PDF, mean, variance, and applications.

  20. Beta-Binomial Distribution

    Definition, probability mass function, mean and variance, and overdispersion.

  21. Skew Normal Distribution

    Definition, probability density function, mean and variance, skewness, and applications.

  22. Log-Logistic Distribution

    Definition, PDF, CDF, mean, and variance.

  23. Generalized Gamma Distribution

    Definition, PDF, mean, variance, and special cases.

  24. Generalized Pareto Distribution

    Definition, PDF, mean, variance, and peaks-over-threshold modeling.

Reading

  • Making Chance Rigorous [Reading]

    For two centuries probability rested on shaky foundations. From Hilbert's sixth problem to Kolmogorov's axioms—the one-line story that "probability is a measure."

  • Why the Bell Curve Is Everywhere [Reading]

    Heights, errors, and test scores all settle into the same bell shape despite their different origins. The story of the central limit theorem, where the sum of small fluctuations gives birth to the normal distribution.

  • When the Average Lies [Reading]

    The expected value becomes infinite, fails to exist, or departs from reality. From the St. Petersburg paradox and heavy-tailed distributions, a look at the danger of summarizing with a single number.

Prerequisites

  • The contents of Probability — Beginner
  • Basics of real analysis (suprema and infima, limits)
  • Basics of Lebesgue integration (helpful)