Proofs of Matrix Calculus Formulas

Proofs of Vector/Matrix Calculus Identities

About This Collection

This page compiles detailed proofs of the formulas listed in the Vector/Matrix Calculus Formula Sheet. The proofs are organized by chapter; use the table of contents below to navigate to each chapter.

Purpose of This Collection

This collection is a research-level supplementary resource aimed at rigorously deriving matrix calculus identities under the denominator layout convention. Each proof has been cross-checked against standard references such as Magnus & Neudecker and the Matrix Cookbook, but may contain original derivations and reorganizations by the author. If you find any errors, please contact us.

Notation and Layout Convention

Throughout this collection, we consistently adopt the denominator layout convention, in which the shape of the variable in the denominator determines the row direction of the result.

Notation Meaning Result Shape
$\dfrac{\partial y}{\partial \mathbf{x}}$ Differentiate scalar $y$ by vector $\mathbf{x} \in \mathbb{R}^n$ $n \times 1$ column vector
$\dfrac{\partial \mathbf{y}}{\partial \mathbf{x}}$ Differentiate vector $\mathbf{y} \in \mathbb{R}^m$ by vector $\mathbf{x} \in \mathbb{R}^n$ $n \times m$ Jacobian matrix
$\dfrac{\partial y}{\partial X}$ Differentiate scalar $y$ by matrix $X \in \mathbb{R}^{m \times n}$ $m \times n$ matrix

The denominator layout is widely used in statistics and machine learning. It is the transpose of the numerator layout used in some engineering fields.

Chapter Dependencies

The diagram below illustrates the logical dependencies among chapters. Arrows indicate prerequisites.

Part I: Foundations Ch.1 Ch.2 Ch.3 Ch.4 Part II: Operations Ch.5 Ch.6 Part III: Functions Ch.7 Ch.8 Part IV: Spectral Ch.9 Ch.10 Part V Advanced Part V-bis Lie Groups Part VI Computation Part VII Applications

Recommended study order: Part I → Part II → Part III → Part IV, then consult Parts V, V-bis, VI, and VII as needed. Part V-bis requires the matrix exponential from Part V.

Table of Contents

Part I: Differential Foundations

Starting from the definition of differentiation, establishing the basics of scalar, vector, and matrix differentiation.

Part II: Matrix Operations

Differentiation of trace, element-wise operations, activation functions, and other operations common in machine learning.

Part III: Matrix Functions

Differentiation of fundamental matrix functions from linear algebra: determinants, inverses, etc.

Part IV: Spectral Theory

Differentiation of eigenvalues, eigenvectors, and analysis of quadratic forms.

Part V: Advanced Topics

More advanced formulas for matrix powers, norms, and structured matrices.

Part V-bis: Differentiation on Lie Groups

Differentiation on Lie groups and manifolds. Essential techniques for robotics and computer vision.

Prerequisites

For the basics of Lie groups and Lie algebras (definitions, exponential map, SO(3), SE(3)), see Lie Algebra. In particular, Intro Ch.6 and Advanced Ch.7 (Rigid Body Transformations) are relevant.

Part VI: Computational Techniques

Matrix chain rule, special matrices, complex matrix differentiation. Connection to automatic differentiation.

Part VII: Applications

Application formulas are classified by mathematical topic. Each category page provides detailed formulas and proofs.

References

Internal Links

Textbooks and Academic References

  • Magnus, J. R., & Neudecker, H. (2019). Matrix Differential Calculus with Applications in Statistics and Econometrics (3rd ed.). Wiley.
  • Absil, P.-A., Mahony, R., & Sepulchre, R. (2008). Optimization Algorithms on Matrix Manifolds. Princeton University Press.
  • Petersen, K. B., & Pedersen, M. S. (2012). The Matrix Cookbook. Technical University of Denmark.

Online Resources