Proofs of Matrix Calculus Formulas

Proofs of Vector/Matrix Calculus Identities

About This Collection

This page compiles detailed proofs of the formulas listed in the Vector/Matrix Calculus Formula Sheet. The proofs are organized by chapter; use the table of contents below to navigate to each chapter.

Purpose of This Collection

This collection is an advanced companion reference aimed at rigorously deriving matrix calculus identities under the denominator layout convention. Each formula has been symbolically verified using computer algebra systems or proof-checking systems wherever possible; for the few formulas where symbolic verification was impractical, we have confirmed the formula's validity numerically using multiple random matrices. Every effort has been made to ensure accuracy, but if you find any errors, please contact us.

Notation and Layout Convention

Throughout this collection, we consistently adopt the denominator layout convention, in which the shape of the variable in the denominator determines the row direction of the result; in components, the $(i, j)$ entry of $\dfrac{\partial \mathbf{y}}{\partial \mathbf{x}}$ is $\dfrac{\partial y_j}{\partial x_i}$.

Notation Meaning Result Shape
$\dfrac{\partial y}{\partial \mathbf{x}}$ Differentiate scalar $y$ by vector $\mathbf{x} \in \mathbb{R}^n$ $n \times 1$ column vector
$\dfrac{\partial \mathbf{y}}{\partial \mathbf{x}}$ Differentiate vector $\mathbf{y} \in \mathbb{R}^m$ by vector $\mathbf{x} \in \mathbb{R}^n$ $n \times m$ matrix (the transpose of the usual $m \times n$ Jacobian)
$\dfrac{\partial y}{\partial X}$ Differentiate scalar $y$ by matrix $X \in \mathbb{R}^{m \times n}$ $m \times n$ matrix

The denominator layout is common in statistics and econometrics, while the numerator layout is also widely used in machine learning and engineering; check the convention of each reference. For derivatives involving vectors the two are transposes of each other, but for matrix-by-matrix derivatives the relation is not a simple transpose, and the arrangement must be made explicit, for example by vectorization.

Chapter Dependencies

The diagram below shows the recommended study order (by Part). The prerequisites of each chapter are listed at the beginning of that chapter (for example, Chapter 7 needs only Chapters 4 and 5, and Chapters 12–14 can be read with Chapters 2–5 alone).

Part I: Foundations Ch.1 Ch.2 Ch.3 Ch.4 Part II: Operations Ch.5 Ch.6 Part III: Functions Ch.7 Ch.8 Part IV: Spectral Ch.9 Ch.10 Part V Advanced Part V-bis Lie Groups Part VI Computation Part VII Applications

Recommended study order: Part I → Part II → Part III → Part IV, then consult Parts V, V-bis, VI, and VII as needed. Part V-bis requires the matrix exponential from Part V.

Table of Contents

Part I: Differential Foundations

Starting from the definition of differentiation, establishing the basics of scalar, vector, and matrix differentiation.

Part II: Matrix Operations

Differentiation of trace, element-wise operations, activation functions, and other operations common in machine learning.

Part III: Matrix Functions

Differentiation of fundamental matrix functions from linear algebra: determinants, inverses, etc.

Part IV: Spectral Theory

Differentiation of eigenvalues, eigenvectors, and analysis of quadratic forms.

Part V: Advanced Topics

More advanced formulas for matrix powers, norms, and structured matrices.

Part V-bis: Differentiation on Lie Groups

Differentiation on Lie groups and manifolds. Important techniques for pose estimation, SLAM, 3D vision, and related areas.

Prerequisites

For the basics of Lie groups and Lie algebras (definitions, exponential map, SO(3), SE(3)), see Lie Algebra. In particular, Intro Ch.6 and Advanced Ch.7 (Rigid Body Transformations) are relevant.

Part VI: Computational Techniques

Matrix chain rule, special matrices, complex matrix differentiation. Connection to automatic differentiation.

Part VII: Applications

Application formulas are classified by mathematical topic. Each category page provides detailed formulas and proofs.

Frequently asked questions

What is the denominator layout convention in matrix calculus?

It is the convention in which the $(i, j)$ entry of $\dfrac{\partial \mathbf{y}}{\partial \mathbf{x}}$ is $\dfrac{\partial y_j}{\partial x_i}$ (the variables in the denominator run along the rows). Differentiating a scalar $y$ by a vector $\mathbf{x} \in \mathbb{R}^n$ gives an $n \times 1$ column vector, and differentiating a vector $\mathbf{y} \in \mathbb{R}^m$ by $\mathbf{x}$ gives an $n \times m$ matrix (the transpose of the usual Jacobian). The numerator layout (its transpose) is also widely used, so check the convention of each reference.

What is the derivative of the determinant?

$\det A$ is differentiable for every square matrix $A$, and if $A$ is nonsingular, $\dfrac{\partial \det A}{\partial A} = \det(A)\, A^{-\top}$ (Jacobi's formula). For the log-determinant, $\dfrac{\partial \log\det A}{\partial A} = A^{-\top}$ when $\det A > 0$; for a general nonsingular matrix, the derivative of $\log|\det A|$ gives the same expression.

How is this proof collection organized?

It has 16 main chapters plus 3 Lie group chapters and 4 application chapters. The recommended study order is Part I (Differential Foundations) → Part II (Matrix Operations) → Part III (Matrix Functions) → Part IV (Spectral Theory), with advanced topics and applications from Part V onward. The prerequisites of each chapter are listed at its beginning.

References

Internal Links

Textbooks and Academic References

  • Magnus, J. R., & Neudecker, H. (2019). Matrix Differential Calculus with Applications in Statistics and Econometrics (3rd ed.). Wiley.
  • Absil, P.-A., Mahony, R., & Sepulchre, R. (2008). Optimization Algorithms on Matrix Manifolds. Princeton University Press.
  • Petersen, K. B., & Pedersen, M. S. (2012). The Matrix Cookbook. Technical University of Denmark.

Online Resources