Proofs Chapter 5: Trace Derivatives

Proof

We recall the definition of trace. The trace is the sum of the diagonal elements of a square matrix.

\begin{equation} \text{tr}(\boldsymbol{X}) = \displaystyle\sum_{i=0}^{N-1} X_{ii} \label{eq:5-1-1} \end{equation}

Differentiating this scalar with respect to the $(j, l)$ entry $X_{jl}$ of $\boldsymbol{X}$:

\begin{equation} \dfrac{\partial}{\partial X_{jl}} \text{tr}(\boldsymbol{X}) = \dfrac{\partial}{\partial X_{jl}} \displaystyle\sum_{i=0}^{N-1} X_{ii} = \displaystyle\sum_{i=0}^{N-1} \dfrac{\partial X_{ii}}{\partial X_{jl}} \label{eq:5-1-2} \end{equation}

$X_{ii}$ and $X_{jl}$ are the same variable only when $i = j$ and $i = l$, i.e., $j = l$. Using the Kronecker delta:

\begin{equation} \dfrac{\partial X_{ii}}{\partial X_{jl}} = \delta_{ij} \delta_{il} \label{eq:5-1-3} \end{equation}

Substituting \eqref{eq:5-1-3} into \eqref{eq:5-1-2} and summing over $i$. Since $\delta_{ij} = 1$ only when $i = j$:

\begin{equation} \displaystyle\sum_{i=0}^{N-1} \delta_{ij} \delta_{il} = \delta_{jl} \label{eq:5-1-4} \end{equation}

Since $\delta_{jl}$ is the $(j, l)$ entry of the identity matrix $\boldsymbol{I}$:

\begin{equation} \delta_{jl} = I_{jl} \label{eq:5-1-5} \end{equation}

Since \eqref{eq:5-1-5} holds for all $(j, l)$, we obtain the final result in matrix form.

\begin{equation} \dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{X}) = \boldsymbol{I} \label{eq:5-1-6} \end{equation}

Proof

Writing out the $(i, i)$ entry (diagonal entry) of the matrix product $\boldsymbol{A}\boldsymbol{X}$:

\begin{equation} (\boldsymbol{A}\boldsymbol{X})_{ii} = \displaystyle\sum_{k=0}^{N-1} A_{ik} X_{ki} \label{eq:5-2-1} \end{equation}

Since the trace is the sum of diagonal elements:

\begin{equation} \text{tr}(\boldsymbol{A}\boldsymbol{X}) = \displaystyle\sum_{i=0}^{M-1} (\boldsymbol{A}\boldsymbol{X})_{ii} = \displaystyle\sum_{i=0}^{M-1} \displaystyle\sum_{k=0}^{N-1} A_{ik} X_{ki} \label{eq:5-2-2} \end{equation}

Differentiating this scalar with respect to the $(j, l)$ entry $X_{jl}$ of $\boldsymbol{X}$. Since $A_{ik}$ is a constant:

\begin{equation} \dfrac{\partial}{\partial X_{jl}} \text{tr}(\boldsymbol{A}\boldsymbol{X}) = \displaystyle\sum_{i=0}^{M-1} \displaystyle\sum_{k=0}^{N-1} A_{ik} \dfrac{\partial X_{ki}}{\partial X_{jl}} \label{eq:5-2-3} \end{equation}

$X_{ki}$ and $X_{jl}$ are the same variable only when $(k, i) = (j, l)$. Using the Kronecker delta:

\begin{equation} \dfrac{\partial X_{ki}}{\partial X_{jl}} = \delta_{kj} \delta_{il} \label{eq:5-2-4} \end{equation}

Substituting \eqref{eq:5-2-4} into \eqref{eq:5-2-3}:

\begin{equation} \dfrac{\partial}{\partial X_{jl}} \text{tr}(\boldsymbol{A}\boldsymbol{X}) = \displaystyle\sum_{i=0}^{M-1} \displaystyle\sum_{k=0}^{N-1} A_{ik} \delta_{kj} \delta_{il} \label{eq:5-2-5} \end{equation}

Since $\delta_{kj} = 1$ only when $k = j$, we get $\displaystyle\sum_{k=0}^{N-1} A_{ik} \delta_{kj} = A_{ij}$. Similarly, since $\delta_{il} = 1$ only when $i = l$:

\begin{equation} \displaystyle\sum_{i=0}^{M-1} A_{ij} \delta_{il} = A_{lj} \label{eq:5-2-6} \end{equation}

$A_{lj}$ is the $(j, l)$ entry of the transpose $\boldsymbol{A}^\top$:

\begin{equation} A_{lj} = (\boldsymbol{A}^\top)_{jl} \label{eq:5-2-7} \end{equation}

Since \eqref{eq:5-2-7} holds for all $(j, l)$, we obtain the final result in matrix form.

\begin{equation} \dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{A}\boldsymbol{X}) = \boldsymbol{A}^\top \label{eq:5-2-8} \end{equation}

Proof

Method 1: Using the cyclic property of trace

By the cyclic property of trace, for matrices $\boldsymbol{P}, \boldsymbol{Q}$ such that both $\boldsymbol{P}\boldsymbol{Q}$ and $\boldsymbol{Q}\boldsymbol{P}$ are defined, we have $\text{tr}(\boldsymbol{P}\boldsymbol{Q}) = \text{tr}(\boldsymbol{Q}\boldsymbol{P})$. Applying this to $\boldsymbol{X}\boldsymbol{A}$:

\begin{equation} \text{tr}(\boldsymbol{X}\boldsymbol{A}) = \text{tr}(\boldsymbol{A}\boldsymbol{X}) \label{eq:5-3-1} \end{equation}

Applying the result of Formula 5.2:

\begin{equation} \dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{X}\boldsymbol{A}) = \dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{A}\boldsymbol{X}) = \boldsymbol{A}^\top \label{eq:5-3-2} \end{equation}

Method 2: Direct computation

Writing out the $(i, i)$ entry of the matrix product $\boldsymbol{X}\boldsymbol{A}$:

\begin{equation} (\boldsymbol{X}\boldsymbol{A})_{ii} = \displaystyle\sum_{k=0}^{M-1} X_{ik} A_{ki} \label{eq:5-3-3} \end{equation}

Since the trace is the sum of diagonal elements:

\begin{equation} \text{tr}(\boldsymbol{X}\boldsymbol{A}) = \displaystyle\sum_{i=0}^{N-1} \displaystyle\sum_{k=0}^{M-1} X_{ik} A_{ki} \label{eq:5-3-4} \end{equation}

Differentiating with respect to $X_{jl}$ and substituting $\displaystyle\dfrac{\partial X_{ik}}{\partial X_{jl}} = \delta_{ij} \delta_{kl}$, then summing:

\begin{equation} \displaystyle\sum_{i=0}^{N-1} \displaystyle\sum_{k=0}^{M-1} \delta_{ij} \delta_{kl} A_{ki} = A_{lj} = (\boldsymbol{A}^\top)_{jl} \label{eq:5-3-5} \end{equation}

Proof

We verify the entries of the transpose. The $(k, i)$ entry of $\boldsymbol{X}^\top$ equals the $(i, k)$ entry of $\boldsymbol{X}$.

\begin{equation} (\boldsymbol{X}^\top)_{ki} = X_{ik} \label{eq:5-4-1} \end{equation}

Writing out the $(i, i)$ entry of the matrix product $\boldsymbol{A}\boldsymbol{X}^\top$:

\begin{equation} (\boldsymbol{A}\boldsymbol{X}^\top)_{ii} = \displaystyle\sum_{k=0}^{N-1} A_{ik} (\boldsymbol{X}^\top)_{ki} = \displaystyle\sum_{k=0}^{N-1} A_{ik} X_{ik} \label{eq:5-4-2} \end{equation}

Since the trace is the sum of diagonal elements:

\begin{equation} \text{tr}(\boldsymbol{A}\boldsymbol{X}^\top) = \displaystyle\sum_{i=0}^{M-1} \displaystyle\sum_{k=0}^{N-1} A_{ik} X_{ik} \label{eq:5-4-3} \end{equation}

Differentiating this expression with respect to $X_{jl}$. Since $A_{ik}$ is a constant:

\begin{equation} \dfrac{\partial}{\partial X_{jl}} \text{tr}(\boldsymbol{A}\boldsymbol{X}^\top) = \displaystyle\sum_{i=0}^{M-1} \displaystyle\sum_{k=0}^{N-1} A_{ik} \dfrac{\partial X_{ik}}{\partial X_{jl}} \label{eq:5-4-4} \end{equation}

Since $\displaystyle\dfrac{\partial X_{ik}}{\partial X_{jl}} = \delta_{ij} \delta_{kl}$ (equals 1 only when $(i, k) = (j, l)$), substituting:

\begin{equation} \dfrac{\partial}{\partial X_{jl}} \text{tr}(\boldsymbol{A}\boldsymbol{X}^\top) = \displaystyle\sum_{i=0}^{M-1} \displaystyle\sum_{k=0}^{N-1} A_{ik} \delta_{ij} \delta_{kl} \label{eq:5-4-5} \end{equation}

Summing over $i$ with $\delta_{ij}$, only the $i = j$ term survives; summing over $k$ with $\delta_{kl}$, only the $k = l$ term survives:

\begin{equation} \dfrac{\partial}{\partial X_{jl}} \text{tr}(\boldsymbol{A}\boldsymbol{X}^\top) = A_{jl} \label{eq:5-4-6} \end{equation}

Since \eqref{eq:5-4-6} holds for all $(j, l)$, we obtain the final result in matrix form.

\begin{equation} \dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{A}\boldsymbol{X}^\top) = \boldsymbol{A} \label{eq:5-4-7} \end{equation}

Proof

Applying the cyclic property of trace. Setting $\boldsymbol{P} = \boldsymbol{X}^\top$ and $\boldsymbol{Q} = \boldsymbol{A}$ in $\text{tr}(\boldsymbol{P}\boldsymbol{Q}) = \text{tr}(\boldsymbol{Q}\boldsymbol{P})$:

\begin{equation} \text{tr}(\boldsymbol{X}^\top\boldsymbol{A}) = \text{tr}(\boldsymbol{A}\boldsymbol{X}^\top) \label{eq:5-5-1} \end{equation}

For $\text{tr}(\boldsymbol{X}^\top\boldsymbol{A})$ to be defined, $\boldsymbol{X}^\top\boldsymbol{A}$ must be a square matrix. Since $\boldsymbol{X}^\top \in \mathbb{R}^{N \times M}$ and $\boldsymbol{A} \in \mathbb{R}^{M \times N}$, we have $\boldsymbol{X}^\top\boldsymbol{A} \in \mathbb{R}^{N \times N}$, so the trace is defined.

In this case, $\boldsymbol{A}\boldsymbol{X}^\top \in \mathbb{R}^{M \times M}$, and by the cyclic property both sides have the same scalar value. Applying the result of Formula 5.4:

\begin{equation} \dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{X}^\top\boldsymbol{A}) = \dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{A}\boldsymbol{X}^\top) = \boldsymbol{A} \label{eq:5-5-2} \end{equation}

Proof

Writing out the $(i, i)$ entry of $\boldsymbol{X}^2 = \boldsymbol{X} \cdot \boldsymbol{X}$:

\begin{equation} (\boldsymbol{X}^2)_{ii} = \displaystyle\sum_{k=0}^{N-1} X_{ik} X_{ki} \label{eq:5-6-1} \end{equation}

Since the trace is the sum of diagonal elements:

\begin{equation} \text{tr}(\boldsymbol{X}^2) = \displaystyle\sum_{i=0}^{N-1} \displaystyle\sum_{k=0}^{N-1} X_{ik} X_{ki} \label{eq:5-6-2} \end{equation}

Differentiating with respect to $X_{jl}$. Since both $X_{ik}$ and $X_{ki}$ are entries of $\boldsymbol{X}$, we apply the product rule (1.25).

\begin{equation} \dfrac{\partial}{\partial X_{jl}} \text{tr}(\boldsymbol{X}^2) = \displaystyle\sum_{i=0}^{N-1} \displaystyle\sum_{k=0}^{N-1} \left( \dfrac{\partial X_{ik}}{\partial X_{jl}} X_{ki} + X_{ik} \dfrac{\partial X_{ki}}{\partial X_{jl}} \right) \label{eq:5-6-3} \end{equation}

Computing the first term. Substituting $\displaystyle\dfrac{\partial X_{ik}}{\partial X_{jl}} = \delta_{ij} \delta_{kl}$, only the $i = j$ and $k = l$ terms survive:

\begin{equation} \displaystyle\sum_{i=0}^{N-1} \displaystyle\sum_{k=0}^{N-1} \delta_{ij} \delta_{kl} X_{ki} = X_{lj} \label{eq:5-6-4} \end{equation}

Computing the second term. Substituting $\displaystyle\dfrac{\partial X_{ki}}{\partial X_{jl}} = \delta_{kj} \delta_{il}$, only the $k = j$ and $i = l$ terms survive:

\begin{equation} \displaystyle\sum_{i=0}^{N-1} \displaystyle\sum_{k=0}^{N-1} X_{ik} \delta_{kj} \delta_{il} = X_{lj} \label{eq:5-6-5} \end{equation}

Combining the first and second terms:

\begin{equation} \dfrac{\partial}{\partial X_{jl}} \text{tr}(\boldsymbol{X}^2) = X_{lj} + X_{lj} = 2X_{lj} \label{eq:5-6-6} \end{equation}

Since $X_{lj}$ is the $(j, l)$ entry of the transpose $\boldsymbol{X}^\top$, we obtain the final result in matrix form.

\begin{equation} \dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{X}^2) = 2\boldsymbol{X}^\top \label{eq:5-6-7} \end{equation}

Proof

Since the $(i, j)$ entry of $\boldsymbol{X}^2$ is $(\boldsymbol{X}^2)_{ij} = \displaystyle\sum_{k=0}^{N-1} X_{ik} X_{kj}$, the trace is:

\begin{equation} \text{tr}(\boldsymbol{X}^2\boldsymbol{A}) = \displaystyle\sum_{i=0}^{N-1} \displaystyle\sum_{j=0}^{N-1} \displaystyle\sum_{k=0}^{N-1} X_{ik} X_{kj} A_{ji} \label{eq:5-7-1} \end{equation}

Differentiating with respect to $X_{pq}$. Since both $X_{ik}$ and $X_{kj}$ are entries of $\boldsymbol{X}$, we apply the product rule (1.25).

\begin{equation} \dfrac{\partial}{\partial X_{pq}} \text{tr}(\boldsymbol{X}^2\boldsymbol{A}) = \displaystyle\sum_{i,j,k} \left( \dfrac{\partial X_{ik}}{\partial X_{pq}} X_{kj} A_{ji} + X_{ik} \dfrac{\partial X_{kj}}{\partial X_{pq}} A_{ji} \right) \label{eq:5-7-2} \end{equation}

Computing the first term. Substituting $\displaystyle\dfrac{\partial X_{ik}}{\partial X_{pq}} = \delta_{ip} \delta_{kq}$ selects $i = p$ and $k = q$:

\begin{equation} \displaystyle\sum_{i,j,k} \delta_{ip} \delta_{kq} X_{kj} A_{ji} = \displaystyle\sum_{j} X_{qj} A_{jp} = (\boldsymbol{X}\boldsymbol{A})_{qp} \label{eq:5-7-3} \end{equation}

Computing the second term. Substituting $\displaystyle\dfrac{\partial X_{kj}}{\partial X_{pq}} = \delta_{kp} \delta_{jq}$ selects $k = p$ and $j = q$:

\begin{equation} \displaystyle\sum_{i,j,k} X_{ik} \delta_{kp} \delta_{jq} A_{ji} = \displaystyle\sum_{i} X_{ip} A_{qi} = (\boldsymbol{A}\boldsymbol{X})_{qp} \label{eq:5-7-4} \end{equation}

Combining the first and second terms:

\begin{equation} \dfrac{\partial}{\partial X_{pq}} \text{tr}(\boldsymbol{X}^2\boldsymbol{A}) = (\boldsymbol{X}\boldsymbol{A})_{qp} + (\boldsymbol{A}\boldsymbol{X})_{qp} \label{eq:5-7-5} \end{equation}

The $(q,p)$ entry is the $(p,q)$ entry of the transpose, and by linearity of the transpose:

\begin{equation} (\boldsymbol{X}\boldsymbol{A})_{qp} + (\boldsymbol{A}\boldsymbol{X})_{qp} = ((\boldsymbol{X}\boldsymbol{A} + \boldsymbol{A}\boldsymbol{X})^\top)_{pq} \label{eq:5-7-6} \end{equation}

Since \eqref{eq:5-7-6} holds for all $(p, q)$, we obtain the final result in matrix form.

\begin{equation} \dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{X}^2\boldsymbol{A}) = (\boldsymbol{X}\boldsymbol{A} + \boldsymbol{A}\boldsymbol{X})^\top \label{eq:5-7-7} \end{equation}

Proof

Writing out the $(i, k)$ entry of $\boldsymbol{A}\boldsymbol{X}$:

\begin{equation} (\boldsymbol{A}\boldsymbol{X})_{ik} = \displaystyle\sum_{j=0}^{N-1} A_{ij} X_{jk} \label{eq:5-8-1} \end{equation}

The $(l, l)$ entry of $\boldsymbol{X}^\top\boldsymbol{A}\boldsymbol{X}$ is:

\begin{equation} (\boldsymbol{X}^\top\boldsymbol{A}\boldsymbol{X})_{ll} = \displaystyle\sum_{i=0}^{N-1} (\boldsymbol{X}^\top)_{li} (\boldsymbol{A}\boldsymbol{X})_{il} \label{eq:5-8-2} \end{equation}

Substituting $(\boldsymbol{X}^\top)_{li} = X_{il}$ and using \eqref{eq:5-8-1}:

\begin{equation} (\boldsymbol{X}^\top\boldsymbol{A}\boldsymbol{X})_{ll} = \displaystyle\sum_{i=0}^{N-1} X_{il} \displaystyle\sum_{j=0}^{N-1} A_{ij} X_{jl} \label{eq:5-8-3} \end{equation}

Rearranging the sums:

\begin{equation} (\boldsymbol{X}^\top\boldsymbol{A}\boldsymbol{X})_{ll} = \displaystyle\sum_{i=0}^{N-1} \displaystyle\sum_{j=0}^{N-1} X_{il} A_{ij} X_{jl} \label{eq:5-8-4} \end{equation}

Since the trace is the sum of diagonal elements, summing over $l$:

\begin{equation} \text{tr}(\boldsymbol{X}^\top\boldsymbol{A}\boldsymbol{X}) = \displaystyle\sum_{l=0}^{M-1} \displaystyle\sum_{i=0}^{N-1} \displaystyle\sum_{j=0}^{N-1} X_{il} A_{ij} X_{jl} \label{eq:5-8-5} \end{equation}

Differentiating with respect to $X_{pq}$. Since both $X_{il}$ and $X_{jl}$ are entries of $\boldsymbol{X}$, we apply the product rule (1.25):

\begin{equation} \dfrac{\partial}{\partial X_{pq}} \text{tr}(\boldsymbol{X}^\top\boldsymbol{A}\boldsymbol{X}) = \displaystyle\sum_{l,i,j} \left( \dfrac{\partial X_{il}}{\partial X_{pq}} A_{ij} X_{jl} + X_{il} A_{ij} \dfrac{\partial X_{jl}}{\partial X_{pq}} \right) \label{eq:5-8-6} \end{equation}

Computing the first term. Substituting $\displaystyle\dfrac{\partial X_{il}}{\partial X_{pq}} = \delta_{ip} \delta_{lq}$:

\begin{equation} \displaystyle\sum_{l,i,j} \delta_{ip} \delta_{lq} A_{ij} X_{jl} = \displaystyle\sum_{j} A_{pj} X_{jq} \label{eq:5-8-7} \end{equation}

Here $\delta_{ip}$ selects $i = p$ and $\delta_{lq}$ selects $l = q$.

Rewriting \eqref{eq:5-8-7} as a matrix product:

\begin{equation} \displaystyle\sum_{j} A_{pj} X_{jq} = (\boldsymbol{A}\boldsymbol{X})_{pq} \label{eq:5-8-8} \end{equation}

Computing the second term. Substituting $\displaystyle\dfrac{\partial X_{jl}}{\partial X_{pq}} = \delta_{jp} \delta_{lq}$:

\begin{equation} \displaystyle\sum_{l,i,j} X_{il} A_{ij} \delta_{jp} \delta_{lq} = \displaystyle\sum_{i} X_{iq} A_{ip} \label{eq:5-8-9} \end{equation}

Here $\delta_{jp}$ selects $j = p$ and $\delta_{lq}$ selects $l = q$.

Transforming \eqref{eq:5-8-9}. Using $A_{ip} = (\boldsymbol{A}^\top)_{pi}$:

\begin{equation} \displaystyle\sum_{i} X_{iq} A_{ip} = \displaystyle\sum_{i} (\boldsymbol{A}^\top)_{pi} X_{iq} \label{eq:5-8-10} \end{equation}

Rewriting as a matrix product:

\begin{equation} \displaystyle\sum_{i} (\boldsymbol{A}^\top)_{pi} X_{iq} = (\boldsymbol{A}^\top\boldsymbol{X})_{pq} \label{eq:5-8-11} \end{equation}

Combining the first term \eqref{eq:5-8-8} and the second term \eqref{eq:5-8-11}:

\begin{equation} \dfrac{\partial}{\partial X_{pq}} \text{tr}(\boldsymbol{X}^\top\boldsymbol{A}\boldsymbol{X}) = (\boldsymbol{A}\boldsymbol{X})_{pq} + (\boldsymbol{A}^\top\boldsymbol{X})_{pq} \label{eq:5-8-12} \end{equation}

Since \eqref{eq:5-8-12} holds for all $(p, q)$, we obtain the final result in matrix form.

\begin{equation} \dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{X}^\top\boldsymbol{A}\boldsymbol{X}) = \boldsymbol{A}\boldsymbol{X} + \boldsymbol{A}^\top\boldsymbol{X} \label{eq:5-8-13} \end{equation}

Proof

By the cyclic property of trace (1.12), $\text{tr}(\boldsymbol{A}\boldsymbol{X}\boldsymbol{X}^\top) = \text{tr}(\boldsymbol{X}^\top\boldsymbol{A}\boldsymbol{X})$, so this gives the same result as 5.8.

Proof

By the cyclic property of trace (1.12), $\text{tr}(\boldsymbol{X}\boldsymbol{X}^\top\boldsymbol{A}) = \text{tr}(\boldsymbol{X}^\top\boldsymbol{A}\boldsymbol{X})$, so this gives the same result as 5.8.

Proof

Writing the trace in component form:

Differentiating with respect to $X_{pq}$, there are two types of terms containing $X_{pq}$.

Case 1: $X_{ij} = X_{pq}$ (i.e., $i = p, j = q$)

Case 2: $X_{ik} = X_{pq}$ (i.e., $i = p, k = q$)

Combining the above:

Proof

By the cyclic property of trace (1.12), $\text{tr}(\boldsymbol{A}\boldsymbol{X}^\top\boldsymbol{X}) = \text{tr}(\boldsymbol{X}\boldsymbol{A}\boldsymbol{X}^\top)$, so this gives the same result as 5.11.

Proof

By the cyclic property of trace (1.12), $\text{tr}(\boldsymbol{X}^\top\boldsymbol{X}\boldsymbol{A}) = \text{tr}(\boldsymbol{X}\boldsymbol{A}\boldsymbol{X}^\top)$, so this gives the same result as 5.11.

Proof

Writing the trace in component form:

Differentiating with respect to $X_{pq}$, there are two types of terms containing $X_{pq}$.

Case 1: $X_{jk} = X_{pq}$ (i.e., $j = p, k = q$)

Case 2: $X_{li} = X_{pq}$ (i.e., $l = p, i = q$)

Combining the above:

Proof

Setting $\boldsymbol{A} = \boldsymbol{I}$ in 5.8 gives $\boldsymbol{I}\boldsymbol{X} + \boldsymbol{I}^\top\boldsymbol{X} = 2\boldsymbol{X}$.

Proof

By the cyclic property of trace (1.12), $\text{tr}(\boldsymbol{X}\boldsymbol{X}^\top) = \text{tr}(\boldsymbol{X}^\top\boldsymbol{X})$, so this gives the same result as 5.15.

Proof

Setting $\boldsymbol{Y} = \boldsymbol{X}\boldsymbol{B}$, we get $\text{tr}(\boldsymbol{B}^\top\boldsymbol{X}^\top\boldsymbol{C}\boldsymbol{X}\boldsymbol{B}) = \text{tr}(\boldsymbol{Y}^\top\boldsymbol{C}\boldsymbol{Y})$. By 5.8, $\displaystyle\dfrac{\partial}{\partial \boldsymbol{Y}} \text{tr}(\boldsymbol{Y}^\top\boldsymbol{C}\boldsymbol{Y}) = (\boldsymbol{C} + \boldsymbol{C}^\top)\boldsymbol{Y}$. Applying the chain rule (matrix version of 1.26):

Proof

Writing the trace in component form, $\text{tr}(\boldsymbol{X}^\top\boldsymbol{B}\boldsymbol{X}\boldsymbol{C}) = \displaystyle\sum_{i,j,k,l} X_{ji} B_{jk} X_{kl} C_{li}$. Differentiating with respect to $X_{pq}$:

Proof

Writing the trace in component form, $\text{tr}(\boldsymbol{A}\boldsymbol{X}\boldsymbol{B}\boldsymbol{X}^\top\boldsymbol{C}) = \displaystyle\sum_{i,j,k,l,m} A_{ij} X_{jk} B_{kl} X_{ml} C_{mi}$. Differentiating with respect to $X_{pq}$:

Proof

Setting $\boldsymbol{Y} = \boldsymbol{A}\boldsymbol{X}\boldsymbol{B}+\boldsymbol{C}$, we get $\text{tr}(\boldsymbol{Y}\boldsymbol{Y}^\top) = \|\boldsymbol{Y}\|_F^2$. By 5.16, $\displaystyle\dfrac{\partial}{\partial \boldsymbol{Y}} \text{tr}(\boldsymbol{Y}\boldsymbol{Y}^\top) = 2\boldsymbol{Y}$. Applying the chain rule (1.26), since $\displaystyle\dfrac{\partial Y_{ij}}{\partial X_{pq}} = A_{ip} B_{qj}$:

Proof

By the trace property of the Kronecker product $\text{tr}(\boldsymbol{A} \otimes \boldsymbol{B}) = \text{tr}(\boldsymbol{A})\text{tr}(\boldsymbol{B})$, we have $\text{tr}(\boldsymbol{X} \otimes \boldsymbol{X}) = [\text{tr}(\boldsymbol{X})]^2$. Differentiating with respect to $X_{pq}$:

Proof

We have $\text{tr}(\boldsymbol{X}^\top \boldsymbol{A} \boldsymbol{X}) = \displaystyle\sum_{i,j,k} X_{ik} A_{ij} X_{jk}$. Differentiating with respect to $X_{pq}$:

Proof

In component form, $\text{tr}(\boldsymbol{A}\boldsymbol{X}\boldsymbol{B}) = \displaystyle\sum_{l,i,j} A_{li} X_{ij} B_{jl}$. Differentiating with respect to $X_{pq}$:

Proof

Writing the trace in component form, $\text{tr}(\boldsymbol{A}\boldsymbol{X}^\top\boldsymbol{B}) = \displaystyle\sum_{i,j,k} A_{ij} X_{kj} B_{ki}$. Differentiating with respect to $X_{pq}$:

Proof

By the trace property of the Kronecker product, $\text{tr}(\boldsymbol{A} \otimes \boldsymbol{X}) = \text{tr}(\boldsymbol{A}) \cdot \text{tr}(\boldsymbol{X})$. Differentiating with respect to $X_{pq}$:

Proof

Differentiating the identity $\boldsymbol{X} \boldsymbol{X}^{-1} = \boldsymbol{I}$ with respect to $X_{pq}$ gives $\displaystyle\dfrac{\partial \boldsymbol{X}^{-1}}{\partial X_{pq}} = -\boldsymbol{X}^{-1} \boldsymbol{E}_{pq} \boldsymbol{X}^{-1}$. Using this to differentiate $\text{tr}(\boldsymbol{X}^{-1}\boldsymbol{A})$ with respect to $X_{pq}$:

Proof

Applying the chain rule (1.26) to the derivative of a matrix power gives $\displaystyle\dfrac{\partial \boldsymbol{X}^k}{\partial X_{pq}} = \displaystyle\sum_{r=0}^{k-1} \boldsymbol{X}^r \boldsymbol{E}_{pq} \boldsymbol{X}^{k-r-1}$. Using the cyclic property of trace:

Proof

Similarly to 5.27, we compute the derivative of the matrix power.

Using $\text{tr}(\boldsymbol{M} \boldsymbol{E}_{pq}) = M_{qp}$:

Performing the substitution $s = k - r - 1$ ($r = k - s - 1$):

Therefore $\displaystyle\dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{A}\boldsymbol{X}^k) = \displaystyle\sum_{r=0}^{k-1} (\boldsymbol{X}^r \boldsymbol{A} \boldsymbol{X}^{k-r-1})^\top$.

Proof

Since $\boldsymbol{X}$ appears in four positions in this compound expression, the derivative is the sum of the results obtained by differentiating at each position. Setting $\boldsymbol{Y} = \boldsymbol{X}\boldsymbol{B}$, the original expression can be written as $\text{tr}(\boldsymbol{Y}^\top\boldsymbol{C}\boldsymbol{X}\boldsymbol{X}^\top\boldsymbol{C}\boldsymbol{Y})$. We differentiate with respect to each occurrence of $\boldsymbol{X}$.

Term 1 (differentiation at the leftmost $\boldsymbol{X}^\top$):

Using the cyclic property of trace and $\text{tr}(\boldsymbol{E}_{qp}\boldsymbol{M}) = M_{pq}$, this term gives $(\boldsymbol{C}\boldsymbol{X}\boldsymbol{X}^\top\boldsymbol{C}\boldsymbol{X}\boldsymbol{B}\boldsymbol{B}^\top)_{pq}$.

Term 2 (differentiation at $\boldsymbol{X}$ in $\boldsymbol{C}\boldsymbol{X}$):

Using the cyclic property of trace, this term gives $(\boldsymbol{C}^\top\boldsymbol{X}\boldsymbol{B}\boldsymbol{B}^\top\boldsymbol{X}^\top\boldsymbol{C}^\top\boldsymbol{X})_{pq}$.

Term 3 (differentiation at $\boldsymbol{X}^\top$ in $\boldsymbol{X}\boldsymbol{X}^\top$):

This term gives $(\boldsymbol{C}\boldsymbol{X}\boldsymbol{B}\boldsymbol{B}^\top\boldsymbol{X}^\top\boldsymbol{C}\boldsymbol{X})_{pq}$.

Term 4 (differentiation at the rightmost $\boldsymbol{X}$):

This term gives $(\boldsymbol{C}^\top\boldsymbol{X}\boldsymbol{X}^\top\boldsymbol{C}^\top\boldsymbol{X}\boldsymbol{B}\boldsymbol{B}^\top)_{pq}$.

Combining all four terms:

Proof

We use the inverse matrix derivative formula derived in 8.2:

where $\boldsymbol{E}_{pq}$ is the matrix with 1 only at position $(p, q)$. Differentiating $\text{tr}(\boldsymbol{A}\boldsymbol{X}^{-1}\boldsymbol{B})$ with respect to $X_{pq}$:

Since $\text{tr}(\boldsymbol{M} \boldsymbol{E}_{pq}) = M_{qp}$:

Therefore $\displaystyle\dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{A}\boldsymbol{X}^{-1}\boldsymbol{B}) = -\boldsymbol{X}^{-\top}\boldsymbol{A}^\top\boldsymbol{B}^\top\boldsymbol{X}^{-\top}$.

Proof

Let $\boldsymbol{W} = \boldsymbol{X}^\top \boldsymbol{C} \boldsymbol{X}$ (an $M \times M$ matrix). First, we compute the derivative of $\boldsymbol{W}$ with respect to $X_{pq}$. Since $W_{ij} = \displaystyle\sum_{k,l} X_{ki} C_{kl} X_{lj}$:

Using the inverse matrix derivative formula $\displaystyle\dfrac{\partial \boldsymbol{W}^{-1}}{\partial W_{ij}} = -\boldsymbol{W}^{-1} \boldsymbol{E}_{ij} \boldsymbol{W}^{-1}$ and the chain rule:

Substituting $\displaystyle\dfrac{\partial}{\partial W_{ij}} \text{tr}(\boldsymbol{W}^{-1}\boldsymbol{A}) = -(\boldsymbol{W}^{-1}\boldsymbol{A}\boldsymbol{W}^{-1})_{ji}$ and using $\boldsymbol{C} = \boldsymbol{C}^\top$:

Therefore $\displaystyle\dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}[(\boldsymbol{X}^\top\boldsymbol{C}\boldsymbol{X})^{-1}\boldsymbol{A}] = -\boldsymbol{C}\boldsymbol{X}(\boldsymbol{X}^\top\boldsymbol{C}\boldsymbol{X})^{-1}(\boldsymbol{A}+\boldsymbol{A}^\top)(\boldsymbol{X}^\top\boldsymbol{C}\boldsymbol{X})^{-1}$.

Proof

Let $\boldsymbol{W} = \boldsymbol{X}^\top \boldsymbol{C} \boldsymbol{X}$ and $\boldsymbol{V} = \boldsymbol{X}^\top \boldsymbol{B} \boldsymbol{X}$. By the product rule:

Term 1 (derivative of $\boldsymbol{W}^{-1}$, with $\boldsymbol{V}$ fixed): Substituting $\boldsymbol{A} = \boldsymbol{V} = \boldsymbol{X}^\top\boldsymbol{B}\boldsymbol{X}$ (symmetric) into the result of 5.31:

Term 2 (derivative of $\boldsymbol{V}$, with $\boldsymbol{W}^{-1}$ fixed): By 5.22, $\displaystyle\dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{W}^{-1}\boldsymbol{X}^\top\boldsymbol{B}\boldsymbol{X}) = 2\boldsymbol{B}\boldsymbol{X}\boldsymbol{W}^{-1}$ (when $\boldsymbol{B}$ is symmetric).

Therefore:

Proof

Let $\boldsymbol{W} = \boldsymbol{A} + \boldsymbol{X}^\top \boldsymbol{C} \boldsymbol{X}$ and $\boldsymbol{V} = \boldsymbol{X}^\top \boldsymbol{B} \boldsymbol{X}$. In the derivative of $\boldsymbol{W}$ with respect to $X_{pq}$, the constant term $\boldsymbol{A}$ vanishes:

This has the same form as in 5.32. The factor 2 in 5.32 came from the symmetry of $\boldsymbol{W}$, so when $\boldsymbol{A}$ is symmetric and $\boldsymbol{W} = \boldsymbol{A} + \boldsymbol{X}^\top\boldsymbol{C}\boldsymbol{X}$ is therefore symmetric (and invertible), the result is obtained simply by replacing the $\boldsymbol{W}$ of 5.32 with this $\boldsymbol{W}$:

Proof

The matrix exponential is defined by its Taylor series:

Taking the trace and differentiating term by term:

Proof

We consider the matrix logarithm when $\boldsymbol{X}$ is positive definite. Using the trace property $\text{tr}(\log(\boldsymbol{X})) = \log(|\boldsymbol{X}|)$, which follows from the diagonalization $\boldsymbol{X} = \boldsymbol{P}\boldsymbol{\Lambda}\boldsymbol{P}^{-1}$: $\text{tr}(\log(\boldsymbol{X})) = \displaystyle\sum_i \log(\lambda_i) = \log(\displaystyle\prod_i \lambda_i) = \log(|\boldsymbol{X}|)$.

Using the determinant derivative formula $\displaystyle\dfrac{\partial}{\partial \boldsymbol{X}} |\boldsymbol{X}| = |\boldsymbol{X}| \boldsymbol{X}^{-\top}$:

Proof

When $\boldsymbol{X}$ is positive definite, a unique positive definite square root $\boldsymbol{X}^{1/2}$ exists. Since $f(x) = x^{1/2}$ is analytic on a region containing $(0, \infty)$, where the eigenvalues of a positive definite matrix lie, the general formula below applies (5.27 is for a positive integer $k$, so $n = 1/2$ cannot be substituted into it directly). Substituting $f'(x) = \tfrac{1}{2}x^{-1/2}$:

Proof

The matrix sine function is defined by its Taylor series:

Taking the trace and applying the formula from 5.27, $\displaystyle\dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{X}^n) = n(\boldsymbol{X}^{n-1})^\top$, term by term:

Proof

The matrix cosine function is defined by its Taylor series:

Taking the trace and differentiating term by term. The $k=0$ term is constant $\boldsymbol{I}$, so its derivative is $\boldsymbol{O}$:

Proof

We first treat the case where $f$ is given by its Taylor series about the origin, $f(x) = \displaystyle\sum_{k=0}^{\infty} c_k x^k$, and the series converges at the eigenvalues of $\boldsymbol{X}$ (other $f$ are treated at the end). The matrix function is defined as $f(\boldsymbol{X}) = \displaystyle\sum_{k=0}^{\infty} c_k \boldsymbol{X}^k$, so:

By the result of 5.27 (derivative of power traces), $\displaystyle\dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{X}^k) = k(\boldsymbol{X}^{k-1})^\top$. This gives the same result whether one differentiates $\text{tr}(\boldsymbol{X}^k) = \displaystyle\sum_i \lambda_i^k$ as a scalar or differentiates each term of the Taylor series directly.

Applying term-by-term differentiation:

Here $f'(x) = \displaystyle\sum_{k=1}^{\infty} k\, c_k x^{k-1}$ is the scalar derivative of $f$, and $f'(\boldsymbol{X})$ is obtained by substituting $\boldsymbol{X}$ into this series.

The version with $\boldsymbol{A}$ follows from the same term-by-term differentiation. We have $\text{tr}(\boldsymbol{A}f(\boldsymbol{X})) = \displaystyle\sum_{k=0}^{\infty} c_k \,\text{tr}(\boldsymbol{A}\boldsymbol{X}^k)$, and by 5.28,

(the second equality uses $\boldsymbol{A}\boldsymbol{X} = \boldsymbol{X}\boldsymbol{A}$ to collect each term of the sum into $\boldsymbol{A}\boldsymbol{X}^{k-1}$). Summing term by term gives $(\boldsymbol{A}f'(\boldsymbol{X}))^\top$. No diagonalization is used, so $\boldsymbol{X}$ need not be diagonalizable.

For an $f$ whose series about the origin does not converge up to the eigenvalues of $\boldsymbol{X}$ (for example $\text{arccosh}$ or $\sqrt{x}$, which are not defined at $x = 0$), use the integral representation $f(\boldsymbol{X}) = \dfrac{1}{2\pi i}\displaystyle\oint_\Gamma f(z)(z\boldsymbol{I}-\boldsymbol{X})^{-1}\,dz$ over a closed curve $\Gamma$ enclosing the eigenvalues of $\boldsymbol{X}$. Integrating $\dfrac{\partial}{\partial \boldsymbol{X}}\text{tr}(\boldsymbol{A}(z\boldsymbol{I}-\boldsymbol{X})^{-1}) = (\boldsymbol{A}(z\boldsymbol{I}-\boldsymbol{X})^{-2})^\top$ (when $\boldsymbol{A}$ and $\boldsymbol{X}$ commute; no condition is needed for $\boldsymbol{A} = \boldsymbol{I}$) over $\Gamma$ gives the same conclusion. $\square$

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\tan(x) = \sec^2(x)$. Substituting $f(x) = \tan(x)$, $f'(x) = \sec^2(x)$ into the general formula:

Here $\sec^2(\boldsymbol{X}) = \cos(\boldsymbol{X})^{-2}$ is defined as the square of the inverse of the matrix cosine.

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\arcsin(x) = \displaystyle\dfrac{1}{\sqrt{1-x^2}}$. Substituting $f(x) = \arcsin(x)$, $f'(x) = (1-x^2)^{-1/2}$ into the general formula:

Here $(\boldsymbol{I}-\boldsymbol{X}^2)^{-1/2}$ is the inverse square root of the matrix $\boldsymbol{I}-\boldsymbol{X}^2$.

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\arccos(x) = -\displaystyle\dfrac{1}{\sqrt{1-x^2}}$. Substituting $f(x) = \arccos(x)$, $f'(x) = -(1-x^2)^{-1/2}$ into the general formula:

Here the matrix $f'(\boldsymbol{X})$ is the same $(\boldsymbol{I}-\boldsymbol{X}^2)^{-1/2}$ as in 5.40, differing only in sign.

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\arctan(x) = \displaystyle\dfrac{1}{1+x^2}$. Substituting $f(x) = \arctan(x)$, $f'(x) = (1+x^2)^{-1}$ into the general formula:

Here $(\boldsymbol{I}+\boldsymbol{X}^2)^{-1}$ is the inverse of the matrix $\boldsymbol{I}+\boldsymbol{X}^2$.

Proof

The matrix hyperbolic sine function is defined by its Taylor series:

Differentiating term by term as in 5.37:

Proof

The matrix hyperbolic cosine function is defined by its Taylor series:

Differentiating term by term as in 5.38:

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\tanh(x) = \text{sech}^2(x)$. Substituting $f(x) = \tanh(x)$, $f'(x) = \text{sech}^2(x)$ into the general formula:

Here $\text{sech}^2(\boldsymbol{X}) = \cosh(\boldsymbol{X})^{-2}$ is defined as the square of the inverse of the matrix hyperbolic cosine.

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\text{arcsinh}(x) = \displaystyle\dfrac{1}{\sqrt{1+x^2}}$. Substituting $f(x) = \text{arcsinh}(x)$, $f'(x) = (1+x^2)^{-1/2}$ into the general formula:

Here $(\boldsymbol{I}+\boldsymbol{X}^2)^{-1/2}$ is the inverse square root of the matrix $\boldsymbol{I}+\boldsymbol{X}^2$.

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\text{arccosh}(x) = \displaystyle\dfrac{1}{\sqrt{x^2-1}}$ ($x > 1$). Substituting $f(x) = \text{arccosh}(x)$, $f'(x) = (x^2-1)^{-1/2}$ into the general formula:

Here $(\boldsymbol{X}^2-\boldsymbol{I})^{-1/2}$ is the inverse square root of the matrix $\boldsymbol{X}^2-\boldsymbol{I}$, which is defined when all eigenvalues are greater than $1$.

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\text{arctanh}(x) = \displaystyle\dfrac{1}{1-x^2}$ ($|x| < 1$). Substituting $f(x) = \text{arctanh}(x)$, $f'(x) = (1-x^2)^{-1}$ into the general formula:

Here $(\boldsymbol{I}-\boldsymbol{X}^2)^{-1}$ is the inverse of the matrix $\boldsymbol{I}-\boldsymbol{X}^2$.

Proof

Using the Taylor series as in 5.37:

Using the formula from 5.28, $\displaystyle\dfrac{\partial}{\partial \boldsymbol{X}} \text{tr}(\boldsymbol{A}\boldsymbol{X}^n) = n(\boldsymbol{A}\boldsymbol{X}^{n-1})^\top$ (when $\boldsymbol{A}$ and $\boldsymbol{X}$ commute):

Proof

Using the Taylor series as in 5.37:

When $\boldsymbol{A}$ and $\boldsymbol{X}$ commute, differentiating term by term:

Proof

Using the Taylor series:

When $\boldsymbol{A}$ and $\boldsymbol{X}$ commute, differentiating term by term:

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\tan(x) = \sec^2(x)$. Substituting $f(x) = \tan(x)$, $f'(x) = \sec^2(x)$ into the version with $\boldsymbol{A}$ of the general formula (requires $\boldsymbol{A}\boldsymbol{X} = \boldsymbol{X}\boldsymbol{A}$):

Here $\sec^2(\boldsymbol{X}) = \cos(\boldsymbol{X})^{-2}$.

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\arcsin(x) = \displaystyle\dfrac{1}{\sqrt{1-x^2}}$. Substituting $f(x) = \arcsin(x)$, $f'(x) = (1-x^2)^{-1/2}$ into the version with $\boldsymbol{A}$ of the general formula (requires $\boldsymbol{A}\boldsymbol{X} = \boldsymbol{X}\boldsymbol{A}$):

Here $(\boldsymbol{I}-\boldsymbol{X}^2)^{-1/2}$ is the inverse square root of the matrix $\boldsymbol{I}-\boldsymbol{X}^2$.

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\arccos(x) = -\displaystyle\dfrac{1}{\sqrt{1-x^2}}$. Substituting $f(x) = \arccos(x)$, $f'(x) = -(1-x^2)^{-1/2}$ into the version with $\boldsymbol{A}$ of the general formula (requires $\boldsymbol{A}\boldsymbol{X} = \boldsymbol{X}\boldsymbol{A}$):

Here the matrix $f'(\boldsymbol{X})$ differs from 5.53 only in sign.

Proof

In the scalar case, $\displaystyle\dfrac{d}{dx}\arctan(x) = \displaystyle\dfrac{1}{1+x^2}$. Substituting $f(x) = \arctan(x)$, $f'(x) = (1+x^2)^{-1}$ into the version with $\boldsymbol{A}$ of the general formula (requires $\boldsymbol{A}\boldsymbol{X} = \boldsymbol{X}\boldsymbol{A}$):

Here $(\boldsymbol{I}+\boldsymbol{X}^2)^{-1}$ is the inverse of the matrix $\boldsymbol{I}+\boldsymbol{X}^2$.

Proof

Using the Taylor series:

When $\boldsymbol{A}$ and $\boldsymbol{X}$ commute, differentiating term by term:

Proof

Using the Taylor series:

When $\boldsymbol{A}$ and $\boldsymbol{X}$ commute, differentiating term by term:

Proof

The matrix hyperbolic tangent is defined by a power series:

More precisely, $\tanh(\boldsymbol{X}) = \sinh(\boldsymbol{X})\cosh(\boldsymbol{X})^{-1}$, which is defined when $\cosh(\boldsymbol{X})$ is invertible.

Apply the version with $\boldsymbol{A}$ of the general formula ($\dfrac{\partial}{\partial \boldsymbol{X}}\text{tr}(\boldsymbol{A}f(\boldsymbol{X})) = (\boldsymbol{A}f'(\boldsymbol{X}))^\top$ when $\boldsymbol{A}$ and $\boldsymbol{X}$ commute) with $f = \tanh$ and $f' = \text{sech}^2$. Since $\tanh$ is analytic near the eigenvalues of $\boldsymbol{X}$ whenever $\cosh(\boldsymbol{X})$ is invertible, the conditions of the general formula are met: