pyrobolearn/tutorials/math/Linear-Algebra.ipynb

Linear Algebra

Vector Space

A vector space V defined on the field K is ...

It satisfies the 10 following properties: 1.

A vector is an element of a vector space.

Linear Transformation

Mapping between 2 vector spaces.

Null and range space

Assume two vector spaces V and W, and a linear mapping \pmb{A}: V \rightarrow W.

• The null (aka kernel) space of a matrix \pmb{A} is defined as the subspace of V such that:

\begin{equation} \mathcal{N}(\pmb{A}) = {\pmb{x} \in V : \pmb{Ax} = \pmb{0}} \subseteq V \end{equation}

• The range (aka column or image) space of a matrix \pmb{A} is defined as the subspace of W such that:

\begin{equation} \mathcal{R}(\pmb{A}) = {\pmb{y} \in W : \pmb{Ax} = \pmb{y}, ; \forall \pmb{x} \in V } \subseteq W \end{equation}

By the Rank–nullity theorem: \begin{equation} dim(\mathcal{N}(\pmb{A})) + dim(\mathcal{R}(\pmb{A})) = dim(V) \end{equation}

Rank of a matrix

Transpose

Determinant

Determinant of a square matrix.

\begin{equation} det : \mathbb{K}^{N \times N} \rightarrow \mathbb{K}: det(\pmb{A}) = |\pmb{A}|
\end{equation}

Properties:

• det(\pmb{A}) = det(\pmb{A}^T)
• det(\pmb{A}^{-1}) = det(\pmb{A})^{-1}
• If \pmb{A} and \pmb{B} are square matrices of same size then det(\pmb{A}\pmb{B}) = det(\pmb{A}) det(\pmb{B}) = det(\pmb{B}) det(\pmb{A}) = det(\pmb{B}\pmb{A})

Invertible matrix

A square matrix \pmb{A} \in \mathbb{R}^{N \times N} is invertible, and if \pmb{A}\pmb{A}^{-1} = \pmb{A}^{-1}\pmb{A} = \pmb{I}.

• \pmb{A} is invertible.
• \pmb{A} is full-rank, i.e.
• The number 0 is not an eigenvalue of \pmb{A}.

Time complexity: O(N^3)

Notes:

• If you want to know if a matrix \pmb{A} (which is at least PSD) is PD, you can check if it is invertible. If that's the case, then 0 can not be an eigenvalue of \pmb{A} and thus the matrix is PD.

Orthogonal Matrices

\pmb{A} is an orthogonal (square) matrix if \pmb{A}^{-1} = \pmb{A}^T, and thus \pmb{A}^T\pmb{A} = \pmb{A}\pmb{A}^T = \pmb{I}. This implies that the columns (resp. rows) of \pmb{A} are othonormal to each other.

Symmetric Matrices

Positive (Semi) Definite

• PSD \rightarrow symmetric.

• All the evals of a PD matrix are positive, and 0 is not one of these.

• All the evals of a PSD matrix are non-negative.

Norm

Trace

\begin{equation} tr(\pmb{A}) = \sum_i a_{ii} \end{equation}

Properties:

• The trace is a linear operator: tr(c_1 \pmb{A}+ c_2 \pmb{B}) = c_1 tr(\pmb{A}) + c_2 tr(\pmb{B})
• The transpose has the same trace: tr(\pmb{A}) = tr(\pmb{A}^T)
• Invariance under cyclic permutation: tr(\pmb{A}\pmb{B}\pmb{C}) = tr(\pmb{B}\pmb{C}\pmb{A}) = tr(\pmb{C}\pmb{A}\pmb{B})
• Product: tr(\pmb{A}\pmb{B}) \neq tr(\pmb{A})tr(\pmb{B})
• Frobenius Norm: ||\pmb{A}||_F = \sqrt{tr(\pmb{A}^T\pmb{A})} = \sqrt{tr(\pmb{A}\pmb{A}^T)}

Matrix Calculus

More information can be found on the matrix cookbook.

First order: \begin{align} \nabla_{\pmb{X}} tr(\pmb{AX}) = \nabla_{\pmb{X}} tr(\pmb{XA}) &= \pmb{A}^T \ \nabla_{\pmb{X}} tr(\pmb{AX}^T) = \nabla_{\pmb{X}} tr(\pmb{X}^T\pmb{A}) &= \pmb{A} \end{align}

Second order: \begin{align} \nabla_{\pmb{X}} tr(\pmb{A}\pmb{X}\pmb{B}\pmb{X}) = \nabla_{\pmb{X}} tr(\pmb{X}\pmb{B}\pmb{X}\pmb{A}) &= (\pmb{BXA})^T + (\pmb{AXB})^T \ \nabla_{\pmb{X}} tr(\pmb{A}\pmb{X}^T\pmb{B}\pmb{X}) = \nabla_{\pmb{X}} tr(\pmb{X}^T\pmb{B}\pmb{X}\pmb{A}) = \nabla_{\pmb{X}} tr(\pmb{B}\pmb{X}\pmb{A}\pmb{X}^T) &= \pmb{BXA} + (\pmb{A}\pmb{X}^T\pmb{B})^T \ \end{align}

Pseudo inverse

This is the generalization of the inverse, in the sense that it can be applied to rectangular matrices. Assume \pmb{A} \in \mathbb{R}^{N \times N}.

Right pseudo-inverse: \begin{equation} \pmb{A}^\dagger = (\pmb{A}^T\pmb{A})^{-1}\pmb{A}^T \end{equation}

Left pseudo-inverse: \begin{equation} ^\dagger\pmb{A} = \pmb{A}^T(\pmb{A}\pmb{A}^T)^{-1} \end{equation}

Linear Regression (LR)

Assume the input is given by \pmb{X} \in \mathbb{R}^{N \times D_x}, the output by \pmb{Y} \in \mathbb{R}^{N \times D_y}, and the weight matrix by \pmb{W} \in \mathbb{R}^{D_x \times D_y}.

The MSE loss is defined as:

\begin{equation} \mathcal{L} = ||\pmb{Y} - \pmb{XW}||^2_F \end{equation}

Taking the gradient of this loss and setting it to zero gives us the minimum:

\begin{align} \nabla_{\pmb{W}} \mathcal{L} &= \nabla_{\pmb{W}} ||\pmb{Y} - \pmb{XW}||^2_F \ &= \nabla_{\pmb{W}} tr((\pmb{Y} - \pmb{XW})^T(\pmb{Y} - \pmb{XW})) \ &= \nabla_{\pmb{W}} [tr(\pmb{Y}^T\pmb{Y}) - tr((\pmb{XW})^T\pmb{Y}) - tr(\pmb{Y}^T\pmb{XW}) + tr(\pmb{W}^T\pmb{X}^T\pmb{X}\pmb{W})] \ &= \nabla_{\pmb{W}} [tr(\pmb{Y}^T\pmb{Y}) - tr(\pmb{Y}^T\pmb{XW}) - tr(\pmb{Y}^T\pmb{XW}) + tr(\pmb{W}^T\pmb{X}^T\pmb{X}\pmb{W})] \ &= -2 \nabla_{\pmb{W}} tr(\pmb{Y}^T\pmb{XW}) + \nabla_{\pmb{W}} tr(\pmb{W}^T\pmb{X}^T\pmb{X}\pmb{W}) \ &= -2 (\pmb{Y}^T\pmb{X})^T + \pmb{X}^T\pmb{X} \pmb{W} + (\pmb{X}^T\pmb{X})^T \pmb{W} \ &= -2 \pmb{X}^T\pmb{Y} + 2 \pmb{X}^T\pmb{X} \pmb{W} \ &= 0 \ \Leftrightarrow & \quad (\pmb{X}^T\pmb{X}) \pmb{W} = \pmb{X}^T\pmb{Y} \end{align}

If the covariance (\pmb{X}^T\pmb{X}) is invertible i.e. is PD, then the best set of weights are given by:

\begin{equation} \pmb{W}^* = (\pmb{X}^T\pmb{X})^{-1} \pmb{X}^T\pmb{Y} = \pmb{X}^\dagger \pmb{Y} = \pmb{\Sigma_{XX}}^{-1}\pmb{\Sigma_{XY}} \end{equation}

Linear Weighted Regression (LWR)

Covariance

The covariance \pmb{C}_{XY} is a positive semi-definite (PSD) matrix which captures linear correlation between 2 random variables X and Y. The PSD implies that it is symmetric by definition.

• If the cov is invertible, then 0 can not be an eigenvalue of \pmb{C}. This means that it is positive definite (PD).

Eigenvalue and Eigenvectors

Diagonalizable and Eigendecomposition

Idempotence

An idempotent matrix \pmb{P} is a square 'matrix which, when multiplied by itself, yields itself'.

\begin{equation} \pmb{P} = \pmb{PP} = \pmb{P}^2 \end{equation}

Properties:

• An idempotent matrix (except the identity) is singular (i.e. not full rank).
• \pmb{I} - \pmb{P} is also idempotent.
• An idempotent matrix is always diagonalizable and its eigenvalues are either 0 or 1.
• The trace of an idempotent matrix equals the rank of the matrix and thus is always an integer.

In linear regression, the optimal solution of \mathcal{L} = ||\pmb{Y} - \pmb{XW}||^2_F with respect to \pmb{W} is \pmb{W}^* = (\pmb{X}^T\pmb{X})^{-1} \pmb{X}^T\pmb{Y} = \pmb{X}^\dagger \pmb{Y}.

The residual error is then given by:

\begin{equation} E = (\pmb{Y} - \pmb{XW}^*) = (\pmb{Y} - \pmb{X}(\pmb{X}^T\pmb{X})^{-1} \pmb{X}^T\pmb{Y}) = [\pmb{I} - \pmb{X}(\pmb{X}^T\pmb{X})^{-1} \pmb{X}^T] \pmb{Y} = [\pmb{I} - \pmb{X}\pmb{X}^\dagger] \pmb{Y} = \pmb{Q}\pmb{Y} \end{equation}

The matrices \pmb{P} = \pmb{X}\pmb{X}^\dagger and \pmb{Q} = [\pmb{I} - \pmb{X}\pmb{X}^\dagger] = [\pmb{I} - \pmb{P}] are symmetric and idempotent matrices.

• An idempotent linear operator \pmb{P} is a projection operator on the range space \mathcal{R}(\pmb{P}) along its null space \mathcal{N}(\pmb{P}).
• \pmb{P} is an orthogonal projection operator \Leftrightarrow it is idempotent and symmetric. Ex: \pmb{P} = \left[\begin{array}{cc} \pmb{I} & \pmb{0} \\ \pmb{0} & \pmb{0} \end{array} \right]

Orthogonal Complement and Projection Matrix

"A projection is a linear transformation \pmb{P} from a vector space V to itself such that \pmb{P}^2 = \pmb{P} (i.e. \pmb{P} is idempotent). That is, whenever \pmb{P} is applied twice to any value, it gives the same result as if it were applied once (idempotent). It leaves its image unchanged.

Let V be a finite dimensional vector space and \pmb{P} be a projection on V. Suppose the subspaces \mathcal{R} and \mathcal{N} are the range and null space (aka kernel) of \pmb{P} respectively. Then \pmb{P} has the following properties:

1. \pmb{P} is idempotent by def (i.e. \pmb{P}^2 = \pmb{P})
2. \pmb{P} is the identity operator \pmb{I} on \mathcal{R} (i.e. \forall \pmb{x} \in \mathcal{R}: \pmb{Px} = \pmb{x})
3. We have a direct sum V = \mathcal{R} \oplus \mathcal{N}. Every vector \pmb{x} \in V may be decomposed uniquely as \pmb{x} = \pmb{r} + \pmb{n} with \pmb{r} = \pmb{Px} \in \mathcal{R} and \pmb{n} = \pmb{x} − \pmb{Px} = (\pmb{I} - \pmb{P}) \pmb{x} \in \mathcal{N}.

The range and kernel of a projection are complementary, as are \pmb{P} and \pmb{Q} = \pmb{I} − \pmb{P}. The operator \pmb{Q} is also a projection, and the range and null spaces of \pmb{P} become the null and range spaces of \pmb{Q} and vice versa. We say \pmb{P} is a projection along \mathcal{N} onto \mathcal{R}, and \pmb{Q} is a projection along \mathcal{R} onto \mathcal{N}."

Orthogonal Projection

When the vector space V has an inner product and is complete (i.e. it is a Hilbert space), the concept of orthogonality can be used. An orthogonal projection is a projection for which the range \mathcal{R} and the null space \mathcal{N} are orthogonal subspaces. That is, \forall \pmb{r} \in \mathcal{R}, \forall \pmb{n} \in \mathcal{N}: \pmb{r} . \pmb{n} = 0.

The orthogonal complement of a subspace W of a vector space V equipped with a bilinear form B is the set W^\perp of all vectors in V that are orthogonal to every vector in W.

SVD

Any rectangular matrices \pmb{A} \in \mathbb{R}^{N \times M} can be decomposed into a product of 3 matrices (which can be seen as the building blocks of \pmb{A}):

\begin{equation} \pmb{A} = \pmb{U_A}\pmb{\Sigma_A}\pmb{V_A}^T \end{equation} where:

• \pmb{U_A} \in \mathbb{R}^{N \times N} is an orthogonal matrix (i.e. \pmb{U_A}^T = \pmb{U_A}^{-1} thus \pmb{U_A}\pmb{U_A}^T = \pmb{U_A}^T\pmb{U_A} = \pmb{I}). The columns of \pmb{U_A} contains the eigenvectors of the PSD (thus symmetric) \pmb{A}\pmb{A}^T, and are also known as the left-singular vectors of \pmb{A}. Its first columns associated with non-zero singular values span the range of \pmb{A}. Note that because \pmb{U_A} is an orthogonal matrix, it is invertible and thus has full rank, i.e. its columns form a basis and span \mathbb{R}^N.
• \pmb{\Sigma_A} \in \mathbb{R}^{N \times M} is a rectangular matrix. The upper left submatrix is a diagonal matrix of size r \times r with r = \min(N,M) while the rest is filled with zeros. The diagonal elements are the singular values ordered by descending order. The singular values (SVs) \sigma_i are equals to the square roots of the eigenvalues \lambda_i of \pmb{A}\pmb{A}^T and \pmb{A}^T\pmb{A}. The rank of \pmb{A} is given by the number of SVs different from 0.
• \pmb{V_A} \in \mathbb{R}^{M \times M} is an orthogonal matrix (i.e. \pmb{V_A}^T = \pmb{V_A}^{-1} thus \pmb{V_A}\pmb{V_A}^T = \pmb{V_A}^T\pmb{V_A} = \pmb{I}). The columns of \pmb{V_A} contains the eigenvectors of the PSD (thus symmetric) \pmb{A}^T\pmb{A}, and are also known as the right-singular vectors of \pmb{A}. Its last columns associated with vanishing singular values \sigma_i = 0 span the null space of \pmb{A}. Note that because \pmb{V_A} is an orthogonal matrix, it is invertible and thus has full rank, i.e. its columns form a basis and span \mathbb{R}^M.

Few notes:

• SVD can be seen as a generalization of eigendecomposition.
• SVD can be compressed such that \pmb{A} = \pmb{\tilde{U}_A}\pmb{\tilde{\Sigma}_A}\pmb{\tilde{V_A}^T} with \pmb{\tilde{U}_A} \in \mathbb{R}^{N \times r}, \pmb{\tilde{\Sigma}_A} \in \mathbb{R}^{r \times r}, \pmb{V_A} \in \mathbb{R}^{M \times r}, and r=\min(N,M).

Some properties:

• Existence:
• Uniqueness:
• Transpose: \pmb{A}^T = (\pmb{U_A}\pmb{\Sigma_A}\pmb{V_A}^T)^T = \pmb{V_A}\pmb{\Sigma_A}^T\pmb{U_A}^T
• If the matrix A is symmetric, then it has real eigenvalues.
• Pseudo-inverse: The pseudo-inverse of \pmb{A} is given by \pmb{A^\dagger} = \pmb{V_A}\pmb{\Sigma_A}^{-1}\pmb{U_A}^T with \pmb{\Sigma_A}^{-1} containing the inverse of singular values on its diagonal.
• Null and range space of \pmb{A}: \mathcal{N}(\pmb{A}) \equiv \mathcal{R}^\perp(\pmb{A}^T) and \mathcal{R}(\pmb{A}) \equiv \mathcal{N}^\perp(\pmb{A}^T). This can be seen by applying SVD on \pmb{A} and \pmb{A}^T. The last columns of \pmb{V_A} associated with vanishing SVs spans the null-space of \pmb{A}, and the first columns of \pmb{V_A} associated with non-zero SVs spans the range space of \pmb{A}^T. Because of the orthogonality property of \pmb{V_A}, we have the r first columns are orthogonals to the M-r last columns. dim(\mathcal{R}(\pmb{A})) + dim(\mathcal{N}(\pmb{A})) = M

Block Matrices

PCA

The (MSE) loss minimized by PCA is:

\begin{equation} \mathcal{L} = ||\pmb{X} - \pmb{XWW}^T||^2_F \end{equation}

where is the \pmb{\tilde{X}} = \pmb{XW} is the projected data on the lower dimensional space, and \pmb{\hat{X}} = \pmb{\tilde{X}W}^T is the data projected back to the original space.

Taking the gradient of this loss with respect to \pmb{W} and setting it to 0 gives us the minimum:

\begin{align} \nabla_{\pmb{W}} \mathcal{L} &= \nabla_{\pmb{W}} tr((\pmb{X} - \pmb{XWW}^T)^T(\pmb{X} - \pmb{XWW}^T)) \ &= \nabla_{\pmb{W}} tr((\pmb{X}^T - \pmb{WW}^T \pmb{X}^T) (\pmb{X} - \pmb{XWW}^T)) \ &= \nabla_{\pmb{W}} [tr(\pmb{X}^T\pmb{X}) - tr(\pmb{WW}^T \pmb{X}^T\pmb{X}) - tr(\pmb{X}^T\pmb{X}\pmb{WW}^T) + tr(\pmb{WW}^T\pmb{X}^T\pmb{XWW}^T)] \ &= \nabla_{\pmb{W}} [- 2 tr(\pmb{W}^T \pmb{\Sigma_{XX}} \pmb{W}) + tr(\pmb{WW}^T \pmb{\Sigma_{XX}} \pmb{WW}^T)] \ &= - 2 (\pmb{\Sigma_{XX}} \pmb{W} + \pmb{\Sigma_{XX}}^T \pmb{W}) + (\pmb{W}^T \pmb{\Sigma_{XX}} \pmb{WW}^T)^T + (\pmb{\Sigma_{XX}} \pmb{WW}^T)\pmb{W} + (\pmb{WW}^T \pmb{\Sigma_{XX}})^T (\pmb{W}^T)^T + (\pmb{WW}^T \pmb{\Sigma_{XX}} \pmb{W}) \ &= - 4 \pmb{\Sigma_{XX}} \pmb{W} + \pmb{WW}^T \pmb{\Sigma_{XX}}^T \pmb{W} + \pmb{\Sigma_{XX}} \pmb{WW}^T\pmb{W} + \pmb{\Sigma_{XX}}^T\pmb{WW}^T \pmb{W} + \pmb{WW}^T \pmb{\Sigma_{XX}} \pmb{W} \ &= - 4 \pmb{\Sigma_{XX}} \pmb{W} + 2 \pmb{WW}^T \pmb{\Sigma_{XX}} \pmb{W} + 2 \pmb{\Sigma_{XX}} \pmb{WW}^T\pmb{W} \ &= 0 \ \Leftrightarrow & \quad \pmb{WW}^T \pmb{\Sigma_{XX}} \pmb{W} + \pmb{\Sigma_{XX}} \pmb{WW}^T\pmb{W} = 2 \pmb{\Sigma_{XX}} \pmb{W} \ \Leftrightarrow & \quad \pmb{WW}^T \pmb{\Sigma_{XX}} + \pmb{\Sigma_{XX}} \pmb{WW}^T = 2 \pmb{\Sigma_{XX}} \ \Leftrightarrow & \quad \pmb{WW}^T + \pmb{\Sigma_{XX}} \pmb{WW}^T \pmb{\Sigma_{XX}}^{-1} = 2 \pmb{I} \ \Leftrightarrow & \quad \pmb{\Sigma_{XX}} = (2 \pmb{I} - \pmb{WW}^T) \pmb{\Sigma_{XX}} (\pmb{WW}^T)^{-1} \ \Leftrightarrow & \quad \pmb{Q\Lambda Q}^T = (2 \pmb{I} - \pmb{WW}^T) \pmb{Q\Lambda Q}^T (\pmb{WW}^T)^{-1} \ \Leftrightarrow & \quad \left{ \begin{array}{l} (2 \pmb{I} - \pmb{WW}^T) \pmb{Q} = \pmb{Q} \ \pmb{Q}^T (\pmb{WW}^T)^{-1} = \pmb{Q}^T : \Leftrightarrow : \pmb{QQ}^T = \pmb{WW}^T \end{array} \right. \ \end{align}

Algo using the covariance:

1. subtract the mean \pmb{X}
2. compute the covariance matrix \pmb{\Sigma_{XX}} = \pmb{X}^T\pmb{X}
3. compute the eigendecomposition of \pmb{\Sigma_{XX}}, i.e. compute \pmb{Q} and \pmb{\Lambda} such that \pmb{\Sigma_{XX}} = \pmb{Q} \pmb{\Lambda} \pmb{Q}^T.
4. return the sorted evals and the corresponding evecs

Algo using SVD:

1. substract the mean from \pmb{X}
2. compute SVD of \pmb{X}, i.e. \pmb{X} = \pmb{U_X \Sigma_X V_X}^T. The eigenvectors are given by \pmb{Q} = \pmb{V_X} and the evals by \pmb{\Lambda} = \pmb{\Sigma_X}^2.

Recursive PCA/SVD

Hierarchical PCA/SVD

Linear FNN with 1 hidden layer

Assume a linear Feedforward Neural Network (FNN) with 1 input, 1 hidden, and 1 output layer. Assume the input data is given by \pmb{X} \in \mathbb{R}^{N \times D_x}, the output data by \pmb{Y} \in \mathbb{R}^{N \times D_y}, the hidden data by \pmb{H} \in \mathbb{R}^{N \times D_h}, the input-hidden weight matrix by \pmb{W_1} \in \mathbb{R}^{D_x \times D_h} and the hidden-output weight matrix \pmb{W_2} \in \mathbb{R}^{D_h \times D_y}. These variables are related by the following relationships:

\begin{equation} \pmb{H} = \pmb{X} \pmb{W_1} \qquad \mbox{and} \qquad \pmb{Y} = \pmb{H} \pmb{W_2} \end{equation}

The MSE loss is thus defined as:

\begin{equation} \mathcal{L} = ||\pmb{Y} - \pmb{XW_1W_2}||^2_F \end{equation}

Taking the gradient of this loss with respect to \pmb{W_1} and \pmb{W_2}, and setting these to zero gives us the minimum:

\begin{align} \nabla_{\pmb{W_1}} \mathcal{L} &= \nabla_{\pmb{W_1}} ||\pmb{Y} - \pmb{XW_1W_2}||^2_F \ &= \nabla_{\pmb{W_1}} tr((\pmb{Y} - \pmb{XW_1W_2})^T(\pmb{Y} - \pmb{XW_1W_2})) \ &= \nabla_{\pmb{W_1}} [tr(\pmb{Y}^T\pmb{Y}) - tr((\pmb{XW_1W_2})^T\pmb{Y}) - tr(\pmb{Y}^T\pmb{XW_1W_2}) + tr(\pmb{W}^T\pmb{X}^T\pmb{X}\pmb{W})] \ &= \nabla_{\pmb{W_1}} [tr(\pmb{Y}^T\pmb{Y}) - tr(\pmb{Y}^T\pmb{XW_1W_2}) - tr(\pmb{Y}^T\pmb{XW_1W_2}) + tr(\pmb{(W_1W_2)}^T\pmb{X}^T\pmb{X}\pmb{(W_1W_2)})] \ &= \nabla_{\pmb{W_1}} [-2 tr(\pmb{Y}^T\pmb{XW_1W_2}) + tr(\pmb{W_2}^T\pmb{W_1}^T\pmb{X}^T\pmb{X}\pmb{W_1}\pmb{W_2})] \ &= \nabla_{\pmb{W_1}} [-2 tr(\pmb{\Sigma_{YX}}\pmb{W_1W_2}) + tr(\pmb{W_2}^T\pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1}\pmb{W_2})] \ &= -2 (\pmb{W_2}\pmb{\Sigma_{YX}})^T + \pmb{\Sigma_{XX}}\pmb{W_1W_2W_2}^T + \pmb{\Sigma_{XX}}^T\pmb{W_1}(\pmb{W_2W_2}^T)^T \ &= -2 \pmb{\Sigma_{XY}} \pmb{W_2}^T + 2 \pmb{\Sigma_{XX}}\pmb{W_1W_2W_2}^T \ &= 0 \ \Leftrightarrow & \quad \pmb{\Sigma_{XX}}\pmb{W_1W_2W_2}^T = \pmb{\Sigma_{XY}} \pmb{W_2}^T \ \Leftrightarrow & \quad \pmb{W_1} = \pmb{\Sigma_{XX}}^{-1} \pmb{\Sigma_{XY}} \pmb{W_2}^T (\pmb{W_2W_2}^T)^{-1} \end{align}

and

\begin{align} \nabla_{\pmb{W_2}} \mathcal{L} &= \nabla_{\pmb{W_2}} [-2 tr(\pmb{\Sigma_{YX}}\pmb{W_1W_2}) + tr(\pmb{W_2}^T\pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1}\pmb{W_2})] \ &= -2 (\pmb{\Sigma_{YX}}\pmb{W_1})^T + \pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1}\pmb{W_2} + (\pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1})^T\pmb{W_2} \ &= -2 \pmb{W_1}^T\pmb{\Sigma_{XY}} + 2 \pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1}\pmb{W_2} \ &= 0 \ \Leftrightarrow & \quad \pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1}\pmb{W_2} = \pmb{W_1}^T \pmb{\Sigma_{XY}} \ \Leftrightarrow & \quad \pmb{W_2} = (\pmb{W_1}^T\pmb{\Sigma_{XX}}\pmb{W_1})^{-1} \pmb{W_1}^T \pmb{\Sigma_{XY}} \ \end{align}

In [ ]: