SVD

Let's walk through an example of Singular Value Decomposition (SVD).

Example: SVD of a 2x2 Matrix

Consider the matrix:

$$A=\left(\begin{array}{cc}3& 4\\ 2& 3\end{array}\right)$$

We will decompose this matrix using SVD.

Step 1: Compute the SVD of $A$

We want to find matrices $U$, $\mathrm{\Sigma }$, and $V^T$ such that:

$$A=U\mathrm{\Sigma }{V}^T$$

Where:
- $U$ is an orthogonal matrix containing the left singular vectors of $A$,
- $\mathrm{\Sigma }$ is a diagonal matrix containing the singular values of $A$,
- $V$ is an orthogonal matrix containing the right singular vectors of $A$.

Step 2: Find $A^{T}A$ and $A{A}^T$

To compute the SVD, we first calculate $A^{T}A$ and $A{A}^T$, as the eigenvalues of these matrices will give us the singular values.

1. $A^{T}A$:

$$A^T=\left(\begin{array}{cc}3& 2\\ 4& 3\end{array}\right)$$

Now, compute $A^{T}A$:

$$A^{T}A=\left(\begin{array}{cc}3& 2\\ 4& 3\end{array}\right)\left(\begin{array}{cc}3& 4\\ 2& 3\end{array}\right)=\left(\begin{array}{cc}9+4& 12+6\\ 12+6& 16+9\end{array}\right)=\left(\begin{array}{cc}13& 18\\ 18& 25\end{array}\right)$$

1. $A{A}^T$:

Now, compute $A{A}^T$:

$$A{A}^T=\left(\begin{array}{cc}3& 4\\ 2& 3\end{array}\right)\left(\begin{array}{cc}3& 2\\ 4& 3\end{array}\right)=\left(\begin{array}{cc}9+16& 6+12\\ 6+12& 4+9\end{array}\right)=\left(\begin{array}{cc}25& 18\\ 18& 13\end{array}\right)$$

Step 3: Compute the Eigenvalues and Eigenvectors

Now, we find the eigenvalues and eigenvectors of $A^{T}A$ and $A{A}^T$, as they will give us the singular values and the corresponding singular vectors.

1. Eigenvalues of $A^{T}A$:
   The eigenvalues of $A^{T}A$ are the singular values squared. We solve the characteristic equation $\text{det}(A^{T}A-\lambda I)=0$:

$$\left|\begin{array}{cc}13-\lambda & 18\\ 18& 25-\lambda \end{array}\right|=0$$

Solving this determinant gives the eigenvalues ${\lambda }_1=36$ and ${\lambda }_2=2$.

The singular values are the square roots of the eigenvalues of $A^{T}A$, so:

$${\sigma }_1=\sqrt{36}=6,{\sigma }_2=\sqrt{2}\approx 1.414$$

Step 4: Find the Singular Vectors

To find the singular vectors, we solve for the eigenvectors corresponding to the eigenvalues of $A^{T}A$.

1. Right singular vectors (columns of $V$):
2. For ${\lambda }_1=36$, the eigenvector can be calculated as $\left(\begin{array}{c}3\\ 4\end{array}\right)$.
3. For ${\lambda }_2=2$, the eigenvector is $\left(\begin{array}{c}-4\\ 3\end{array}\right)$.

After normalization, these eigenvectors are:

$$v_1=\frac{1}{\sqrt{25}}\left(\begin{array}{c}3\\ 4\end{array}\right)=\left(\begin{array}{c}0.6\\ 0.8\end{array}\right),v_2=\frac{1}{\sqrt{25}}\left(\begin{array}{c}-4\\ 3\end{array}\right)=\left(\begin{array}{c}-0.8\\ 0.6\end{array}\right)$$

So, the matrix $V$ is:

$$V=\left(\begin{array}{cc}0.6& -0.8\\ 0.8& 0.6\end{array}\right)$$

1. Left singular vectors (columns of $U$):
2. The left singular vectors can be computed by finding the eigenvectors of $A{A}^T$. After similar calculations, we find the left singular vectors:

$$u_1=\frac{1}{\sqrt{25}}\left(\begin{array}{c}3\\ 4\end{array}\right)=\left(\begin{array}{c}0.6\\ 0.8\end{array}\right),u_2=\frac{1}{\sqrt{25}}\left(\begin{array}{c}-4\\ 3\end{array}\right)=\left(\begin{array}{c}-0.8\\ 0.6\end{array}\right)$$

So, the matrix $U$ is:

$$U=\left(\begin{array}{cc}0.6& -0.8\\ 0.8& 0.6\end{array}\right)$$

Step 5: Construct the SVD

Now, we can construct the Singular Value Decomposition of the matrix $A$:

$$A=U\mathrm{\Sigma }{V}^T$$

where:

• $U=\left(\begin{array}{cc}0.6& -0.8\\ 0.8& 0.6\end{array}\right)$
• $\mathrm{\Sigma }=\left(\begin{array}{cc}6& 0\\ 0& \sqrt{2}\end{array}\right)$
• $V^T=\left(\begin{array}{cc}0.6& 0.8\\ -0.8& 0.6\end{array}\right)$

Thus, the SVD of $A$ is:

$$A=\left(\begin{array}{cc}0.6& -0.8\\ 0.8& 0.6\end{array}\right)\left(\begin{array}{cc}6& 0\\ 0& \sqrt{2}\end{array}\right)\left(\begin{array}{cc}0.6& 0.8\\ -0.8& 0.6\end{array}\right)$$

Conclusion

This decomposition reveals the singular values and vectors that describe how the matrix $A$ acts in terms of scaling and rotation. You can apply the SVD to reconstruct $A$, approximate $A$ for dimensionality reduction, or use it for solving linear systems, among other applications.