Example 1

To demonstrate how to do a singular value decomposition; suppose we start with

$A = \begin{pmatrix} 1.04 & 0.656 & 1.808 & 1.52 \\ -0.28 & 2.008 & 1.144 & 2.36 \\ -0.5 & -0.26 & -1.18 & 1.1 \end{pmatrix}$

We calculate both $AA^\textrm{T}$ and $A^\textrm{T}A$:

$AA^\textrm{T} = \begin{pmatrix} 7.0912 & 6.6816 & -1.152 \\ 6.6816 & 10.9888 & 0.864 \\ -1.152 & 0.864 & 2.92 \end{pmatrix}$

$A^\textrm{T}A = \begin{pmatrix} 1.41 & 0.25 & 2.15 & 0.37 \\ 0.25 & 4.53 & 3.79 & 5.45 \\ 2.15 & 3.79 & 5.97 & 4.15 \\ 0.37 & 5.45 & 4.15 & 9.09 \end{pmatrix}$

Calculating the eigenvalues of these matrices is a little more difficult, but we only need to do so for the smaller one:

$\textrm{det}(\lambda I_3 - AA^\textrm{T}) = \begin{pmatrix} \lambda - 7.0912 & -6.6816 & 1.152 \\ -6.6816 & \lambda - 10.9888 & -0.864 \\ 1.152 & -0.864 & \lambda - 2.92 \end{pmatrix}$

Fortunately, the matrix is symmetric (which is not true with the eigendecomposition), so it is a little easier:

$(\lambda - 7.0912)(\lambda - 10.9888)(\lambda - 2.92) + 2(-6.6816)(1.152)(-0.864) - (\lambda - 7.0912) 0.864^2 - (\lambda - 10.9888) 1.152^2 - (\lambda - 2.92) 6.6816^2$
\lambda^3 - 21\lambda^2 + 84\lambda - 64

which has the roots $16$, $4$ and $1$, and thus the singular values are $4$, $2$ and $1$.

We must find eigenvectors of the matrices $AA^\textrm{T}$ and $A^\textrm{T}A$ where the eigenvalues of the second matrix are $4$, $2$, $1$ and $0$. You can use mathematical software to check that the eigenvectors are those indicated above.

Example 2

Using one of the nicest $2 \times 3$ matrices, we have

$A = \begin{pmatrix} 1.2 & 0.48 & 0.64 \\ 1.6 & -0.36 & -0.48 \end{pmatrix}$

We calculate both $AA^\textrm{T}$ and $A^\textrm{T}A$:

$AA^\textrm{T} = \begin{pmatrix} 2.08 & 1.44 \\ 1.44 & 2.92 \end{pmatrix}$

$A^\textrm{T}A = \begin{pmatrix} 4 & 0 & 0 \\ 0 & 0.36 & 0.48 \\ 0 & 0.48 & 0.64 \end{pmatrix}$

With these nicest matrices, calculating the eigenvalues is easier: $(\lambda - 2.08)(\lambda - 2.92) - 1.44^2 = (\lambda^2 - 5\lambda + 6.0736) - 2.0736$ which equals $\lambda^2 - 5\lambda + 4 = (\lambda - 4)(\lambda - 1)$, which has the eigenvalues $4$ and $1$ and thus we know the singular values are $2$ and $1$ (the square roots of the eigenvalues). Recall that the convention is to list the singular values from largest to smallest as all singular values are guaranteed to be non-negative real numbers. The eigenvalues of $A^\textrm{T}A$ are $4$, $1$ and $0$, and you can almost see this as $A^\textrm{T}A$ is block diagonal, so $4$ is an eigenvalue, and the other $2 \times 2$ matrix on the diagonal is singular, as the third row is a scalar multiple ($\frac{4}{3}$) of the second.

For $U$, to find a first eigenvector corresponding to $\lambda_1 = 4$, we find a basis for the null space by solving $4 \cdot I_3 - AA^\textrm{T} = \textbf{0}_3$ or:

$\begin{pmatrix} 4 - 2.08 & -1.44 \\ -1.44 & 4 - 2.92 \end{pmatrix} = \begin{pmatrix} 1.92 & -1.44 \\ -1.44 & 1.08 \end{pmatrix}$

Because the matrix must be singular, it's necessary the second row is a scalar multiple of the first, so we can just eliminate it:

$\sim \begin{pmatrix} 1.92 & -1.44 \\ 0 & 0 \end{pmatrix}$

and so $\alpha_2$ is free and $\alpha_1 = 0.75 \alpha_2$, so an eigenvector is $\textbf{u}_1 = \begin{pmatrix} 0.75 \\ 1 \end{pmatrix}$ and $\|\textbf{u}_1\|_2 = 1.25$, so $\hat{\textbf{u}}_1 \begin{pmatrix} 0.6 \\ 0.8 \end{pmatrix}$.

For $U$, to find a second eigenvector corresponding to $\lambda_2 = 1$, we find a basis for the null space by solving $1 \cdot I_3 - AA^\textrm{T} = \textbf{0}_3$ or:

$\begin{pmatrix} 1 - 2.08 & -1.44 \\ -1.44 & 1 - 2.92 \end{pmatrix} = \begin{pmatrix} -1.08 & -1.44 \\ -1.44 & -1.92 \end{pmatrix}$

Because the matrix must be singular, it's necessary the second row is a scalar multiple of the first, so we can just eliminate it:

$\sim \begin{pmatrix} -1.08 & -1.44 \\ 0 & 0 \end{pmatrix}$

and so $\alpha_2$ is free and $\alpha_1 = -1.3333 \alpha_2$, so an eigenvector is $\textbf{u}_2 = \begin{pmatrix} -1.3333 \\ 1 \end{pmatrix}$ and $\|\textbf{u}_2\|_2 = 1.6666$, so $\hat{\textbf{u}}_2 \begin{pmatrix} -0.8 \\ 0.6 \end{pmatrix}$.

Thus, the matrix $U = \begin{pmatrix} 0.6 & -0.8 \\ 0.8 & 0.6 \end{pmatrix}$.

For $V$, to find a first eigenvector corresponding to $\lambda_1 = 4$, we find a basis for the null space by solving $4 \cdot I_3 - A^\textrm{T}A = \textbf{0}_3$ or:

$\begin{pmatrix} 4 - 4 & 0 & 0 \\ 0 & 4 - 0.36 & -0.48 \\ 0 & -0.48 & 4 - 0.64 \end{pmatrix} = \begin{pmatrix} 0 & 0 & 0 \\ 0 & 3.64 & -0.48 \\ 0 & -0.48 & 3.36 \end{pmatrix}$

The first row being all zeros immediately indicate that the matrix is singular, and as the dimension of the null space is only $1$, then we are guaranteed that Row 3 cannot be a scalar multiple of Row 2. Thus, if we add an appropriate multiple of Row 2 onto Row 3, we get:

$\sim \begin{pmatrix} 0 & 0 & 0 \\ 0 & 3.64 & -0.48 \\ 0 & 0 & * \end{pmatrix}$

The asterisk signals that this is a non-zero value, but we don't care what it is, we only care that it is non-zero, for we now have that $\alpha_3 = 0$, and substituting this into Row 2, we have that $\alpha_2 = 0$, as well, and the only free variable is $\alpha_1$, so an eigenvector is $\hat{\textbf{v}}_1 = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}$, which is normalized.

For $V$, to find a second eigenvectors corresponding to $\lambda_2 = 1$, we find a basis for the null space by solving $1 \cdot I_3 - A^\textrm{T}A = \textbf{0}_3$ or:

$\begin{pmatrix} 1 - 4 & 0 & 0 \\ 0 & 1 - 0.36 & -0.48 \\ 0 & -0.48 & 1 - 0.64 \end{pmatrix} = \begin{pmatrix} -3 & 0 & 0 \\ 0 & 0.64 & -0.48 \\ 0 & -0.48 & 0.36 \end{pmatrix}$

Because the matrix must be singular, it's necessary the third row is a scalar multiple of the second, so we can just eliminate it (or you can do the arithmetic necessary to ensure you are correct):

$\sim \begin{pmatrix} -3 & 0 & 0 \\ 0 & 0.64 & -0.48 \\ 0 & 0 & 0 \end{pmatrix}$

and so $\alpha_3$ is free and $\alpha_2 = 0.75 \alpha_3$ ($64 = 4 \times 16$ and $48 = 3 \times 16$), and $\alpha_1 = 0$, so an eigenvector is $\textbf{v}_2 = \begin{pmatrix} 0 \\ 0.75 \\ 1 \end{pmatrix}$ and $\|\textbf{v}_2\|_2 = 1.25$, so $\hat{\textbf{v}}_2 \begin{pmatrix} 0 \\ 0.6 \\ 0.8 \end{pmatrix}$.

For $V$, to find a third eigenvectors corresponding to $\lambda_3 = 0$, we find a basis for the null space by solving $0\cdot I_3 - A^\textrm{T}A = \textbf{0}_3$ or:

$\begin{pmatrix} 0 - 4 & 0 & 0 \\ 0 & 0 - 0.36 & -0.48 \\ 0 & -0.48 & 0 - 0.64 \end{pmatrix} = \begin{pmatrix} -4 & 0 & 0 \\ 0 & -0.36 & -0.48 \\ 0 & -0.48 & -0.64 \end{pmatrix}$

Because the matrix must be singular, it's necessary the third row is a scalar multiple of the second, so we can just eliminate it (or you can do the arithmetic necessary to ensure you are correct):

$\sim \begin{pmatrix} -3 & 0 & 0 \\ 0 & -0.36 & -0.48 \\ 0 & 0 & 0 \end{pmatrix}$

and so $\alpha_3$ is free and $\alpha_2 = -1.3333 \alpha_3$ ($36 = 3 \times 12$ and $48 = 4 \times 12$), and $\alpha_1 = 0$, so an eigenvector is $\textbf{v}_3 = \begin{pmatrix} 0 \\ -1.3333 \\ 1 \end{pmatrix}$ and $\|\textbf{v}_3\|_2 = 1.6666$, so $\hat{\textbf{v}}_3 \begin{pmatrix} 0 \\ -0.8 \\ 0.6 \end{pmatrix}$.

Thus, the matrix $V = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 0.6 & -0.8 \\ 0 & 0.8 & 0.6 \end{pmatrix}$.

Example 2

Using the transpose of the nicest $3 \times 4$ matrices, we have

$A = \begin{pmatrix} 0.4 & 1.7 & 0.6 \\ 2 & -1.3& 0.6 \\ 2 & 1.1 & 1.8 \\ 2.8 & -0.1 & -1.8 \end{pmatrix}$

We calculate both $AA^\textrm{T}$ and $A^\textrm{T}A$:

$AA^\textrm{T} = \begin{pmatrix} 3.41 & -1.05 & 3.75 & -0.13 \\ -1.05 & 6.05 & 3.65 & 4.65 \\ 3.75 & 3.65 & 8.45 & 2.25 \\ -0.13 & 4.65 & 2.25 & 11.09 \end{pmatrix}$

$A^\textrm{T}A = \begin{pmatrix} 16 & 0 & 0 \\ 0 & 5.8 & 2.4 \\ 0 & 2.4 & 7.2 \end{pmatrix}$

With these nicest matrices, calculating the eigenvalues is easier: $(\lambda - 16)((\lambda - 5.8)(\lambda - 7.2) - 2.4^2) = (\lambda - 16)(\lambda^2 - 13\lambda + 41.76) - 5.76$ which equals $(\lambda - 16)(\lambda^2 - 13\lambda + 36) = (\lambda - 16)(\lambda - 9)(\lambda - 4)$, which has the eigenvalues $16$, $9$ and $4$ and thus we know the singular values are $4$, $3$ and $2$ (the square roots of the eigenvalues). Recall that the convention is to list the singular values from largest to smallest as all singular values are guaranteed to be non-negative real numbers. The eigenvalues of $AA^\textrm{T}$ are $4$, $3$, $2$ and $0$.

For $U$, to find a first eigenvector corresponding to $\lambda_1 = 4$, we find a basis for the null space by solving $4 \cdot I_4 - AA^\textrm{T} = \textbf{0}_4$ or:

$\begin{pmatrix} 4-3.41 & 1.05 & -3.75 & 0.13 \\ 1.05 & 4-6.05 & -3.65 & -4.65 \\ -3.75 & -3.65 & 4-8.45 & -2.25 \\ 0.13 & -4.65 & -2.25 & 4-11.09 \end{pmatrix} = \begin{pmatrix} 0.59 & 1.05 & -3.75 & 0.13 \\ 1.05 & -2.05 & -3.65 & -4.65 \\ -3.75 & -3.65 & -4.45 & -2.25 \\ 0.13 & -4.65 & -2.25 & -7.09 \end{pmatrix}$

This requires a calculator, but we get:

$\sim \begin{pmatrix} 0.59 & 1.05 & -3.75 & 0.13 \\ 0 & -3.91864 & 3.02373 & -4.88136 \\ 0 & 0 &-25.95156 & -5.19031 \\ 0 & 0 & 0 & 0 {pmatrix}$

and so $\alpha_4$ is free, so $\alpha_3 = -0.2 \alpha_4$, so $\alpha_2 = -1.4 \alpha_4$ and $\alpha_1 = \alpha_4$, so an eigenvector is $\textbf{u}_1 = \begin{pmatrix} 1 \\ -1.4 \\ -0.2 \\ 1 \end{pmatrix}$ and $\|\textbf{u}_1\|_2 = 2$, so $\hat{\textbf{u}}_1 \begin{pmatrix} 0.5 \\ -0.7 \\ -0.1 \\ 0.5 \end{pmatrix}$.

To be continued...