Singular Values and Spectral Decomposition

Consider two matrices:

$$A_1=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right],A_2=\left[\begin{array}{cc}1& 0\\ 0& {10}^{-3}\end{array}\right].$$

Both have rank two. Yet they do not preserve a two-dimensional signal equally well. $A_1$ leaves the unit circle unchanged, while $A_2$ squeezes it into a needle: the second coordinate still exists in exact arithmetic, but a small amount of noise can overwhelm it.

Rank cannot express this difference. It tells us whether a direction has vanished, but not how close that direction is to vanishing. What we want instead is a description of a matrix in terms of the input directions it acts on, the amount of stretching along each direction, and the output directions those inputs become.

That description is the singular value decomposition. We will build it from the two-dimensional picture, then use the same picture to understand the singular-value spectrum, $A^{T}A$, low-rank approximation, spectral decomposition, and the spectral norm.

Singular value decomposition

For a real matrix $A\in {\mathbb{R}}^{m\times n}$, the singular value decomposition is

$$A=U\mathrm{\Sigma }{V}^T.$$

$U$ and $V$ are orthogonal matrices, and $\mathrm{\Sigma }$ is rectangular and diagonal. Its diagonal entries are the singular values,

$${\sigma }_1\ge {\sigma }_2\ge \cdots \ge 0.$$

The product becomes useful when we read it in the order in which it acts on a vector:

$$x\stackrel{V^T}{\to }{V}^{T}x\stackrel{\mathrm{\Sigma }}{\to }\mathrm{\Sigma }{V}^{T}x\stackrel{U}{\to }U\mathrm{\Sigma }{V}^{T}x.$$

Let $v_i$ be the $i$-th column of $V$. The $i$-th entry of $V^{T}x$ is

$$(V^{T}x{)}_i=v_i^{T}x,$$

so $V^T$ reads the coordinate of $x$ along the input direction $v_i$. The diagonal matrix $\mathrm{\Sigma }$ multiplies that coordinate by ${\sigma }_i$, and $U$ places the result along the output direction $u_i$. Equivalently,

$$A{v}_i={\sigma }_i{u}_i.$$

This equation contains the geometry of the whole decomposition. The $v_i$ form orthogonal input directions; the $u_i$ form orthogonal output directions; and ${\sigma }_i$ records how much length survives between them. If ${\sigma }_i=0$, the entire $v_i$ direction is erased.

Move ${\theta }_V$, ${\sigma }_1$, ${\sigma }_2$, and ${\theta }_U$ below; the panels follow a fixed vector through $x$, $V^{T}x$, $\mathrm{\Sigma }{V}^{T}x$, and $U\mathrm{\Sigma }{V}^{T}x$.

θ_V

30°

Input-basis angleσ₁

2.10

First singular valueσ₂

0.65

Second singular valueθ_U

-20°

Output-basis angle

x

Vᵀx

ΣVᵀx

UΣVᵀx = Ax

The unit sphere now has a simple fate. $V^T$ and $U$ preserve lengths because they are orthogonal. Only $\mathrm{\Sigma }$ changes length, so $A$ maps the unit sphere to an ellipsoid whose semiaxis lengths are the singular values and whose semiaxis directions are the columns of $U$.

What singular values measure

The opening example has singular values $(1,1)$ for $A_1$ and $(1,{10}^{-3})$ for $A_2$. Rank sees only that both pairs contain two nonzero numbers:

$$\mathrm{rank}(A)=\mathrm{\#}\{i:{\sigma }_i>0\}.$$

The singular values tell us what rank leaves out. A value of zero means that a direction is gone. A very small value means that the direction survives algebraically but is fragile numerically, because recovering it requires division by a small number.

For a full-rank square matrix, this fragility is summarized by the two-norm condition number,

$${\kappa }_2(A)=\frac{{\sigma }_1}{{\sigma }_n}.$$

The condition number of $A_2$ is ${10}^3$: an error aligned with its weak direction can be amplified by a factor on that scale when the map is inverted. If ${\sigma }_n=0$, the inverse does not exist and the condition number is infinite.

Singular-value spectrum

The ordered list $({\sigma }_1,{\sigma }_2,\dots )$ is called the singular-value spectrum. The qualifier matters. The spectrum of a square matrix usually means its eigenvalues, which may be negative or complex; singular values exist for rectangular matrices and are always nonnegative real numbers.

In high dimensions, the tail of this list is often more informative than exact rank. Given a threshold $\epsilon$, we can define an effective rank

$$r_{\epsilon }=\mathrm{\#}\{i:{\sigma }_i\ge \epsilon \}.$$

Given a truncation point $k$, we can also measure how much squared singular-value mass is retained:

$$E_k=\frac{\sum _{i=1}^k{\sigma }_i^2}{\sum _i{\sigma }_i^2}.$$

A flat spectrum spreads scale across many directions. A decaying spectrum concentrates it near the front, so a matrix may have full exact rank while behaving like a much lower-dimensional map.

The example below uses ${\sigma }_i=e^{-\alpha (i-1)}$; adjust $\alpha$, $\epsilon$, and $k$ to see how spectral decay changes effective rank, retained energy, and truncation error.

Decay α

0.24

Threshold ε

0.08

Truncation rank k

5

ε

σ₁

σ₂

σ₃

σ₄

σ₅

σ₆

σ₇

σ₈

σ₉

σ₁₀

σ₁₁

σ₁₂

σ₁₃

σ₁₄

σ₁₅

σ₁₆

Effective rank

r_ε = 11

Retained energy

E₅ = 91.0%

Truncation error

σ₆ = 0.301

This is the quantitative advantage of singular values: they turn the binary statement “a direction exists” into a graded description of how strongly that direction survives.

SVD from AᵀA

The SVD tells us that the important input directions are the columns of $V$. How can we recover those directions from $A$?

The useful observation is that squared output length is a quadratic form:

$$\Vert Ax{\Vert }_2^2=x^T{A}^{T}Ax.$$

The matrix $A^{T}A$ therefore records how much $A$ stretches every input direction, without retaining the final orientation of the output. Substituting the SVD makes this cancellation explicit:

$$\begin{array}{rl}{A}^{T}A& =(U\mathrm{\Sigma }{V}^T{)}^T(U\mathrm{\Sigma }{V}^T)\\ & =V{\mathrm{\Sigma }}^T{U}^{T}U\mathrm{\Sigma }{V}^T\\ & =V{\mathrm{\Sigma }}^T\mathrm{\Sigma }{V}^T.\end{array}$$

Because $U^{T}U=I$, the left singular directions disappear. What remains is an orthogonal spectral decomposition of $A^{T}A$:

$$A^{T}A{v}_i={\sigma }_i^2{v}_i.$$

Thus the right singular vectors are eigenvectors of $A^{T}A$, and the singular values are the square roots of its eigenvalues. For every positive singular value we can then recover the corresponding left singular vector from

$$u_i=\frac{A{v}_i}{{\sigma }_i}.$$

When ${\sigma }_i=0$, this formula is undefined; the remaining columns of $U$ are supplied by an orthonormal completion. The same argument on the output side gives

$$A{A}^T{u}_i={\sigma }_i^2{u}_i.$$

There is one common shortcut worth avoiding. For an arbitrary vector $x$,

$$A^{T}Ax=V{\mathrm{\Sigma }}^T\mathrm{\Sigma }{V}^{T}x,$$

not ${\mathrm{\Sigma }}^{2}x$. The diagonal action appears only after $x$ has been expressed in the $V$ basis, or when $x$ itself is one of the $v_i$.

This eigendecomposition route explains the mathematics, but it is not the preferred numerical implementation of SVD. Explicitly forming $A^{T}A$ squares the condition number and can destroy information in weak directions; stable algorithms work on $A$ more directly, typically through bidiagonalization.

Spectral decomposition

The relation $A{v}_i={\sigma }_i{u}_i$ describes what $A$ does to one special direction. To recover its action on an arbitrary vector, expand

$$x=\sum _i(v_i^{T}x)v_i.$$

Applying $A$ gives

$$Ax=\sum _i{\sigma }_i(v_i^{T}x)u_i.$$

Since this holds for every $x$, the matrix itself can be written as

$$A=\sum _i{\sigma }_i{u}_i{v}_i^T.$$

Each rank-one term is a directional channel: $v_i^T$ reads one input coordinate, ${\sigma }_i$ scales it, and $u_i$ writes it into the output space. Keeping only the first $k$ channels gives the truncated SVD,

$$A_k=\sum _{i=1}^k{\sigma }_i{u}_i{v}_i^T.$$

The Eckart–Young theorem says that $A_k$ is the best rank-$k$ approximation to $A$ in both the spectral and Frobenius norms. In the spectral norm, the error has an especially clean form:

$$\Vert A-A_k{\Vert }_2={\sigma }_{k+1}.$$

This is why the tail of the bar chart matters: the first discarded bar is exactly the worst remaining directional error.

The rank-one expansion is sometimes loosely called a spectral decomposition, but the distinction is useful. A real symmetric matrix has an orthogonal eigendecomposition

$$A=Q\mathrm{\Lambda }{Q}^T=\sum _i{\lambda }_i{q}_i{q}_i^T.$$

Here the same direction $q_i$ appears on both sides. For a general matrix, the input and output directions $v_i$ and $u_i$ need not agree, so its SVD is not an orthogonal spectral decomposition of $A$ itself.

If $A$ is symmetric positive semidefinite, then its eigenvalues are nonnegative and its eigendecomposition aligns with its SVD. If $A$ is symmetric but indefinite, its singular values are $|{\lambda }_i|$ and the signs are absorbed into one set of singular vectors. For an arbitrary matrix, it is $A^{T}A$ and $A{A}^T$ that always have genuine orthogonal spectral decompositions:

$$A^{T}A=\sum _i{\sigma }_i^2{v}_i{v}_i^T,A{A}^T=\sum _i{\sigma }_i^2{u}_i{u}_i^T.$$

Spectral norm

One final quantity falls out of the same picture. The spectral norm asks for the largest factor by which $A$ can stretch a unit vector:

$$\Vert A{\Vert }_2=\underset{\Vert x{\Vert }_2=1}{max}\Vert Ax{\Vert }_2.$$

Set $y=V^{T}x$. Orthogonality gives $\Vert y{\Vert }_2=\Vert x{\Vert }_2=1$, and therefore

$$\begin{array}{rl}\Vert Ax{\Vert }_2^2& =\Vert U\mathrm{\Sigma }{V}^{T}x{\Vert }_2^2\\ & =\Vert \mathrm{\Sigma }y{\Vert }_2^2\\ & =\sum _i{\sigma }_i^2{y}_i^2\\ & \le {\sigma }_1^2\sum _i{y}_i^2\\ & ={\sigma }_1^2.\end{array}$$

The bound is attained by choosing $x=v_1$, so

$$\boxed{{\displaystyle \Vert A{\Vert }_2={\sigma }_1}}.$$

We can now answer the question posed at the start. A matrix is best understood as a collection of orthogonal directional channels,

$$v_i\stackrel{{\sigma }_i}{\to }{u}_i.$$

Rank counts how many channel gains are nonzero. The singular-value spectrum records all their strengths. Truncation removes the weak channels in the best possible way, and the spectral norm is the gain of the strongest channel. This picture applies to every real matrix; only when the matrix is symmetric do its input and output directions align into an orthogonal spectral decomposition of the matrix itself.