SVD, pseudoinverse, and PCA: SVD: singular value decomposition
Singular value decomposition
The singular value decomposition of a matrix, abbreviated as SVD, is widely used in statistics (for example, in principal components analysis and in the least-squares method for data regression), in signal processing (e.g., to reduce noise in a signal), and in even more disciplines, such as computer vision and related fields. In this theory page we only discuss what the singular value decomposition is and what it has to do with eigenvalues, but not how to calculate the SVD. In practice you do the computations anyway with numerical software.
We consider an matrix . Then is a symmetric matrix, and, according to the spectral theorem for symmetric matrices, diagonalizable with real eigenvalues. But in this case holds even more:
The eigenvalues of are nonnegative.
So you can take square roots of the eigenvalues.
Let be an matrix. Then the singular values of are the square roots of the eigenvalues of .
Singular values is commonly denoted by and arranged in descending order, that is, .
Thus, the singular values of are .
Before we can formulate the singular value decomposition we must define the notion of orthogonal matrix.
A real square matrix is orthogonal if and only if
The following properties of a matrix are equivalent:
- is orthogonal.
- the rows of have length 1 and are mutually perpendicular to each other with respect to the standard dotproduct.
- the columns of have length 1 and are mutually perpendicular to each other with respect to the standard dotproduct.
Now we can formulate the singular value decomposition in the following way.
Singular value decomposition Let be an matrix with rank and with singular values and . Then, there is an orthogonal matrix , an orthogonal matrix , and an matrix with the singular values on the main diagonal in descending order and zeros elsewhere, such that is uniquely determined, but the orthogonal matrices and not.
You can also write the decomposition with the first column vectors of the matrix and the first column vectors of the matrix . Then: If and/or , this gives a more compact description of the singular value decomposition.
We also have the following more compact decomposition: