Singular Value Decomposition by itself produces an exact fit. But you can use the decomposed form to find a lower-rank matrix which approximates the original matrix.
Given the decomposition A=UDV* (where U and V are unitary matrices), D is a diagonal matrix whose entries are the “singular values” of A, and has the same rank as A. But if we truncate D to contain just the largest k values on its diagonal, that is the best approximation of A that has rank k (according to a least-squares goodness of fit.)
Thus, SVD followed by elimination of some singular values reduces the dimension, but this is often just referred to as “SVD” in context.
Many machine learning toolkits include SVD, but we can use Wolfram Alpha to show a simple example. Given the input matrix
its SVD is
If we zero the lowest value in the diagonal matrix Σ and multiply back out (remember to transpose V!) we get
You can eyeball this and see it's pretty close to the original value M, but it only has rank 2 instead of rank 3 (though since we proceeded numerically, Wolfram Alpha would give a small but nonzero determinant for the result.) Dropping the dimension all the way down to one by eliminating another of the singular values gives us:
which is a pretty crude approximation, but it is the best single-dimensional (rank 1) approximation of the original matrix that we can make.
Originally answered on Quora (without the example): https://www.quora.com/How-is-SVD-reducing-the-data-dimension-We-are-taking-one-matrix-and-splitting-it-into-three-after-all/answer/Mark-Gritter
