Singular value decomposition (SVD) unsupervised generative method is considered an algebra-driven tool for decomposing a data matrix, X, to extract its singular values Σ.
The singular values, Σ, and eigenvalues, Λ, play a substantial role in model reduction, where the reduced version of the data is the result of transforming the original data from one vector space to another space.
The singular values measure the spectrum of the eigenvalues and are used to find the data rank, r, which indicates how many feature vectors are linearly independent (the number of columns, or rows in a compact space).
For a given data matrix X , SVD generates three matrices (U, Σ, and V) such that: X = UΣV, where V should be transposed matrix. U and V are orthonormal matrices, called the left and right singular vectors, respectively, that represent the eigenvectors of XX⊺ and X⊺X, respectively.
The figure depicts the graphical representation of SVD decomposition, where the original data is represented by X. Additionally, Σ is a diagonal matrix that contains the singular values {δi |i = 1 . . . , D} of the matrix X and are sorted in decreasing order. If the rank of X is r, the singular values after the first r values are equal to zero (or neglectable).
In the figure, data X is formed in colored cells that reflect specific intensities of image pixels, or they give numeric measurement values when data comes in a tabular form. Moreover, X is the original data, X˜ is the reconstructed data, and the rank r reflects the dimensionality in the mapped space Z and can be obtained by counting the highest singular values of the matrix Σ.
The exact decomposition can be approximated when the nonzero singular values are neglected based on a threshold of small value, i.e., truncated SVD.
This practice is useful in addressing numerical instability and noise and in reducing the d-dimensionality of the data.
However, dimensionality reduction to a d-dimensional space is obtained from the rank, r; the rank gives the dimension of the column space of a matrix U, and it gives the dimension of the row space of a matrix V.
Furthermore, the dimensionality reduction can be realized by defining Ud as the matrix with the first d columns of U, Σd as the d × d diagonal matrix containing the first d singular values, and Vd as the first d rows of V.
Also, Ud represents the distribution of the data points around the directions Vd when they are multiplied by the singular values as UdΣd.
Various extensions have been proposed in the literature to generalize SVD, e.g., generalized SVD (GSVD), in which the joint decomposition of two different matrices is performed. This approach is extended in restricted SVD (RSVD), which decomposes a given matrix with special constraints encoded in two other matrices.
𝓑𝓎: 𝓜𝓸𝒽𝒶𝓃𝒶𝒹 𝒜𝒷𝓊𝓀𝓂𝓮𝒾𝓁 𝒫𝒽.𝒟. 𝓮𝓍𝒸𝓮𝓇𝓅𝓉𝓈
