Five Perspectives on PCA: From Variance Maximization to SVD

The article presents PCA from five perspectives, starting with variance maximization: the first principal component is the direction maximizing projected variance, found as the eigenvector of the covariance matrix with the largest eigenvalue. Subsequent components are orthogonal and correspond to lower eigenvalues. The covariance matrix is symmetric and positive semi-definite, leading to nonnegative eigenvalues. PCA decorrelates data because the principal components have a diagonal covariance matrix. Whitening extends PCA by scaling components to unit variance, though it may amplify noise. Another perspective frames PCA as a special case of Singular Value Decomposition (SVD) of the centered data matrix, relating singular vectors to eigenvectors of covariance matrices. The article also touches on spectral decomposition, orthogonal projectors, and affine transformations, all supporting a unified geometric view. The discussion is inspired by a Skoltech course on geometrical methods of machine learning by Alexander Bernstein.