Principal Component Analysis Explained, with a Python Example

Datasets with dozens of correlated columns are common: a panel of asset returns, a set of customer attributes, a block of risk factors. Principal component analysis (PCA) is a standard way to summarise them with a handful of new variables. This post explains what it does, how to use it in Python and where it goes wrong. It was written with Bohdan Boiprav.

The idea

PCA finds a new set of axes for your data. The first axis, the first principal component, points in the direction along which the data varies most. The second is the direction of greatest remaining variance that is perpendicular to the first, and so on. Each component is a weighted combination of the original columns, the components are uncorrelated with one another, and they are ordered by how much variance they capture.

If several original columns move together, one component can stand in for all of them. You keep the first few components and drop the rest, which gives you fewer columns while retaining most of the variation.

How it works

1. Centre each column by subtracting its mean. Usually also scale it to unit variance.
2. Compute the covariance (or correlation) matrix of the columns.
3. Find its eigenvectors and eigenvalues. The eigenvectors are the principal directions, and each eigenvalue is the variance along its direction.
4. Project the data onto the top k eigenvectors.

Libraries normally get the same result from a singular value decomposition of the centred data matrix, which is more numerically stable. Jonathon Shlens’ tutorial on PCA gives a readable derivation of both views.

An example with simulated returns

The code below builds eight synthetic return series driven by two hidden factors plus noise, standardises them and runs PCA. The data are made up for illustration.

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(1)
n_days, n_assets = 750, 8
factors = rng.normal(size=(n_days, 2))                 # two hidden drivers
loadings = rng.normal(size=(2, n_assets))
returns = factors @ loadings + rng.normal(scale=0.5, size=(n_days, n_assets))

Z = StandardScaler().fit_transform(returns)            # centre and scale first
pca = PCA().fit(Z)

print(pca.explained_variance_ratio_.round(3))
print(pca.explained_variance_ratio_.cumsum().round(3))

pca2 = PCA(n_components=0.90).fit(Z)                   # keep 90% of variance
print(pca2.n_components_)
print(pca2.transform(Z).shape)

On my run the explained-variance ratios were 0.620, 0.272, 0.045, 0.027 and smaller values after that. The first two components account for 89.2% of the variance, which matches the two factors built into the data. Asking scikit-learn to keep 90% of the variance then selects three components, because 89.2% falls just short. That is a useful reminder that a variance threshold is a convention, not a law. Look at the whole curve, often plotted as a scree plot, and choose with the use case in mind.

The same technique is used on real market data. Principal components of a yield-curve history, for example, are commonly interpreted as broad shifts in level, slope and curvature. Treat that as an interpretation you check against the loadings, not something PCA guarantees.

What PCA is good for

• Reducing correlated inputs before a model that struggles with multicollinearity, such as an unregularised regression.
• Visualising high-dimensional data by plotting the first two or three components.
• Compressing data and filtering out low-variance directions that are often noise.
• Summarising risk: seeing how many independent drivers explain most of the movement in a set of assets.

Common mistakes

Several claims in the earlier version of this post were looser than they should have been, so here are the corrections and caveats.

• PCA is feature extraction, not feature selection. The components are new variables built from all of the original ones. No original column is dropped or kept as such, which is why the output is harder to read.
• It does not automatically improve a model. Reducing dimensions can speed up training and reduce overfitting, but PCA ignores the target. A direction with low variance can still be the most predictive one. Compare with and without PCA using cross-validation.
• Scale matters. Without standardisation, a column measured in large units dominates the first component simply because its variance is bigger.
• PCA is linear. It cannot capture curved structure. Other methods such as kernel PCA or manifold learning exist for that.
• Fit it on training data only. Fitting on the full dataset before splitting leaks information from the test set. Put it in a scikit-learn Pipeline so this happens automatically.
• Interpretation takes work. Check the loadings (pca.components_) to see which original columns drive each component, and be wary of naming components too confidently.

Summary

PCA rotates the data onto uncorrelated axes ordered by variance, so you can keep a few and discard the rest. Standardise first, choose the number of components by looking at the explained-variance curve, fit inside cross-validation, and keep in mind that variance is not the same thing as relevance. The scikit-learn PCA documentation covers the parameters and solvers.