1 Factors with units

Shocks with units: \(A=C\Delta\)

From the Introduction, \(\mathbf x_t=A\boldsymbol\varepsilon_t\). But the shocks \(\boldsymbol\varepsilon_t\) have variance one and no units. So split \(A\) in two:

\[\mathbf x_t=A\boldsymbol\varepsilon_t=C\underbrace{\Delta\boldsymbol\varepsilon_t}_{\mathbf f_t}=C\mathbf f_t\]

  • \(\Delta\) is diagonal. It turns the shocks into factors \(\mathbf f_t\) in km/h and tonnes.
  • \(C\) says what one unit of each factor does to the data. Then \(S=C\Delta^2C'\).

Structural: the flights

  1. Identify \(A\). The manualโ€™s fuel flow fixes the direction of the headwind arrow.
  2. Normalise with one fact per factor. Each km/h of headwind adds 0.06 min, each tonne adds 0.1 min.

This gives sds of 20 km/h and 2 t, and 6 kg of fuel per km/h and 65 kg per tonne.

Cholesky: a regression

  • Restriction \(a_{12}=0\). Normalisation: ones on the diagonal of \(C\).
  • Factor 1 is education itself, with an sd of 2.5 years.
  • Factor 2 is the earnings residual, with an sd of $15k. The slope \(\beta=4\) links the two.

Symmetric root: roommates

  • Restriction \(a_{12}=a_{21}\). Normalisation: ones on the diagonal again.
  • Each factor is a studentโ€™s own GPA before the peer effect, with an sd of 0.5.
  • The peer effect is 0.2 in both directions.

PCA: Coke and Pepsi

  • Restriction: the first column is as long as possible. Normalisation: columns of length one, written \(C=\Gamma\).
  • One unit of a factor then moves the data by a distance of one.
  • The factor variances add up to the total variance, \(\operatorname{tr}(S)=\sum_l \delta_l^2\).

Four normalisations

restriction on \(A\) normalisation of \(C\) factors
Structural the manual one outside fact per factor headwind (km/h), load (t)
Cholesky \(a_{12}=0\) ones on the diagonal regressor, residual
Symmetric root \(a_{12}=a_{21}\) ones on the diagonal each roommateโ€™s own GPA
PCA first column longest columns of length one, \(C=\Gamma\) scores (data units)

\(\Delta\) says how important each factor is, \(C\) how the variables combine them. A restriction picks \(A\), a normalisation splits it.

2 The direction of greatest variance

Two arrows: who carries the variance?

Every \(A\) with \(AA'=S\) fits the data. The total variance \(\operatorname{tr}(S)=\lVert a_1\rVert^2+\lVert a_2\rVert^2\) stays fixed, and turning the arrows only redistributes it. PCA picks the angle at which \(a_1\) is longest.

Building the data: \(\mathbf x_t=\Gamma\Delta\boldsymbol\varepsilon_t\)

  • At the PCA angle the arrows are perpendicular, so \(\Gamma\) is a rotation, \(\Gamma'\Gamma=I\).
  • Read right to left. \(\Delta\) stretches the shocks into the scores \(\mathbf f_t\), and \(\Gamma\) turns them into the data.

Scores: take \(A\) apart one layer at a time

  • Undo the rotation with \(\Gamma'\). The scores \(\mathbf f_t=\Gamma'\mathbf x_t\) are projections onto the components.
  • Undo the stretch with \(\Delta^{-1}\). This gives the standardised scores \(\boldsymbol\varepsilon_t\).
  • The middle panel is the biplot. Each purple arrow is where one unit of a variable lands.

Reading the biplot

The biplot draws \(\mathbf x_t=\Gamma\mathbf f_t\). Points are the scores, arrows are the rows of \(\Gamma\).

  • Point and arrow. Project a point onto an arrow to read that variableโ€™s value.
  • Distances. Distances between points are Euclidean distances.
  • Arrow lengths. Each arrow has length one when all components are shown. With fewer components, a short arrow means the variable lies mostly in the dropped ones.

The entries of \(\Gamma\) are the weights (prcomp()$rotation). With standardised scores, the arrows become the rows of \(A\), the loadings. biplot(pc, scale = 0) draws the scores version.

Second arrow off: predictions

Dimension reduction. Drop the short arrow and keep one number per week, \(\mathbf x_t\approx f_{t1}\gamma_1\). This keeps 95% of Coca-Colaโ€™s and 96% of Pepsiโ€™s variance.

The same answer from the data side

Project the data onto any unit direction \(\gamma_1\). The scores \(f_{t1}=\gamma_1'\mathbf x_t\) have variance \(\gamma_1'S\gamma_1\), the same curve as \(\lVert a_1\rVert^2\).

The principal components

\[\max_{\gamma_1}\ \gamma_1'S\gamma_1\quad\text{subject to}\quad\gamma_1'\gamma_1=1\]

  • Without the length constraint there is no maximum. Doubling \(\gamma_1\) quadruples the variance.
  • \(\gamma_2\) has the largest variance among directions perpendicular to \(\gamma_1\), and so on.
  • The scores \(f_{tl}=\gamma_l'\mathbf x_t\) are the new variables.

By Pythagoras, the largest variance along the line means the smallest perpendicular distances to it.

Not the regression line

  • Regression minimises vertical distances and treats one variable as given.
  • PCA minimises perpendicular distances and treats both variables alike.

3 Eigenvectors and eigenvalues

Vectors that stay on their line

\(S\) moves the plane. The slider goes from doing nothing to applying \(S\).

  • Points on the grey lines leave their lines.
  • Points on the blue and red lines stay and are only stretched or squished. These are the eigenvectors, \(Sv=\kappa v\).

They point at 46.5ยฐ and 136.5ยฐ, the PCA directions, with eigenvalues \(\kappa_1=2.00\) and \(\kappa_2=0.10\).

Why the PCA directions are eigenvectors

\[S=AA'=(\Gamma\Delta)(\Gamma\Delta)'=\Gamma\Delta^2\Gamma'\] \[S\Gamma=\Gamma\Delta^2\Gamma'\Gamma=\Gamma\Delta^2\quad\Rightarrow\quad S\gamma_l=\delta_l^2\,\gamma_l\]

  • Each PCA direction is an eigenvector of \(S\), and its eigenvalue is the factor variance \(\delta_l^2\).
  • The first component has the largest eigenvalue.

In general \(S=VKV^{-1}\). Because \(S\) is symmetric, \(V^{-1}=V'\), so \(S=VKV'\) is the PCA split with \(\Gamma=V\) and \(\Delta^2=K\).

In R, eigen(S) replaces the slider.

4 Covariance or correlation matrix?

PCA depends on the units

Measure Pepsi in other units. The covariance-matrix arrows swing towards the variable with more variance. The correlation matrix gives the same answer for every unit.

More variables: the states

  • With the covariance matrix, PC1 is income alone, because its sd is 614 dollars.
  • With different units, standardise first. That is PCA of the correlation matrix.

The states in a biplot

A horizontal arrow belongs to PC1, a vertical one to PC2. Opposite arrows form a contrast.

PC1 (59%) contrasts illiteracy with life expectancy. Call it well-being.

PC2 (16%) is mainly income.

Life expectancy has a short arrow, so the map shows it less well. These labels are interpretations, not facts.

5 How many components?

Rules of thumb

  • Keep 70 to 90% of the variance, or the eigenvalues above average (Kaiser 1, Jolliffe 0.7), or the components before the elbow.
  • For the states the rules disagree. Choose with the purpose in mind.

Reconstruction from \(k\) components

\(\hat{\mathbf x}_t=\sum_{l\le k}f_{tl}\gamma_l\). One component reconstructs Alaska and Hawaii badly.

6 What PCA is, and what it is not

A convenient rotation, not the true shocks

  • For the flights, PC1 explains 91% of the variance but mixes headwind and load.
  • “No mathematical method is, or could be, designed to give physically meaningful results” (Marriott 1974).

Practical points

  • Only correlated variables can be compressed.
  • Units matter. Standardise unless all variables share a unit.
  • Outliers move the components. Look at the data first.
  • Signs are arbitrary, and components with close eigenvalues are unstable.
  • In a regression on components, a low-variance component can still predict the outcome.

7 Two sets of variables

Canonical correlation

Choose weights within each set that maximise the correlation between the two combinations. The next pair must be uncorrelated with the first.

First pair, \(r_1=0.71\): \[\text{education}\propto-0.98\,\text{Illiteracy}+0.19\,\text{HSGrad},\quad \text{health}\propto0.30\,\text{LifeExp}-0.95\,\text{Murder}\] Second pair, \(r_2=0.37\): \[\text{education}\propto0.60\,\text{Illiteracy}+0.80\,\text{HSGrad},\quad \text{health}\propto0.75\,\text{LifeExp}+0.66\,\text{Murder}\]

Summary

Takeaways

  • \(A=C\Delta\): \(\Delta\) says how important each factor is, \(C\) how the variables respond. PCA gives the columns of \(C\) length one.
  • PCA turns the arrows until the first carries the most variance.
  • The components are the eigenvectors of \(S\), and their variances are the eigenvalues.
  • With different units, use the correlation matrix. How many components to keep is a judgement.
  • The biplot shows scores and variables together. Short arrows are poorly represented.
  • PCA describes and compresses. It does not find the true forces.

PCA keeps the directions in which the data vary most, and drops the rest.