From the Introduction, \(\mathbf x_t=A\boldsymbol\varepsilon_t\). But the shocks \(\boldsymbol\varepsilon_t\) have variance one and no units. So split \(A\) in two:
\[\mathbf x_t=A\boldsymbol\varepsilon_t=C\underbrace{\Delta\boldsymbol\varepsilon_t}_{\mathbf f_t}=C\mathbf f_t\]
This gives sds of 20 km/h and 2 t, and 6 kg of fuel per km/h and 65 kg per tonne.
| restriction on \(A\) | normalisation of \(C\) | factors | |
|---|---|---|---|
| Structural | the manual | one outside fact per factor | headwind (km/h), load (t) |
| Cholesky | \(a_{12}=0\) | ones on the diagonal | regressor, residual |
| Symmetric root | \(a_{12}=a_{21}\) | ones on the diagonal | each roommateโs own GPA |
| PCA | first column longest | columns of length one, \(C=\Gamma\) | scores (data units) |
\(\Delta\) says how important each factor is, \(C\) how the variables combine them. A restriction picks \(A\), a normalisation splits it.
Every \(A\) with \(AA'=S\) fits the data. The total variance \(\operatorname{tr}(S)=\lVert a_1\rVert^2+\lVert a_2\rVert^2\) stays fixed, and turning the arrows only redistributes it. PCA picks the angle at which \(a_1\) is longest.
The biplot draws \(\mathbf x_t=\Gamma\mathbf f_t\). Points are the scores, arrows are the rows of \(\Gamma\).
The entries of \(\Gamma\) are the weights (prcomp()$rotation). With standardised scores, the arrows become the rows of \(A\), the loadings. biplot(pc, scale = 0) draws the scores version.
Dimension reduction. Drop the short arrow and keep one number per week, \(\mathbf x_t\approx f_{t1}\gamma_1\). This keeps 95% of Coca-Colaโs and 96% of Pepsiโs variance.
Project the data onto any unit direction \(\gamma_1\). The scores \(f_{t1}=\gamma_1'\mathbf x_t\) have variance \(\gamma_1'S\gamma_1\), the same curve as \(\lVert a_1\rVert^2\).
\[\max_{\gamma_1}\ \gamma_1'S\gamma_1\quad\text{subject to}\quad\gamma_1'\gamma_1=1\]
By Pythagoras, the largest variance along the line means the smallest perpendicular distances to it.
\(S\) moves the plane. The slider goes from doing nothing to applying \(S\).
They point at 46.5ยฐ and 136.5ยฐ, the PCA directions, with eigenvalues \(\kappa_1=2.00\) and \(\kappa_2=0.10\).
\[S=AA'=(\Gamma\Delta)(\Gamma\Delta)'=\Gamma\Delta^2\Gamma'\] \[S\Gamma=\Gamma\Delta^2\Gamma'\Gamma=\Gamma\Delta^2\quad\Rightarrow\quad S\gamma_l=\delta_l^2\,\gamma_l\]
In general \(S=VKV^{-1}\). Because \(S\) is symmetric, \(V^{-1}=V'\), so \(S=VKV'\) is the PCA split with \(\Gamma=V\) and \(\Delta^2=K\).
In R, eigen(S) replaces the slider.
Measure Pepsi in other units. The covariance-matrix arrows swing towards the variable with more variance. The correlation matrix gives the same answer for every unit.
A horizontal arrow belongs to PC1, a vertical one to PC2. Opposite arrows form a contrast.
PC1 (59%) contrasts illiteracy with life expectancy. Call it well-being.
PC2 (16%) is mainly income.
Life expectancy has a short arrow, so the map shows it less well. These labels are interpretations, not facts.
\(\hat{\mathbf x}_t=\sum_{l\le k}f_{tl}\gamma_l\). One component reconstructs Alaska and Hawaii badly.
Choose weights within each set that maximise the correlation between the two combinations. The next pair must be uncorrelated with the first.
First pair, \(r_1=0.71\): \[\text{education}\propto-0.98\,\text{Illiteracy}+0.19\,\text{HSGrad},\quad \text{health}\propto0.30\,\text{LifeExp}-0.95\,\text{Murder}\] Second pair, \(r_2=0.37\): \[\text{education}\propto0.60\,\text{Illiteracy}+0.80\,\text{HSGrad},\quad \text{health}\propto0.75\,\text{LifeExp}+0.66\,\text{Murder}\]
PCA keeps the directions in which the data vary most, and drops the rest.