1 Why do the points move together?
200 flights on one route
Each point is one flight: flight time and fuel burned, as deviations from a typical flight. The cloud is tilted (correlation 0.79).
- Why do the two move together?
- Is there a common movement behind both?
- How many movements, and how strongly does each affect each variable?
Multivariate analysis: explain a cloud of correlated data by a few simpler, unobserved movements.
A world without relationships
Two independent shocks per flight, a round cloud:
- headwind \(\varepsilon_1\): stronger wind than usual
- load \(\varepsilon_2\): heavier plane than usual
Press Play: the aircraft turns these shocks into flight time and fuel, and we get the tilted cloud of the 200 flights.
Correlated data can come from independent forces: the aircraft combines them into flight time and fuel.
2 From shocks to data
Each shock moves both measurements
- Headwind (1 sd): 1.20 min longer; at the manual’s 100 kg/min, 120 kg more fuel.
- Load (1 sd, 2 t): 0.20 min and 130 kg more.
\[ \begin{aligned} X_1 = \text{time} &= 1.20\,\varepsilon_1 + 0.20\,\varepsilon_2\\ X_2 = \text{fuel} &= 1.20\,\varepsilon_1 + 1.30\,\varepsilon_2 \end{aligned} \]
Each variable: a weighted sum of the same two shocks.
In matrix form: \(X = A\varepsilon\)
\[ \underset{X}{\begin{pmatrix} X_1\\ X_2\end{pmatrix}} = \underset{A}{\begin{pmatrix} 1.20 & 0.20\\ 1.20 & 1.30\end{pmatrix}} \underset{\varepsilon}{\begin{pmatrix} \varepsilon_1\\ \varepsilon_2\end{pmatrix}} \]
Row \(i\) of \(A\): how \(X_i\) responds to each shock.
Column \(j\): how shock \(j\) moves all variables.
Capitals (\(X_1\), \(X_2\)): random, they change from flight to flight. Lower case (\(x_1\), \(x_2\)): the values recorded for one flight.
A matrix moves space
\(A\) moves the whole plane: lines stay straight, grid lines parallel and evenly spaced, the origin stays put.
The basis vectors land on the columns of \(A\): \[Ae_1=\begin{pmatrix}1.20\\1.20\end{pmatrix},\quad Ae_2=\begin{pmatrix}0.20\\1.30\end{pmatrix}\]
Every other point follows: \(X = \varepsilon_1\,Ae_1 + \varepsilon_2\,Ae_2\).
Idea from 3Blue1Brown, Essence of Linear Algebra.
And back again: whitening
The inverse matrix moves the plane back: \[\varepsilon = A^{-1}X\]
For every flight: how much headwind, how much load?
Whitening: turning correlated variables into uncorrelated, unit-variance shocks (“white noise”).
3 The same data, many stories
Many matrices fit the same cloud
What the data can tell us: \(\Sigma\)
The data pin down only three numbers, the covariance matrix:
\[ \Sigma=\begin{pmatrix}\sigma_1^2 & \sigma_{12}\\ \sigma_{12} & \sigma_2^2\end{pmatrix} =\begin{pmatrix}1.48 & 1.70\\ 1.70 & 3.13\end{pmatrix} \]
- \(\sigma_1^2, \sigma_2^2\): variances of flight time and fuel
- \(\sigma_{12}\): covariance; \(\rho = \sigma_{12}/(\sigma_1\sigma_2)\)
The black ellipse is a picture of \(\Sigma\).
What a candidate \(A\) implies: \(\Sigma = AA'\)
\[ \begin{aligned} \sigma_1^2 &= a_{11}^2+a_{12}^2\\ \sigma_2^2 &= a_{21}^2+a_{22}^2\\ \sigma_{12} &= a_{11}a_{21}+a_{12}a_{22} \end{aligned} \]
or in short \(\;\boxed{\Sigma=AA'}\;\): 3 equations, 4 unknowns.
All four \(A\)’s fit the same cloud.
One extra restriction is needed: which \(A\) is useful?
4 Choosing a useful \(A\)
Symmetric root: \(A = A'\)
Uncorrelated coordinates that stay closest to the original variables.
- Roommates’ grades. Random pairs influence each other; neither comes first: \(a_{12} = a_{21}\).
Cholesky: \(a_{12} = 0\)
Order the variables; \(X_1\) responds only to the first shock.
Regression (\(X_1\) education, \(X_2\) earnings): \[ \begin{aligned} X_2 &= \beta X_1 + \sigma_2\varepsilon_2\\ X_1 &= \sigma_1\varepsilon_1 \end{aligned} \] \[ \Rightarrow\quad \begin{pmatrix}X_1\\X_2\end{pmatrix}= \begin{pmatrix}\sigma_1 & 0\\ \beta\sigma_1 & \sigma_2\end{pmatrix} \begin{pmatrix}\varepsilon_1\\\varepsilon_2\end{pmatrix} \]
- Timing. Earlier variables first (recursive VARs).
PCA: first column as long as possible
Squared column length = that shock’s share of the total variance. PCA makes the first column longest; the second is perpendicular.
- Coke and Pepsi. Major movement: the soft-drink market. Minor: brand switching.
Simultaneous equations: a known coefficient
Engineering or economics, not statistics, supplies the restriction.
- Flights. Manual: 100 kg per extra cruise minute fixes the headwind direction. The data give the rest: a load shock adds 130 kg but only 12 s.
Choose latent coordinates that make some feature of the data simple or meaningful.
5 Structural form, reduced form, sample
Three layers: nature one way, analysis the other
- Structural form \(\varepsilon\), \(A\): why the variables move together (a circle).
- Reduced form \(X = A\varepsilon\), \(\Sigma = AA'\): only how (an ellipse).
- Sample \(\mathbf X\): the 200 flights we recorded, draws of \(X\).
How nature produces the data
The shocks generate the random vector \(X = A\varepsilon\); we sample 200 flights from it.
How we analyse the data
We only ever see the orange points.
6 Fewer coordinates: dimension reduction
Coke and Pepsi: one number per week
- One big movement (market), one small (brand switching): keep one number, not two.
- Many variables, few movements (yield curves, surveys, stock returns): a few numbers per observation describe almost everything.
7 Other latent structures
Factor analysis: two food critics
\[ \begin{aligned} X_1&=\sigma_1\varepsilon_1+\lambda_1 f\\ X_2&=\sigma_2\varepsilon_2+\lambda_2 f \end{aligned} \] \[ A=\begin{pmatrix}\sigma_1 & 0 & \lambda_1\\ 0 & \sigma_2 & \lambda_2\end{pmatrix} \]
3 reduced + 1 free = 4 par. (\(\lambda_1, \lambda_2, \sigma_1, \sigma_2\)). Open: how the noise is split.
All one-factor stories that fit the critics
Four natural restrictions
- Critic 1 never errs (\(\sigma_1 = 0\)): her rating is the quality; critic 2 is a regression on her. Exactly Cholesky.
- Critic 2 never errs: the same, critic 2 first.
- Equally reliable (equal communalities): quality explains the same share of each critic’s rating variance.
- Equal noise (\(\sigma_1 = \sigma_2\)): both critics’ quirks are equally large.
A discrete latent variable: clustering
- Factor model: the center \(\lambda f\) slides. Clustering: it jumps among groups.
- Theory (CFA, SEM) fixes entries of \(A\) to zero, like Cholesky and the headwind.
A different distance: MDS on a loop road
What multidimensional scaling does
MDS turns travel times into coordinates whose straight-line distances reproduce them.
A construction-site wait opens the ring; beyond 27.9 min nobody drives through it, and one dimension suffices.
First turn the distances we care about into coordinates, then treat them like any data.
8 The road ahead
Coming lectures: structural models
First the sample (looking at data), then the reduced form (distributions), then structural models, each with its own identifying assumptions.
Summary
Takeaways
- Independent shocks form a round cloud; a system \(A\) combines them into a tilted cloud. Undoing \(A\) (whitening) recovers the shocks.
- The data reveal only the reduced form \(\Sigma\). Every rotation of \(A\) gives the same \(\Sigma = AA'\), so one extra restriction picks a structural form: symmetry, an ordering, the longest first column, or a known coefficient.
- Infer (sample → reduced form) + identify (reduced form → structural form) = estimate.
- A structural model explains the data and can compress them.
- \(X = A\varepsilon\) is one structural model. Factors and clusters change the latent structure. MDS first rebuilds coordinates from the distances we care about, then the same ideas apply.
Which latent coordinates make the multivariate data easier to understand?