From Data to Distributions

1 Random variables

Experiments and random variables

Random experiment: result unknown in advance (roll a die).
Random variable $X$: a number for each outcome, “the number shown”.
Capital $X$ before the roll; its realisation $x$ after. Data are realisations.

Discrete distribution, two equivalent descriptions:

pmf $p(x)=P(X=x)$, $\sum_x p(x)=1$
cdf $F(x)=P(X\le x)$, a staircase

2 Random vectors

Two dice: first die and sum

$X_1$ = first die, $X_2$ = sum: a random vector $X=(X_1,X_2)'$.

Joint: $P(X_1=x_1,X_2=x_2)$, $1/36$ per point
Marginal: add over the other variable (grey bars)
Conditional: one column, rescaled: given $X_1=3$, the sum is $4,\dots,9$

Independence

Independent: knowing $X_1$ tells nothing about $X_2$:

\[P(X_1=x_1,X_2=x_2)=P(X_1=x_1)\,P(X_2=x_2)\quad\text{for all pairs.}\]

The two dice are independent: $\frac16\cdot\frac16=\frac1{36}$.
The first die and the sum are dependent: $P(X_1=1,X_2=12)=0$, but $P(X_1=1)P(X_2=12)>0$.

Dependence between variables is what multivariate analysis is about.

3 From random vectors to a data matrix

Repeating the experiment

Repetition $t$: random vector $X^{(t)}$, realisation $x^{(t)}$ = row $t$ of $\mathbf X$.

\[X^{(1)},\dots,X^{(n)}\;\longrightarrow\;\mathbf X=\begin{pmatrix}x^{(1)\prime}\\ \vdots\\ x^{(n)\prime}\end{pmatrix}\]

Superscript $(t)$: the row. Subscript $j$: the variable. $X_1$ is the first die, $X^{(1)}$ the first roll.

Random sample: rows iid, independent and identically distributed.

Then every $X^{(t)}$ has the distribution of one $X$:

$\mathbf X$ is $n$ independent draws of the same $X$, not one draw each of $n$ different random vectors.

The data come to look like the distribution

Blue: share of the $n$ rows at each pair. Circles: the probability $1/36$.
More rows, closer match: the data tell us about the unknown distribution.

4 Continuous random vectors

Densities: probabilities are areas

$P(a\le X\le b)=\int_a^b f(x)\,dx$, and $F(x)=P(X\le x)$ is the area to the left of $x$.
$f(x)$ is not a probability: it is high where values are crowded.

The multivariate normal

Flight time $X_1$, fuel $X_2$. Drag to rotate.

\[f(x)\propto\exp\!\Big(-\tfrac12(x-\mu)'\Sigma^{-1}(x-\mu)\Big)\]

One hill, fully described by $\mu$ and $\Sigma$.

From 3D to 2D: the contour plot

Look at the hill from above: lines of equal height, like a hiking map.

Equal height = equal Mahalanobis distance $d$.
So every contour is an ellipse.

How much probability inside each ellipse?

If $X\sim N(\mu,\Sigma)$ with $p$ variables, the squared Mahalanobis distance:

\[d^2=(X-\mu)'\Sigma^{-1}(X-\mu)\sim\chi^2_p\]

With $p=2$: 39%, 86% and 98.9% within $d=1,2,3$.

A distance is surprising if it is far in the tail of $\chi^2_p$.

Conditional distributions are regressions

Fix flight time $X_1=x_1$, look at fuel $X_2$ (blue slices).

Each slice is normal, all with the same spread.
Centres on a straight line: $E[X_2\mid X_1=x_1]$, the regression line.

Correlation shapes the ellipse

Standardised normal, contours $d=1,2,3$. Move $\rho$:

$\rho=0$: circles, independent
$|\rho|\to 1$: thin ellipses, almost a line

Not every distribution is one hill

Penguin bill length and depth, smoothed into a density (a kernel density estimate). Drag to rotate.

Three peaks. A normal has only one.

The hills are the species

The density found the groups without being told the species.

Several hills suggest a mixture of groups: the idea behind cluster analysis.

5 Moments

Two examples, side by side

Moments are defined the same way for every distribution; only the arithmetic differs.

	Discrete: pmf, sums	Continuous: pdf, integrals
one variable	one die	daily sales
two variables	two dice: first die $X_1$, sum $X_2$	flights: time $X_1$, fuel $X_2$ (normal)

Left column: always the dice and a pmf. Right column: always a continuous variable and a pdf.

Mean and variance

$\mu=\sum_x x\,p(x)=3.5$ (never rolled: the balance point)
$\sigma^2=\sum_x (x-\mu)^2p(x)=35/12$

$\mu=\int x\,f(x)\,dx=\$1{,}200$
$\sigma^2=\int (x-\mu)^2f(x)\,dx$, $\sigma=\$600$

Covariance and correlation

\[\sigma_{12}=E\big[(X_1-\mu_1)(X_2-\mu_2)\big],\qquad \rho_{12}=\frac{\sigma_{12}}{\sigma_1\sigma_2}\in[-1,1]\]

Two dice (sum over the joint pmf)

$\sigma_{12}=\sum\sum(x_1-\mu_1)(x_2-\mu_2)\,p(x_1,x_2)$

Sum = first die + independent second die: $\sigma_{12}=\operatorname{Var}(X_1)=35/12$, $\rho_{12}=1/\sqrt2\approx0.71$

Flights (integral over the joint pdf)

$\sigma_{12}=\iint(x_1-\mu_1)(x_2-\mu_2)\,f(x_1,x_2)$

$\sigma_{12}=1.70$, $\rho_{12}=0.79$: longer flights burn more fuel

Mean vector and covariance matrix

$\mu=E[X]$, $\Sigma=E\big[(X-\mu)(X-\mu)'\big]$: centre (red) and shape (one-sd ellipse).

$\mu=\begin{pmatrix}3.5\\7\end{pmatrix}$, $\Sigma=\frac{35}{12}\begin{pmatrix}1&1\\1&2\end{pmatrix}$

$\mu=\begin{pmatrix}0\\0\end{pmatrix}$, $\Sigma=\begin{pmatrix}1.48&1.70\\ 1.70&3.13\end{pmatrix}$

For the normal (right), $\mu$ and $\Sigma$ describe the distribution completely; for the dice (left), they only summarise it.

Population and sample moments

	Population (the distribution)	Sample (the data matrix)
mean	$\mu=E[X]$	$\bar x=\frac1n\sum_t x^{(t)}$
variance	$\sigma_j^2=E[(X_j-\mu_j)^2]$	$s_j^2=\frac{1}{n-1}\sum_t (x_{tj}-\bar x_j)^2$
covariance matrix	$\Sigma=E[(X-\mu)(X-\mu)']$	$S=\frac{1}{n-1}\sum_t (x^{(t)}-\bar x)(x^{(t)}-\bar x)'$
correlation	$\rho_{jk}=\sigma_{jk}/(\sigma_j\sigma_k)$	$r_{jk}=s_{jk}/(s_js_k)$

Population moments are fixed numbers. Sample moments change from sample to sample, and estimate them (Section 6).

6 Learning the distribution from data

Every sample gives a different estimate

20 months of sales, 30 days each.

◆ sample mean $\bar x$, — $\bar x\pm s$
Black: the true $\mu=\$1{,}200$.

$\bar X=\frac1n\sum_t X^{(t)}$ is itself random.

Law of large numbers

Rows iid draws of $X$. Press Play.

\[\bar x=\frac1n\sum_{t=1}^n x^{(t)}\;\longrightarrow\;\mu\]

Band $\mu\pm 2\sigma/\sqrt n$: four times the data, half the error.

Likewise $S\to\Sigma$.

Central limit theorem

Daily sales are skewed ($n=1$). Averages of $n$ days become bell-shaped.
For large $n$: $\bar X\approx N\big(\mu,\;\sigma^2/n\big)$, whatever the distribution of $X$.

When the rows are not identically distributed

Shop A: one distribution every day (black). Shop B: growing, so $X^{(t)}$ depends on $t$ (red), and there is no single curve to settle on.

No single $\mu$ to estimate

Shop A: $\bar x\to\mu$. Shop B: each day has its own mean $\mu^{(t)}$.

$\bar x$ and $S$ estimate $\mu$ and $\Sigma$ only if all rows share one distribution: the reduced form.

Back to the states: is Alaska surprising?

States as iid draws of one $X\sim N(\mu,\Sigma)$. Squared Mahalanobis distance, with $\bar x$, $S$ for $\mu$, $\Sigma$:

\[d_t^2=(x^{(t)}-\bar x)'S^{-1}(x^{(t)}-\bar x)\approx\chi^2_6\]

Q-Q plot: sorted $d_t^2$ against $\chi^2_6$ quantiles; on the line if the distribution is right.
Alaska and Hawaii exceed the 97.5% cut-off (14.4).
Caveats: one distribution? normal? only 50 states?

7 From the reduced form to structural models

Linear transformations: $\operatorname{Var}(AX)=A\Sigma A'$

Everything so far describes the observed $X$: the reduced form. Structural models explain it as a linear transformation of simpler variables. If $\operatorname{Var}(X)=\Sigma$ and $Y=a+AX$:

\[E[Y]=a+A\mu,\qquad \operatorname{Var}(Y)=A\Sigma A'\]

Two dice $D$, $\operatorname{Var}(D)=\frac{35}{12}I$:

\[\begin{aligned}X&=\begin{pmatrix}1&0\\1&1\end{pmatrix}D\\ \operatorname{Var}(X)&=\tfrac{35}{12}\begin{pmatrix}1&1\\1&2\end{pmatrix}\end{aligned}\]

Independent dice, combined by $A$: dependent variables.

Flights: shocks $\varepsilon$, $\operatorname{Var}(\varepsilon)=I$:

\[X=A\varepsilon,\qquad \Sigma=AA'\]

Normal shocks give normal $X$. The structural form determines the reduced form.

Many $A$, one $\Sigma$

The structural form gives the reduced form, $\Sigma=AA'$. Can we go back, from $\Sigma$ to $A$? Not uniquely:

Any rotation $Q=\begin{pmatrix}\cos\varphi&-\sin\varphi\\ \sin\varphi&\cos\varphi\end{pmatrix}$, $QQ'=I$:

\[(AQ)(AQ)'=AQQ'A'=AA'=\Sigma\]

$\tilde\varepsilon=Q'\varepsilon\sim N(0,I)$: the same distribution of $X$.
$\Sigma$: 3 numbers; $A$: 4; one free angle $\varphi$ (Introduction: turning the arrows).

Picking one $\varphi$ needs an extra restriction: identification (symmetric root, Cholesky, PCA, a known coefficient).

Coming lectures: structural models

Each method maps the reduced form ($\mu$, $\Sigma$) to structural parameters, with its own identifying assumptions.

Starting with PCA, then MDS, factor analysis, cluster analysis, CFA and SEM.

Summary

Takeaways

Random variable: a number for each outcome; random vector $X=(X_1,\dots,X_p)'$: several from one experiment.
Rows $x^{(t)}$ are realisations of $X^{(t)}$; in a random sample, $n$ iid draws of one $X$.
pmf (heights), pdf (areas), cdf; joint, marginal, conditional; independence.
Multivariate normal: one hill, elliptical contours, $d^2\sim\chi^2_p$, linear conditional means.
Moments $\mu$, $\Sigma$; $\operatorname{Var}(AX)=A\Sigma A'$.
LLN and CLT: $\bar x$, $S$ estimate $\mu$, $\Sigma$, if the rows share one distribution. Alaska and Hawaii are surprising.

All of this describes the observed $X$. Structural models, from PCA on, try to explain it.

	Population (the distribution)	Sample (the data matrix)
mean	\(\mu=E[X]\)	\(\bar x=\frac1n\sum_t x^{(t)}\)
variance	\(\sigma_j^2=E[(X_j-\mu_j)^2]\)	\(s_j^2=\frac{1}{n-1}\sum_t (x_{tj}-\bar x_j)^2\)
covariance matrix	\(\Sigma=E[(X-\mu)(X-\mu)']\)	\(S=\frac{1}{n-1}\sum_t (x^{(t)}-\bar x)(x^{(t)}-\bar x)'\)
correlation	\(\rho_{jk}=\sigma_{jk}/(\sigma_j\sigma_k)\)	\(r_{jk}=s_{jk}/(s_js_k)\)