1 The data matrix
The data: 50 states, 6 variables
| Income $ per capita | Illiteracy % | Life exp. years | Murder per 100k | HS grad % | Frost days | |
|---|---|---|---|---|---|---|
| 1 Alabama | 3,624 | 2.1 | 69.0 | 15.1 | 41.3 | 20 |
| 2 Alaska | 6,315 | 1.5 | 69.3 | 11.3 | 66.7 | 152 |
| 3 Arizona | 4,530 | 1.8 | 70.5 | 7.8 | 58.1 | 15 |
| 4 Arkansas | 3,378 | 1.9 | 70.7 | 10.1 | 39.9 | 65 |
| 5 California | 5,114 | 1.1 | 71.7 | 10.3 | 62.6 | 20 |
| ⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ |
| 43 Texas | 4,188 | 2.2 | 70.9 | 12.2 | 47.4 | 35 |
| ⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ | ⋮ |
| 50 Wyoming | 4,566 | 0.6 | 70.3 | 6.9 | 62.9 | 173 |
- Row \(t\): one state, the column vector \(x^{(t)}=(x_{t1},\dots,x_{tp})'\) ( Texas). Column \(j\): one variable ( murder).
- Entry \(x_{tj}\): Texas has \(x_{43,4}=12.2\) murders per 100,000.
Superscript \((t)\): the observation. Plain subscript \(j\): the variable, so \(x_1\) is the first variable and \(x^{(1)}\) the first observation. The prime \('\) is the transpose.
A \(50\times 6\) data matrix \(\mathbf X\) (Rโs state.x77, 1970s): the sample.
Which US state is the most unusual?
All 300 numbers at once, standardised \(z_{tj}=(x_{tj}-\bar x_j)/s_j\): how many standard deviations a state is above (purple) or below (orange) the average.
Which state stands out? Single extreme values are easy to spot; unusual combinations are not. We answer at the end.
Observations are points
- One variable: a point on a line. Two: in the plane. Three: in space. \(p\): a point in \(p\) dimensions.
- The data matrix is a cloud of \(n\) points; this course is about its shape.
Distance between two observations
Euclidean distance: lay a ruler between the two points. By Pythagoras:
\[\lVert x^{(s)}-x^{(t)}\rVert=\sqrt{\sum_{j=1}^p (x_{sj}-x_{tj})^2}\]
In vector form, with \(a'b=\sum_j a_jb_j\) (inner product):
\[\lVert x^{(s)}-x^{(t)}\rVert=\sqrt{(x^{(s)}-x^{(t)})'(x^{(s)}-x^{(t)})}\]
Same formula for any \(p\). It depends on the units: the variable with the widest range dominates.
2 Types of variables
Not every column is a number on a line
- Nominal: labels without order (island, species).
- Ordinal: ordered labels, gaps not comparable (diamond cut, survey answers).
- Interval: equal steps, arbitrary zero (temperature in ยฐF).
- Ratio: true zero, so ratios make sense (body mass, income, murder rate).
Binary: a nominal variable with two categories (coded 0/1, its mean is a share). Count: ratio-scaled, whole numbers only.
Why the type matters
Numbers on a scale: means, covariances, axes, sizes. Categories: count, colour or split.
Categorical variables in R: factors
island <- factor(c("Dream", "Biscoe",
"Dream", "Torgersen"))
levels(island) # stored once
## [1] "Biscoe" "Dream" "Torgersen"
as.integer(island) # codes per row
## [1] 2 1 2 3A factor stores the categories once (its levels) and a whole-number code per observation.
- Smaller and faster than repeated text
- A fixed set of categories: typos become
NA - Level order sets table and legend order;
ordered = TRUEfor ordinal - Models create the dummy variables automatically
3 A plot for each type
One plot does not fit all types
Two categorical: a scatterplot shows 333 penguins as 5 dots. Show counts.
Categorical + numeric: points pile up on lines. Use boxplots (plus jittered points).
Two numeric: the scatterplot is the right tool.
Many numeric variables: the scatterplot matrix
\(p\) variables, \(p(p-1)/2\) pairs: every pair once below the diagonal, correlations above.
- Illiteracy, murder, life expectancy and graduation all move together.
- Frost goes with low illiteracy (northern states).
Pairs are 2D. What about three variables at once?
4 A third variable, in two dimensions
Why not just plot in 3D?
- 333 penguins: bill (beak) length and depth, flipper length. Three angles, three different stories. Depth is guessed, points hide, values cannot be read off.
Humans read 2D well. Keep two axes; show the third variable in a way that fits its type: categorical โ colour or panels, numeric โ size or colour gradient.
The same three variables in 2D
Two numeric variables on the axes; the third (flipper length, also numeric) as a colour gradient.
One flat picture, values can be read off: three groups, which no single 3D view showed clearly.
Categorical third variable: colour
- Overall \(r = -0.23\), but positive within every species: Simpsonโs paradox.
- Grouped scatterplot: one hue per category,
aes(colour = species). Hues have no order: fine for nominal.
Numeric third variable: size
Hues cannot show how much. A numeric variable needs an order and a scale: size or a colour gradient.
Bubble plot: position = illiteracy, murder; size = population; gradient = income.
aes(size = Population, fill = Income). Crowds quickly; areas are hard to compare.
Categorical third variable: panels
Conditioning plot: one panel per category (species ร sex), same axes everywhere.
- Within each panel: heavier penguins have longer flippers.
- Across panels: Gentoo are larger, males heavier.
Also called trellis plots (lattice), facet plots (ggplot2: facet_wrap(), facet_grid()) or small multiples.
Same idea in other tools
Facet plot, one variable:facet_wrap(~ species)
Trellis plot (lattice):xyplot(mass ~ flipper | species * sex)
5 Summaries of the cloud
Mean and standard deviation
Sample mean of variable \(j\) and the mean vector, the centre of the cloud:
\[\bar x_j=\frac1n\sum_{t=1}^n x_{tj},\qquad \bar x=\begin{pmatrix}\bar x_1\\ \vdots\\ \bar x_p\end{pmatrix}=\frac1n\sum_{t=1}^n x^{(t)}\]
Sample variance and standard deviation, the typical distance from the mean:
\[s_j^2=\frac{1}{n-1}\sum_{t=1}^n (x_{tj}-\bar x_j)^2,\qquad s_j=\sqrt{s_j^2}\]
Murder: \(\bar x=7.4\), \(s=3.69\) per 100,000. \(s\) is in the units of the variable.
Covariance: the average product
\[ s_{jk}=\frac{1}{n-1}\sum_{t=1}^n (x_{tj}-\bar x_j)(x_{tk}-\bar x_k) \qquad\text{illiteracy and murder: } s = 1.58 \]
The sample covariance matrix
All variances (diagonal) and covariances (off the diagonal):
\[ S=\begin{pmatrix} s_1^2 & s_{12} & \cdots & s_{1p}\\ s_{21} & s_2^2 & \cdots & s_{2p}\\ \vdots & & \ddots & \vdots\\ s_{p1} & s_{p2} & \cdots & s_p^2\end{pmatrix} =\frac{1}{n-1}\sum_{t=1}^n (x^{(t)}-\bar x)(x^{(t)}-\bar x)' \]
- A column times a row, \(ab'\), is a \(p\times p\) matrix with entries \(a_jb_k\) (outer product): entry \((j,k)\) of each term is \((x_{tj}-\bar x_j)(x_{tk}-\bar x_k)\).
- \(S\) is symmetric. It describes the shape of the cloud, as \(\bar x\) describes its centre.
Correlation: covariance without units
Covariance carries the units of both variables (dollars times daysโฆ). Divide by the standard deviations:
\[r_{jk}=\frac{s_{jk}}{s_j\,s_k}\in[-1,1]\]
Same as the covariance of standardised values \(z=(x-\bar x)/s\).
All correlations form the matrix \(R\), ones on the diagonal. With \(D=\operatorname{diag}(s_1,\dots,s_p)\):
\[R=D^{-1}SD^{-1}\]
Correlation is an average of products
Move the slider. Each point is coloured by its product \(z_{t1}z_{t2}\):
- blue: same side of the mean on both variables, pulls \(r\) up
- red: opposite sides, pulls \(r\) down
The thinner the ellipse, the larger \(|r|\).
What correlation misses
- \(r\) measures linear association only: a perfect curve can have \(r\approx 0\).
- A single point can create a correlation. Always look at the scatterplot.
6 Unusual combinations
Hawaii: ordinary on each variable
Hawaii: \(z = 1.20\) and \(1.09\), within the whiskers, so no outlier. But high illiteracy usually comes with few graduates (\(r=-0.66\)); Hawaii has many.
From Euclidean to Mahalanobis distance
Euclidean distance from the centre treats every direction alike:
\[\sqrt{(x^{(t)}-\bar x)'(x^{(t)}-\bar x)}\]
Mahalanobis distance puts \(S^{-1}\) in the middle:
\[d_t=\sqrt{(x^{(t)}-\bar x)'S^{-1}(x^{(t)}-\bar x)}\]
- The cloud is an ellipse: a step along its long axis is normal, the same step across it is far out.
- \(S^{-1}\) (with \(S^{-1}S=I\), R:
solve(S)) shrinks differences where the cloud is wide and stretches them where it is narrow. - Uncorrelated, standardised variables: \(S=I\), and \(d_t\) is the Euclidean distance.
Mahalanobis distance: two variables
Distance from the centre in units of the cloudโs own shape.
Equal \(d_t\): one ellipse. Further-out ellipse, larger \(d_t\): same order as in the scatterplot.
Hawaii and Alaska lie across the cloud; Louisiana beyond its end.
Mahalanobis distance: all six variables
The same formula with \(p=6\): no picture, but the same idea.
Alaska is farthest, then Hawaii, New Mexico and Nevada.
Is a distance of 4.7 surprising? That needs a model of where the data come from: next lecture.
Summary
Takeaways
- A data set is an \(n\times p\) data matrix: a cloud of \(n\) points in \(p\) dimensions.
- Types of variables: nominal, ordinal, interval, ratio. Means, covariances and distances need numbers on a scale.
- The type decides the plot: counts, boxplots, scatterplots, scatterplot matrices.
- 3D is hard to read: show a third variable in 2D, fitted to its type: colour or panels (categorical), size (numeric).
- Each observation is a point; the Euclidean distance is the ruler between two points.
- Mean vector \(\bar x\), covariance matrix \(S\), correlation matrix \(R\) summarise the cloud; correlation captures only linear co-movement, so always look at the plot.
- The Mahalanobis distance measures distance relative to the shape of the cloud and finds unusual combinations: Hawaii, and with six variables Alaska.