Statistics on DataLeaf

Least squares, three ways

Tue, 18 Aug 2026 00:00:00 +0000

Polynomial fits make the difference visible: with twelve columns, the normal equations lose every digit while QR and the SVD keep eight or nine. One identity about condition numbers explains it, and also says when the fast method is good enough.

Notebook: rerunning an analysis from two years ago

Fri, 14 Nov 2025 00:00:00 +0000

A reviewer asked for one more column in a table produced two years ago. Adding a column means rerunning the analysis, and rerunning it means first showing that the old numbers come back. This entry records what that took.

Checklist

Check out the tagged commit that produced the submitted table
Recreate the environment from the lock file
Verify the input data against the recorded checksums
Rerun the pipeline end to end and compare every cell
Record the seed of every random step
Add the new column and log the rerun

The first four steps took an afternoon:

Why the sample variance divides by n − 1

Mon, 19 Feb 2024 00:00:00 +0000

Every introductory course defines the sample variance with $n - 1$ in the denominator, and most students accept it on trust. The reason fits in three lines, and so does the caveat that usually goes unmentioned.

Definition — Sample variance

For observations

x_1, \dots, x_n

with mean

\bar x = \frac{1}{n} \sum_{i} x_i

, let

S = \sum_{i=1}^{n} (x_i - \bar x)^2

. The sample variance is

s^2 = S / (n - 1)

; the plug-in variance is

\hat\sigma^2 = S / n

The bias, and its correction

Theorem — Unbiasedness

X_1, \dots, X_n

are independent with mean

\mu

and variance

\sigma^2 < \infty

, then

\mathbb{E}[S] = (n - 1)\,\sigma^2

. Consequently

\mathbb{E}[s^2] = \sigma^2

, while

\mathbb{E}[\hat\sigma^2] = \frac{n-1}{n}\,\sigma^2

Proof.

Split each deviation from the true mean into a deviation from the sample mean and the error of the sample mean. The cross terms sum to zero, which gives the identity $\sum_i (X_i - \mu)^2 = S + n (\bar X - \mu)^2$ . Taking expectations,

\begin{aligned} \mathbb{E}[S] &= \sum_{i=1}^{n} \mathbb{E}\bigl[(X_i - \mu)^2\bigr] - n\, \mathbb{E}\bigl[(\bar X - \mu)^2\bigr] \\ &= n \sigma^2 - n \operatorname{Var}(\bar X) = n\sigma^2 - \sigma^2 = (n - 1)\, \sigma^2, \end{aligned}

because independence gives $\operatorname{Var}(\bar X) = \sigma^2 / n$ .

Two series, one axis each

Fri, 08 Sep 2023 00:00:00 +0000

Few chart forms are as tempting, or as quietly misleading, as the line chart with two vertical axes. The motivation is honest: two quantities with different units seem to move together, and putting them on one chart invites the reader to compare them. The trouble is that the comparison the chart makes is not in the data. It is in the choice of axes.

The crossing is a design decision

A line chart maps values to vertical positions. With one axis, the mapping is shared, and every visual relationship between the lines is a relationship between the numbers: one line above another means one value exceeds the other. With two axes, each series gets its own mapping, chosen independently. Where the lines sit relative to each other, whether they cross, and how steep they look are all consequences of two arbitrary pairs of numbers: the limits of the right-hand axis.