Posts on DataLeaf

Least squares, three ways

Tue, 18 Aug 2026 00:00:00 +0000

Polynomial fits make the difference visible: with twelve columns, the normal equations lose every digit while QR and the SVD keep eight or nine. One identity about condition numbers explains it, and also says when the fast method is good enough.

Notebook: rerunning an analysis from two years ago

Fri, 14 Nov 2025 00:00:00 +0000

A reviewer asked for one more column in a table produced two years ago. Adding a column means rerunning the analysis, and rerunning it means first showing that the old numbers come back. This entry records what that took.

Checklist

Check out the tagged commit that produced the submitted table
Recreate the environment from the lock file
Verify the input data against the recorded checksums
Rerun the pipeline end to end and compare every cell
Record the seed of every random step
Add the new column and log the rerun

The first four steps took an afternoon:

Adding up ten million numbers

Tue, 03 Jun 2025 00:00:00 +0000

A floating-point sum looks like the most innocent computation in a program. It is computed one addition at a time, $\hat s_k = \operatorname{fl}(\hat s_{k-1} + x_k)$ , and every addition rounds its result to the nearest representable number. Each rounding is tiny, but a long sum performs millions of them, and their errors do not have to cancel.

How large can the error get?

For recursive summation, the classic bound is¹

A note on log-sum-exp

Wed, 02 Oct 2024 00:00:00 +0000

Log-likelihoods, mixture models, and softmax layers all need the same quantity,

\operatorname{LSE}(x_1, \dots, x_n) = \log \sum_{i=1}^{n} e^{x_i},

and the direct formula fails at the first large argument. In double precision, $e^{x}$ overflows to infinity once $x$ exceeds about 709.78, and underflows to zero once $x$ falls below about $-745$ . Log-likelihoods of a few thousand observations easily reach either range.

The fix is one line of algebra. For any constant $m$ ,

\log \sum_{i=1}^{n} e^{x_i} = m + \log \sum_{i=1}^{n} e^{x_i - m},

and choosing $m = \max_i x_i$ makes the largest exponent exactly zero. Every term is then at most one, so nothing overflows, and the largest term is exactly one, so the sum is at least one and its logarithm is finite.

Why the sample variance divides by n − 1

Mon, 19 Feb 2024 00:00:00 +0000

Every introductory course defines the sample variance with $n - 1$ in the denominator, and most students accept it on trust. The reason fits in three lines, and so does the caveat that usually goes unmentioned.

Definition — Sample variance

For observations

x_1, \dots, x_n

with mean

\bar x = \frac{1}{n} \sum_{i} x_i

, let

S = \sum_{i=1}^{n} (x_i - \bar x)^2

. The sample variance is

s^2 = S / (n - 1)

; the plug-in variance is

\hat\sigma^2 = S / n

The bias, and its correction

Theorem — Unbiasedness

X_1, \dots, X_n

are independent with mean

\mu

and variance

\sigma^2 < \infty

, then

\mathbb{E}[S] = (n - 1)\,\sigma^2

. Consequently

\mathbb{E}[s^2] = \sigma^2

, while

\mathbb{E}[\hat\sigma^2] = \frac{n-1}{n}\,\sigma^2

Proof.

Split each deviation from the true mean into a deviation from the sample mean and the error of the sample mean. The cross terms sum to zero, which gives the identity $\sum_i (X_i - \mu)^2 = S + n (\bar X - \mu)^2$ . Taking expectations,

\begin{aligned} \mathbb{E}[S] &= \sum_{i=1}^{n} \mathbb{E}\bigl[(X_i - \mu)^2\bigr] - n\, \mathbb{E}\bigl[(\bar X - \mu)^2\bigr] \\ &= n \sigma^2 - n \operatorname{Var}(\bar X) = n\sigma^2 - \sigma^2 = (n - 1)\, \sigma^2, \end{aligned}

because independence gives $\operatorname{Var}(\bar X) = \sigma^2 / n$ .

Two series, one axis each

Fri, 08 Sep 2023 00:00:00 +0000

Few chart forms are as tempting, or as quietly misleading, as the line chart with two vertical axes. The motivation is honest: two quantities with different units seem to move together, and putting them on one chart invites the reader to compare them. The trouble is that the comparison the chart makes is not in the data. It is in the choice of axes.

The crossing is a design decision

A line chart maps values to vertical positions. With one axis, the mapping is shared, and every visual relationship between the lines is a relationship between the numbers: one line above another means one value exceeds the other. With two axes, each series gets its own mapping, chosen independently. Where the lines sit relative to each other, whether they cross, and how steep they look are all consequences of two arbitrary pairs of numbers: the limits of the right-hand axis.

What a research notebook is for

Fri, 21 Apr 2023 00:00:00 +0000

Laboratory scientists learn early that an experiment not written down did not happen. The bound notebook, with numbered pages and dated entries in ink, is one of the oldest instruments of experimental science, and it survives because it does something no other record does: it captures what the experimenter knew and intended at the moment of acting.

Computational work has inherited the experiments but not, in most places, the notebook. A modern analysis leaves behind a repository full of code, a directory full of outputs, and perhaps a commit history. It is easy to believe that this is a complete record. It is a record of what was done. It is almost never a record of why.