Notebook: rerunning an analysis from two years ago

A reviewer asked for one more column in a table produced two years ago. Adding a column means rerunning the analysis, and rerunning it means first showing that the old numbers come back. This entry records what that took.

• Check out the tagged commit that produced the submitted table
• Recreate the environment from the lock file
• Verify the input data against the recorded checksums
• Rerun the pipeline end to end and compare every cell
• Record the seed of every random step
• Add the new column and log the rerun

The first four steps took an afternoon:

git switch --detach analysis-2023-v2
sha256sum --check data/SHA256SUMS
python -m venv .venv
.venv/bin/pip install --requirement requirements.lock
.venv/bin/python run.py --table 3 --output rerun/

The checksums matched, the environment installed, and every point estimate in the table came back identical to the last printed digit. The confidence intervals did not.

The pipeline drew its bootstrap resamples from a generator seeded from the operating system. That is a reasonable default for a single run and a poor one for a result that must be reproduced: the original seed was never written down, so the original resamples cannot be recovered.

The question worth answering is whether the change matters. The bootstrap interval is itself a Monte Carlo estimate, so it has its own sampling error, which shrinks as the number of resamples $B$ grows. To measure it, I took a sample like the one in the table, 60 skewed observations with mean 1.061, and computed the same 95 % percentile interval for the mean with thirty different seeds.

With $B = 1000$, the upper endpoint ranged from 1.250 to 1.287 across seeds. Its standard deviation falls roughly as $1 / \sqrt{B}$:

The original analysis used 1,000 resamples, a common recommendation for percentile intervals.¹ The differences between the published and the rerun intervals were all within one Monte Carlo standard deviation. The conclusions stand; the third decimal never meant anything.

Seed

The integer that initializes a random generator. Write it down with the name of the generator and the version of the library, and pass it explicitly to every function that draws random numbers.

Monte Carlo error

The variability of a result that comes from simulation rather than from the data. Report it, or choose enough resamples that it is smaller than the precision you print.

Lock file

A list of the exact version of every package an analysis used, including indirect dependencies. It reproduces the environment; it does not reproduce randomness.

The two open items on the checklist are now done: every random step takes a seed from the configuration file, and the configuration file is part of the commit.