Adding up ten million numbers

Tue, 03 Jun 2025 00:00:00 +0000

A floating-point sum looks like the most innocent computation in a program. It is computed one addition at a time, $\hat s_k = \operatorname{fl}(\hat s_{k-1} + x_k)$ , and every addition rounds its result to the nearest representable number. Each rounding is tiny, but a long sum performs millions of them, and their errors do not have to cancel.

How large can the error get?

For recursive summation, the classic bound is¹

A note on log-sum-exp

Wed, 02 Oct 2024 00:00:00 +0000

Log-likelihoods, mixture models, and softmax layers all need the same quantity,

\operatorname{LSE}(x_1, \dots, x_n) = \log \sum_{i=1}^{n} e^{x_i},

and the direct formula fails at the first large argument. In double precision, $e^{x}$ overflows to infinity once $x$ exceeds about 709.78, and underflows to zero once $x$ falls below about $-745$ . Log-likelihoods of a few thousand observations easily reach either range.

The fix is one line of algebra. For any constant $m$ ,

\log \sum_{i=1}^{n} e^{x_i} = m + \log \sum_{i=1}^{n} e^{x_i - m},

and choosing $m = \max_i x_i$ makes the largest exponent exactly zero. Every term is then at most one, so nothing overflows, and the largest term is exactly one, so the sum is at least one and its logarithm is finite.

Floating Point on DataLeaf

Adding up ten million numbers

How large can the error get?

A note on log-sum-exp