Numerical Methods on DataLeaf

Least squares, three ways

Tue, 18 Aug 2026 00:00:00 +0000

Polynomial fits make the difference visible: with twelve columns, the normal equations lose every digit while QR and the SVD keep eight or nine. One identity about condition numbers explains it, and also says when the fast method is good enough.

Adding up ten million numbers

Tue, 03 Jun 2025 00:00:00 +0000

A floating-point sum looks like the most innocent computation in a program. It is computed one addition at a time, $\hat s_k = \operatorname{fl}(\hat s_{k-1} + x_k)$ , and every addition rounds its result to the nearest representable number. Each rounding is tiny, but a long sum performs millions of them, and their errors do not have to cancel.

How large can the error get?

For recursive summation, the classic bound is¹

A note on log-sum-exp

Wed, 02 Oct 2024 00:00:00 +0000

Log-likelihoods, mixture models, and softmax layers all need the same quantity,

\operatorname{LSE}(x_1, \dots, x_n) = \log \sum_{i=1}^{n} e^{x_i},

and the direct formula fails at the first large argument. In double precision, $e^{x}$ overflows to infinity once $x$ exceeds about 709.78, and underflows to zero once $x$ falls below about $-745$ . Log-likelihoods of a few thousand observations easily reach either range.

The fix is one line of algebra. For any constant $m$ ,

\log \sum_{i=1}^{n} e^{x_i} = m + \log \sum_{i=1}^{n} e^{x_i - m},

and choosing $m = \max_i x_i$ makes the largest exponent exactly zero. Every term is then at most one, so nothing overflows, and the largest term is exactly one, so the sum is at least one and its logarithm is finite.