A note on log-sum-exp

Wed, 02 Oct 2024 00:00:00 +0000

Log-likelihoods, mixture models, and softmax layers all need the same quantity,

\operatorname{LSE}(x_1, \dots, x_n) = \log \sum_{i=1}^{n} e^{x_i},

and the direct formula fails at the first large argument. In double precision, $e^{x}$ overflows to infinity once $x$ exceeds about 709.78, and underflows to zero once $x$ falls below about $-745$ . Log-likelihoods of a few thousand observations easily reach either range.

The fix is one line of algebra. For any constant $m$ ,

\log \sum_{i=1}^{n} e^{x_i} = m + \log \sum_{i=1}^{n} e^{x_i - m},

and choosing $m = \max_i x_i$ makes the largest exponent exactly zero. Every term is then at most one, so nothing overflows, and the largest term is exactly one, so the sum is at least one and its logarithm is finite.

Why the sample variance divides by n − 1

Mon, 19 Feb 2024 00:00:00 +0000

Every introductory course defines the sample variance with $n - 1$ in the denominator, and most students accept it on trust. The reason fits in three lines, and so does the caveat that usually goes unmentioned.

Definition — Sample variance

For observations

x_1, \dots, x_n

with mean

\bar x = \frac{1}{n} \sum_{i} x_i

, let

S = \sum_{i=1}^{n} (x_i - \bar x)^2

. The sample variance is

s^2 = S / (n - 1)

; the plug-in variance is

\hat\sigma^2 = S / n

The bias, and its correction

Theorem — Unbiasedness

X_1, \dots, X_n

are independent with mean

\mu

and variance

\sigma^2 < \infty

, then

\mathbb{E}[S] = (n - 1)\,\sigma^2

. Consequently

\mathbb{E}[s^2] = \sigma^2

, while

\mathbb{E}[\hat\sigma^2] = \frac{n-1}{n}\,\sigma^2

Proof.

Split each deviation from the true mean into a deviation from the sample mean and the error of the sample mean. The cross terms sum to zero, which gives the identity $\sum_i (X_i - \mu)^2 = S + n (\bar X - \mu)^2$ . Taking expectations,

\begin{aligned} \mathbb{E}[S] &= \sum_{i=1}^{n} \mathbb{E}\bigl[(X_i - \mu)^2\bigr] - n\, \mathbb{E}\bigl[(\bar X - \mu)^2\bigr] \\ &= n \sigma^2 - n \operatorname{Var}(\bar X) = n\sigma^2 - \sigma^2 = (n - 1)\, \sigma^2, \end{aligned}

because independence gives $\operatorname{Var}(\bar X) = \sigma^2 / n$ .

Probability on DataLeaf

A note on log-sum-exp

Why the sample variance divides by n − 1

The bias, and its correction