← Monday, September 28, 2026Models & trainingDeep dive

Least-squares estimation addresses exploding variance in log-sum-exp

francisbach.com 21 min read Editor score 6/10

The piece examines why sampling estimates of log-sum-exp can have high variance, especially when the potential function takes large values, and motivates a least-squares approach. These functions appear in probabilistic models, transformers, and entropy-regularized reinforcement learning.

What

Log-sum-exp estimates aggregate exponentials and appear in probabilistic-model normalization, smooth maxima, transformers, and reinforcement learning.

How

For Gaussian samples, the relative squared error of the exponential mean grows as (e^σ²−1)/n; taking the logarithm does not remove the exploding variance.

Why it matters

This matters in machine learning systems that estimate or optimize log-sum-exp from samples, where high variance can make computation difficult.

Read on francisbach.com → Hacker News

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.