Exploding variance of means of exponentials: least-squares to the rescue

Estimating log-sum-exp quantities like log ∫ e^{v(x)} dq(x) by sampling is common but statistically fragile: exponentiating amplifies variance so severely that sample averages of e^{z} (for Gaussian z) have relative error scaling (e^{σ^2}-1)/n, and taking the logarithm does not cure the explosion. The proposed remedy is to exploit the stability of least-squares by rewriting the Kullback-Leibler f(t)=t log t - t +1 (and other f-divergences) as integrals of “quadratic-over-affine” functions. That identity turns the original convex variational problem with log-sum-exp terms into a supremum over quadratic objectives, which correspond to weighted least-squares prediction problems with controlled variance.

Technically, for each mixing parameter ρ∈(0,1) the weighted chi-square fρ(t)=½ (t−1)^2/(ρ t+1−ρ) admits a quadratic variational form whose optimizer u(ρ,x) is explicit. Integrating these representations against a nonnegative measure over ρ recovers general f and produces two potentials v(x), w(x) as integrals of simple quadratic expressions in u(ρ,x). This yields a least-squares-based framework for relative density and KL estimation that preserves the benefits of log-sum-exp models (smoothness, normalization) while avoiding unstable empirical exponential averages, and it extends naturally to related quantities like mutual information.