In machine learning, estimating or optimizing "log-sum-exp" functions is a common task, used for normalizing probabilistic models, approximating the maximum, and in reinforcement learning via entropy regularization. However, a major challenge arises when the variance of these estimates explodes, particularly when the potential function takes large values.
New Framework for Relative Density Estimation Circumvents Exploding Variance in Log-Sum-Exp Functions
Researcher Francis Bach has introduced a new framework for relative density estimation that addresses this exploding variance. The method employs a continuum of stable least-squares problems, making it computationally feasible through a single generalized eigenvalue decomposition.
While the variational approach is effective when there are large numbers of observations, the new spectral method becomes preferable when the number of observations is small. This approach simplifies feature learning by removing the need for normalized features. Additionally, the framework provides a closed-form estimator for softmax regression in cases where the sum or integral can be computed.
Sources
- Exploding variance of means of exponentials: least-squares to the rescue （Hacker News Frontpage, 2026-09-25）