Selection bias (also called the “winner’s curse” or regression to the mean) is the phenomenon where extreme observed values z_i systematically overestimate their corresponding true parameters mu_i. When z_i ~ N(mu_i, sigma^2) and we select the largest z_i values, these are large partly because mu_i is large (signal) and partly because the noise term pushed them upward (luck). The evanescence of this luck factor causes the selected z_i’s to be biased upward relative to their true mu_i’s.
Generalized Tweedie’s formula provides an elegant correction: E{mu|z} = z + sigma^2 * l’(z), where the score l’(z) = d/dz log f(z) is always negative in the tails, automatically shrinking extreme values back toward the center. The magnitude of the correction depends on the shape of the marginal density — steeper log-density tails produce larger corrections. This is exactly the denoising operation performed by diffusion models: the score function pulls noisy samples back toward the data manifold, correcting for the “selection bias” induced by the forward noising process.
The connection to diffusion bridges is direct: in a Brownian random bridge from x to y over [0,T], the drift at time t decomposes as (xi_t - x)/t + sigma^2 * nabla log p_t — a source-push plus a score-based correction. The score term corrects for the “bias” of the noisy intermediate state xi_t relative to the target distribution, functioning as a continuous-time analogue of Tweedie’s selection bias correction.
Key Details
- Extreme z_i values overestimate mu_i due to favorable noise realizations
- Tweedie’s correction l’(z) is always negative for extreme z — provides automatic shrinkage
- Correction magnitude: for the 100 largest out of 5000 in Efron’s example, corrections range from -0.97 to -1.40
- Bayes rule is immune to selection bias (Dawid, 1994); empirical Bayes inherits this approximately
- Direct analogue in diffusion models: score-based denoising corrects the “bias” of noisy states