Tweedie’s formula is a Bayesian estimation result that expresses the posterior mean of a latent variable in terms of the score function of the observed marginal density. In its classical Gaussian form (Robbins, 1956, attributed to Tweedie): if μ ~ g(·) and Z|μ ~ N(μ, σ²I), then E[μ|Z=z] = z + σ²∇_z log f(z), where f is the marginal density of Z. This formula is foundational in score-based generative modelling as it connects the score ∇log p_t to the denoising objective.
Shelley & Mengütürk (2025) generalize Tweedie’s formula to the generalized exponential family (GE). If η ~ g and Z|η ~ GE(h, ϕ, F, ψ; η) with ∇_z F(z) = 1/c for constant c ≠ 0, then:
E[ϕ(η)|Z=z] = (∇_z log f(z) - E[∇_z log h(z,η)|Z=z]) · c⁻¹
This extends beyond distributions where h(z,η) = h(z) (the standard exponential family), handling cases where the base measure depends on the natural parameter — precisely the situation arising with Poisson random bridges, whose product-binomial distribution belongs to GE but not the standard exponential family.
For the standard exponential family subcase (h independent of η), the formula simplifies to E[ϕ(η)|Z=z] = ∇_z log f(z) - ∇_z log h(z), recovering the classical Tweedie result when applied to Gaussians.
Exponential Family Derivation (Efron 2011)
Efron derives Tweedie’s formula from the general exponential family: η ~ g(·), z|η ~ f_η(z) = exp(ηz - ψ(η))f₀(z). The posterior of η given z is itself an exponential family with CGF λ(z) = log(f(z)/f₀(z)), yielding posterior mean E{η|z} = λ’(z) and variance Var{η|z} = λ”(z). For the normal translation family (η = μ/σ²), this gives E{μ|z} = z + σ²l’(z) and Var{μ|z} = σ²(1 + σ²l”(z)). Log-concavity of f(z) guarantees shrinkage of uncertainty below σ². For Poisson data: E{μ|z} = (z+1)f(z+1)/f(z) (Robbins’ prediction formula). Variable σ²: when σ² depends on μ, the posterior ratio g(μ|z₀)/g₀(μ|z₀) involves a variance-ratio scaling (Theorem 7.1).
Key Details
- Classical form: E[μ|Z=z] = z + σ²∇log f(z), used in denoising score matching
- Decomposition: E{μ|z} = unbiased estimate + Bayes correction — same structure as denoising in diffusion models
- GE extension handles distributions where h(z,η) depends on η, as in Poisson bridges
- For Poisson bridges: the formula gives the drift in terms of ∇log p_t, ∇log h, and a log-ratio involving t/(T-t)
- The variance extension: V[ϕ(η)|Z=z] = ∇²_z log f(z) - ∇²_z log h(z) (for standard exponential family)
- The formula establishes the link between score functions and optimal Bayesian estimation across the exponential family