score matching

Score matching is the estimation of the score function ${\nabla }_x\log p(x)$ — the gradient of the log probability density — which is the fundamental building block of score-based generative models. Since the score does not require computing the normalizing constant, it can be estimated from data without knowing $p(x)$ explicitly.

Denoising score matching (Vincent, 2011) trains a neural network $s_{\theta }(x,t)$ to predict the score of noisy data distributions by minimizing ${\mathbb{E}}_{p(x_t|x_0)}[\Vert s_{\theta }(x_t,t)-{\nabla }_{x_t}\log p(x_t|x_0){\Vert }^2]$. For Gaussian perturbation kernels, the score of the perturbation is $\nabla \log p(x_t|x_0)=-(x_t-x_0)/{\sigma }_t^2$, making the denoising objective equivalent to predicting the clean data from noisy observations (connecting to Tweedie’s formula).

In the SDE framework (Song et al. 2021), a time-dependent network is trained via continuous weighted denoising score matching integrated over all noise levels. The learned score enables: (1) reverse-time SDE simulation for generation; (2) probability flow ODE for deterministic sampling and likelihood; (3) predictor-corrector sampling combining SDE solvers with Langevin MCMC.

Key Details

• ${\nabla }_x\log p(x)$ does not require normalizing constant
• Denoising score matching connects to Tweedie’s formula
• Enables reverse-time SDE via Anderson (1982)
• Continuous version integrates over all noise levels
• Foundation of SMLD, DDPM, and all SDE-based generative models
• Hyvärinen (2005) introduced score matching, Vincent (2011) established denoising connection

---

concept