Path integral control (Kappen, 2005) provides an analytic expression for the optimal path distribution of a stochastic optimal control problem, given any sampling distribution with sufficient support. In the context of GSBM, it is used to debias the Gaussian path approximation of the CondSOC solution.
Given a reference process with drift , the optimal path-integral solution to the CondSOC problem (Proposition 4 of GSBM) is , where the importance weight is:
The key insight is the connection to information-theoretic stochastic optimal control (Theodorou et al., 2010): the -norm control cost allows the KL divergence between controlled and uncontrolled processes to be computed analytically via Girsanov’s theorem, yielding the importance weight formula. When , the optimal solution reduces to the Brownian bridge — the Brownian motion conditioned on reaching , which is exactly the reference process used in DSBM.
In GSBM, the Gaussian path approximation (optimized via splines) serves as the reference distribution , and path integral resampling draws samples proportionally to . This improves performance particularly at low noise (), at the cost of ~8% additional runtime and requiring sequential simulation.
Key Details
- Provides exact optimal path distribution via importance reweighting of any reference process
- Requires (stochastic processes only) and differentiable
- Lower variance when reference is closer to optimal (motivates using optimized Gaussian paths as )
- Reduces to Brownian bridge conditioning when
- Connected to linearly-solvable MDPs (Todorov, 2007) and probabilistic inference formulation of control (Levine, 2018)
- Optional step in GSBM (Alg. 4): empirically helps most at low noise