Off-Policy Continuous Control: DDPG, TD3, and SAC

Where we are. Weeks 7–8 optimized stochastic policies on-policy. This chapter — the last of the RL foundations — turns to off-policy continuous control, where two ideas dominate the model-free baselines. First, the deterministic policy gradient (DDPG) makes continuous actions tractable by pushing the gradient through a differentiable critic instead of sampling them. Second, maximum-entropy RL (SAC) augments the reward with policy entropy, which both stabilizes learning and reveals a deep identity: maximum-entropy RL is KL-regularized optimal control — the bridge out of learning and into Part II. Between them, TD3 fixes the overestimation that Chapter 6 first diagnosed, now arising from the critic’s own bootstrap.

The deterministic policy gradient

With a continuous action space the stochastic policy gradient (Chapter 7) must average the score over actions — expensive and high-variance. A deterministic policy $\mu_\theta:\statespace\to\actionspace$ avoids the action integral, and its gradient flows through the critic by the chain rule.

This is the continuous-action analogue of greedy improvement: where a discrete agent takes $\argmax_a \qfn$, the deterministic actor takes a gradient step in $a$ toward larger $\qfn$.

Overestimation returns: TD3

DDPG inherits the deadly-triad fragility and the overestimation bias of Chapter 6 — now produced by the critic bootstrapping on its own optimistic estimates. TD3 Fujimoto et al. (2018) applies three fixes, the first echoing Double DQN:

1. Twin critics, take the minimum. Train two critics $\qfn_{\phi_1},\qfn_{\phi_2}$ and form the target with $\min(\qfn_{\phi_1},\qfn_{\phi_2})$ — clipped double Q-learning, a continuous cousin of Double DQN that caps the upward bias of Proposition 6.1.
2. Delayed policy updates. Update the actor (and targets) less often than the critics, so the policy chases a more settled value.
3. Target policy smoothing. Add clipped noise to the target action, regularizing the critic against sharp peaks it could otherwise exploit.

The first is the load-bearing one: taking the minimum of two independent critics systematically underestimates, which is far safer for a bootstrapped target than the overestimate the single-critic max produces.

Maximum-entropy RL and SAC

A different idea reshapes the objective itself. Maximum-entropy RL adds the policy’s entropy $\mathcal{H}(\policy(\cdot\mid s))$ to the reward, scaled by a temperature $\alpha$:

The agent is rewarded for acting well and for staying stochastic — sustaining exploration and robustness. Soft actor-critic Haarnoja et al. (2018) is the off-policy actor-critic for this objective, with an automatically tuned temperature Haarnoja et al. (2019) . The one-step soft problem has a clean optimum.

The entropy bonus of Chapter 7 has been promoted from a heuristic to the objective, and its optimum is a temperature-controlled softmax over the action-value.

The dynamic-programming bridge

Maximum-entropy RL is where learning rejoins control. The soft value $\alpha\log\sum_a\exp(\qfn/\alpha)$ is the optimal cost-to-go of a KL-regularized control problem: maximizing reward minus $\alpha\,\mathrm{KL}(\policy\,\|\,\policy_0)$ against a reference $\policy_0$ yields exactly the Boltzmann policy of Proposition 9.1, and Todorov’s linearly-solvable MDPs exploit precisely this — under the exponential transform $z = \exp(\valuefn_{\text{soft}}/\alpha)$ the soft Bellman equation becomes linear. Three threads close Part I:

• Continuous-action improvement is the deterministic actor (DPG) or the soft Boltzmann policy (SAC), replacing the discrete $\argmax$ — approximate policy iteration (Chapter 1) in a continuum.
• Overestimation control (TD3’s twin-min) is the same bias management as Double DQN (Chapter 6), now load-bearing because the bootstrap runs through a critic.
• To Part II. KL-regularized control, the maximum-entropy LQR with its closed form, and the deterministic limit ($\alpha\to0$) that is classical optimal control are the entry points to LQR (Week 13) and MPC (Week 15) — the model-based, deterministic side of the same fixed point.

What’s next

• Week 10 steps back to ask where RL has actually worked in the real world — a survey of robotics successes (locomotion, manipulation, flight, plasma control) and the conditions that made them possible.
• Part II (Week 11+) then changes register entirely, to control theory: stability, LQR, and model predictive control — met now from the RL side, and rejoined with it in Part III.

Exercises

1. (Derive) Starting from $J(\theta)=\E_{s\sim\rho}[\qfn_{\mu_\theta}(s,\mu_\theta(s))]$, derive the deterministic policy gradient by the chain rule (Theorem 9.1).

   Solution

   $\nabla_\theta\qfn(s,\mu_\theta(s)) = \nabla_\theta\mu_\theta(s)\,\nabla_a\qfn(s,a)|_{a=\mu_\theta(s)}$ by the chain rule (the explicit $s$-dependence of $\qfn$ is held fixed); taking the expectation over $s\sim\rho$ gives Theorem 9.1. The state distribution $\rho$ may be off-policy, which is why DDPG can learn from a replay buffer.

2. (Prove) Show the maximum-entropy one-step optimal policy is $\policy^*(a\mid s)\propto\exp(\qfn(s,a)/\alpha)$ with soft value $\alpha\log\sum_a\exp(\qfn(s,a)/\alpha)$ (Prop. 9.1).

   Solution

   Maximize $\sum_a\policy(\qfn-\alpha\log\policy)$ under $\sum_a\policy=1$; stationarity gives $\qfn-\alpha\log\policy-\alpha-\lambda=0$, so $\policy\propto\exp(\qfn/\alpha)$. Substituting the normalized policy back yields the log-sum-exp soft value. The $\alpha\to0$ limit recovers the hard $\max$.

3. (Compute) A target state has twin-critic estimates $\qfn_{\phi_1}=2.0$, $\qfn_{\phi_2}=1.4$ for the target action. What value does TD3 use, and why is the minimum the safer choice for a bootstrap target?

   Solution

   TD3 uses $\min(2.0,1.4)=1.4$. The single-critic max/overestimate (Prop. 6.1) compounds through bootstrapping; taking the minimum of two independent estimates biases downward, and a slight underestimate does not amplify across the backup the way an overestimate does.

4. (Implement) In the companion, verify the twin-critic minimum lowers the target versus a single critic; that target-policy-smoothing noise is clipped to range; the Polyak soft target update; and that minimal TD3 learns Pendulum above the random return.

   Solution

   See experiments/python/week09/test_td3.py: the clipped-double-Q target equals the per-sample minimum of the twin critics (≤ either); the smoothing noise and target action respect their clip bounds; the Polyak update matches its closed form; and a seeded TD3 run on Pendulum-v1 clears the random-return baseline by a wide margin.

5. (Extend) Sweep the SAC temperature $\alpha$ and relate the limit $\alpha\to0$ (greedy) and large $\alpha$ (uniform). (The roadmap’s JAX/Brax SAC baseline is deferred to the dedicated JAX track.)

   Solution

   Small $\alpha$ concentrates the Boltzmann policy on the $\argmax$ (exploitation, recovering ordinary RL); large $\alpha$ flattens it toward uniform (maximal exploration). Automatic temperature tuning Haarnoja et al. (2019) adjusts $\alpha$ to hold a target entropy rather than fixing it by hand.

Companion code

The Week-9 companion lives at experiments/python/week09/ and is a minimal TD3 on Pendulum-v1 (the chapter’s testable centerpiece), with Stable-Baselines3 named as the reference baseline.

• td3.py — a continuous-action ReplayBuffer, a deterministic Actor and twin Critics, the exposed td3_target (clipped double-Q with target-policy smoothing), Polyak soft_update, and the training loop. Pure PyTorch.
• test_td3.py — component-correctness tests (the twin-critic minimum lowers the target; smoothing-noise and target-action clipping; the Polyak update’s closed form) plus a seeded Pendulum-v1 run learning well above the random return.

# component tests + a seeded Pendulum learning check (PyTorch; ~1-2 min on CPU)
PYTHONPATH=. pytest experiments/python/week09/test_td3.py -q

# worked minimal-TD3 training run on Pendulum
PYTHONPATH=. python experiments/python/week09/td3.py --steps 40000