RL Surrogate Losses
in 2026

For response reward \(R_i\), group mean \(\bar R\), and a small stabilizer \(\epsilon\):

Identical group rewards give zero relative signal. Recipes differ in standard-deviation conventions and further normalization; our experiment uses the sample standard deviation.

The denominator must be explicit. Here \(q\) is the recorded rollout probability. Some recipes instead use a trainer snapshot and add a separate correction for inference mismatch. A token ratio alone does not fully correct the distribution of earlier prefixes.

Source snapshots checked September 15, 2026. Support does not establish relative throughput.

Our Miles experiment uses the custom hook. Check the reduction separately: Miles calls its default “sample mean”; verl and SkyRL also expose alternative reductions.

Implemented in verl and SkyRL, dual-clip limits unusually large negative-advantage terms. For a constant \(\kappa>1\):

At \(r=4,\ A=-1,\ \kappa=3\), ordinary PPO keeps \(c=-4\). Dual-clip selects the constant objective \(-3\), so \(c=0\). Our runs use ordinary PPO.

Take \(A=-1,\ q=0.90,\ p=0.74\). PPO keeps \(c\approx-0.822\), inside its 0.8–1.2 bounds. DPPO rejects it: the decrease of 0.16 exceeds 0.15. Mass-based gating can be stricter for common tokens.

The paper also studies binary Kullback–Leibler divergence and top-\(K\) approximations. Here DPPO means the binary-TV variant of Divergence PPO.

The pinned verl path additionally caps a detached importance weight: \(c=A\min(r,C)m\), for cap \(C\). The pinned SkyRL path and our experiment use the uncapped coefficient above, subject to the numerical guard documented with our results.

Let \(j_{igt}\) be token \(t\)’s objective in response \(g\) to prompt \(i\). There are \(B\) prompts, \(G\) responses per prompt, and \(L_{ig}\) valid tokens per response.

A trajectory gradient naturally sums token terms. Dividing by length deliberately reweights it; equal reduction weights do not guarantee equal gradient norms. Check the denominator and distributed normalization, because library names vary.

Other methods change different layers: RLOO changes the reward baseline; REINFORCE++ changes baseline and normalization choices; Dr. GRPO removes particular normalization biases.

Countdown

DeepMath

Six baseline evaluations per dataset used the same starting checkpoint. Countdown ranged from 48.44% to 50.39%; DeepMath from 67.97% to 73.44%. Average pairwise task-reward disagreement was 15.94% and 11.93%, respectively. Greedy inference was not bitwise repeatable.

The same 256 test tasks are reused across seeds. The paired task-bootstrap intervals condition on the realized models and outputs; they exclude training-seed and inference-runtime uncertainty. Disjoint study splits do not rule out base-model pretraining exposure.

Apply all three gates to the same PPO-run token probabilities. The share of nonzero-advantage, sequence-normalized token weight gated off was:

These are counting weights, not fractions of gradient norm. A rare token with a large ratio can matter disproportionately. Different updates then change future responses and rewards.

• Zero-advantage groups were common: 27.34%–62.89% of minibatch steps had zero fresh gradient. Adam momentum can still move weights.
• The full audit includes six rules. DAPO, CISPO, and SAPO columns are diagnostics, not trained arms.
• Guarded extremes need care: counterfactual weight columns omit log-ratio-clamp derivative saturation. The PPO traces above never activated that guard.

Countdown repetition. At 256 updates, GLM-5 seed 42 scored 10.55%; 89.06% of outputs hit the cap. Its first failed evaluation response repeated 145/ ? = ? until truncation.

All three methods failed that selected task. The excerpt illustrates the behavior; the curves measure its broader impact. Both GLM runs continued to their scheduled endpoints.

DeepMath grader limits. We inspected the first three zero-to-one reward transitions in evaluation order for GLM-5 seed 42. The recorded cases show:

1. An oscillating limit labeled 1 by the dataset; the rewarded response also boxed 1.
2. A response using Taylor expansions despite instructions forbidding them.
3. A baseline that derived 44 but truncated before boxing it; the trained response finished.

No labels changed after inspection. These three cases cannot estimate the prevalence of grader errors.

The rollout anchor stays fixed for all 16 minibatch updates. Then weights synchronize and generation resumes. Later minibatches therefore see a changed trainer even though collection is synchronous.

Data and rewards

• Countdown: four-number puzzles, checked with a restricted parser and exact rational arithmetic.
• DeepMath: integer-answer problems from the first shard, checked against the final boxed integer.
• Deduplicated, disjoint pilot/training/test splits. Binary rewards; no solution traces, partial credit, or special truncation penalty.

Validation and protocol changes

• Full-parameter audit: Adam state covers 8,953,803,264 language-policy parameters; weights changed in all 32 decoder layers. Unused vision and multi-token-prediction components are outside these text tasks.
• Gradient checks: analytic derivatives for six token surrogates; custom PPO matches native Miles loss and gradients on 1,000 randomized cases.
• Numerical guard: log ratios clamp to [−20,20]. It activated only in GLM runs, where those tokens were already rejected.
• Grader correction: a pilot exposed valid LaTeX arithmetic rejected by the parser. We fixed it, restarted from the starting checkpoint, and froze training code before the comparison.
• Runtime adjustment: DeepMath’s timing probe led to 28 collections for every arm. Its interrupted probe is excluded; the revised schedule used runtime, not algorithm scores.

Trainer/inference agreement

Before minibatch updates, run-mean absolute log-probability differences averaged 0.0047–0.0124 natural-log units per response. Token-level 99.9th percentiles were 0.155–0.250. Small averages did not eliminate outliers.

Countdown GLM-5 seed 43 had six differences above 5 among 4,039,764 positions, maximum 14.491. They appeared after 320 updates, after scores had fallen. Their cause remains unresolved.

Runtime includes initialization, compilation, generation, evaluation, and checkpointing. Shared caches and six-rule diagnostics prevent treating these timings as a loss-kernel speed comparison.

1. Schulman et al., Proximal Policy Optimization Algorithms (2017). Clipped surrogate.
2. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024). GRPO and its original sequence-mean objective.
3. DAPO: An Open-Source LLM Reinforcement Learning System at Scale (2025). Clip-higher and the broader training recipe.
4. MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention (2025), §3. CISPO.
5. Group Sequence Policy Optimization (2025). Sequence ratios and clipping.
6. Soft Adaptive Policy Optimization (2025). Smooth token gradients and Qwen3-VL practice.
7. Rethinking the Trust Region in LLM Reinforcement Learning (2026). DPPO; Binary and top-K divergence gates.
8. GLM-5: from Vibe Coding to Agentic Engineering (2026), §4.1.2. Direct rollout importance ratios and token rejection.
9. The Art of Scaling Reinforcement Learning Compute for LLMs (2025). ScaleRL; Controlled study including prompt, sequence, and token aggregation.
10. verl policy losses and actor defaults, pinned September 15, 2026.
11. SkyRL policy losses, pinned September 15, 2026. Executable operations, rather than comments alone, determine the gradients.
12. Miles source used for training.
13. Qwen3.5-9B checkpoint; Countdown-Tasks-3to4; DeepMath-103K. Exact data revisions and split hashes accompany the experiment.

@article{dong2026rlsurrogates,
  title = {RL Surrogate Losses in 2026},
  author = {Dong, Simon},
  year = {2026},
  url = {https://simondong1.github.io/rl-surrogates-2026.html}
}