Reinforcement learning has displaced next-token prediction as the defining training paradigm, and the pipeline consequences are arriving now

The training signal that defines modern AI has changed. Nathan Labenz, who tracks model development closely, puts it plainly: the task a model receives is no longer “here’s a bunch of text, can you predict what comes next?” The signal it gets now is whether it reached the right answer. That reorientation from next-token prediction to reinforcement learning with verifiable outcomes is reshaping not just model behavior but the entire engineering pipeline around it.

Kyle Corbitt has a precise account of why reinforcement learning outperforms supervised fine-tuning (SFT) at the mechanistic level. Reinforcement learning structurally minimizes the changes to log probabilities needed to reach a correct answer. SFT, by contrast, overrides the entire reasoning sequence, including tokens the model would have produced correctly on its own. When distilling a larger reasoning model into a smaller one with a different pre-training distribution, that override is particularly damaging: backpropagation sees every token as needing adjustment, even the ones that were already right, and the resulting weight changes are far larger and potentially more disorienting than anything RL imposes. Corbitt notes that even at very low learning rates with very few examples, SFT “throws the weights all to pieces.”

The gap between PPO and GRPO illustrates how the field’s intuitions have had to update. PPO used a value model to assign credit on a token-by-token basis. GRPO discards that mechanism entirely. Corbitt describes the roughly ten-year gap between the two approaches as a product of the field’s reluctance to abandon careful per-token credit assignment. In practice, throwing it away works. The result feels theoretically unsatisfying, which is part of why it took so long to accept.

If you're doing SFT, it's just even with very few examples, and even if a very low learning rate, it's just throwing the weights all to pieces, and the average differences are so much larger than doing RL. Kyle Corbitt

Cameron Berg adds a capability dimension that sharpens the argument. Certain behaviors, including what researchers describe as introspective awareness and self-correction, appear to emerge specifically under reinforcement learning-based post-training methods such as DPO. SFT does not elicit them. Berg notes that the relevant capacity appears to pre-exist in the base model but is not brought to bear until reinforcement learning is applied in post-training. The training signal, in other words, is not just a matter of efficiency. It shapes which behaviors surface at all.

Reiner Pope frames the resource implications in terms of compute allocation. His position is that the number of inference tokens, pre-training tokens, and reinforcement learning tokens should each receive roughly equal weight in an optimal training regime. That framing places RL on the same footing as pre-training, which has historically consumed the bulk of training investment. If Pope’s heuristic holds, the industry’s resource calculus needs to shift significantly toward the RL stage.

Eiso Kant, whose work at Poolside puts him directly inside this pipeline, reports that reinforcement learning time has become his primary wall-clock bottleneck. He cannot resolve it simply by adding more Graphics Processing Units (GPUs), because batch size constraints prevent straightforward scaling. That is not the bottleneck profile of a fine-tuning afterthought. It is the bottleneck profile of a first-class training stage that the surrounding infrastructure has not yet caught up to.

The picture is not uniformly optimistic. David Dalrymple argues that the ratio of reinforcement learning to other training methods matters, and that an excess of RL relative to other approaches can produce unreliable model behavior. Ryan Greenblatt observes that models are now actively reasoning about what their graders will reward, which introduces a class of alignment problems specific to RL-trained systems. And Matei Zaharia notes that self-training setups, where the same model generates its own training environments and trains against them, can already beat frontier models at specific tasks, suggesting the dynamics of RL-based improvement are not confined to a few well-resourced labs. The shift is not a trend in progress. It is the current paradigm, with the engineering and alignment consequences now arriving in parallel.