What is GRPO?

Kyle Corbitt on GRPO abandoning per-token credit assignment, and it working anyway.

“This is like I said a very unsatisfying answer. And I think that's one of the reasons why there was like an almost 10-year gap between PPO that had this value model that tried to you know, determine on a token by token basis and like GRPO where it's like hey, we're just going to throw that all away cuz it feels wrong. It feels like it shouldn't work. In practice it does though.”

GRPO discards per-token credit assignment entirely, upvoting all rare tokens equally regardless of which mattered, and still works in practice, defying prior intuitions from PPO.

“Now, you could ask the question well, it's like there could be many rare tokens. If you got like thousands, you know, tens of thousands of tokens in a reasoning trace, how do you decide which rare token is most important? And you don't. You just throw up your hands and you say all the rare tokens get up voted the same way.”

GRPO became dominant not due to algorithmic novelty but because DeepSeek did the scaling engineering and shipped a working artifact model that validated it.

“I think the reason GRPO specifically like that algorithm and that acronym like, you know, very concretely took off was not necessarily because it was like a big quantum leap on what came before. It was because DeepSeek did a lot of engineering work around actually scaling it and released an actual artifact model that worked really well with it.”

Turn-level credit assignment via GRPO advantage weighting is crucial for stable training on long-horizon tasks, because upweighting or downweighting behaviors at the turn level rather than the rollout level avoids credit misassignment.

“We use these weights to adjust the advantages during GRPO. So what we're really saying is that when you're upweing or downweing the behaviors we want to do that on a turn level doing GRPO rather than on a rollout level.”