← Wednesday, October 14, 2026Models & trainingPaper

TPPO tracks token-level value changes for language-model reasoning

huggingface.co by Futing Wang et al. Editor score 8/10

TPPO builds value estimates from additive position-wise outputs and reports aggregate reasoning-score gains of 7.0 and 8.4 points over PPO on two Qwen3 base models, with results extending to other models and tasks. Its released implementation makes the alternative value parameterization available for testing in RL training.

What: - Tracking-Value PPO (TPPO) is a cumulative value parameterization for language-model reasoning.
Method: - It uses a Bayesian decomposition of successive prefix values, adds position-wise scalar outputs in log-odds space, and trains reconstructed values with terminal-outcome supervision.
Why it matters: - TPPO improves aggregate mathematical reasoning scores over PPO by 7.0 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base, respectively, and yields more reliable advantage estimates.

Terminal-outcome supervision trains scalar outputs that accumulate into value logits. Sketched from the article's own text.

Read on huggingface.co → Code · 2 stars

Discussion

Discussion is not switched on yet. Use Note above to keep your own thoughts on this story.

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.