← Saturday, October 3, 2026AgentsBenchmark

GPT-6 Luna was right 68% of the time when it claimed 99% confidence

anth.us 11 min read Editor score 7/10

In 3,600 reasoning problems, Luna’s answers marked at least 99% confident were correct 68% of the time; the test is a proxy, not an evaluation of the Decisions API itself. Teams routing agent actions by confidence should scrutinize calibration before relying on thresholds.

What: - Decisions API selects one answer from a developer-defined list so software can branch on classifications, routes, or agent actions.
How: - In 3,600 ProofWriter problems, Luna was correct only 68% of the time when reporting at least 99% confidence.
Why it matters: - Engineers routing uncertain cases to people need calibrated confidence to set thresholds and direct work reliably.

A confidence-calibration pipeline checks model probabilities against labeled answers before routing. Sketched from the article's own text.

Read on anth.us → Hacker News · 15 comments

Discussion

Discussion is not switched on yet. Use Note above to keep your own thoughts on this story.

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.