← Monday, October 19, 2026AgentsPaper

SciΠvot finds four of five frontier models below chance on research judgments

huggingface.co by Youran Sun et al. Editor score 8/10

SciΠvot tests 248 decisions drawn from six months of agent-assisted research and finds four of five frontier models perform below chance on selected cases. It highlights how agent evaluations can expose failures in context-dependent judgment, not just task execution.

What

SciΠvot is a 248-case benchmark of scientific judgment built from human corrections to LLM agents’ research decisions.

Method

Models choose between an original LLM plan and a human-corrected alternative using the project record; four of five frontier models scored below chance on selected cases.

Why it matters

The results show that autonomous research agents can struggle with pivotal decisions, even when given project context.

Read on huggingface.co →

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.