← Sunday, October 4, 2026AgentsPaper
Strict tool-call checks expose false positives in small-model benchmarks
Two models score similarly on lenient tool-use metrics, but verbatim checks find valid generalized calls in 6/6 examples for one and 0/6 for the other; targeted SFT raises the latter’s valid emissions from 0.100 to 0.959 on corpus rows. The diagnostic ladder and repair recipe are directly useful for evaluating tool-using models.
- What
- The study finds that keyword-matching benchmarks can falsely credit small language models for tool use they do not perform.
- Method
- Verbatim reproduction checks and a first-token probe expose a tool-call failure; targeted supervised fine-tuning repairs it, with gains on corpus rows and unseen prompts.
- Why it matters
- Strict, cheap diagnostics can help engineers distinguish genuine tool-use capability from benchmark false positives in small models.
Discussion
Discussion is not switched on yet. Use Note above to keep your own thoughts on this story.
Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.
