Diffbrief Engineering daily

← Friday, October 9, 2026Models & trainingBenchmark

Jevman compares decision models across 100 Pac-Man games each

opper.ai Editor score 6/10

The benchmark has jev, kev, clef, and several GPT models play 100 games each, with results on a leaderboard and code available to run other models. It offers a reproducible game-based comparison, though its value for broader agent evaluation is unclear.

What
Jevman benchmarks AI decision models playing Pac-Man against the classic scripted ghosts.
How
The game sends maze state as JSON to a model endpoint, which returns direction probabilities; late answers use a backup rule.
Why it matters
Recorded, replayable games and CI score checks give engineers a way to inspect and verify model evaluations.
maze JSON decision requ… probabilities direction probabilities over 2 seconds fallback move game HTTP endpoint backup rule model
The game sends decisions to a model endpoint and falls back to a rule when an answer is late. Sketched from the article's own text.

Read on opper.ai → Hacker News · 11 comments

Discussion

Discussion is not switched on yet. Use Note above to keep your own thoughts on this story.

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.

Your note

Private to you. Stored in this browser, and synced to your account when you are signed in.