← Thursday, October 8, 2026AgentsPaper

SWE-Game benchmarks coding agents on 247 executable game tasks

huggingface.co by Xiaoyu Chen et al. Editor score 8/10

SWE-Game evaluates agents on 247 tasks across 41 Godot games, using execution checks, replay, and feature demonstrations; the best construction-task scores remain below 60/100. Its task design and failure analysis provide a concrete way to assess coding-agent capabilities.

What

SWE-Game benchmarks coding agents on 247 tasks from 41 executable Godot games across 13 2D and 3D categories.

Method

Evaluation combines engine-state checks, reference-input replay, agent demos, and visual rubrics; reviewed failures often omit requirements or contain logic errors.

Why it matters

Executable checks reached 92.59% balanced accuracy on human-labeled behaviors, versus 78.41% for a video-based VLM judge.

Read on huggingface.co →

Headline, summary, brief and diagram are written by a model from the source's own text; the link, byline and numbers come from the source. Read the original before relying on a claim.