Nitrode Contact

GameEngineBench

GameEngineBench evaluates whether frontier coding agents can make scoped C++ changes that compile and behave correctly inside real, running Unreal Engine 5 projects.

110 UE5 C++ tasks
9 Source projects
12 Model configs evaluated
55.5% Best pass rate

Leaderboard

How frontier coding agents perform on real UE5 tasks: judge-calibrated pass@1 for each model and effort setting on the active task set.

What it measures

Correctness in Unreal depends on the engine, not just the diff. A patch can compile, look plausible, and still fail because it misses replicated state, mishandles an actor lifecycle transition, or breaks a gameplay ability system it never touched directly.

Each task gives the agent a buildable start state, a behavior specification, and a scoped set of editable files, then programmatic tests and judge auditing check whether the resulting behavior is actually correct at runtime.

Key takeaways

  • The strongest configuration, Claude Fable 5 (max), now solves a majority of the task set at 55.5% calibrated pass@1, but every other evaluated configuration stays under 30%.
  • The hard part is runtime integration, not syntax: strong models usually compile and reach execution; the remaining failures are authority mistakes, replication/state-sync errors, actor lifecycle bugs, and incomplete integration with surrounding gameplay systems.
  • Failure is not evenly spread: save/persistence has the largest unresolved share, with weapons/combat, serialization, and AI/world orchestration also retaining unsolved tasks.
  • 79 of 110 tasks are solved by at least one configuration; 31 remain unsolved by every configuration evaluated, so model capabilities are complementary rather than strictly nested.

Where agents struggle

Dataset & task design

Core gameplay Multiplayer & replication Weapons & combat Inventory & interaction Gameplay ability systems Save / load & persistence AI & world orchestration Animation & movement UI / session systems XR / spatial Rendering & plugins

Methodology

01

Start state

Each task begins from a buildable Unreal project; no scaffolding to invent.

02

Scoped edit

The agent edits a constrained set of native C++ files against a behavior spec.

03

Programmatic tests

Tests run programmatically against the real Unreal build and runtime after the agent finishes.

04

Judge audit

LLM-as-judge checks requested behavior, not a match to a reference diff.

Failure analysis

Task 01, Unsolved by all

Zombie System

Requires AI control, round-state updates, server-authoritative damage, and replicated feedback to all agree; models solve pieces but rarely the whole coordination.

Task 19, Unsolved by all

Map Orchestrator

Procedural generation, actor pooling, and readiness signaling; failures are almost always ordering and lifecycle bugs, not generation logic itself.

Tasks 76–80, SPUD persistence

Save / load lifecycle

Plausible serialization code that still loses actor identity or state across streaming levels and engine-managed teardown boundaries.