Benchmarks
What OpenGameEval says about writing better Roblox AI tasks
July 25, 2026
The useful part of an AI benchmark is not the winner. It is the failure pattern. Roblox's latest OpenGameEval comparison of Claude models shows that Fable 5 does best when the task has a named target, a narrow scope, and a behavior that can be checked.
That is actionable whether you use Fable, Gemini, GPT, or a local model. Better task boundaries improve the conditions under which an agent can search, edit, and verify a Roblox game.
The headline scores
| Model | Code Pass@1 | Debug Pass@1 | Tool calls |
|---|---|---|---|
| Claude Opus 4.6 | 48.05% | 50.67% | 9.12 |
| Claude Opus 4.7 | 43.45% | 52.67% | 5.53 |
| Claude Fable 5 | 50.34% | 64.67% | 5.41 |
Fable's 6.9-point code-generation lead over Opus 4.7 did not reach statistical significance in this sample (p=0.08). Its debugging gain is larger: 12 points over Opus 4.7 and 14 points over Opus 4.6. Roblox describes that result as borderline significant on the 30-task debug set, with the consistency gain over Opus 4.6 reaching significance.
The safe reading is not “Fable is always 14 points better.” It is that the model's advantage becomes clearest when the task is constrained.
Fable searches differently, not more
Opus 4.7 cut average tool use from 9.12 calls per task to 5.53. Fable stayed at essentially the same budget: 5.41 calls. The change happened inside that budget.
| Search tool | Opus 4.6 | Opus 4.7 | Fable 5 |
|---|---|---|---|
search_game_tree | 2.11 | 1.31 | 0.90 |
script_grep | 1.31 | 0.48 | 0.82 |
inspect_instance | 1.17 | 0.72 | 0.48 |
Roblox summarizes the behavioral change neatly: Opus walks the scene graph; Fable greps the code. Fable reduced structural traversal again but brought textual code search back up. It found several cases that Opus 4.7 missed without spending more calls overall.
Why bounded debugging tasks win
A debug eval usually supplies one buggy script and one observable failure. That removes three expensive decisions: where to look, how much of the game is in scope, and what “done” means.
On those tasks, Fable improved six evals by at least 30 percentage points compared with Opus 4.7 and regressed on one. On the broader code set, it improved 15 and regressed on eight. The model was still better overall, but the result was less predictable when the instruction was open-ended.
Turn the finding into better prompts
Replace a broad request like this:
Fix the checkpoint system. It sometimes does not work.
With a bounded task:
Inspect ServerScriptService.Checkpoints. After a player touches Stage3, resetting sometimes sends them to Stage1. Find the root cause, change only the checkpoint system, then playtest by touching Stage3 and resetting twice. Report the file changed and the observed spawn each time.
The second version supplies four things:
- Target: the script or system to inspect
- Failure: the exact behavior that is wrong
- Boundary: what the agent may change
- Proof: the test that demonstrates success
A benchmark is not the whole game
OpenGameEval measures automated task success inside a controlled Studio environment. It does not directly score game design, long-term code maintainability, art direction, player retention, or whether a human team would accept the implementation.
The repository also changes. On July 24, Roblox corrected checks in three evals: one debug task and two code-generation tasks. Roblox has not published a rerun of the leaderboard since those changes, so the current table remains the June result. The fixes are a reason to keep watching the benchmark, not a reason to fabricate adjusted scores.
The practical recommendation
Use the leaderboard to choose a starting model. Use bounded tasks and verification to make that model reliable in your own game. A weaker model with a precise target and a test can be more useful than the benchmark leader facing a vague request across an unfamiliar project.
Frequently asked questions
What does OpenGameEval test?
OpenGameEval tests AI models inside Roblox Studio on 87 code-generation tasks and 30 debugging tasks using automated checks and five runs per task.
Which model currently leads OpenGameEval?
Claude Fable 5 leads Pass@1 on both published leaderboards: 50.34% on the 87 code-generation tasks and 64.67% on the 30 debugging tasks.
How should I write Roblox AI tasks based on the benchmark?
Name the target, define the expected behavior, constrain the scope, and ask the agent to test the result. Fable 5's clearest gains appeared on bounded debugging tasks with one named script and one specific failure.