Local model coding tests · Oct 29 – Nov 8, 2026

Coding Test Bench

About this page

The by-eye verdicts on this page are Claude's, an AI model made by Anthropic: for each run it read the screenshots, the grader's output and the code, and wrote the one-line note on the card. The owner did not judge each run, and no human panel was involved. One row (Ripple-65B Preview + dsh + kit) was also cross-checked by a second Claude reviewer, who first judged all 25 runs without seeing the first verdicts, then saw the first reviewer's notes and could keep or change its own verdicts. After that it agreed on 23 of the 25 runs and disagreed on 2 (note 8 under “What ran where” gives the outcome). Of the 220 runs the script graded, 17 (7.7%) were overruled by the eye against the script's verdict, and each of those is marked “grader overruled”. Most prompts ran once per configuration (the four KGP Talkie video tests ran three times), so gaps of one to three runs between configurations are within noise. The rows marked Round-3 kit or dsh harness + kit were given per-test reference notes (checklists, facts and a list of mistakes earlier models had made on these same tests) and extra helpers, and were revised between batches, so they are not like-for-like with the stock rows. Time limits, context windows and hardware also differed between rows, and the two rows that ran fewer than 25 runs are marked incomplete and left out of the ranking. Every prompt is credited to its source on its card: the four video tests come from a KGP Talkie YouTube video, and the others are credited to the people and benchmarks they came from, or marked new.

Good to know: 33 of the live previews load three.js from a public CDN (cdn.jsdelivr.net, or unpkg.com) and show a blank page if your browser or network blocks those hosts. Apart from those previews, this page makes no requests to other sites, and it uses the fonts on your own device.

On this page: Terms · What ran where · What was tested on what · Leaderboard · Credits

Terms used on this page

Claude Code: Anthropic's command-line coding agent. It is the harness for the stock rows (and, with extra tools, for the round-3 kit row): it reads the prompt, then writes files and runs commands in an empty folder until it decides it is done.
stock: One prompt and no add-ons: a single Claude Code call with no kit, no helper agents and no web tools.
Maya: Project Maya, a local inference runtime for very large models on a mix of GPU and system memory (description from the owner's notes). It served the GLM rows and one DeepSeek build (Maya-S).
Strata: The llama.cpp-derived inference engine that Maya is built on (description from the owner's notes). The two DeepSeek V4.1 Flash Coder rows ran on its Linux package for AMD GPUs, version 0.2.10.
dsh: An agent harness: a program that gives a model a shell, a look tool and helper agents, and keeps it working on a goal instead of one prompt and one answer. Used for two rows.
kit: Per-test reference notes handed to the model: checklists, facts and a list of mistakes earlier models made on the same test, shipped with tools and helper agents. The “round-3 kit” and the “dsh kit” are two different kits.
sub-agent: A helper that the main model starts for one job, such as testing or reviewing its own work. In the 27B and DeepSeek kit rows the helpers are the same model as the main one.
Bolt: NVIDIA Nemotron 3.5 Bolt, a 74-billion-parameter mixture-of-experts model with about 9 billion parameters active at a time. Run here at 2-bit and at 4-bit.
quantization: How far a model's weights are compressed (2-bit, 4-bit, IQ3_S, Q4_K_M and so on). Fewer bits make the file smaller and easier to fit, usually at some cost in quality. The letters name the scheme.
context window, seat: A seat is one conversation a server can hold at once. The context window is how many tokens (words and word pieces) a seat can hold: 131,072 is 128K.
expert offload: A mixture-of-experts model uses only some of its experts for each token. When the GPUs cannot hold all of them, the rest wait in system RAM or on disk and are fetched when needed, which is slower.
by eye: The verdict made by looking at the result itself: the screenshots, the running program and the code. On this page that look was Claude's (an AI model made by Anthropic), not a human panel's, and the owner did not judge each run.
grader: The script that re-runs each program in a container (or a headless browser), checks it automatically and saves the pictures.
overruled: A run where the by-eye verdict and the grader's verdict differ. Its card carries a “grader overruled” chip, and the by-eye verdict is the one counted.

The same prompts, word for word, given to GLM 5.3 Flash and to Nemotron Bolt at 2-bit and 4-bit, each working alone in stock Claude Code. Every result was graded by a script and then checked by eye (Claude's reading of the evidence, see About this page); the badges are the by-eye verdict. Bolt ran on a Linux server with two AMD Radeon AI PRO R9700 cards, the 2-bit on one and the 4-bit on both. For the first 21 of the 25 tests of the 4-bit run, the second card was held to a 210 W power limit (clock held at 2,350 MHz) because of a temporary power-supply limit^{† (from the owner’s notes)}; the last four tests ran at full power (note 3). Added Oct 31: Qwen 3.8 27B on that server's second card ( passed), a GLM retry of the seven tests it failed with no time limit and nothing else on GLM ( flipped), and DeepSeek V4.1 Flash on a Windows workstation with three NVIDIA GPUs ( passed; it finished on Nov 1). Added Nov 1: Qwen 3.8 27B again, this time inside the round-3 kit (sub-agents, a look tool that reports numbers and text thumbnails because the model cannot see pictures, fact packs, medium reasoning effort): the visual tests plus one run of each video test ( passed). Added Nov 3: DeepSeek V4.1 Flash Coder, ISTA's 2-bit build that keeps 192 of the 384 experts, on that server's two R9700 cards under Strata 0.2.10, in stock Claude Code like the full DeepSeek ( passed; it finished on Nov 4). The DeepSeek V4.1 Flash Coder + dsh + kit row ran all 25 again through the dsh agent harness with its kit (checklists, a look tool, tester and reviewer helpers; web tools off): it kept all 13 of the passes of the DeepSeek V4.1 Flash Coder · 2-bit, pruned row and fixed 8 of its 12 failures, so 21 of 25 by eye with no run getting worse, level with the full DeepSeek's 21 in stock Claude Code. The four that still fail are solar run 3 and all three chess runs. A further re-run of solar run 3 with a lighting note added to the kit passed but is not in the grid; counting it, 9 of the 12 failures are fixed and the row reads 22 of 25. Added Nov 8: Ripple-65B Preview, Ricursive's new dense 65B model, at 4-bit on the same two R9700 cards through the same dsh harness and kit ( passed). It plans in single steps of 15,000 to 31,000 tokens, and at about 13.5 tokens a second that is about 18 to 38 minutes before it writes anything, so six runs ended with no file at the time limit; where the task is small (all three cinema runs, done in about 12 minutes each) or the plan fits the clock, the work is good. Two calls are the owner's to overrule: its ray tracer (the grader passes it, but the required glass ball renders dark) and its black hole (scored a pass but contested: the second reviewer, before it saw the first reviewer's notes, failed it for the seam and stripes; the first reviewer kept it as a pass because the “DeepSeek V4.1 Flash Coder + dsh + kit” run passed with the same seam and stripes). Its tetris r2 first try was cut short and re-run in full; only the full run counts.

What ran where

Which model, at which quantization, on which runtime and hardware, under which harness and limits, for the 10 configurations on this page. It comes from the run scripts, the server settings, the logs kept for the tests and the owner's notes; “not recorded” means the setting was not captured at the time. Times are the machines' local times. The last column uses the same by-eye verdicts as the leaderboard.

† Taken from the owner’s notes. Most other details come from the run scripts, server settings, logs and live checks; a few more rest on the owner’s notes without a marker.

Windows workstation

Windows 11; AMD Ryzen Threadripper PRO 7975WX (32 cores); 192 GB DDR5^{† (from the owner’s notes)} RAM; two NVIDIA GeForce RTX 4090 (24 GB each) and one RTX 5090 (32 GB), 80 GB of GPU memory in all.

Served the model for 3 rows: DeepSeek V4.1 Flash · Maya-S; GLM 5.3 Flash; GLM 5.3 Flash · retry. It also ran the test harness, the sandbox containers and the graders for all 10 rows: Claude Code for 8 of them (the 7 stock rows and the round-3 kit row) and the dsh harness for the 2 dsh rows.

Linux server

AMD Ryzen 9 9950X3D2 (16 cores); 96 GB DDR5 RAM^{† (from the owner’s notes)} (89 GB usable); two AMD Radeon AI PRO R9700 cards with 32 GB each, 64 GB of GPU memory in all.

Served the model for 7 rows: DeepSeek V4.1 Flash Coder + dsh + kit; DeepSeek V4.1 Flash Coder · 2-bit, pruned; Ripple-65B Preview + dsh + kit; Qwen 3.8 27B · 4-bit; Nemotron Bolt 74B · 4-bit; Nemotron Bolt 74B · 2-bit; Qwen 3.8 27B · round-3 kit. The test harness ran on the Windows workstation and reached this server over the local network. Bolt 2-bit and the two Qwen 27B rows used one card; the other rows used both.

No other machine took part in any of the 10 rows.

The table is wide: scroll it sideways for all columns (the configuration name stays in view). Small numbers beside a name refer to the notes below the table.

What ran where: the setup of each of the 10 configurations, in leaderboard order
Configuration	Model and quantization	Runtime	Hardware used	Context window and seats	Harness	Per-test time limit	When it ran	Thinking and sampling	Result, by eye
DeepSeek V4.1 Flash · Maya-S⁵	DeepSeek V4.1 Flash The 1017 build, a mixture-of-experts model; total parameter count not verified. No picture input (vision off). Maya-S pack IQ2_XXS experts (4 files, about 178 GB) plus two Q6_K “Engram” table files (161 GB together): 339.5 GB in all. Bits per weight: not recorded.	Project Maya v1.1.28 DeepSeek engine build of Oct 25. It does not shorten the answer length near the context limit: the server returns a hard HTTP 400 error instead.	Windows workstation, all 3 GPUs The two RTX 4090 as a tensor-parallel pair, the RTX 5090 for later layers. Engine log: 5,301 experts in GPU memory; 11,911 on the CPU side, of which 11,401 copied into 125 GB of system RAM and 510 read in place from disk. About 4 GB of RAM left free.	131,072 tokens One stream, one test at a time (the seat setting was not recorded; presumed 1). The context wall (a prompt of about 99K) stopped 12 of 25 runs.	Stock Claude Code 2.1.302, one claude -p call per test, no add-ons: no web tools, no sub-agents. Video tests: no shell (cinema: its unit-test command only). Visual tests: only the sandbox runner (sbx-run), a throwaway offline container.	45 min per video test, 50 min per visual test	Oct 31 – Nov 1, 2026 From 17:34 on Oct 31 to 08:59 on Nov 1.	Reasoning effort high; speculative drafter off. Temperature 1.0, top_p 1.0 (config).	21/25video 12/12 · visual 9/13
DeepSeek V4.1 Flash Coder + dsh + kit⁷	Same model as the “DeepSeek V4.1 Flash Coder · 2-bit, pruned” row. Same files and quantization as the “DeepSeek V4.1 Flash Coder · 2-bit, pruned” row.	Strata 0.2.10, engine 0.1.56 As the “DeepSeek V4.1 Flash Coder · 2-bit, pruned” row; 1 seat, 131,072 tokens.	Linux server, both R9700s As the “DeepSeek V4.1 Flash Coder · 2-bit, pruned” row. The agent's sandbox container ran on the Windows workstation, on a slow disk.	131,072 tokens 1 seat; one test at a time.	dsh harness + kit dsh 0.2.1-rc.5 in goal mode, plus the kit: checklists and reference notes, a look tool, tester and reviewer sub-agents (the same model), a shell, Godot 4.7.2, Java and gcc. Picture reading on, web tools off. Prompts word for word. Kit version 2 for the first 6 runs, version 3 (Nov 5) for the rest.	45 min per video test, 50 min per visual test, then the run is cancelled 9 of 12 video runs and 11 of 13 visual runs ended at the limit.	Nov 4–5, 2026 From 22:44 on Nov 4 to 19:26 on Nov 5.	Not set by the harness: it sends a 32,768-token output length and no temperature or top_p. Sampling: not recorded (server defaults).	21/25video 8/12 · visual 13/13
GLM 5.3 Flash¹	GLM 5.3 Flash Mixture-of-experts, 321B total / 18B active^{† (from the owner’s notes)}. Takes picture input. GLM-5.3-Flash-Maya-L-IQ3_S Maya “L” mixed build, IQ3_S class, 4 files, 156.3 GB. Bits per weight: not recorded.	Project Maya v1.1.28 Build of Oct 25. A llama.cpp-derived engine built on Strata^{† (from the owner’s notes)}, with an Anthropic/OpenAI-style server.	Windows workstation, all 3 GPUs Both RTX 4090 and the RTX 5090. The most-used experts sit in GPU memory (about 75% of them in an Oct 25 bench^{† (from the owner’s notes)}); the rest are in system RAM (96–121 GB in use) and on disk. The cache beyond 32K tokens streams to RAM. So RAM and disk hold part of the model, by design.	131,072 tokens per seat (128K) 4 seats; one test at a time.	Stock Claude Code 2.1.302, one claude -p call per test, no add-ons: no web tools, no sub-agents. Video tests: no shell (cinema: its unit-test command only). Visual tests: only the sandbox runner (sbx-run), a throwaway offline container.	45 min per video test, 50 min per visual test	Oct 29–30, 2026 Video tests on Oct 29; visual tests overnight into Oct 30.	Reasoning effort high. Temperature 1.0, top_p 0.95 (server config; any per-request override not recorded).	18/25video 9/12 · visual 9/13
DeepSeek V4.1 Flash Coder · 2-bit, pruned⁶	DeepSeek V4.1 Flash “Coder” ISTA-DASLab GSQ-RCO Coder, pruned to 192 of 384 experts (perplexity 4.645 vs 3.140 unpruned). Picture input is switched on in the runtime; whether stock Claude Code used it was not checked. IQ2_XXS experts + IQ4_NL Engram tables + Q8_0 drafter 6 expert files of about 72 GB, tables 110.6 GB, drafter 8 GB: 192.1 GB in all.	Strata 0.2.10 Linux package for AMD GPUs, engine strata-dsv41 0.1.56 (HIP, RDNA4). It shortens the answer length near the context limit instead of failing.	Linux server, both R9700s Layers split; 6 experts used per token (full quality). 1,988 experts (17 GB) sit on the CPU side, all copied into system RAM; the rest are in GPU memory. The Bolt and 27B models were stopped for the run.	131,072 tokens 1 seat; one test at a time.	Stock Claude Code 2.1.302, one claude -p call per test, no add-ons (as the “GLM 5.3 Flash” row). It ran on the Windows workstation and reached the server over the local network.	45 min per video test, 50 min per visual test	Nov 3–4, 2026 From 12:57 on Nov 3 to 00:29 on Nov 4.	Runtime default thinking, with all past thinking kept in every request; drafter on (auto). Sampling: not recorded (the runtime's defaults for this model were not read).	13/25video 7/12 · visual 6/13
Ripple-65B Preview + dsh + kit⁸	Ricursive Ripple-65B Preview Dense 65B (80 blocks), Apache-2.0. Text-only: no picture input. AtomicChat AD-IQ4_XS 38.7 GB, 4.78 bits per weight.	llama.cpp b12177 Prebuilt ROCm 10.0 build on the pinned ROCm 7.2 runtime; flash attention on.	Linux server, both R9700s Layers split (about 24 + 24 GB of GPU memory), no CPU/RAM offload. The Bolt and 27B models were stopped.	131,072 tokens 1 seat; cache at full precision (f16).	dsh harness + kit The same dsh 0.2.1-rc.5 and kit as the “DeepSeek V4.1 Flash Coder + dsh + kit” row (the kit unchanged since Nov 5). Picture reads return a failed result because the model is text-only.	45 min per video test, 50 min per visual test, then the run is cancelled All 13 visual runs and 9 of 12 video runs ended at the limit.	Nov 7–8, 2026 From 19:53 on Nov 7 to 16:28 on Nov 8; verdicts cross-checked by a second reviewer on Nov 8 (note 8).	Default thinking (a “budget” variant was written but not applied). Server flags: temperature 0.6, top_p 0.95, top_k 20, min_p 0. The harness sends none.	13/25video 5/12 · visual 8/13
Qwen 3.8 27B · 4-bit	Qwen 3.8 27B Dense, with a built-in multi-token-prediction head. No picture input: 5 visual runs ended on an image read. Q4_K_M 16.8 GB.	llama.cpp b11718 The same server build as the Bolt rows. Its chat template was overridden by a one-line fix on Oct 31 so that Claude Code's requests are accepted.	Linux server, one R9700 (the second card) The whole model in GPU memory (about 28 GB with 2 seats), no offload. A Bolt model stayed loaded on the other card, idle.	131,072 tokens per seat (128K) 2 seats; one test at a time. Cache at 8-bit.	Stock Claude Code 2.1.302, one claude -p call per test, no add-ons (as the “GLM 5.3 Flash” row). It ran on the Windows workstation and reached the server over the local network.	45 min per video test, 50 min per visual test	Oct 31, 2026 About 08:38 to 20:28.	Server thinking budget 32,768 tokens; Claude Code asks for adaptive thinking at effort high. Temperature 1.0, top_p 0.95, top_k 20, min_p 0 (server flags). Multi-token-prediction drafting on.	11/25video 6/12 · visual 5/13
Nemotron Bolt 74B · 4-bit²³	Same Bolt 74B-A9B as the Bolt 2-bit row. unsloth UD-Q4_K_XL 3 files, 50.2 GB. Picture projector in BF16.	llama.cpp b11718 llama-server, a ROCm 10.0 build on a pinned ROCm 7.2 runtime.	Linux server, both R9700s Layers split across the two cards (about 27 + 28 GB of GPU memory), no CPU/RAM offload.	262,144 tokens per seat (256K) 2 seats; one test at a time. Cache at 8-bit. Claude Code still assumed a 200K window.	Stock Claude Code 2.1.302, one claude -p call per test, no add-ons (as the “GLM 5.3 Flash” row). It ran on the Windows workstation and reached the server over the local network.	45 min per video test, 50 min per visual test	Oct 30–31, 2026 From about 11:00 on Oct 30 to 00:36 on Oct 31; interrupted once (note 3).	Server thinking budget 32,768 tokens, then a stop message; Claude Code sent no budget of its own. Temperature 1.0, top_p 0.95 (server flags; per-request values not recorded).	8/25video 4/12 · visual 4/13
Nemotron Bolt 74B · 2-bit²	NVIDIA Nemotron 3.5 Bolt 74B-A9B Mamba-2/Transformer hybrid mixture-of-experts, 74B total, about 9B active. Takes picture input. NVIDIA's own quantization-aware “24GB” GGUF IQ2_XXS class: 91.7% of weights IQ2_XXS, 5.9% NVFP4, 2.3% Q8_0. 22.02 GB.	llama.cpp b11718 llama-server, a ROCm 10.0 build on a pinned ROCm 7.2 runtime.	Linux server, one R9700 The whole model in GPU memory (about 25 GB used), no CPU/RAM offload. The second R9700 was installed but idle.	262,144 tokens per seat (256K) 2 seats; one test at a time. Cache at 8-bit. Claude Code still assumed a 200K window.	Stock Claude Code 2.1.302, one claude -p call per test, no add-ons (as the “GLM 5.3 Flash” row). It ran on the Windows workstation and reached the server over the local network.	45 min per video test, 50 min per visual test	Oct 29–30, 2026 Video tests from the evening of Oct 29; visual tests overnight to about 06:00 on Oct 30.	Server thinking budget 32,768 tokens, then a stop message; Claude Code sent no budget of its own. Temperature 1.0, top_p 0.95 (server flags; per-request values not recorded).	7/25video 4/12 · visual 3/13
Qwen 3.8 27B · round-3 kit⁹	Qwen 3.8 27B, as the 27B 4-bit row. No picture input. Q4_K_M The same file as the 27B 4-bit row.	llama.cpp b11718, as the 27B 4-bit row Reached through a local pass-through proxy that sets the reasoning effort on every request.	Linux server, one R9700 (the second card) As the 27B 4-bit row. The harness and sandbox ran on the Windows workstation.	131,072 tokens per seat 2 seats, shared with the sub-agents. Claude Code was told a 131,072 limit, a 20,000-token output cap and auto-compaction at 110,000.	Round-3 kit Claude Code 2.1.302 plus sub-agents (tester, reviewer, researcher, builder; all the same 27B), run-and-look tools, reference packs (checklist, facts and mistakes to avoid; not code) and guard hooks. No vision, so the look tool reports numbers and text thumbnails. One run per test.	50 min for every test, video tests included	Oct 31 – Nov 1, 2026 From 23:29 on Oct 31 to 06:47 on Nov 1.	Reasoning effort forced to medium by the proxy; no run thought until the limit. Server flags: temperature 1.0, top_p 0.95, top_k 20, min_p 0.	16/17video 4/4 · visual 12/13 · incomplete, 17 of 25 runs
GLM 5.3 Flash · retry⁴	GLM 5.3 Flash, as in the “GLM 5.3 Flash” row (the original attempt). GLM-5.3-Flash-Maya-L-IQ3_S Same files as the “GLM 5.3 Flash” row.	Project Maya v1.1.28 Same build and configuration as the “GLM 5.3 Flash” row.	Windows workstation, all 3 GPUs As the “GLM 5.3 Flash” row: experts offloaded to system RAM and disk.	131,072 tokens per seat (128K) 4 seats; one test at a time. The context wall stopped 4 of the 7 retries.	Stock As the “GLM 5.3 Flash” row, but with no time limit. Only the 7 runs GLM failed were retried; run numbers do not map to the original ones.	None (the longest run took 62 min)	Oct 31, 2026 11:29 to 15:08.	As the “GLM 5.3 Flash” row.	2/7video 2/3 · visual 0/4 · incomplete, 7 of 25 runs

Notes on individual rows

Round 1 shared the machine with another job and a few short chats until 20:17 on Oct 29: another request was in flight about half the time and speed dropped (median 20.6 tokens/s against 29.8 when quiet). The run diagnosis is that this slowed the model and did not cause failures.
The run notes say Bolt was the only model on its server during these runs, but other traffic that week was not recorded, so it cannot be ruled out.
Interrupted part-way: the cut-short runs were set aside and dominoes, Mandelbrot, Rubik and FPS were run again. For the first 21 of the 25 tests of the 4-bit run, the server's second graphics card was held to a 210 W power limit with its clock held at 2,350 MHz (unheld, it boosts to about 3.3 GHz), because of a temporary power-supply limit^{† (from the owner’s notes)}. The new power supply was fitted on Oct 30, and the last four tests (dominoes, Mandelbrot, Rubik, FPS) were run at full power (300 W limit, clock not held)^{† (from the owner’s notes)}. How much this changed speed or time-limit outcomes was not measured.
A retry of only the 7 runs GLM failed (solar, tetris and chess once each; Game of Life, iso city, Mandelbrot and FPS). The other 18 slots were deliberately not run, so this row is incomplete and not ranked.
GLM was switched off on the machine for the whole run, and the graders were deferred to the end so they would not take CPU from the expert cache. One chess run was interrupted and re-run from scratch.
The sandbox was slow in this row: the visual round spent 41.5% of its time waiting on sandbox calls (34 s per call against 12 s for the full DeepSeek).
Most runs ended at the time limit by design, because goal mode keeps checking its own work. The sandbox disk was slow in this row. Solar run 3 fails by eye; a further re-run of it with a lighting note added to the kit (not in the grid) passed, which would make the row 22 of 25.
A second Claude reviewer first judged all 25 runs from the pictures, videos, grader output and code without seeing the first verdicts, then saw the first reviewer's notes and could keep or change its own verdicts (it changed the cinema app from a fail to a pass, and on solar run 3 it adopted the first reviewer's cause). After that it agreed with the first verdicts on 23 of the 25. It disagreed on 2: Game of Life, changed to a fail (the generation counter does not reset), and the black hole, kept as a pass for consistency with the “DeepSeek V4.1 Flash Coder + dsh + kit” row and so contested. A third call is also contested: the ray tracer (the grader passes it, the glass ball renders dark, so it fails by eye). Six of the 12 failures wrote no file before the limit. One tetris run was cut short and re-run in full; only the full run counts.
One run per test (17 runs, not 25), and 50 minutes for every test including the video tests (the others got 45). It is a kit row with a forced reasoning effort, so it is not like-for-like with the stock rows.

Why rows are not like-for-like

Harness. Seven rows ran in stock Claude Code with almost no tools (the GLM rows, Bolt, the 27B, DeepSeek and DeepSeek Coder). Two rows ran in the dsh harness with a kit that gave them a shell, tester and reviewer sub-agents, checklists and, for one of them, picture reading. One row ran in Claude Code with the round-3 kit: sub-agents, a look tool, reference notes and a forced reasoning effort. Scores are not like-for-like across these three groups.
Hardware. GLM and DeepSeek on the Windows workstation, and the two DeepSeek V4.1 Flash Coder rows on the Linux server, kept part of the model in system RAM or on disk; Bolt, the 27B and Ripple sat entirely in GPU memory. Bolt 2-bit used one 32 GB card, the 27B rows used the second card alone, and the other Linux rows used both cards.
Time limits. 45 minutes per video test and 50 per visual test everywhere except the GLM retry (no limit, runs up to 62 minutes) and the round-3 kit row (50 minutes for the video tests too). Many dsh runs ended at the limit by design.
Number of runs. The round-3 kit row is one run per test (17 runs), the GLM retry is the 7 retried runs, and the other eight rows have 25. Each visual test is a single sample per row.
Context windows. 131,072 tokens per seat for GLM, the 27B, DeepSeek and Ripple, 262,144 for Bolt. Claude Code assumes 200K for models it does not know, so stock rows ran into the 131K wall (12 of DeepSeek's 25 runs, 4 of the 7 GLM retries) while Bolt did not. DeepSeek's server answered with a hard error where the DeepSeek Coder runtime shortened the answer length, so the same prompt behaves differently.
Load on the machine. GLM's round 1 shared its machine (note 1); everything else ran alone or nearly alone. That affects times, not verdicts, according to the run diagnosis.
Picture input. GLM (both rows), Bolt (both rows) and DeepSeek V4.1 Flash Coder + dsh + kit could look at pictures. DeepSeek, the 27B (both rows) and Ripple could not. For DeepSeek Coder (stock) it was switched on in the runtime, but whether the harness used it was not checked.
Sampling and thinking. Not uniform: temperature and top_p are 1.0 / 0.95 for GLM, Bolt and the 27B (top_k 20 for the 27B), 1.0 / 1.0 for DeepSeek, 0.6 / 0.95 for Ripple, and not recorded for the two DeepSeek V4.1 Flash Coder rows. Bolt and the 27B had a 32,768-token thinking cap at the server, only the round-3 kit row forced a medium reasoning effort, and a speculative drafter was off for DeepSeek and on for the Coder rows.
Quantization. The classes differ widely: IQ3_S for GLM, IQ2_XXS experts for DeepSeek with 6-bit or 4-bit Engram tables, IQ2_XXS and NVFP4 against UD-Q4_K_XL for Bolt, Q4_K_M for Qwen, IQ4_XS for Ripple. The DeepSeek Coder is also a pruned model (192 of 384 experts), not the full DeepSeek.
Sandbox speed. Tool calls were slower in some rows than others (note 6 and note 7), which cuts the thinking time left inside the same limit.
Verdicts. By eye means Claude's reading of the screenshots, the grader's output and the code (see About this page); the owner did not judge each run. One row, Ripple, was also cross-checked by a second Claude reviewer, who judged first without seeing the first verdicts and then saw the first reviewer's notes (note 8). Some calls are contested, and the DeepSeek V4.1 Flash Coder + dsh + kit row has a re-run that passed but is not in the grid (note 7).
Settings changed afterwards. Settings on the machines were changed after some of these runs, so the settings in use today are not all the ones that produced these rows.

What was tested on what

Every configuration against every prompt. Each cell is the by-eye verdict. The four video tests ran three times, so their cells show how many of the runs that were run passed; the visual tests ran once. Click a test name to jump to its card.

passed (every run) some runs passed failed (no run passed) not run

By-eye verdict of each of the 10 configurations on each of the 17 prompts, with totals; rows in leaderboard order
Configuration	4 tests from the KGP Talkie video · 3 runs each				13 visual tests · 1 run each													Passed / run
Configuration	Solar systemJavaScript · Three.js	TetrisJavaScript · canvas	ChessJavaScript · canvas	Cinema appPython · unittest	PelicanSVG by hand	DonutRust	Ray tracerC	Game of LifeHTML · CSS · JavaScript	HeptagonPython · tkinter physics	MarsPython · matplotlib 3D	SmokeGo · fluid simulation	Iso cityJava 2D	Black holeGLSL shader	DominoesGodot · GDScript	MandelbrotJavaScript · canvas (Chinese prompt)	Rubik's cubeJavaScript · Three.js	FPSJavaScript · canvas raycaster	Passed / run
DeepSeek V4.1 Flash · Maya-S	3 of 3 runs passed	3 of 3 runs passed	3 of 3 runs passed	3 of 3 runs passed	passed	passed	passed	passed	passed	failed	passed	passed	passed	passed	failed	failed	failed	21/25
DeepSeek V4.1 Flash Coder + dsh + kit	2 of 3 runs passed	3 of 3 runs passed	0 of 3 runs passed	3 of 3 runs passed	passed	passed	passed	passed	passed	passed	passed	passed	passed	passed	passed	passed	passed	21/25
GLM 5.3 Flash	2 of 3 runs passed	2 of 3 runs passed	2 of 3 runs passed	3 of 3 runs passed	passed	passed	passed	failed	passed	passed	passed	failed	passed	passed	failed	passed	failed	18/25
DeepSeek V4.1 Flash Coder · 2-bit, pruned	2 of 3 runs passed	2 of 3 runs passed	0 of 3 runs passed	3 of 3 runs passed	passed	failed	passed	failed	passed	failed	failed	passed	failed	failed	passed	passed	failed	13/25
Ripple-65B Preview + dsh + kit	1 of 3 runs passed	1 of 3 runs passed	0 of 3 runs passed	3 of 3 runs passed	passed	passed	failed	failed	passed	failed	passed	passed	passed	failed	passed	passed	failed	13/25
Qwen 3.8 27B · 4-bit	3 of 3 runs passed	0 of 3 runs passed	0 of 3 runs passed	3 of 3 runs passed	passed	passed	failed	failed	passed	passed	failed	failed	passed	failed	failed	failed	failed	11/25
Nemotron Bolt 74B · 4-bit	1 of 3 runs passed	0 of 3 runs passed	0 of 3 runs passed	3 of 3 runs passed	passed	passed	failed	failed	passed	passed	failed	failed	failed	failed	failed	failed	failed	8/25
Nemotron Bolt 74B · 2-bit	0 of 3 runs passed	1 of 3 runs passed	0 of 3 runs passed	3 of 3 runs passed	passed	passed	failed	failed	failed	passed	failed	failed	failed	failed	failed	failed	failed	7/25
Qwen 3.8 27B · round-3 kit	1 of 1 run passed (2 of 3 run slots not run)	1 of 1 run passed (2 of 3 run slots not run)	1 of 1 run passed (2 of 3 run slots not run)	1 of 1 run passed (2 of 3 run slots not run)	passed	passed	passed	passed	passed	passed	passed	passed	passed	failed	passed	passed	passed	16/178 not run
GLM 5.3 Flash · retry	0 of 1 run passed (2 of 3 run slots not run)	1 of 1 run passed (2 of 3 run slots not run)	1 of 1 run passed (2 of 3 run slots not run)	not run	not run	not run	not run	failed	not run	not run	not run	failed	not run	not run	failed	not run	failed	2/718 not run
All configurations: passed / run	15/26	14/26	7/26	25/25	9/9	8/9	5/9	3/10	8/9	6/9	5/9	5/10	6/9	3/9	4/10	5/9	2/10	130/224
Slots not run	4	4	4	5	1	1	1	0	1	1	1	0	1	1	0	1	0	26

130 passes in 224 runs, and 26 run slots not run (the GLM retry ran 7 of 25 slots, the round-3 kit row 17 of 25). Totals count the runs that were run. Hardest: FPS passed in 2 of 10 runs, both from kit rows (DeepSeek V4.1 Flash Coder + dsh + kit and Qwen 3.8 27B · round-3 kit); chess 7 of 26; Game of Life 3 of 10; dominoes 3 of 9. Easiest: the cinema app passed 25 of 25 and the pelican 9 of 9.

Leaderboard

Every configuration, ranked by pass rate over the runs it attempted; a configuration that ran fewer than all 25 runs is marked incomplete and not ranked. Click a column heading to sort by it; click again to reverse.

How it was tested

Each model got the prompt in an empty folder inside Claude Code with no add-ons and no internet. The time limit was 45 minutes on the four video tests and 50 minutes on the 13 visual tests; the GLM retry had none. The round-3 kit runs had 50 minutes on every test, so its chess run had five minutes more than the stock chess runs. For the visual tests its only shell command was sbx-run, which runs code in a throwaway offline Linux container, so it could compile and test its own C, Rust, Go, Java and Python. Godot was not available, so the dominoes were written blind.

Graders then re-ran every program in the same kind of container (or a headless browser), checked it automatically, and saved the pictures. Where the grader and the picture disagreed, the picture won: the solar system's dark planets, the Mars and smoke code a pattern check misread.

GLM shared its machine with another job until 20:17 on Oct 29, so its times are slow. Claude Code assumes a 200K-token window for these models, but GLM's seat holds 128K, which is how the Java city ran out of room. Bolt's server lets it think for up to 32,768 tokens, more than Claude Code's 32,000-token answer limit, so long tasks often ended with nothing written.

The chips on each run say how it ended: clean exit (Claude Code finished by itself), time limit (stopped while still working), time limit, nothing written (it thought or tested the whole run and never saved the file), context wall (the next request no longer fit the model's window), output cap (one answer, thinking included, ran past Claude Code's output limit), vision error (a screenshot went to a server that cannot read images). Grader overruled marks a run where the by-eye verdict and the script's verdict differ; open its Grader line to see why, and press Code to see the line of the program that explains it. Token counts are the output tokens Claude Code reported (sub-agents included). A run the time limit cut off reports nothing, so its count (marked ~) is estimated from the thinking stream: it can be off by about a third either way and it never sees a sub-agent's share.

Credits

Where the prompts came from, as credited on each test card:

KGP Talkie (a YouTube video): the four video tests (solar system, Tetris, chess and the cinema app), word for word.
Simon Willison: the pelican on a bicycle, word for word.
Artificial Analysis: the Game of Life with the glider gun, word for word.
KCORES arena: the 20 balls in a spinning heptagon, Earth to Mars and back (the Grok 3 launch demo) and the Mandelbrot in Li Bai's poem, word for word.
Written new for this bench, after: a1k0n's donut.c (the donut), the Ray Tracing in One Weekend scene (the ray tracer), Jos Stam's Stable Fluids (the smoke), the Kimi K3 demo (the black hole) and @_akhaliq's prompt (the Rubik's cube). The iso city and the dominoes are new; the first-person shooter is original, written in place of a “recreate DOOM” prompt.

Prompts marked “word for word” are reproduced as published by their authors and remain theirs. The programs and pictures on this page were generated by the models; who owns them is set by each model's licence, not by this page.