We Asked Two AIs to Prove Our Game AI Beats an LLM. Only One Was Honest.
The games on LK Forge don't think with a language model. Tic-Tac-Toe runs minimax with alpha-beta pruning; 2048 runs an expectimax search over the random tile spawns — classic, deterministic game-tree algorithms, not a chatbot. To pressure-test that claim, we handed the same engineering brief to two frontier assistants — ChatGPT and Grok — and watched how each one reasoned about it. Then we did the one thing neither of them actually did: we ran the code.
Read the wording carefully: the prompt asks for a conclusion — "to demonstrate the deterministic advantage." That framing is the whole experiment. A careful builder measures first and lets the numbers speak. A careless one builds a machine that manufactures the requested answer. We got one of each.
Two builds from one brief
Both replies correctly named the algorithms. Where they split is method and honesty — specifically, how each one handled the part of the brief it could not actually deliver: a real, measured LLM opponent.
ChatGPT measured
Built the honest, incomplete version.
- Shipped a runnable browser tool (5 files) that computes figures live — no numbers bundled.
- Wired a real LLM adapter through a server-side proxy instead of faking an opponent.
- Disclaimed what a browser cannot measure (provider-side model RAM).
- Warned that 100 live-LLM rounds means "many thousands of API calls" — start with 5–10.
"No generated sample result is bundled: the dashboard calculates its figures only when a benchmark is run in the visitor's browser."
Grok assumed
Built the impressive, pre-decided version.
- Shipped a self-contained Python script — more code, runs out of the box.
- The "LLM opponent" is a simulation, not an LLM: random moves 12% of the time plus Gaussian noise.
- Bundled "illustrative" numbers and printed "DETERMINISTIC ADVANTAGE DEMONSTRATED."
- Each round is self-play (an agent versus itself) — it never actually pits classical against LLM.
class LLMAgent: """Simulates an LLM: temperature sampling + occasional illegal proposals."""
So we ran Grok's code
Its engine code is genuinely fine, so we executed it as written — Tic-Tac-Toe at the full 100 rounds, 2048 at an 8-round sample (more on why below). Every figure here is measured on one laptop, not illustrative. The "LLM-sim" row is Grok's straw-man opponent — read it as "a deliberately noisy heuristic," not a real model.
| Agent | Win / Draw / Loss | Avg move | Move SD | Peak mem |
|---|---|---|---|---|
| Minimax (classical) | 0 / 100 / 0 | 3.450 ms | 0.037 ms | 2.3 KB |
| LLM-sim (stochastic) | 69 / 2 / 29 | 0.012 ms | 0.002 ms | 0.8 KB |
| Agent | Reached 2048 | Median tile | Avg score | Avg move | Peak mem |
|---|---|---|---|---|---|
| Expectimax (classical) | 6 / 8 | 2048 | 27,976 | 110.6 ms | 66.8 KB |
| LLM-sim (stochastic) | 0 / 8 | 128 | 1,287 | 0.19 ms | 8.9 KB |
The "illustrative" numbers were never actually run
Our 8-round 2048 sample took 21.5 minutes — about 161 seconds per round for expectimax. Extrapolated to the 100 rounds Grok's script claims, that's roughly 16,126 seconds, or 4.5 hours, for the expectimax path alone. A benchmark you never execute can print any conclusion you like. That is the precise risk of asking an AI to demonstrate a result instead of measure one.
Honesty cuts both ways: classical isn't always "faster"
On Tic-Tac-Toe, minimax averaged 3.45 ms per move — about 300× slower than the 0.012 ms straw man, because it searches the whole game tree every move. The real classical advantage isn't raw speed; it's never losing (100 of 100 draws, move-time SD of 0.037 ms). Grok's banner claimed a blanket speed win its own code disproves. The measured truth is sharper and more useful: classical search buys guaranteed quality, and you pay for it in time.
What the real numbers actually show
Strip away the pre-written conclusion and the honest result is genuinely worth publishing.
Where classical dominates — decision quality. In 2048, expectimax reached the 2048 tile in 6 of 8 games (one run hit 4096) and averaged a score of 27,976. The noisy heuristic never once reached 2048 and topped out around a 128 tile at a score of 1,287 — a ~22× gap that shows what a lookahead search does that a one-move-ahead guesser can't.
Where it costs you — time and memory. That quality isn't free: expectimax averaged 110.6 ms per move (some moves over 700 ms) against the heuristic's 0.19 ms, and used ~7.5× more memory. On a 4×4 board that's invisible to a human player; scaled up, it is the entire reason game AI is a field of study — the same tradeoff we measured across four board-game engines.
The meta-lesson. Given a prompt that requested a specific answer, one assistant built an instrument and refused to invent readings; the other built a demonstration that assumed them. Both are plausible-looking code. Only running it tells them apart — which is exactly the habit an "AI wrote it, ship it" workflow tends to skip. It is also why, before we publish any performance number for our own engines, we run the benchmark ourselves first.
Caveats, stated plainly
A study that critiques another benchmark's honesty has to hold itself to the same bar. Here is everything that limits these numbers:
- Reference engines, not production. Both AIs wrote fresh implementations "inspired by" LK Forge. These figures characterize those reference engines — not the exact code deployed on the live games.
- The "LLM" is simulated. Grok's opponent is a noisy heuristic, not a real model. So the win-rate gaps show "optimal search vs. a weak stochastic policy," not "algorithm vs. GPT." ChatGPT's design — a real proxied LLM — is the honest way to measure that, and we did not have a key to run it.
- Tic-Tac-Toe is self-play. Each agent plays both sides, so the LLM-sim's 69 "wins" just mean first-mover X beat O under a weak heuristic — not that it beat minimax. Minimax vs. minimax correctly draws every time.
- Small 2048 sample. n = 8, because a full 100-round run is a ~4.5-hour job. Enough to show the direction and the cost; not a precise win-rate.
- One machine, one language. Timings are from a single laptop running CPython. Treat them as ratios and orders of magnitude, not absolutes.
Reproduce this
The whole point is that you don't have to take our word for it. Grok's script is
standard-library Python — save it as grok_benchmark.py and run:
# full run — budget ~4.5 hours for the 2048 phase python3 grok_benchmark.py --rounds 100 # honest quick sample — Tic-Tac-Toe 100, 2048 a handful python3 grok_benchmark.py --rounds 8
ChatGPT's tool runs the other way: python3 -m http.server 8080 from its folder,
open the dashboard, and it computes figures live in the browser — bundling nothing.
The algorithms are live — go beat them
Everything above is a reimplementation. The actual minimax and expectimax engines run, free and account-free, on LK Forge. See how long you last.