Proving It's Actually Unbeatable: How We Benchmark a Game AI Before Publishing a Number

"Unbeatable" is a testable claim, not a marketing word. Before that word goes anywhere near the tic-tac-toe page, the shipped AI has to survive a headless self-play harness — the same functions the browser runs, played thousands of times with no human at the keyboard. Here is what that harness measured, including the part that didn't look good at first.

July 19, 2026  ·  8 min read  ·  all numbers from re-runnable headless benchmarks

The Numbers

Win / draw / loss, two opponents

Win Draw Loss — the bar that matters is the one that never appears. Zero losses in 1,000 games against a random opponent, zero losses in 200 games against a copy of itself playing perfectly.

What "Unbeatable" Has to Mean

"Unbeatable" is a claim about the worst case, not the average case, so testing it against one kind of opponent proves nothing. We ran two. Against 1,000 games of a random-move opponent, the AI went 825 wins, 175 draws, 0 losses — the wins come from a weak opponent occasionally handing over a forced win. Against 200 games of the AI playing a perfect copy of itself, every single game ended in a draw: 0 wins, 0 losses, 200 draws. That is the correct result for perfect tic-tac-toe play, and it is the harder claim to satisfy — a program that merely beats random players a lot could still have a hole a perfect opponent would find.

Across both runs — 1,200 games total — the loss count is 0. That is the number the word "unbeatable" is actually standing on.

The Harness

The benchmark is headless self-play: the exact move-selection functions from the shipped page, imported into a script with no DOM and no human, playing thousands of games back to back. That distinction matters more than it sounds. A benchmark written against a re-implementation of the algorithm proves the algorithm is sound in theory; it says nothing about the code a browser actually executes. Pulling the real functions out of the production page and driving them from a script means the numbers describe the shipped build, not a clean-room stand-in that happens to share a name with it.

It also has to be re-runnable. A number that came from one afternoon of manual play-testing is an anecdote; a script that plays 1,000 games against a random opponent and 200 games against itself, and produces the same win/draw/loss counts every time it's re-run, is a measurement. Mean move time came out at about 0.3 ms across 500 sampled positions — cheap enough that running the full 1,200-game suite costs seconds, not minutes, which is what makes it realistic to re-run on every change instead of trusting the last time someone checked.

Depth Is the Proof

The shipped engine searches to depth 9 — the full game tree for a 3x3 board — which is what guarantees the zero-loss result above. To find out whether that depth is actually necessary, the harness re-ran the same 200-game vs-random test with the search artificially capped at each depth from 1 to 9:

The honest finding is in the depth-2 column: at a search depth of 2, the AI lost 2 of 200 games. A shallow search is not automatically safe just because it wins most of the time — 87% win rate at depth 2 looks fine on its own, until you notice the two losses hiding next to it.

Win percentage also doesn't rise monotonically with depth (it peaks at depth 3, at 90.5%, then drifts down before settling at 81% at full depth 9), because deeper search doesn't just chase more wins — past the point where a win is already forced, it starts steering into the safest won-or-drawn line instead of the flashiest one, which trades a few wins-against-random for the zero-loss guarantee. Depth 9 is the only depth in the table with a 0-loss guarantee behind it; every shallower depth is a different, weaker program that happens to share the same UI.

That is the entire argument for benchmarking instead of reasoning from the algorithm's name. "It's minimax, so it's unbeatable" is true only at one specific depth setting, and the only way to know which setting that is, is to run all of them and look for the losses.

The 2048 Case

Tic-tac-toe has a small enough state space that "unbeatable" is checkable in an absolute sense. Most games don't — 2048 has no realistic path to "solved," so the only honest claim is a measured reach-rate. The same harness pattern applies: 250 headless self-play games using the shipped expectimax-plus-corner-snake solver, no human involved.

The result is 69.6% of games reach the 2048 tile, with reach-rate falling off sharply after that (30% reach 4096, 0% reach 8192 in this run) — a strength profile, not a single number, and one that would be indistinguishable from a coin flip if we'd only run it 10 times instead of 250. The solver internals are covered separately in How the 2048 AI Solver Works; this post is only about the measurement discipline that number came from.

The Rule This Enforces

This is the same rule the 26-repos build retrospective landed on after a month of shipping pages with an AI pipeline: benchmark before you publish a number. Any performance or strength claim on this site has to come from a measurement that can be re-run, not a description of what the algorithm should do in theory. The depth-2 result above is exactly the kind of thing that rule is for — a plausible-sounding claim ("it's minimax, it can't lose") that turns out to be depth-dependent, and would have shipped wrong without a script that actually played the games.

Related reading: the three search algorithms these numbers come from are broken down in Six Games, Three Classic Algorithms, and the frame-budget work that keeps them running client-side is in Search Algorithms Inside One Browser Tab.

Try to Beat It

1,200 self-play games came up with zero losses. See if a human can do better against the same full-depth engine.