Two ways a scoreboard can lie

14 September 2026 · FourShades

The CPU alone could not run a single test ROM. It needed the rest of the machine: the memory map, a cartridge chip, the timer, interrupts, the serial port and the direct-memory-access unit that copies sprites. That is now built, and the second scoreboard line has moved off zero:

cpu instructions  499 / 500
test roms          85 / 165

165, not 1,300, and not 167. Both of those corrections are part of the story.

What actually got built

A Game Boy with no screen. Every one of those parts advances one machine cycle at a time, in step with the CPU, because that is the only way the timing tests can pass. The whole thing runs all 165 test ROMs in 5.6 seconds, which matters: a test suite you can run in the time it takes to read this sentence gets run after every change.

Five groups are now complete:

The first of those is the one I care about. Blargg’s cpu_instrs was written by someone other than the author of the CPU tests from the last post. If the CPU had been quietly shaped to fit one test suite, this is where it would show. It didn’t.

The rest are waiting on hardware that doesn’t exist yet: the graphics chip (piece 3), the other cartridge chips (piece 4) and sound (piece 5). They were in the runner from day one, failing with a reason, so the number can only rise through real work.

The first way a scoreboard can lie

Blargg’s tests report their result by writing it into the cartridge’s memory: a three-byte signature, then a status byte. My checker looked for the signature and read the status.

The trouble is the order the test writes them in. The signature goes down first, and the status a moment later. In between, the status byte is still zero, and zero means “passed”. A check landing in that window would read a pass from a test that had not run a single instruction of its own.

It never did land there. The checker ran every 8,192 cycles, and the window is four instructions wide, so it stepped over it every time. The reviewer found it by checking at every instruction boundary instead, where all 23 of those tests report a false pass. The fix was to refuse any result until the test has first been seen in its “running” state. The score afterwards: identical, the same 68 tests. It had never actually lied. It just could have.

The second way

Mooneye’s tests signal failure by sending six bytes over the serial port. Mine was looking for a failure signal in the CPU’s registers instead, which these builds don’t set. So 14 tests that had failed, and said so within a quarter of a second, were published as “timeout after 6.5 seconds”.

The score was right, because a failure is a failure either way. The reason was false, and the reasons are published beside every test. Fixed, and those 14 now carry what the test actually said.

Both of these came from the same place: I wrote the checker before the tests were run against it. Neither would have shown up in the score. Both were found by a reviewer whose only job was to ask whether a pass could be claimed without being earned.

The denominator was wrong twice

The scoreboard used to read 0 / 1300. That number came from an early planning note, where it was an example, not a count. Nobody counted anything. When I finally did: the Shootout runs 167 original-Game-Boy tests, and two of them have no “correct” reference image at all — the Shootout itself treats those as informational, so no emulator can pass them. That leaves 165 with a pass condition.

Both corrections are on the site and in the repository, with notes saying what was wrong. A number that only ever goes up because the denominator shrinks is the oldest trick there is, so it’s worth being explicit: 1,300 → 167 → 165 happened before the line had moved off zero, and the 85 was scored against 165.

When the documentation is the thing that’s wrong

Nine timing tests failed for a single reason. Pan Docs, the reference everyone builds against, says that while the sprite-copy unit is running, “the CPU can access only HRAM”. I implemented that sentence literally, so the tests’ own code stopped being readable and they hung.

The tests say otherwise, and they say it with authority: each one is marked verified on real DMG, MGB, SGB, SGB2, CGB, AGB and AGS hardware. They show that only the sprite memory and the one bus the copy is reading from are blocked. The Pan Docs sentence is advice to programmers, not a description of the chip.

So the project rule got sharper. It used to be “the documentation beats the tests”. Now:

• the documentation beats emulator-generated tests, which is why STOP is still deliberately failing at 499 / 500;
• hardware-verified tests beat a documentation simplification.

That change is written down with its evidence, and it took the score from 76 to 85.

What is checked on every push

• The test ROMs are pinned to one version of the Shootout and fingerprinted; the runner refuses to start if a byte differs.
• The list of 165 is generated from the Shootout’s own definitions, and the build now regenerates it and fails if it drifted — including if a new test appeared in a file the generator wasn’t looking at.
• Time limits come from a formula, never per test by hand, and the build rejects a list where any limit doesn’t match.
• Both scores are recomputed from scratch and the published numbers must match exactly, in either direction.
• The emulator code is forbidden from mentioning the tests at all: no names, no pass patterns, no result signatures.

Next

The graphics chip. It’s the largest group of failing tests, and it’s the piece that finally produces a picture: the boot logo, then dmg-acid2, then a real game.

The code, the specs, the plans and the reviews are all in the repository, and the divergences, with the evidence behind each decision, are in known-divergences.md.

← All posts