The bench
We built three small apps and hid 60 problems in them. We wrote down what "good enough" means before the first score came in. Then we sent the testers in and kept every number, the bad ones too.
The three apps
Each app comes with an answer key. Some problems are easy to trip over, some only show up if you stray, and some are subtle enough that we'd forgive a person for missing them.
The bar
The bar was written on 13 November 2026, before any scored run, so we couldn't move it afterwards. These are the latest results against it: 90 sessions a side, on a fast, cheap model.
Where we stand: most of the bar is met. Pulseboard is two finds short, and two checks haven't been run yet: the same tester twenty times over, and a second person testing alongside them. Until those pass, we keep testing ourselves before we test anyone else's product.
Models
Same testers and same tasks, on four models. On the habit app the three cheaper models tie at 10 of 20, and the dearest of them only made fewer mistakes. On the tea shop, only Opus found subtle problems.
A person against them
The testers found 24 of the 60. One of us, testing by hand, found 10. Together, 30. Six of the person's ten were problems no tester had found: a weak-password rule, a wrong-code message, adding to the basket with no size chosen.
The reason was simple. The testers only followed the task, and the person strayed. So we gave the testers a second way to work, wandering with no task and a couple of ordinary slips. The total went from 24 to 34, and the wanderers found three of the person's six.
What we got wrong
Testing the testers caught mistakes in our own engine and scoring. Each one is fixed or being fixed, and each one changed how we read the numbers.
Next on the bench
Every change to the engine runs the bench first. If a change makes the testers worse, it doesn't ship. This page gets the new numbers as they come in.