The bench

We mark our own testers.Here's the whole report card.

We built three small apps and hid 60 problems in them. We wrote down what "good enough" means before the first score came in. Then we sent the testers in and kept every number, the bad ones too.

Hiding 60 problemsLatest full pass · 15 Nov 2026

The three apps

Three apps, twenty problems each,from obvious to sneaky.

Each app comes with an answer key. Some problems are easy to trip over, some only show up if you stray, and some are subtle enough that we'd forgive a person for missing them.

The bar

We set the bar first.We haven't cleared it yet.

The bar was written on 13 November 2026, before any scored run, so we couldn't move it afterwards. These are the latest results against it: 90 sessions a side, on a fast, cheap model.

Where we stand: most of the bar is met. Pulseboard is two finds short, and two checks haven't been run yet: the same tester twenty times over, and a second person testing alongside them. Until those pass, we keep testing ourselves before we test anyone else's product.

Models

Cheap models find the obvious.Only the strongest find the subtle.

Same testers and same tasks, on four models. On the habit app the three cheaper models tie at 10 of 20, and the dearest of them only made fewer mistakes. On the tea shop, only Opus found subtle problems.

Bars: planted problems found · coins: cost per testerLoop: about 20 sessions a model · Kettle: 30

A person against them

One of us tested the same apps.We learned the most from that.

Planted problems found, of 6014 Nov 2026

The testers found more. The person found different things.

The testers found 24 of the 60. One of us, testing by hand, found 10. Together, 30. Six of the person's ten were problems no tester had found: a weak-password rule, a wrong-code message, adding to the basket with no size chosen.

The reason was simple. The testers only followed the task, and the person strayed. So we gave the testers a second way to work, wandering with no task and a couple of ordinary slips. The total went from 24 to 34, and the wanderers found three of the person's six.

24testers
10one person
34testers + wander

What we got wrong

Our own bugs,found the same way.

Testing the testers caught mistakes in our own engine and scoring. Each one is fixed or being fixed, and each one changed how we read the numbers.

Next on the bench

We'll keep marking them.You'll see every score.

Every change to the engine runs the bench first. If a change makes the testers worse, it doesn't ship. This page gets the new numbers as they come in.