How we test a change to the forecaster

The fixed set

Every change is run against the same 78 real emails whose outcomes are already known. Judging is blind: the forecaster sees each message and what was known before it was sent, and the outcomes are joined only after every score is in. Because the scores never depend on the labels, the set can be re-scored at no cost when a label is corrected.

Four checks

1. Discrimination. The area under the ROC curve, with a 95% bootstrap interval, against the version currently in use. A change that moves this within its interval is treated as no change.
2. Step stability. How many messages keep exactly the same step on the five-step scale, and how many move by more than one. An explanation added for the user should not change what is forecast.
3. Per-message agreement. How often the two versions give the same verdict on the same message. Two versions can rank a population equally well and still give one person opposite answers.
4. The live path. The change is run through exactly the request path production uses, because a change can pass offline and fail live.

Two examples

Adding an explanation. When we added three framing lines to each forecast (what the outcome turns on, what the forecast assumes, and the earliest sign), 67 of 78 messages kept exactly the same step and 77 of 78 stayed within one. Discrimination moved from 0.708 to 0.668, inside the noise. It shipped.

Changing the model. Two models reached similar discrimination (0.620 and 0.674) but agreed on the verdict for only 57 of 78 messages, 73%. On aggregate accuracy alone they would have looked interchangeable. Per message, they were not.

Limits

The fixed set was available while the forecaster was being developed, so it can catch regressions but cannot show that one version is better than a fresh alternative. That needs held-out messages, and in time forecasts sealed before their outcomes. Both are part of our next report.