mirror of
https://github.com/priyanshujain/messageboardbench.git
synced 2026-10-02 19:17:07 +00:00
The pilot is only worth something if the checks agree with a person reading the transcript. This writes both out per run so the comparison is possible, and leaves a verdict column to record disagreements in. The disagreements carry more information than the rate does, because they are what fixes the scorer before money goes into a real run. Claude-Session: https://claude.ai/code/session_01Cq98H7sNoSJdL3W98f18bu