fix(analyze): censor a clean run at the steps it ran, and refuse mismatched budgets

A run stops at whichever comes first, the step budget or --duration, so
a clean run that reached the wall clock exited with fewer steps than the
budget and was still credited with the whole of it. The model arm pays a
network call and a screenshot per step, so it reaches the wall sooner and
was handed exposure it never had.

Nothing checked that two arms shared a budget either. Thirty identical
clean runs under budgets of 400 and 100 read a12 0.000 and p 1.685e-14
from the rank-sum while the log-rank in the same report read p 1.0000.
groupArms already refused this within one arm.

The claims the old convention left in comments and report lines are
corrected rather than left standing beside the new behaviour.
This commit is contained in:
pj committed 2026-08-18 14:11:28 +05:30
1 parent ff6c66a74b
commit 1da0c3e118
10 files changed
+141 -41

No files matched your search

+2 -2
View File
@@ -99,7 +99,7 @@ func writeReport(result analysis, out io.Writer) {
}
if len(result.Pairwise) > 0 {
fmt.Fprintln(out, "\npairwise wilcoxon rank-sum, censored runs held at the budget")
fmt.Fprintln(out, "\npairwise wilcoxon rank-sum, censored runs held at the steps they ran")
fmt.Fprintln(out, "a12 above 0.5 means the first arm takes more steps to its first violation")
writeTable(out, []string{"comparison", "n1", "n2", "u", "a12", "p", "holm p"}, func(add func(...string)) {
for _, pair := range result.Pairwise {
@@ -130,7 +130,7 @@ func writeReport(result analysis, out io.Writer) {
}
func writePaired(out io.Writer, comparison pairedComparison) {
fmt.Fprintf(out, "\npaired per-seed difference, %s minus %s, censored runs held at the budget\n",
fmt.Fprintf(out, "\npaired per-seed difference, %s minus %s, censored runs held at the steps they ran\n",
comparison.First, comparison.Second)
fmt.Fprintf(out, "%d seed pair(s): %s sooner in %d, %s sooner in %d, tied in %d\n",
comparison.Pairs, comparison.First, comparison.FirstSooner,