fix(analyze): compare arms on censored runs, not on flattened step counts

stepTimes threw the censoring flag away and handed the rank-sum a plain
number per run, so a run the wall clock stopped at step 12 was ranked as one
that violated at step 12. That was defensible while every clean run sat at
the budget, the largest value any run could take, and it stopped being
defensible when a clean run started being censored where it stopped.

Twenty runs clean at step 12 against twenty violations at step 100 read a12
0.000 and p 4.683e-10 from the rank-sum, in the same report as a log-rank
reading p 1.0000. The pairwise comparison is now the Gehan test over the
observations themselves, and the report says how many run pairs censoring
left with no order between them, which is how much of the effect size is the
null value rather than an observation.
This commit is contained in:
pj committed 2026-08-18 20:13:12 +05:30
1 parent b7ee23942e
commit f0726a1e61
6 files changed
+144 -52

No files matched your search

+5 -3
View File
@@ -107,9 +107,10 @@ func writeReport(result analysis, out io.Writer) {
}
if len(result.Pairwise) > 0 {
fmt.Fprintln(out, "\npairwise wilcoxon rank-sum, censored runs held at the steps they ran")
fmt.Fprintln(out, "a12 above 0.5 means the first arm takes more steps to its first violation")
writeTable(out, []string{"comparison", "n1", "n2", "u", "a12", "p", "holm p"}, func(add func(...string)) {
fmt.Fprintln(out, "\npairwise gehan generalized wilcoxon, which reads a censored run as the bound it is")
fmt.Fprintln(out, "a12 above 0.5 means the first arm takes more steps to its first violation; u counts the run")
fmt.Fprintln(out, "pairs the first arm outlived, and the unordered pairs count as half in both u and a12")
writeTable(out, []string{"comparison", "n1", "n2", "u", "a12", "unordered", "p", "holm p"}, func(add func(...string)) {
for _, pair := range result.Pairwise {
add(
pair.First+" vs "+pair.Second,
@@ -117,6 +118,7 @@ func writeReport(result analysis, out io.Writer) {
strconv.Itoa(pair.SecondSize),
fmt.Sprintf("%.1f", pair.Statistic),
fmt.Sprintf("%.3f", pair.A12),
fmt.Sprintf("%d of %d", pair.Unordered, pair.FirstSize*pair.SecondSize),
formatPValue(pair.PValue),
formatPValue(pair.HolmPValue),
)