mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
fix(analyze): censor a clean run at the steps it ran, and refuse mismatched budgets
A run stops at whichever comes first, the step budget or --duration, so a clean run that reached the wall clock exited with fewer steps than the budget and was still credited with the whole of it. The model arm pays a network call and a screenshot per step, so it reaches the wall sooner and was handed exposure it never had. Nothing checked that two arms shared a budget either. Thirty identical clean runs under budgets of 400 and 100 read a12 0.000 and p 1.685e-14 from the rank-sum while the log-rank in the same report read p 1.0000. groupArms already refused this within one arm. The claims the old convention left in comments and report lines are corrected rather than left standing beside the new behaviour.
This commit is contained in:
1 parent
ff6c66a74b
commit
1da0c3e118
10 files changed
+141
-41
No files matched your search
@@ -1,6 +1,6 @@
|
||||
// Command analyze reduces campaign directories to the statistics the
|
||||
// evaluation reports. The primary outcome is steps to first violation, with
|
||||
// clean runs right-censored at the step budget rather than discarded: defect
|
||||
// clean runs right-censored where they stopped rather than discarded: defect
|
||||
// yield per run is a binary that would need on the order of eighty runs an arm
|
||||
// to separate, while survival analysis uses every run, including the clean ones.
|
||||
package main
|
||||
@@ -24,7 +24,7 @@ Usage:
|
||||
|
||||
Each directory is one produced by the campaign tool and must hold campaign.json
|
||||
and runs.jsonl. Directories sharing an arm label are pooled and must agree on
|
||||
the step budget.
|
||||
the step budget, and arms compared against each other must agree on it too.
|
||||
|
||||
One invocation is one research question: Holm corrects across the comparisons it
|
||||
produces and across nothing else.
|
||||
@@ -87,7 +87,10 @@ func run(arguments []string, stdout, stderr io.Writer) error {
|
||||
return err
|
||||
}
|
||||
} else {
|
||||
result = analyse(arms, time.Now().UTC())
|
||||
result, err = analyse(arms, time.Now().UTC())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
result.Question = question
|
||||
writeReport(result, stdout)
|
||||
|
||||
Reference in new issue
Block a user