fix(analyze): a run that never explored is excluded, not counted as evidence

The campaign has written precondition_failures since it learned to count them
and neither reader declared it, so a run whose scope guard could not bring the
app back entered the survival analysis with its whole budget as exposure the
app survived. A strict majority of steps is the line: below it the failures
were transient and the run still explored.
This commit is contained in:
pj committed 2026-08-22 21:12:27 +05:30
1 parent 52625d95a7
commit 3e4633f588
2 files changed
+67 -5

No files matched your search

+21 -5
View File
@@ -65,17 +65,30 @@ type runRecord struct {
// field, and reading that silence as none would let a denominator of unknown
// provenance pass for one that excludes the setup's login.
UnattributedActions *int `json:"unattributed_actions"`
// PreconditionFailures is how many of the run's steps never had the app under
// test in front of them: the startup gate's verdict and every later step the
// scope guard could not bring the app back for. The campaign omits the field
// when it is zero, so absence and zero mean the same thing here.
PreconditionFailures int `json:"precondition_failures"`
}
// Exclusion reasons. A run that failed or timed out is missing data, not a
// censored observation: it broke off, so its step count is not exposure the app
// survived and counting it as one would bias the survival estimate downward.
const (
reasonLaunchError = "launch error"
reasonTimedOut = "timed out"
reasonNonzeroExit = "nonzero exit"
reasonTraceError = "unreadable trace"
reasonMalformedStep = "violation step outside the budget"
reasonLaunchError = "launch error"
reasonTimedOut = "timed out"
reasonNonzeroExit = "nonzero exit"
reasonTraceError = "unreadable trace"
// reasonPreconditionFailures is the run that exited cleanly having spent its
// budget failing preconditions. The scope guard records one on every step it
// could not bring the app back for and lets the run finish, so nothing else
// here separates it from a run that explored the whole budget and found
// nothing. A majority is the line: below it the failures are transient and
// the run still explored, above it the step count the analysis would censor
// at is mostly steps the app was never there for.
reasonPreconditionFailures = "precondition failures"
reasonMalformedStep = "violation step outside the budget"
)
type classifiedRun struct {
@@ -201,6 +214,9 @@ func classify(record runRecord, budget int) classifiedRun {
case record.TraceError != "":
item.ExcludedBecause = reasonTraceError
return item
case record.PreconditionFailures*2 > record.Steps:
item.ExcludedBecause = reasonPreconditionFailures
return item
}
if record.FirstViolationOriginStep == nil {
if len(record.ViolatedProperties) > 0 {