mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
fix(analyze): a run that never explored is excluded, not counted as evidence
The campaign has written precondition_failures since it learned to count them and neither reader declared it, so a run whose scope guard could not bring the app back entered the survival analysis with its whole budget as exposure the app survived. A strict majority of steps is the line: below it the failures were transient and the run still explored.
This commit is contained in:
1 parent
52625d95a7
commit
3e4633f588
2 files changed
+67
-5
No files matched your search
@@ -65,17 +65,30 @@ type runRecord struct {
|
||||
// field, and reading that silence as none would let a denominator of unknown
|
||||
// provenance pass for one that excludes the setup's login.
|
||||
UnattributedActions *int `json:"unattributed_actions"`
|
||||
// PreconditionFailures is how many of the run's steps never had the app under
|
||||
// test in front of them: the startup gate's verdict and every later step the
|
||||
// scope guard could not bring the app back for. The campaign omits the field
|
||||
// when it is zero, so absence and zero mean the same thing here.
|
||||
PreconditionFailures int `json:"precondition_failures"`
|
||||
}
|
||||
|
||||
// Exclusion reasons. A run that failed or timed out is missing data, not a
|
||||
// censored observation: it broke off, so its step count is not exposure the app
|
||||
// survived and counting it as one would bias the survival estimate downward.
|
||||
const (
|
||||
reasonLaunchError = "launch error"
|
||||
reasonTimedOut = "timed out"
|
||||
reasonNonzeroExit = "nonzero exit"
|
||||
reasonTraceError = "unreadable trace"
|
||||
reasonMalformedStep = "violation step outside the budget"
|
||||
reasonLaunchError = "launch error"
|
||||
reasonTimedOut = "timed out"
|
||||
reasonNonzeroExit = "nonzero exit"
|
||||
reasonTraceError = "unreadable trace"
|
||||
// reasonPreconditionFailures is the run that exited cleanly having spent its
|
||||
// budget failing preconditions. The scope guard records one on every step it
|
||||
// could not bring the app back for and lets the run finish, so nothing else
|
||||
// here separates it from a run that explored the whole budget and found
|
||||
// nothing. A majority is the line: below it the failures are transient and
|
||||
// the run still explored, above it the step count the analysis would censor
|
||||
// at is mostly steps the app was never there for.
|
||||
reasonPreconditionFailures = "precondition failures"
|
||||
reasonMalformedStep = "violation step outside the budget"
|
||||
)
|
||||
|
||||
type classifiedRun struct {
|
||||
@@ -201,6 +214,9 @@ func classify(record runRecord, budget int) classifiedRun {
|
||||
case record.TraceError != "":
|
||||
item.ExcludedBecause = reasonTraceError
|
||||
return item
|
||||
case record.PreconditionFailures*2 > record.Steps:
|
||||
item.ExcludedBecause = reasonPreconditionFailures
|
||||
return item
|
||||
}
|
||||
if record.FirstViolationOriginStep == nil {
|
||||
if len(record.ViolatedProperties) > 0 {
|
||||
|
||||
Reference in new issue
Block a user