fix(confusion-matrix): a campaign that died is missing data, not a true negative

The sweep-level loop excluded a run on launch_error alone, while
excludedBecause already checked the campaign process's exit code. An
interrupted campaign wrote exit_code -1 with an empty launch_error, so
its one completed seed scored the implementation as a clean cell on a
tenth of the planned evidence.

The fixture builder wrote one exit code into both the sweep record and
the campaign run record, which is why no test could tell the two levels
apart.
This commit is contained in:
pj committed 2026-08-18 14:08:59 +05:30
1 parent 1d684a98eb
commit ff6c66a74b
3 files changed
+55 -5

No files matched your search

@@ -16,8 +16,12 @@ import (
type fixtureRun struct {
Seed int64
ExitCode int
TimedOut bool
Violated []string
// CampaignExitCode is the campaign process's own exit status, which the
// sweep records beside the campaign directory. It is not the exit status of
// the run inside that campaign.
CampaignExitCode int
TimedOut bool
Violated []string
// Surfaces is the locatableSurfaces reading the trace records. A nil map
// with NoTrace false still writes a reading of every surface false.
Surfaces map[string]bool
@@ -141,7 +145,7 @@ func writeImplementation(t *testing.T, built fixture, implementation fixtureImpl
}
runs = append(runs, map[string]any{
"seed": run.Seed,
"exit_code": run.ExitCode,
"exit_code": run.CampaignExitCode,
"campaign_directory": campaignDirectory,
})
}