experiment instrumentation: step budgets, arm labels, campaign runner (#72)

* feat(cli): add --max-steps for step-bounded runs

runner.Options.MaxSteps already worked but was unreachable from the command
line. A step budget is what makes two generators comparable: one making a
model call per step and one drawing from a PRNG are not comparable per second.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* feat(trace): record arm membership and host in meta.json

meta.json recorded the seed but not which picker ran, how it was configured,
what budget it was given, or which machine produced it. A directory of runs
cannot be attributed to an experiment cell without those, which makes any
factorial computed from such a directory unanalysable after the fact.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* feat(cli): add --arm and populate run meta from it

Model and instructions are recorded only when the LLM picker is the one that
will actually run, so a spec declaring generator = llm() that is run under the
seeded picker does not label its trace with a model it never called.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* feat(campaign): sweep seeds for one experiment cell

campaign.json lists the seeds a sweep intended to run and is written before
the first run, so a host that dropped runs shows up as missing seeds rather
than as a smaller sample. Seed 0 is rejected: sanderling test reads it as
"derive a seed from the clock", which is why conformance/gates.sh controls
nothing today.

Each run contributes one runs.jsonl line carrying steps to first violation by
origin step, the step that armed the failed obligation, so the survival
analysis never reopens a trace.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* fix(runner): no silent generator fallback, and llm on web

--generator llm against a spec declaring no generator = llm(...) logged a
warning and ran the seeded picker. For a comparison campaign that is silent
arm corruption: the run completes, the directory looks correct, and the wrong
policy drove it. It is now fatal.

pickSources also returned the V8 source for both action and extractor on web
before it looked at the generator, so the llm policy was unreachable there.
The two axes are now independent: the driver picks the extractor source, the
flag picks the action source, and llmSource composes with either because the
runner populates the candidate list and screenshot on every platform.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* fix(chrome): make the hierarchy dump agree with the web runtime

Three facts differed between the dump the goja host reads and the DOM the V8
host reads, so the two enumerated different candidates on one page.

scrollable was never emitted, and worker.go reads exactly that attribute while
targets.ts requires it for scrolls, so the goja host could not offer a single
web scroll. clickable tested el.onclick, which React assigns to its root
container for event delegation, making the whole viewport a tap target here and
in no other enumeration. Both now resolve through the selector sets in
pkg/spec/src/web-runtime.ts.

The dump also rooted at body while collectTargets walks querySelectorAll("*"),
so the goja host never saw html, where page-level scrolling lives. It now roots
at documentElement and skips the head subtree, which is all zero-bounds and
would otherwise carry script and title text into the trace.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* fix(conformance): give the gate reproducible seeds

SEED defaulted to 0 and sanderling test reads --seed 0 as "derive a seed from
the clock", so the tunable controlled nothing and a gate failure could not be
re-run. SEEDS now takes one explicit non-zero seed per run, recorded in the
results table so a failing row names its stream.

The five runs stay on five different streams: a gate that scored one path five
times would catch less than one that scores five.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* fix(chrome): emit editable as a plain boolean

editable was emitted as `isEditable || null`, and an absent field sends
internal/hierarchy into the native fallback, which reads any class name
containing "EditText" as an Android text widget. On web that is just a CSS
class, so a page styling a div with it was editable to the goja host and not to
the web runtime, and the model policy could be offered typing into a div.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* fix(spec): leave the head subtree out of the web target walk

collectTargets walked querySelectorAll("*") while the hierarchy dump skips head,
so the two hosts enumerated different element sets on every page with a <head>.
No candidate changes: builtinCandidates pushes only for targets acceptsTarget
admits, and head elements have no positive bounds, so the list the draw ranges
over is untouched. What changes is that targetIndex now means the same thing on
both hosts.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* test(chrome): compare the facts both hosts derive from one DOM

The existing parity harness hand-authors the facts on both sides, so it proves
that given identical facts both hosts select identical candidates, and says
nothing about the two code paths that derive those facts from a real page. Four
divergences lived in that blind spot and it passed throughout.

This drives one real page and compares clickable, enabled, editable, scrollable
and positiveBounds element by element, plus the element sets themselves, which
is what catches a host that omits html or includes head. Reverting any of the
four fixes makes it fail naming the element and the fact.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* chore(make): run the browser packages one at a time

Both launch Chrome and launching two at once has failed with "Launch: context
canceled".

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* style: remove every em-dash and en-dash

Eighteen occurrences across fourteen files. Each sentence was repunctuated to
suit what the dash was doing rather than swapped for a hyphen, which produces
comma splices. The minus sign in folio-web's ledger is a minus sign and stays.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* fix(chrome): honor the caller context in Launch

Launch and clearState ran against d.tabCtx, so a target that accepts the
connection and never answers wedged the process past its own --duration and
through SIGTERM, needing SIGKILL. Unattended that is a campaign worker lost for
the rest of the sweep with no diagnostic.

The browser is still allocated against d.tabCtx first, because chromedp starts
Chrome under whichever context calls Run first and allocating under a caller
deadline would kill the browser when Launch returns. Everything after
allocation goes through runCtx.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* fix(sidecarassets): publish the extracted jar through a rename

Extract wrote a 96 MB jar with a plain WriteFile into a temp path every
sanderling process on the host shares. On a cold host several concurrent
workers all miss the checksum and all write the same path, and O_TRUNC lets one
spawn a JVM against another's half-written archive. A fresh experiment host is
exactly a cold host.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* feat(campaign): kill a run that outlives --run-timeout

A wedged run holds its worker for the rest of the sweep, and on an unattended
host nothing else will send it a signal. Defaults to three times --duration and
must exceed it. A killed run is recorded as timed_out rather than as a generic
failure, so the analysis can tell a lost cell from a real crash.

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX

* style(test): gofmt browser_test.go

Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
This commit is contained in:
pj authored and GitHub committed 2026-08-12 22:20:31 +05:30
1 parent 26b49b379a
commit 76dce1a75e
44 files changed
+3021 -87

No files matched your search

+343
View File
@@ -0,0 +1,343 @@
//go:build browser
package chrome
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"path/filepath"
"runtime"
"slices"
"testing"
"time"
"github.com/chromedp/chromedp"
"github.com/priyanshujain/sanderling/internal/bundler"
"github.com/priyanshujain/sanderling/internal/hierarchy"
)
// One page, two fact producers.
//
// Sanderling enumerates candidate actions from two independent hosts. The goja
// host reads the hierarchy dump this driver builds; the V8 host reads the live
// DOM through collectTargets in pkg/spec/src/web-runtime.ts. Both feed the SAME
// eligibility rule (pkg/spec/src/targets.ts), so the two enumerations agree only
// as far as the two producers agree on the facts.
//
// internal/verifier/host_parity_test.go and pkg/spec/test/host-parity.test.ts
// pin the other half: given identical facts, both hosts select identical
// candidates. They hand-author those facts on both sides, so neither producer is
// on their path, and three producer bugs lived in that gap: the dump emitted no
// `scrollable` attribute at all, it derived `clickable` from el.onclick (which
// React assigns to its root container, making the whole viewport a tap target
// here and nowhere else), and it rooted at document.body while the web runtime
// walks the whole document, hiding `html` and with it every page-level scroll.
//
// This test starts from one real DOM in a real browser and compares what the two
// producers derive from it, element by element. It found a fourth: the dump left
// `editable` unset rather than false, and an unset field sends the parser into
// the native EditText class-name heuristic, which reads a CSS class as an
// Android text widget.
// elementFacts is one element as a producer reports it: the tag, for readable
// failures, and every fact acceptsTarget consults.
type elementFacts struct {
tag string
clickable bool
enabled bool
editable bool
scrollable bool
positiveBounds bool
}
// factRow pairs an element's facts with the id both producers key on, kept in
// enumeration order because the order is part of the picker's parity contract.
type factRow struct {
id string
facts elementFacts
}
func TestHierarchy_DerivesTheSameFactsAsTheWebRuntime(t *testing.T) {
server := httptest.NewServer(http.FileServer(http.Dir("testdata")))
defer server.Close()
d := New()
defer d.Terminate(context.Background())
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
if err := d.Launch(ctx, server.URL+"/fact-parity.html", false, nil); err != nil {
t.Fatalf("Launch: %v", err)
}
dump, err := d.Hierarchy(ctx)
if err != nil {
t.Fatalf("Hierarchy: %v", err)
}
fromDump := factsFromHierarchyDump(t, dump)
fromWebRuntime := factsFromWebRuntime(ctx, t, d)
requireEveryElementNamed(t, "the hierarchy dump", fromDump)
requireEveryElementNamed(t, "the web runtime", fromWebRuntime)
requireBothPolarities(t, fromWebRuntime)
compareEnumeratedElements(t, fromDump, fromWebRuntime)
compareDerivedFacts(t, fromDump, fromWebRuntime)
}
// factsFromHierarchyDump reads the dump the way the goja host does: parse it,
// then take exactly the fields internal/verifier/worker.go targets() reads off
// each element.
func factsFromHierarchyDump(t *testing.T, dump string) []factRow {
t.Helper()
tree, err := hierarchy.Parse(dump)
if err != nil {
t.Fatalf("parse hierarchy: %v", err)
}
rows := make([]factRow, 0, len(tree.Elements))
for _, element := range tree.Elements {
rows = append(rows, factRow{
id: element.ResourceID,
facts: elementFacts{
tag: element.Attributes["tag"],
clickable: element.Clickable,
enabled: element.Enabled,
editable: element.Editable,
scrollable: element.Attributes["scrollable"] == "true",
positiveBounds: hasPositiveBounds(
element.Bounds.Width(),
element.Bounds.Height(),
),
},
})
}
return rows
}
// factsFromWebRuntime installs pkg/spec/test/dom-facts-probe.ts into the page
// and calls it. The probe is bundled through the production web bundler and
// reports what the production collectTargets derives, so the facts come from the
// shipped runtime rather than a copy of it; only the id-carrying readback is
// test-only.
func factsFromWebRuntime(
ctx context.Context,
t *testing.T,
d *Driver,
) []factRow {
t.Helper()
specSource := filepath.Join(repoRootDir(t), "pkg", "spec")
probe, err := bundler.BundleWeb(bundler.WebOptions{
EntryFile: filepath.Join(specSource, "test", "dom-facts-probe.ts"),
WebRuntimeFile: filepath.Join(specSource, "src", "web-runtime.ts"),
})
if err != nil {
t.Fatalf("bundle dom facts probe: %v", err)
}
if err := d.InstallBundle(ctx, probe.JavaScript); err != nil {
t.Fatalf("install dom facts probe: %v", err)
}
var encoded string
script := `JSON.stringify(window.__sanderlingDomFacts__())`
if err := chromedp.Run(d.tabCtx, chromedp.Evaluate(script, &encoded)); err != nil {
t.Fatalf("read web runtime facts: %v", err)
}
var wire []struct {
ID string `json:"id"`
Tag string `json:"tag"`
Clickable bool `json:"clickable"`
Enabled bool `json:"enabled"`
Editable bool `json:"editable"`
Scrollable bool `json:"scrollable"`
Width int `json:"width"`
Height int `json:"height"`
}
if err := json.Unmarshal([]byte(encoded), &wire); err != nil {
t.Fatalf("decode web runtime facts: %v", err)
}
rows := make([]factRow, 0, len(wire))
for _, item := range wire {
rows = append(rows, factRow{
id: item.ID,
facts: elementFacts{
tag: item.Tag,
clickable: item.Clickable,
enabled: item.Enabled,
editable: item.Editable,
scrollable: item.Scrollable,
positiveBounds: hasPositiveBounds(item.Width, item.Height),
},
})
}
return rows
}
// hasPositiveBounds is the positiveBounds fact of pkg/spec/src/targets.ts,
// applied to both producers so the geometry comparison cannot drift from the
// rule it stands in for.
func hasPositiveBounds(width, height int) bool {
return width > 0 && height > 0
}
// requireEveryElementNamed fails when a producer enumerates an element the
// fixture gave no id, because such an element cannot be joined and would sit in
// the comparison invisibly.
func requireEveryElementNamed(t *testing.T, producer string, rows []factRow) {
t.Helper()
for index, row := range rows {
if row.id == "" {
t.Fatalf(
"%s enumerated an unnamed <%s> at position %d; every element in "+
"testdata/fact-parity.html must carry an id to be comparable",
producer,
row.facts.tag,
index,
)
}
}
}
// requireBothPolarities keeps the comparison from going vacuous. A fixture, or
// an emulated viewport, that leaves a fact constant would compare false against
// false on every element and pass while proving nothing about that fact.
func requireBothPolarities(t *testing.T, rows []factRow) {
t.Helper()
for _, fact := range []struct {
name string
read func(elementFacts) bool
}{
{"clickable", func(f elementFacts) bool { return f.clickable }},
{"enabled", func(f elementFacts) bool { return f.enabled }},
{"editable", func(f elementFacts) bool { return f.editable }},
{"scrollable", func(f elementFacts) bool { return f.scrollable }},
{"positiveBounds", func(f elementFacts) bool { return f.positiveBounds }},
} {
var sawTrue, sawFalse bool
for _, row := range rows {
if fact.read(row.facts) {
sawTrue = true
} else {
sawFalse = true
}
}
if !sawTrue || !sawFalse {
t.Errorf(
"testdata/fact-parity.html no longer exercises %s both ways (the web "+
"runtime saw true=%v, false=%v), so comparing it proves nothing",
fact.name,
sawTrue,
sawFalse,
)
}
}
}
// compareEnumeratedElements is the check that the two producers walk the same
// document. It is what notices a producer that roots at body and never sees
// `html`, or one that enumerates the head subtree the other drops.
func compareEnumeratedElements(
t *testing.T,
fromDump, fromWebRuntime []factRow,
) {
t.Helper()
dumpIDs := enumeratedIDs(fromDump)
webIDs := enumeratedIDs(fromWebRuntime)
for _, row := range fromWebRuntime {
if !slices.Contains(dumpIDs, row.id) {
t.Errorf(
"the hierarchy dump never enumerated %q (<%s>); the web runtime does, "+
"so the goja host cannot offer a single action on it",
row.id,
row.facts.tag,
)
}
}
for _, row := range fromDump {
if !slices.Contains(webIDs, row.id) {
t.Errorf(
"the web runtime never enumerated %q (<%s>); the hierarchy dump does, "+
"so the V8 host cannot offer a single action on it",
row.id,
row.facts.tag,
)
}
}
if len(dumpIDs) == len(webIDs) && !slices.Equal(dumpIDs, webIDs) {
t.Errorf(
"the two producers enumerate the same elements in different order\n"+
" dump=%v\n web=%v",
dumpIDs,
webIDs,
)
}
}
// compareDerivedFacts compares the facts fact by fact, over every element both
// producers saw, so a failure names the element and the fact that diverged.
func compareDerivedFacts(t *testing.T, fromDump, fromWebRuntime []factRow) {
t.Helper()
webByID := map[string]elementFacts{}
for _, row := range fromWebRuntime {
webByID[row.id] = row.facts
}
for _, row := range fromDump {
web, ok := webByID[row.id]
if !ok {
continue
}
dump := row.facts
if dump.tag != web.tag {
t.Errorf(
"%q: the hierarchy dump calls it <%s>, the web runtime <%s>; the id "+
"join is resolving to different elements",
row.id,
dump.tag,
web.tag,
)
}
for _, fact := range []struct {
name string
dump bool
web bool
}{
{"clickable", dump.clickable, web.clickable},
{"enabled", dump.enabled, web.enabled},
{"editable", dump.editable, web.editable},
{"scrollable", dump.scrollable, web.scrollable},
{"positiveBounds", dump.positiveBounds, web.positiveBounds},
} {
if fact.dump != fact.web {
t.Errorf(
"%q (<%s>): the hierarchy dump derives %s=%v, the web runtime "+
"derives %s=%v",
row.id,
dump.tag,
fact.name,
fact.dump,
fact.name,
fact.web,
)
}
}
}
}
func enumeratedIDs(rows []factRow) []string {
ids := make([]string, 0, len(rows))
for _, row := range rows {
ids = append(ids, row.id)
}
return ids
}
func repoRootDir(t *testing.T) string {
t.Helper()
_, thisFile, _, ok := runtime.Caller(0)
if !ok {
t.Fatal("cannot resolve caller path")
}
return filepath.Clean(
filepath.Join(filepath.Dir(thisFile), "..", "..", ".."),
)
}