fix ltl semantics and unify action enumeration (#71)

* fix(ltl): give every thunk a construction identity

Two distinct unnamed predicates both described as "Thunk(...)", so obligation
collapse merged their residuals and could drop a live violation. Identity is
assigned at construction and the fields are unexported, so a thunk cannot be
built without one.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(ltl): reduce a thrown-predicate residual instead of panicking

The verifier substitutes an ErrorFormula for the residual of a property whose
predicate threw, and that residual is fed back in on the next step. reduce had
no case for it, so the run crashed. It re-reports the same failure now.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(ltl): make a bounded always the dual of a bounded eventually

G<=n(f) and not F<=n(not f) disagreed on traces where the inner was still
pending when the window closed, so nnf's negation normal form was not semantics
preserving. Both sides now range over the observations at which their inner can
definitely resolve: the eventually keeps a pending inner as a disjunct, and the
always discharges vacuously at window close.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(ltl): arm a one-shot root once per run

A root that carries its own horizon is one obligation for the whole run, not one
per observation. Re-instantiating a top-level eventually monitored G F<=n(p)
instead of F<=n(p) and left one live obligation per step behind; a bounded
always restarted its window every step and never closed.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(verifier): stop wrapping a top-level eventually in always

`eventually(p).within(300, "seconds")` as a property meant "within 300 seconds
of every step", which spawned an obligation per step with its own resolved
deadline. A 553-step run carried 553 of them and serialized a 75 KB residual.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(ltl): serialize the resolved deadline of a bounded window

Two obligations spawned at different steps from one duration-bounded formula
differ only in the deadline the evaluator resolved for them, so they serialized
identically and the trace erased a distinction the evaluator makes. The authored
window stays in amount/unit; the resolved deadline rides alongside.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(verifier): split a witness's origin step from its detection step

A deferred obligation spans two steps: the one that armed it and the one whose
reduction failed. They were conflated under one index, so the extractor snapshot
(which is the detecting step's state) was reported against the origin step.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(runner): record a witness's detection step in the trace

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* feat(replay-ui): show the step a violation was detected at

The witness evidence is the detecting step's state, so say which step that is
and let a reader jump to it.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(verifier): record the extractor state the predicates actually read

On the web path extractor bodies are evaluated in V8 and injected here, but only
the goja value was replaced. The trace diff and the violation witness therefore
described a state no property ever saw.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* refactor(spec): one candidate producer over one target-eligibility rule

Both hosts routed verbs themselves and both policies enumerated their own
actions, and all four drifted. Web sent `swipes` to scrollable containers only,
so swipe-to-dismiss on a list row was reachable on native and unreachable on
web; the model policy folded gestures its own way and could not reach what the
seeded picker drew.

A host now reports facts about every element and never decides which verb may
act on it: targets.ts acceptsTarget owns that for both. pick.ts builtinCandidates
is the single enumeration, and the model policy reads it through
__sanderlingEnumerateBuiltin__ instead of reimplementing it in Go.

Gesture verbs change with it: scrolls stay vertical over scrollable containers,
swipes go free-form in all four directions from any element with real bounds.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(runner): name a builtin scroll by its drag origin

A builtin gesture carries endpoints and no selector, so every scroll rendered as
"Scroll down " in the prompt's recent-action memory and two scrollable regions
were indistinguishable.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(chrome): clear storage over cdp instead of scripting an opaque origin

Launch runs while the tab is still on about:blank, whose opaque origin denies
storage access, so localStorage.clear() threw SecurityError and every web run
died at launch. Storage.clearDataForOrigin needs no navigation. The exception
helper lands here because "Uncaught" is what hid this for so long.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(chrome): enable the swiftshader webgl fallback

Headless Chrome runs with --disable-gpu, and without this flag it refuses the
software WebGL backend: getContext returns null, so a canvas-rendered app paints
nothing and every screenshot is identical black.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(web): resolve testTag through data-testid or id

Compose Multiplatform emits its testTag into the element id, which the native
table already accepts via the resource-id alias. The two web selector tables
were the only place that rejected it.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* test(spec): type-check the spec api as part of make test

The fake runtime in api.test.ts did not return a chainable handle from extract,
so the file had not type-checked since named() was added. Wiring the check into
make test stops it drifting again.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* docs(manual): one-shot eventually and the gesture verbs

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J
This commit is contained in:
pj authored and GitHub committed 2026-08-12 18:06:04 +05:30
1 parent 7343085614
commit 26b49b379a
48 files changed
+2604 -712

No files matched your search

+5 -2
View File
@@ -29,7 +29,7 @@ WEB_DIST := replay-ui/dist
GOLINES := $(shell $(GO) env GOPATH)/bin/golines
.PHONY: bootstrap proto sidecar sanderling install test test-go test-browser test-companion test-kotlin test-spec-api web-test web-typecheck web-build web-dev replay-dev docs clean release-cli release-npm-dry fmt fmt-go fmt-kotlin fmt-ts fmt-swift
.PHONY: bootstrap proto sidecar sanderling install test test-go test-browser test-companion test-kotlin test-spec-api spec-typecheck web-test web-typecheck web-build web-dev replay-dev docs clean release-cli release-npm-dry fmt fmt-go fmt-kotlin fmt-ts fmt-swift
bootstrap:
$(GO) mod download
@@ -101,7 +101,7 @@ fmt-ts:
fmt-swift:
xcrun swift-format format -i -r companion/Sources
test: test-go test-spec-api web-typecheck web-test
test: test-go spec-typecheck test-spec-api web-typecheck web-test
test-go:
$(GO) test $(GO_PACKAGES)
@@ -126,6 +126,9 @@ test-kotlin:
test-spec-api:
cd pkg/spec && npm test --silent
spec-typecheck:
cd pkg/spec && npm run check --silent
docs: $(DOCS_OUT) build/site/_assets
@echo "built $(words $(DOCS_OUT)) pages to build/site"
+13 -3
View File
@@ -171,6 +171,8 @@ Extractors are evaluated before properties and action generators. Use `.previous
| `now(f)` | `f` evaluated at the current step (for use inside `always`/`next` bodies) |
| `next(f)` | `f` evaluated at the step immediately after the current one |
A property that is an `eventually` at the top level is one goal for the whole run: it is armed at the first step and discharged for good the first time it holds. Written inside `always(...)` the same formula is armed again at every step, which asks for the window to be met from every step in the run.
**Formula combinators** - available on every `Formula`:
| Method | Meaning |
@@ -207,11 +209,19 @@ On web, `"back"` maps to Backspace and `"home"` is not supported. All other keys
| `doubleTaps` | Random double tap on a clickable element |
| `longPresses` | Random long press on a clickable element |
| `typing` | Types a value from the edge-case corpus into a random editable field |
| `scrolls` | Random scroll gesture |
| `swipes` | Random swipe gesture |
| `scrolls` | Scrolls a random scrollable container up or down |
| `swipes` | Random up, down, left, or right swipe from any visible element |
| `waitOnce` | Idles one step |
| `pressKeys` | Presses a random supported key |
`scrolls` anchors on a scrollable container and moves its content up or down. It never
scrolls sideways, because every scrollable container on screen gets a candidate and a
sideways pair doubles that list for little return; write `Scroll({ in, direction })` when
you need one.
`swipes` is a free drag of 200 to 600 px from any element with real bounds, in any of the
four directions. The sideways ones are what reach swipe-to-dismiss and swipe-to-delete on
a list row.
### `actions(generator)`
Wraps a callback that returns `Action[]`. The callback runs each step the generator is eligible.
@@ -280,7 +290,7 @@ export const generator = llm({
Set `OPENROUTER_API_KEY` or `OPENAI_API_KEY` (OpenRouter wins if both are set). With a plain OpenAI key, drop the vendor prefix from the model id. The model needs image input and strict `json_schema` structured output.
Each step it gets a screenshot plus a numbered list of the concrete actions your tree yields right now, each tagged with its weight, and picks one number. `instructions` are appended to the prompt: say what the app is, not how to test it — the model works that part out. Everything else is unchanged. Setup actions still run first, typing still falls back to the edge-case corpus when the model supplies no text, and the trace records the reasoning, the chosen number, and `source: "llm"` so the replay UI can show why each pick happened.
Each step it gets a screenshot plus a numbered list of the concrete actions your tree yields right now, each tagged with its weight, and picks one number. That list is the seeded picker's own candidate enumeration, so both modes explore the same action space and only the choice differs. `instructions` are appended to the prompt: say what the app is, not how to test it; the model works that part out. Everything else is unchanged. Setup actions still run first, typing still falls back to the edge-case corpus when the model supplies no text, and the trace records the reasoning, the chosen number, and `source: "llm"` so the replay UI can show why each pick happened.
It is one model call per step, so keep `--duration` modest.
+69 -11
View File
@@ -5,6 +5,7 @@ import (
"context"
"encoding/json"
"fmt"
"net/url"
"strconv"
"strings"
"sync"
@@ -14,6 +15,7 @@ import (
"github.com/chromedp/cdproto/network"
"github.com/chromedp/cdproto/page"
"github.com/chromedp/cdproto/runtime"
"github.com/chromedp/cdproto/storage"
"github.com/chromedp/chromedp"
"github.com/chromedp/chromedp/kb"
@@ -37,6 +39,11 @@ func New() *Driver {
append(chromedp.DefaultExecAllocatorOptions[:],
chromedp.Flag("headless", true),
chromedp.Flag("disable-gpu", true),
// Chrome refuses to fall back to the SwiftShader WebGL backend
// without this flag, so with --disable-gpu a canvas app (Compose
// for Web, Flutter web, anything on WebGL) gets a null context and
// paints nothing: black screenshots and an empty accessibility DOM.
chromedp.Flag("enable-unsafe-swiftshader", true),
chromedp.NoSandbox,
// CI runners give Chrome a tiny /dev/shm; without this the browser
// process hangs on startup and never reports its DevTools socket.
@@ -91,17 +98,8 @@ func New() *Driver {
func (d *Driver) Launch(ctx context.Context, bundleID string, clearState bool, _ map[string]string) error {
if clearState {
if err := chromedp.Run(d.tabCtx, network.ClearBrowserCookies()); err != nil {
return fmt.Errorf("clear cookies: %w", err)
}
if err := chromedp.Run(d.tabCtx, chromedp.ActionFunc(func(ctx context.Context) error {
_, exp, err := runtime.Evaluate(`localStorage.clear(); sessionStorage.clear();`).Do(ctx)
if exp != nil {
return fmt.Errorf("clear storage: %s", exp.Text)
}
if err := d.clearState(bundleID); err != nil {
return err
})); err != nil {
return fmt.Errorf("clear storage: %w", err)
}
}
if err := chromedp.Run(d.tabCtx, chromedp.Navigate(bundleID)); err != nil {
@@ -125,6 +123,66 @@ func (d *Driver) Launch(ctx context.Context, bundleID string, clearState bool, _
return nil
}
// clearState wipes the target's stored data before the application loads.
// Script cannot do it: the tab still sits on about:blank, whose opaque origin
// denies storage access, so `localStorage.clear()` throws SecurityError and
// every web run dies at launch. The Storage domain clears by origin instead,
// which needs no navigation. sessionStorage is per-tab and outside that
// domain's reach; it only survives when a relaunch reuses a tab already on
// the target origin, which is the one case where script can reach it.
func (d *Driver) clearState(bundleID string) error {
if err := chromedp.Run(d.tabCtx, network.ClearBrowserCookies()); err != nil {
return fmt.Errorf("clear cookies: %w", err)
}
origin := securityOrigin(bundleID)
if origin == "" {
return nil
}
clearForOrigin := storage.ClearDataForOrigin(origin, string(storage.TypeAll))
if err := chromedp.Run(d.tabCtx, clearForOrigin); err != nil {
return fmt.Errorf("clear storage for %s: %w", origin, err)
}
script := fmt.Sprintf(
`location.origin === %q && (sessionStorage.clear(), true)`, origin)
return chromedp.Run(d.tabCtx, chromedp.ActionFunc(func(ctx context.Context) error {
_, exception, err := runtime.Evaluate(script).Do(ctx)
if err != nil {
return fmt.Errorf("clear session storage: %w", err)
}
if exception != nil {
return fmt.Errorf("clear session storage: %s", exceptionMessage(exception))
}
return nil
}))
}
// securityOrigin returns the scheme://host[:port] the Storage domain keys data
// by, or "" for a target that has no such origin (data:, file:, about:blank),
// where there is no per-origin storage to clear.
func securityOrigin(bundleID string) string {
parsed, err := url.Parse(bundleID)
if err != nil || parsed.Host == "" {
return ""
}
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return ""
}
return parsed.Scheme + "://" + parsed.Host
}
// exceptionMessage renders a page exception for an error string. The
// description carries the actual message ("SecurityError: Failed to read the
// 'localStorage' property..."); Text alone is the useless "Uncaught".
func exceptionMessage(exception *runtime.ExceptionDetails) string {
if exception == nil {
return ""
}
if exception.Exception != nil && exception.Exception.Description != "" {
return exception.Exception.Description
}
return exception.Text
}
func (d *Driver) Terminate(_ context.Context) error {
d.tabCancel()
d.allocCancel()
@@ -483,7 +541,7 @@ func (d *Driver) InstallBundle(ctx context.Context, source []byte) error {
return fmt.Errorf("evaluate bundle: %w", err)
}
if exception != nil {
return fmt.Errorf("bundle threw: %s", exception.Text)
return fmt.Errorf("bundle threw: %s", exceptionMessage(exception))
}
return nil
}),
+74
View File
@@ -6,10 +6,84 @@ import (
"context"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/chromedp/chromedp"
)
// TestLaunch_ClearStateWipesStorageForTheTargetOrigin covers the CLI's default
// path (--clear-data). The tab sits on about:blank when Launch runs, an opaque
// origin that denies storage access, so clearing by script there throws
// SecurityError and kills every web run before the app loads.
func TestLaunch_ClearStateWipesStorageForTheTargetOrigin(t *testing.T) {
const page = `<body><script>
const visits = Number(localStorage.getItem("visits") ?? "0") + 1;
localStorage.setItem("visits", String(visits));
sessionStorage.setItem("tab", "dirty");
</script></body>`
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "text/html")
_, _ = w.Write([]byte(page))
}))
defer server.Close()
d := New()
defer d.Terminate(context.Background())
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
if err := d.Launch(ctx, server.URL, true, nil); err != nil {
t.Fatalf("Launch with clearState on a fresh tab: %v", err)
}
if err := d.Launch(ctx, server.URL, false, nil); err != nil {
t.Fatalf("Launch: %v", err)
}
var visits string
if err := chromedp.Run(d.tabCtx,
chromedp.Evaluate(`localStorage.getItem("visits")`, &visits)); err != nil {
t.Fatalf("read localStorage: %v", err)
}
if visits != "2" {
t.Fatalf("visits = %q, want 2 (two loads, storage kept)", visits)
}
if err := d.Launch(ctx, server.URL, true, nil); err != nil {
t.Fatalf("Launch with clearState on the target origin: %v", err)
}
if err := chromedp.Run(d.tabCtx,
chromedp.Evaluate(`localStorage.getItem("visits")`, &visits)); err != nil {
t.Fatalf("read localStorage: %v", err)
}
if visits != "1" {
t.Errorf("visits = %q, want 1 (storage cleared before the app loaded)", visits)
}
}
// TestLaunch_WebGLContextIsAvailable pins the SwiftShader fallback. Headless
// Chrome runs with --disable-gpu, and without --enable-unsafe-swiftshader it
// refuses the software WebGL backend: getContext returns null, so a
// canvas-rendered app paints nothing and every screenshot is identical black.
func TestLaunch_WebGLContextIsAvailable(t *testing.T) {
d := New()
defer d.Terminate(context.Background())
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
if err := d.Launch(ctx, "data:text/html,<body></body>", false, nil); err != nil {
t.Fatalf("Launch: %v", err)
}
var hasContext bool
if err := chromedp.Run(d.tabCtx, chromedp.Evaluate(
`!!document.createElement("canvas").getContext("webgl2")`, &hasContext)); err != nil {
t.Fatalf("evaluate: %v", err)
}
if !hasContext {
t.Error("webgl2 context is null; a canvas-rendered app would render nothing")
}
}
// TestActionMethods_HonorCallerCancellation confirms the DeviceDriver action
// methods route through runCtx so a cancelled caller context aborts the CDP
// round-trip instead of blocking on d.tabCtx. Without this a hung browser would
+47
View File
@@ -0,0 +1,47 @@
package chrome
import (
"testing"
"github.com/chromedp/cdproto/runtime"
)
func TestSecurityOrigin(t *testing.T) {
cases := map[string]string{
"http://localhost:8088": "http://localhost:8088",
"http://localhost:5173/": "http://localhost:5173",
"https://app.example.com/a/b?c=d": "https://app.example.com",
"data:text/html,<body>hi</body>": "",
"file:///tmp/index.html": "",
"about:blank": "",
"": "",
"://not a url": "",
}
for target, want := range cases {
if got := securityOrigin(target); got != want {
t.Errorf("securityOrigin(%q) = %q, want %q", target, got, want)
}
}
}
// TestExceptionMessage_PrefersDescription pins the useful half of a page
// exception. ExceptionDetails.Text is "Uncaught" for every throw; the
// description is where the error a caller can act on lives.
func TestExceptionMessage_PrefersDescription(t *testing.T) {
withDescription := &runtime.ExceptionDetails{
Text: "Uncaught",
Exception: &runtime.RemoteObject{
Description: "SecurityError: Access is denied for this document.",
},
}
if got := exceptionMessage(withDescription); got != withDescription.Exception.Description {
t.Errorf("exceptionMessage = %q, want the description", got)
}
textOnly := &runtime.ExceptionDetails{Text: "Uncaught SyntaxError"}
if got := exceptionMessage(textOnly); got != "Uncaught SyntaxError" {
t.Errorf("exceptionMessage = %q, want the text fallback", got)
}
if got := exceptionMessage(nil); got != "" {
t.Errorf("exceptionMessage(nil) = %q, want empty", got)
}
}
+7 -1
View File
@@ -42,7 +42,13 @@ func TranslateStringSelector(selector string) (string, bool, error) {
return `[aria-label="` + cssEscape(value) + `"]`, false, nil
case "descPrefix":
return `[aria-label^="` + cssEscape(value) + `"]`, false, nil
case "testTag", "testID", "testid", "data-testid":
case "testTag":
// Mirrors the in-page table and the native resource-id alias: a
// testTag reaches the DOM as data-testid or as an id, depending on
// the toolkit. `:is()` keeps this one compound selector.
escaped := cssEscape(value)
return `:is([data-testid="` + escaped + `"], [id="` + escaped + `"])`, false, nil
case "testID", "testid", "data-testid":
return `[data-testid="` + cssEscape(value) + `"]`, false, nil
case "placeholder", "placeholderValue", "hintText":
return `[placeholder="` + cssEscape(value) + `"]`, false, nil
+1 -1
View File
@@ -21,7 +21,7 @@ func TestTranslateStringSelector_KnownKeys(t *testing.T) {
{"accessibilityLabel:logout", `[aria-label="logout"]`, false},
{"aria-label:Sign in", `[aria-label="Sign in"]`, false},
{"descPrefix:account:", `[aria-label^="account:"]`, false},
{"testTag:submit", `[data-testid="submit"]`, false},
{"testTag:submit", `:is([data-testid="submit"], [id="submit"])`, false},
{"testID:submit", `[data-testid="submit"]`, false},
{"placeholder:Email", `[placeholder="Email"]`, false},
}
+170
View File
@@ -0,0 +1,170 @@
package ltl
import (
"math/rand/v2"
"testing"
"testing/quick"
"time"
)
// dualityTrace holds the per-step truth of each predicate. The thunks read the
// current step out of the trace rather than counting their own invocations, so
// a formula and its negation see the same values no matter how many times each
// predicate is reduced.
type dualityTrace struct {
step int
values [2][]bool
}
func newDualityTrace(random *rand.Rand, steps int) *dualityTrace {
trace := &dualityTrace{}
for index := range trace.values {
trace.values[index] = make([]bool, steps)
for step := range trace.values[index] {
trace.values[index][step] = random.IntN(2) == 0
}
}
return trace
}
func (t *dualityTrace) atoms() []Formula {
return []Formula{
ThunkNamed("p0", func() (bool, error) { return t.values[0][t.step], nil }),
ThunkNamed("p1", func() (bool, error) { return t.values[1][t.step], nil }),
Pure(true),
Pure(false),
}
}
// randomFormula draws a formula of at most the given depth over the atoms.
// Every operator the evaluator can reduce is reachable, including the bounded
// Always that only nnf produces.
func randomFormula(random *rand.Rand, depth int, atoms []Formula) Formula {
if depth == 0 {
return atoms[random.IntN(len(atoms))]
}
inner := func() Formula { return randomFormula(random, depth-1, atoms) }
bound := random.IntN(3) + 1
switch random.IntN(12) {
case 0, 1:
return atoms[random.IntN(len(atoms))]
case 2:
return Not(inner())
case 3:
return Next(inner())
case 4:
return Now(inner())
case 5:
return And(inner(), inner())
case 6:
return Or(inner(), inner())
case 7:
return Implies(inner(), inner())
case 8:
return Always(inner())
case 9:
return Eventually(inner())
case 10:
if random.IntN(2) == 0 {
return EventuallyWithinSteps(inner(), bound)
}
return EventuallyWithin(inner(), time.Duration(bound)*time.Second)
default:
if random.IntN(2) == 0 {
return AlwaysFormula{Inner: inner(), StepBound: bound, HasStepBound: true}
}
return AlwaysFormula{Inner: inner(), Duration: time.Duration(bound) * time.Second}
}
}
// TestNNF_ExcludedMiddleHoldsOnRandomTraces is the semantic counterpart to the
// describe()-comparing laws in nnf_test.go: those check that nnf produces the
// syntax the dual laws predict, this checks that the syntax it produces means
// the same thing. If any operator pair reduces non-dually then some trace makes
// a formula and its own nnf-negation both violate, so their disjunction
// violates and their conjunction holds.
func TestNNF_ExcludedMiddleHoldsOnRandomTraces(t *testing.T) {
const steps = 8
law := func(seed uint64) bool {
random := rand.New(rand.NewPCG(seed, 0x9e3779b97f4a7c15))
trace := newDualityTrace(random, steps)
formula := randomFormula(random, 3, trace.atoms())
tautology := NewEvaluator(Or(formula, nnf(Not(formula))))
contradiction := NewEvaluator(And(formula, nnf(Not(formula))))
for index := range steps {
trace.step = index
now := time.Unix(int64(index), 0)
if tautology.ObserveAtStep(now, index+1) == VerdictViolated {
t.Logf("seed %d: %s violated at step %d", seed, Describe(formula), index+1)
return false
}
if contradiction.ObserveAtStep(now, index+1) == VerdictHolds {
t.Logf("seed %d: %s held at step %d", seed, Describe(formula), index+1)
return false
}
}
return tautology.Finalize() != VerdictViolated
}
if err := quick.Check(law, &quick.Config{MaxCount: 2000}); err != nil {
t.Error(err)
}
}
// TestNNF_BoundedAlwaysOverNextIsDual is the counterexample that showed nnf was
// not semantics preserving: phi = G<=1(X p) with p true only at the first step.
// Before the bounded operators were made dual, phi violated at step 2 and
// nnf(not phi) violated at step 1, so both a formula and its negation failed on
// one trace. Exactly one of the pair may violate.
func TestNNF_BoundedAlwaysOverNextIsDual(t *testing.T) {
step := 0
predicate := ThunkNamed("p", func() (bool, error) { return step == 0, nil })
formula := AlwaysFormula{Inner: Next(predicate), StepBound: 1, HasStepBound: true}
negated := nnf(Not(formula))
run := func(root Formula) Verdict {
step = 0
evaluator := NewEvaluator(root)
for index := range 3 {
step = index
if evaluator.ObserveAtStep(time.Unix(int64(index), 0), index+1) == VerdictViolated {
return VerdictViolated
}
}
return evaluator.Finalize()
}
if got := run(formula); got == VerdictViolated {
t.Errorf("G<=1(X p) = %v; the window closes on a deferred check, which is not a breach", got)
}
if got := run(negated); got != VerdictViolated {
t.Errorf("nnf(not G<=1(X p)) = %v, want violated", got)
}
if got := run(Or(formula, negated)); got == VerdictViolated {
t.Errorf("excluded middle violated: %v", got)
}
if got := run(And(formula, negated)); got != VerdictViolated {
t.Errorf("phi and not phi = %v, want violated", got)
}
}
// TestEventually_PendingInnerSurvivesAsDisjunct locks the conjunct/disjunct
// mirror the duality rests on: Always keeps a pending inner as a conjunct of
// its residual, so Eventually must keep one as a disjunct. Dropping it made an
// inner that can only discharge on a later step unsatisfiable.
func TestEventually_PendingInnerSurvivesAsDisjunct(t *testing.T) {
values := []bool{false, true, false}
step := 0
formula := EventuallyWithinSteps(Next(ThunkNamed("p", func() (bool, error) {
return values[step], nil
})), 2)
evaluator := NewEvaluator(formula)
if got := evaluator.ObserveAtStep(time.Unix(0, 0), 1); got != VerdictPending {
t.Fatalf("step 1: got %v, want pending", got)
}
step = 1
if got := evaluator.ObserveAtStep(time.Unix(1, 0), 2); got != VerdictHolds {
t.Errorf("step 2: got %v, want holds (X p armed at step 1 discharged here)", got)
}
}
+85 -24
View File
@@ -37,6 +37,10 @@ type Evaluator struct {
violated bool
steps int
violation *Violation
// oneShot marks a root that is armed once at the first observation rather
// than re-asserted at every one; armed records that it has been.
oneShot bool
armed bool
}
// obligation pairs a residual formula with the step that spawned it, so a
@@ -62,7 +66,8 @@ type Violation struct {
}
func NewEvaluator(formula Formula) *Evaluator {
return &Evaluator{root: nnf(formula)}
normalized := nnf(formula)
return &Evaluator{root: normalized, oneShot: isOneShotRoot(normalized)}
}
// Observe evaluates the formula against the current state and returns the
@@ -89,8 +94,11 @@ func (e *Evaluator) ObserveAtStep(now time.Time, step int) Verdict {
}
e.steps = step
fresh := obligation{formula: rootObligation(e.root), origin: step}
obligations := append(e.pending, fresh)
obligations := make([]obligation, 0, len(e.pending)+1)
obligations = append(obligations, e.pending...)
if formula, ok := e.instantiateRoot(); ok {
obligations = append(obligations, obligation{formula: formula, origin: step})
}
e.pending = e.pending[:0]
for _, entry := range obligations {
@@ -121,8 +129,11 @@ func (e *Evaluator) ObserveAtStep(now time.Time, step int) Verdict {
// collapse removes structurally-identical obligations, keeping the first
// occurrence in order so the surviving entry carries the earliest origin step.
// Distinct predicates never merge because ThunkFormula's name participates in
// its describe() key, so deduping cannot hide a violation.
// Equal describe() keys mean the same operators over the same predicates with
// the same remaining bounds, so the merged obligations reduce identically on
// every future and dropping one cannot hide a violation. Distinct predicates
// never merge because every thunk's construction-time identity is part of its
// key, whether or not the caller named it.
func collapse(obligations []obligation) []obligation {
if len(obligations) < 2 {
return obligations
@@ -264,10 +275,48 @@ func (e *Evaluator) Residual() Formula {
return combined
}
// rootObligation returns the formula to instantiate at each step. An outer
// Always is stripped so its inner is re-evaluated every step; any other root
// formula is itself re-instantiated each step (matching the v0.1 semantics
// where a bare Thunk is re-observed on every call).
// instantiateRoot returns the obligation to register for this observation, and
// whether there is one at all. A one-shot root is armed only at the first
// observation; a recurring root is re-asserted at every one.
func (e *Evaluator) instantiateRoot() (Formula, bool) {
if !e.oneShot {
return rootObligation(e.root), true
}
if e.armed {
return nil, false
}
e.armed = true
return e.root, true
}
// isOneShotRoot reports whether a root formula is a single obligation for the
// whole run rather than one instance per observation.
//
// A root that carries its own horizon is one-shot: an eventually is a
// reachability goal ("this happens at some point"), and a bounded always is a
// single window. Re-instantiating either at every step would monitor a
// different property -- G F<=n(p) instead of F<=n(p), and G(p) instead of
// G<=n(p), the latter because a re-instantiated window restarts and never
// closes -- and would leave one live obligation per step behind.
//
// Every other root keeps the implicit-always reading: an unbounded always
// re-instantiates its inner (which is what gives each instance its own origin
// step), and a bare predicate or connective is re-asserted each observation.
func isOneShotRoot(root Formula) bool {
switch concrete := root.(type) {
case EventuallyFormula:
return true
case AlwaysFormula:
return concrete.HasStepBound || concrete.HasDeadline || concrete.Duration > 0
default:
return false
}
}
// rootObligation returns the formula a recurring root instantiates at each
// step. An outer Always is stripped so its inner is re-evaluated every step;
// any other root formula is itself re-instantiated each step (matching the
// v0.1 semantics where a bare Thunk is re-observed on every call).
func rootObligation(root Formula) Formula {
if always, ok := root.(AlwaysFormula); ok {
return always.Inner
@@ -333,8 +382,14 @@ func reduce(formula Formula, now time.Time) reduceResult {
}
return violatedWith(concrete, "pure false")
case ErrorFormula:
// A thrown predicate substituted into a residual at the trace
// boundary. Reducing it re-reports the same failure rather than
// crashing the run.
return violatedByError(concrete, concrete.Message)
case ThunkFormula:
result, err := concrete.Func()
result, err := concrete.predicate()
if err != nil {
return violatedByError(concrete, err.Error())
}
@@ -363,6 +418,9 @@ func reduce(formula Formula, now time.Time) reduceResult {
if innerResult.status == statusHolds {
return holds()
}
// The window is measured in observations at which the inner could have
// discharged, so an inner that is merely pending has not discharged and
// the window closing on it is a violation.
if concrete.HasStepBound && concrete.StepBound <= 1 {
return violatedFrom(innerResult, concrete, "eventually bound exhausted")
}
@@ -373,6 +431,14 @@ func reduce(formula Formula, now time.Time) reduceResult {
if concrete.HasStepBound {
next.StepBound = concrete.StepBound - 1
}
// F(inner) unrolls to inner or X F(inner). A pending inner is a
// deferred way of satisfying the promise, so it is kept as a disjunct
// rather than dropped; dropping it is what made an inner that only
// resolves on a later step unsatisfiable, and it is the mirror of the
// conjunct Always keeps below.
if innerResult.status == statusPending {
return pending(OrFormula{Left: innerResult.formula, Right: next})
}
return pending(next)
case ImpliesFormula:
@@ -444,25 +510,20 @@ func reduce(formula Formula, now time.Time) reduceResult {
if innerResult.status == statusViolated {
return violatedFrom(innerResult, concrete, "always inner violated")
}
// A bounded Always is the dual of a bounded Eventually: once the window
// closes without a breach it is vacuously satisfied. A pending inner at
// the closing step is a deferred obligation (a strong next, or an inner
// liveness that has not discharged); it must be carried so a later step
// or Finalize resolves it, never dropped to holds.
// A bounded Always must reduce exactly as its dual does, so that
// G<=n(f) and not F<=n(not f) agree on every trace. The dual of "the
// window closed on an inner that never definitely held, so violate" is
// "the window closed on an inner that was never definitely breached, so
// hold". A pending inner has not been breached inside the window, so it
// discharges vacuously here exactly as its negation violates on the
// Eventually side.
if concrete.HasStepBound && concrete.StepBound <= 1 {
if innerResult.status == statusHolds {
return holds()
}
return pending(innerResult.formula)
return holds()
}
if concrete.HasDeadline && !now.Before(concrete.Deadline) {
if innerResult.status == statusHolds {
return holds()
}
return pending(innerResult.formula)
return holds()
}
next := concrete
next.Inner = concrete.Inner
if concrete.HasStepBound {
next.StepBound = concrete.StepBound - 1
}
+24 -6
View File
@@ -70,14 +70,32 @@ func TestImplies_TemporalAntecedent_FalseAntecedentHolds(t *testing.T) {
}
}
// A bounded Always whose inner is a still-pending deferred Next obligation must
// carry that obligation past the window, not drop it to holds. The pre-fix
// window-close branch returned holds() unconditionally and lost the violation.
func TestBoundedAlways_PendingInnerCarried(t *testing.T) {
inner := AlwaysFormula{Inner: Next(Thunk(thunkSeq(false))), StepBound: 1, HasStepBound: true}
// A bounded Always whose inner is definitely false inside the window violates.
// The pre-fix window-close branch returned holds() unconditionally and lost
// this; the breach check runs before the window check and must stay there.
func TestBoundedAlways_ViolatedInnerInsideWindow(t *testing.T) {
inner := AlwaysFormula{Inner: Thunk(thunkSeq(false)), StepBound: 1, HasStepBound: true}
formula := Always(inner)
if verdict, _ := runAndFinalize(formula, 3); verdict != VerdictViolated {
t.Fatalf("Always(boundedAlways(Next(false),1)): got %v, want violated", verdict)
t.Fatalf("Always(boundedAlways(false,1)): got %v, want violated", verdict)
}
}
// A bounded Always whose inner is still a deferred Next obligation when the
// window closes discharges vacuously: nothing was breached inside the window.
// This is the exact dual of the bounded Eventually violating when its inner has
// not held by the time the window closes
// (TestEventuallyWithinSteps_NextInnerHitsBoundFirstStep), and the pair is what
// makes nnf's G/F dualisation semantics preserving. It costs the deferred check
// the window closed on: G<=n and F<=n both range over the observations at which
// their inner can definitely resolve, never past them.
func TestBoundedAlways_PendingInnerDischargesAtWindowClose(t *testing.T) {
inner := AlwaysFormula{Inner: Next(Thunk(thunkSeq(false))), StepBound: 1, HasStepBound: true}
if verdict, _ := runAndFinalize(Always(inner), 3); verdict != VerdictHolds {
t.Fatalf("Always(boundedAlways(Next(false),1)): got %v, want holds", verdict)
}
if verdict, _ := runAndFinalize(Always(nnf(Not(inner))), 3); verdict != VerdictViolated {
t.Fatalf("its negation: got %v, want violated", verdict)
}
}
+59 -1
View File
@@ -152,7 +152,7 @@ func TestViolationLatchIsMonotonic(t *testing.T) {
// holds/violated at run end, making sanderling lie about pass/fail.
func TestFinalize_KleeneConnectives(t *testing.T) {
pure := func(v bool) Formula { return PureFormula{Value: v} }
pendingThunk := ThunkFormula{Name: "t", Func: func() (bool, error) { return true, nil }}
pendingThunk := ThunkNamed("t", func() (bool, error) { return true, nil })
eventuallyViolated := EventuallyFormula{Inner: PureFormula{Value: false}}
nextPending := NextFormula{Inner: PureFormula{Value: true}}
alwaysHolds := AlwaysFormula{Inner: PureFormula{Value: true}}
@@ -224,3 +224,61 @@ func TestCollapse_NamedThunkLeakBoundsPendingSet(t *testing.T) {
t.Errorf("pending set leaked to %d obligations", len(evaluator.pending))
}
}
// TestCollapse_UnnamedPredicatesDoNotMerge is the lost-violation counterexample
// from the attribution analysis, run with unnamed thunks. Every unnamed thunk
// used to print "Thunk(...)", so the four Eventually residuals below shared one
// collapse key and the obligation spawned at step 2 was dropped: the run
// reported holds while a genuine violation was outstanding.
//
// root = And(Or(F a, c), Or(F b, d)), d = not c
// a never true, b true from step 6, c true except at steps 2 and 4
//
// At steps 1, 3 and 5 the left disjunct discharges via c and the right spawns
// F b; at steps 2 and 4 the right discharges via d and the left spawns F a.
// F a can never discharge, so the run violates with origin 2.
func TestCollapse_UnnamedPredicatesDoNotMerge(t *testing.T) {
step := 0
a := Thunk(func() (bool, error) { return false, nil })
b := Thunk(func() (bool, error) { return step >= 6, nil })
c := Thunk(func() (bool, error) { return step != 2 && step != 4, nil })
d := Thunk(func() (bool, error) { return step == 2 || step == 4, nil })
evaluator := NewEvaluator(And(Or(Eventually(a), c), Or(Eventually(b), d)))
for index := 1; index <= 10; index++ {
step = index
if got := evaluator.ObserveAtStep(time.Unix(int64(index), 0), index); got == VerdictViolated {
t.Fatalf("step %d violated early", index)
}
}
if got := evaluator.Finalize(); got != VerdictViolated {
t.Fatalf("Finalize = %v, want violated (F a can never discharge)", got)
}
witness := evaluator.Violation()
if witness == nil {
t.Fatal("Violation = nil, want non-nil")
}
if witness.Step != 2 {
t.Errorf("origin = %d, want 2", witness.Step)
}
}
// TestReduce_ErrorFormulaViolates: the verifier substitutes an ErrorFormula for
// the residual of a property whose predicate threw, and that residual is fed
// back into the evaluator on the next step. Reducing one used to panic.
func TestReduce_ErrorFormulaViolates(t *testing.T) {
evaluator := NewEvaluator(Always(ErrorFormula{Message: "boom"}))
if got := evaluator.Observe(); got != VerdictViolated {
t.Fatalf("got %v, want violated", got)
}
witness := evaluator.Violation()
if witness == nil {
t.Fatal("Violation = nil, want non-nil")
}
if witness.Reason != "boom" {
t.Errorf("Reason = %q, want %q", witness.Reason, "boom")
}
if !witness.IsError {
t.Error("IsError = false, want true")
}
}
+70 -34
View File
@@ -4,6 +4,7 @@ import (
"encoding/json"
"fmt"
"strings"
"sync/atomic"
"time"
)
@@ -13,9 +14,9 @@ type Formula interface {
describe() string
}
// PredicateLabel lets a ThunkFormula expose the identity of the closure it
// wraps. ThunkFormula satisfies it through its Name field; an empty name
// serializes without a name.
// PredicateLabel lets a ThunkFormula expose the caller's label for the closure
// it wraps. ThunkFormula satisfies it through the name passed to ThunkNamed;
// an unnamed thunk serializes without a name.
type PredicateLabel interface {
PredicateName() string
}
@@ -50,17 +51,26 @@ type PureFormula struct {
Value bool
}
// ThunkFormula wraps an opaque predicate closure. Func returns the predicate's
// boolean result and a non-nil error when the predicate threw; a thrown
// predicate is a witnessed violation distinct from a plain false. Name carries
// the predicate's identity so two distinct predicates produce distinct
// describe() keys and are never merged during obligation collapse.
// ThunkFormula wraps an opaque predicate closure. The closure returns the
// predicate's boolean result and a non-nil error when the predicate threw; a
// thrown predicate is a witnessed violation distinct from a plain false.
//
// Every thunk carries an identity assigned at construction, and the identity
// is part of its describe() key. Two thunks are therefore equal keys only when
// they are copies of the same constructed value, which is what lets obligation
// collapse merge residuals without ever merging distinct predicates. The
// fields are unexported so a thunk cannot be built without one.
type ThunkFormula struct {
Func func() (bool, error)
Name string
predicate func() (bool, error)
name string
identity uint64
}
func (t ThunkFormula) PredicateName() string { return t.Name }
func (t ThunkFormula) PredicateName() string { return t.name }
// thunkIdentities hands out the per-thunk identity. It only has to separate
// thunks within one process, so a counter is enough.
var thunkIdentities atomic.Uint64
// NowFormula marks its inner formula for evaluation at the current step only.
// Primarily used so that now(...).implies(...) parses unambiguously.
@@ -113,10 +123,16 @@ func Always(inner Formula) Formula { return AlwaysFormula{Inner: inner} }
func Pure(value bool) Formula { return PureFormula{Value: value} }
func Thunk(function func() (bool, error)) Formula { return ThunkFormula{Func: function} }
func Thunk(function func() (bool, error)) Formula {
return ThunkFormula{predicate: function, identity: thunkIdentities.Add(1)}
}
func ThunkNamed(name string, function func() (bool, error)) Formula {
return ThunkFormula{Func: function, Name: name}
return ThunkFormula{
predicate: function,
name: name,
identity: thunkIdentities.Add(1),
}
}
func Now(inner Formula) Formula { return NowFormula{Inner: inner} }
@@ -172,10 +188,7 @@ func (a AlwaysFormula) describe() string {
}
func (p PureFormula) describe() string { return fmt.Sprintf("Pure(%t)", p.Value) }
func (t ThunkFormula) describe() string {
if t.Name != "" {
return "Thunk(" + t.Name + ")"
}
return "Thunk(...)"
return fmt.Sprintf("Thunk(%s#%d)", t.name, t.identity)
}
func (n NowFormula) describe() string { return "Now(" + n.Inner.describe() + ")" }
func (n NextFormula) describe() string { return "Next(" + n.Inner.describe() + ")" }
@@ -206,10 +219,43 @@ func (n NotFormula) describe() string { return "Not(" + n.Inner.describe() + ")"
func Describe(formula Formula) string { return formula.describe() }
// withinNode mirrors the optional `within` clause attached to bounded
// Eventually nodes in the JSON AST.
// Always/Eventually nodes in the JSON AST.
type withinNode struct {
Amount int64 `json:"amount"`
Unit string `json:"unit"`
// Deadline is the absolute instant the window closes, in unix
// milliseconds, present once the evaluator resolved a relative duration
// against an observation. Two obligations spawned at different steps from
// the same duration differ only here, so without it they serialize
// identically and the trace erases the distinction the evaluator makes.
Deadline int64 `json:"deadline,omitempty"`
}
// withinFor renders the bound clause of a bounded Always or Eventually. The
// authored window (steps or duration) stays in amount/unit so readers keep
// seeing what the spec asked for; the resolved deadline rides alongside.
func withinFor(
hasStepBound bool,
stepBound int,
duration time.Duration,
hasDeadline bool,
deadline time.Time,
) *withinNode {
var node *withinNode
switch {
case hasStepBound:
node = &withinNode{Amount: int64(stepBound), Unit: "steps"}
case duration > 0:
node = &withinNode{Amount: duration.Milliseconds(), Unit: "milliseconds"}
case hasDeadline:
return &withinNode{Amount: deadline.UnixMilli(), Unit: "deadline"}
default:
return nil
}
if hasDeadline {
node.Deadline = deadline.UnixMilli()
}
return node
}
func (a AlwaysFormula) MarshalJSON() ([]byte, error) {
@@ -218,14 +264,9 @@ func (a AlwaysFormula) MarshalJSON() ([]byte, error) {
Arg Formula `json:"arg"`
Within *withinNode `json:"within,omitempty"`
}{Op: "always", Arg: a.Inner}
switch {
case a.HasStepBound:
payload.Within = &withinNode{Amount: int64(a.StepBound), Unit: "steps"}
case a.Duration > 0:
payload.Within = &withinNode{Amount: a.Duration.Milliseconds(), Unit: "milliseconds"}
case a.HasDeadline:
payload.Within = &withinNode{Amount: a.Deadline.UnixMilli(), Unit: "deadline"}
}
payload.Within = withinFor(
a.HasStepBound, a.StepBound, a.Duration, a.HasDeadline, a.Deadline,
)
return json.Marshal(payload)
}
@@ -256,14 +297,9 @@ func (e EventuallyFormula) MarshalJSON() ([]byte, error) {
Arg Formula `json:"arg"`
Within *withinNode `json:"within,omitempty"`
}{Op: "eventually", Arg: e.Inner}
switch {
case e.HasStepBound:
payload.Within = &withinNode{Amount: int64(e.StepBound), Unit: "steps"}
case e.Duration > 0:
payload.Within = &withinNode{Amount: e.Duration.Milliseconds(), Unit: "milliseconds"}
case e.HasDeadline:
payload.Within = &withinNode{Amount: e.Deadline.UnixMilli(), Unit: "deadline"}
}
payload.Within = withinFor(
e.HasStepBound, e.StepBound, e.Duration, e.HasDeadline, e.Deadline,
)
return json.Marshal(payload)
}
+29
View File
@@ -288,3 +288,32 @@ func TestResidual_HoldsViolatedPending(t *testing.T) {
t.Errorf("pending residual:\n got: %s\nwant: %s", body, want)
}
}
// TestMarshalJSON_ResolvedDeadlineDistinguishesObligations: obligations spawned
// at different steps from one duration-bounded eventually differ only in the
// deadline the evaluator resolved for them, and collapse keeps them apart on
// exactly that. The serialized AST has to keep them apart too, or the trace
// shows N copies of one node where the evaluator has N different obligations.
func TestMarshalJSON_ResolvedDeadlineDistinguishesObligations(t *testing.T) {
base := time.UnixMilli(1700000000000)
first := EventuallyFormula{
Inner: PureFormula{Value: false},
Duration: 300 * time.Second,
Deadline: base.Add(300 * time.Second),
HasDeadline: true,
}
second := first
second.Deadline = base.Add(301 * time.Second)
firstBody, _ := json.Marshal(first)
secondBody, _ := json.Marshal(second)
if string(firstBody) == string(secondBody) {
t.Errorf("obligations with different deadlines serialize identically: %s", firstBody)
}
if !strings.Contains(string(firstBody), `"unit":"milliseconds"`) {
t.Errorf("authored window lost: %s", firstBody)
}
if !strings.Contains(string(firstBody), `"deadline":1700000300000`) {
t.Errorf("resolved deadline missing: %s", firstBody)
}
}
+106
View File
@@ -0,0 +1,106 @@
package ltl
import (
"testing"
"time"
)
// TestRoot_BoundedEventuallyStaysOneObligation locks the cost of the one-shot
// root. A duration-bounded top-level eventually is one reachability goal, so
// the pending set holds one obligation for the whole run. Re-instantiating it
// every step monitored G F<=n(p) instead and left one live obligation per step
// behind: a 553-step run carried 553 of them and re-ran the predicate once per
// obligation per step.
func TestRoot_BoundedEventuallyStaysOneObligation(t *testing.T) {
const steps = 600
calls := 0
formula := EventuallyWithin(ThunkNamed("p", func() (bool, error) {
calls++
return false, nil
}), 300*time.Second)
evaluator := NewEvaluator(formula)
base := time.Unix(0, 0)
for index := range steps {
if got := evaluator.ObserveAtStep(base.Add(time.Duration(index)*110*time.Millisecond), index+1); got != VerdictPending {
t.Fatalf("step %d: got %v, want pending", index+1, got)
}
if len(evaluator.pending) != 1 {
t.Fatalf("step %d: %d pending obligations, want 1", index+1, len(evaluator.pending))
}
}
if calls != steps {
t.Errorf("predicate ran %d times over %d steps, want one call per step", calls, steps)
}
}
// TestRoot_TopLevelEventuallyIsSatisfiedOnce pins the semantics behind that
// bound: a top-level eventually is discharged for good the first time it is
// satisfied. Under the old implicit-always reading it was re-armed at every
// step, so a property that had already been reached could still violate later.
func TestRoot_TopLevelEventuallyIsSatisfiedOnce(t *testing.T) {
reached := false
evaluator := NewEvaluator(EventuallyWithinSteps(ThunkNamed("p", func() (bool, error) {
return reached, nil
}), 2))
if got := evaluator.ObserveAtStep(time.Unix(0, 0), 1); got != VerdictPending {
t.Fatalf("step 1: got %v, want pending", got)
}
reached = true
if got := evaluator.ObserveAtStep(time.Unix(1, 0), 2); got != VerdictHolds {
t.Fatalf("step 2: got %v, want holds", got)
}
reached = false
for index := 3; index <= 6; index++ {
if got := evaluator.ObserveAtStep(time.Unix(int64(index), 0), index); got != VerdictHolds {
t.Fatalf("step %d: got %v, want holds (the goal was already reached)", index, got)
}
}
if got := evaluator.Finalize(); got != VerdictHolds {
t.Errorf("Finalize = %v, want holds", got)
}
}
// TestRoot_BoundedAlwaysKeepsItsBound: a bounded root Always is a single
// window, not a recurrence. Stripping it and re-instantiating its inner every
// step dropped the bound, so G<=1(p) behaved as G(p) and a false p after the
// window closed still violated.
func TestRoot_BoundedAlwaysKeepsItsBound(t *testing.T) {
values := []bool{true, true, false}
step := 0
formula := AlwaysFormula{
Inner: ThunkNamed("p", func() (bool, error) { return values[step], nil }),
StepBound: 1,
HasStepBound: true,
}
evaluator := NewEvaluator(formula)
for index := range values {
step = index
if got := evaluator.ObserveAtStep(time.Unix(int64(index), 0), index+1); got == VerdictViolated {
t.Fatalf("step %d: violated outside the 1-step window", index+1)
}
}
}
// TestRoot_UnboundedAlwaysStillReInstantiates: the one-shot rule must not touch
// the recurrence root every spec property is built on. Each step gets its own
// instance of the inner, which is what gives a deferred failure the origin step
// that armed it.
func TestRoot_UnboundedAlwaysStillReInstantiates(t *testing.T) {
values := []bool{true, true, false}
step := 0
evaluator := NewEvaluator(Always(ThunkNamed("p", func() (bool, error) {
return values[step], nil
})))
for index := range 2 {
step = index
if got := evaluator.ObserveAtStep(time.Unix(int64(index), 0), index+1); got != VerdictHolds {
t.Fatalf("step %d: got %v, want holds", index+1, got)
}
}
step = 2
if got := evaluator.ObserveAtStep(time.Unix(2, 0), 3); got != VerdictViolated {
t.Errorf("step 3: got %v, want violated", got)
}
}
+8 -1
View File
@@ -302,7 +302,14 @@ func describeAction(action verifier.Action) string {
case verifier.ActionKindInputText:
return fmt.Sprintf("InputText %s = %q", actionTarget(action), action.Text)
case verifier.ActionKindScroll:
return fmt.Sprintf("Scroll %s %s", action.Direction, action.On)
// A builtin gesture carries endpoints rather than a selector, so name the
// container by where the drag starts; that is what tells two scrollable
// regions apart in the recent-action memory.
target := action.On
if target == "" {
target = fmt.Sprintf("(%d,%d)", action.FromX, action.FromY)
}
return fmt.Sprintf("Scroll %s %s", action.Direction, target)
case verifier.ActionKindSwipe:
// Coordinates make a repeated identical swipe recognizable in the
// prompt's recent-action memory.
+14
View File
@@ -62,6 +62,20 @@ func TestActionForCandidatePassesNonTypingThrough(t *testing.T) {
}
}
func TestDescribeActionNamesGesturesByOrigin(t *testing.T) {
builtin := verifier.Action{
Kind: verifier.ActionKindScroll, Direction: "down",
FromX: 200, FromY: 500, ToX: 200, ToY: 340,
}
if got := describeAction(builtin); got != "Scroll down (200,500)" {
t.Errorf("builtin gesture described as %q, want the drag origin", got)
}
authored := verifier.Action{Kind: verifier.ActionKindScroll, Direction: "up", On: "id:List"}
if got := describeAction(authored); got != "Scroll up id:List" {
t.Errorf("authored scroll described as %q, want its selector", got)
}
}
func TestActionForCandidateUsesModelText(t *testing.T) {
source := &llmSource{}
candidate := verifier.ActionCandidate{
+13 -6
View File
@@ -1006,7 +1006,9 @@ func violationRecords(properties []string, witnesses map[string]trace.Witness, d
// collectWitnesses gathers the violation witness for each newly-violated
// property, logs its cause, and returns them keyed by property name for the
// trace. Properties without a captured witness are skipped.
// trace. Properties without a captured witness are skipped. stepIndex is the
// trace line the witness lands on, and stands in as the detection step for a
// verifier that observed no labeled step (a run-end finalize).
func collectWitnesses(verifierInstance *verifier.Verifier, properties []string, logger *slog.Logger, stepIndex int) map[string]trace.Witness {
if len(properties) == 0 {
return nil
@@ -1017,14 +1019,19 @@ func collectWitnesses(verifierInstance *verifier.Verifier, properties []string,
if witness == nil {
continue
}
detectedStep := witness.DetectedStep
if detectedStep == 0 {
detectedStep = stepIndex
}
logger.Warn("property violated",
"step", witness.Step, "detected_step", stepIndex,
"step", witness.Step, "detected_step", detectedStep,
"property", name, "reason", witness.Reason, "error", witness.IsError)
witnesses[name] = trace.Witness{
Reason: witness.Reason,
IsError: witness.IsError,
Step: witness.Step,
Extractors: witness.Extractors,
Reason: witness.Reason,
IsError: witness.IsError,
Step: witness.Step,
DetectedStep: detectedStep,
Extractors: witness.Extractors,
}
}
if len(witnesses) == 0 {
+5
View File
@@ -363,6 +363,11 @@ globalThis.actions = actions(() => []);
if witness.Step != 2 {
t.Errorf("witness step: got %d, want 2 (causing step)", witness.Step)
}
// The two indices the witness spans are recorded separately: the
// extractor snapshot it carries is step 3's state, not step 2's.
if witness.DetectedStep != 3 {
t.Errorf("witness detected step: got %d, want 3", witness.DetectedStep)
}
}
if err := scanner.Err(); err != nil {
t.Fatalf("scan trace: %v", err)
+118
View File
@@ -0,0 +1,118 @@
package runner
import (
"bufio"
"context"
"encoding/json"
"os"
"path/filepath"
"testing"
"time"
mockdriver "github.com/priyanshujain/sanderling/internal/driver/mock"
"github.com/priyanshujain/sanderling/internal/trace"
)
// engineDisagreementSpec makes the two runtimes disagree on purpose: the
// extractor body returns "goja" when it runs in-process, and the web driver
// below reports "v8" for the same extractor. The property is true of one value
// and false of the other, so the verdict names the engine the verdict used.
const engineDisagreementSpec = `
import { actions, always, extract } from "@sanderling/spec";
const engine = extract("engine", () => "goja");
globalThis.properties = {
ranInGoja: always(() => engine.current === "goja"),
};
globalThis.actions = actions(() => []);
`
// webMockDriver presents the mock device driver as a web target so the runner
// takes the V8 path, where extractor values come from the page rather than
// from goja.
type webMockDriver struct {
*mockdriver.Driver
overrides map[int]json.RawMessage
}
func (d *webMockDriver) InstallBundle(context.Context, []byte) error { return nil }
func (d *webMockDriver) EvaluateExtractors(context.Context) (map[int]json.RawMessage, error) {
return d.overrides, nil
}
func (d *webMockDriver) NextActionFromV8(context.Context) (json.RawMessage, error) {
return nil, nil
}
// TestRunner_TraceRecordsTheValueTheVerdictUsed fails if the trace and the
// verdict disagree about an extractor. A witness is only an explanation of a
// violation if it holds the state the violated property was evaluated against.
func TestRunner_TraceRecordsTheValueTheVerdictUsed(t *testing.T) {
state := newHarnessWithSpec(t, engineDisagreementSpec)
const pageValue = `"v8"`
web := &webMockDriver{
Driver: state.mock,
overrides: map[int]json.RawMessage{0: json.RawMessage(pageValue)},
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
summary, err := Run(ctx, Options{
Duration: 100 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
Driver: web,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
if !containsProperty(summary.Violations, "ranInGoja") {
t.Fatalf("ranInGoja did not violate, so the page value never reached the verdict: %v",
summary.Violations)
}
file, err := os.Open(filepath.Join(state.writer.Directory(), "trace.jsonl"))
if err != nil {
t.Fatal(err)
}
defer file.Close()
type traceLine struct {
Step int `json:"step"`
ExtractorChanges map[string]trace.ExtractorChange `json:"extractor_changes"`
Witnesses map[string]trace.Witness `json:"witnesses"`
}
changes, witnesses := 0, 0
scanner := bufio.NewScanner(file)
scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024)
for scanner.Scan() {
var line traceLine
if err := json.Unmarshal(scanner.Bytes(), &line); err != nil {
t.Fatalf("trace line decode: %v", err)
}
if change, ok := line.ExtractorChanges["engine"]; ok {
changes++
if got := string(change.Curr); got != pageValue {
t.Errorf("step %d: trace records engine=%s, verdict used %s",
line.Step, got, pageValue)
}
}
for name, witness := range line.Witnesses {
witnesses++
if got := string(witness.Extractors["engine"]); got != pageValue {
t.Errorf("step %d: %s witness records engine=%s, verdict used %s",
line.Step, name, got, pageValue)
}
}
}
if err := scanner.Err(); err != nil {
t.Fatalf("scan trace: %v", err)
}
if changes == 0 {
t.Error("no extractor change reached the trace; nothing was compared")
}
if witnesses == 0 {
t.Error("no witness reached the trace; nothing was compared")
}
}
+9 -7
View File
@@ -40,17 +40,19 @@ type Step struct {
Witnesses map[string]Witness `json:"witnesses,omitempty"`
}
// Witness is the trace-side record of a property violation: why it fired and a
// snapshot of every extractor's value at the violating step.
// Witness is the trace-side record of a property violation: why it fired, the
// two steps a deferred obligation spans, and the extractor values behind it.
type Witness struct {
Reason string `json:"reason,omitempty"`
IsError bool `json:"is_error,omitempty"`
// Step is the step the failed obligation originated at: the step that
// caused the violation. For a deferred obligation (a next, an eventually)
// this is earlier than the step whose record carries the witness, which is
// where the failure was detected.
Step int `json:"step,omitempty"`
Extractors map[string]json.RawMessage `json:"extractors,omitempty"`
// armed it. For a deferred obligation (a next, an eventually) this is
// earlier than the step at which the failure was detected.
Step int `json:"step,omitempty"`
// DetectedStep is the observation whose evaluation produced the violation.
// Extractors is that step's state, not Step's.
DetectedStep int `json:"detected_step,omitempty"`
Extractors map[string]json.RawMessage `json:"extractors,omitempty"`
}
// ExtractorChange records the prev/curr JSON values of an extractor whose
+19 -14
View File
@@ -91,7 +91,7 @@ func (v *Verifier) installRuntimeBindings() error {
// installHost exposes globalThis.__sanderlingHost__ for the goja runtime entry.
// The shared picker (pick.ts) draws against it: platform() drives the verb
// matrix and press-key pool; seedHi/seedLo construct its Pcg; queryCandidates
// matrix and press-key pool; seedHi/seedLo construct its Pcg; queryTargets
// enumerates targets over the hierarchy tree; reportUnsupported records the
// verb for the run report.
func (v *Verifier) installHost() error {
@@ -111,7 +111,7 @@ func (v *Verifier) installHost() error {
}); err != nil {
return err
}
if err := host.Set("queryCandidates", v.bindQueryCandidates); err != nil {
if err := host.Set("queryTargets", v.bindQueryTargets); err != nil {
return err
}
if err := host.Set("reportUnsupported", func(call goja.FunctionCall) goja.Value {
@@ -133,20 +133,25 @@ func (v *Verifier) recordUnsupported(verb string) {
v.unsupported = append(v.unsupported, verb)
}
// bindQueryCandidates returns the host-enumerated targets for a verb as an
// array of {x, y, selector, width, height}, in tree order.
func (v *Verifier) bindQueryCandidates(call goja.FunctionCall) goja.Value {
verb := call.Argument(0).String()
candidates := v.candidatesForVerb(verb)
// bindQueryTargets returns every host-enumerated target as an array of
// {x, y, selector, width, height, clickable, enabled, editable, scrollable}, in
// tree order. It takes no verb: the shared rule in targets.ts decides which of
// these a verb may act on.
func (v *Verifier) bindQueryTargets(goja.FunctionCall) goja.Value {
targets := v.targets()
array := v.runtime.NewArray()
for index, candidate := range candidates {
for index, target := range targets {
item := v.runtime.NewObject()
_ = item.Set("x", candidate.x)
_ = item.Set("y", candidate.y)
_ = item.Set("width", candidate.width)
_ = item.Set("height", candidate.height)
if candidate.selector != "" {
_ = item.Set("selector", candidate.selector)
_ = item.Set("x", target.x)
_ = item.Set("y", target.y)
_ = item.Set("width", target.width)
_ = item.Set("height", target.height)
_ = item.Set("clickable", target.clickable)
_ = item.Set("enabled", target.enabled)
_ = item.Set("editable", target.editable)
_ = item.Set("scrollable", target.scrollable)
if target.selector != "" {
_ = item.Set("selector", target.selector)
}
_ = array.Set(fmt.Sprintf("%d", index), item)
}
+52
View File
@@ -4,6 +4,7 @@ import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/priyanshujain/sanderling/internal/ltl"
)
@@ -116,3 +117,54 @@ func TestWithin_InvalidUnitPanics(t *testing.T) {
}
}
}
// TestTopLevelEventually_IsOneReachabilityObligation drives the folio-web shape
// `eventually(p).within(300, "seconds")` as a top-level property over a run of
// the same length and cadence as the 553-step run that exposed this: 60 seconds
// of steps, a predicate that never fires, a window far longer than the run.
//
// The property is one reachability goal, so the run leaves one obligation and
// one residual node behind. Wrapping the root in Always made it "within 300
// seconds of EVERY step" instead, which spawned an obligation per step (each
// with its own resolved deadline, so none of them collapsed), re-ran the
// predicate once per obligation per step, and serialized a 75 KB residual.
func TestTopLevelEventually_IsOneReachabilityObligation(t *testing.T) {
const source = `
globalThis.seen = __sanderling__.extract(state => state.snapshots["seen"] ?? false, "seen");
globalThis.properties = {
reachable: __sanderling__.eventually(() => seen.current).within(300, 'seconds'),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, source)
base := time.Unix(1780000000, 0)
const steps = 553
for index := range steps {
if err := verifier.PushSnapshot(SnapshotInput{
Snapshots: Snapshots{"seen": json.RawMessage(`false`)},
StepIndex: index + 1,
StepTime: base.Add(time.Duration(index) * 108 * time.Millisecond),
RunStart: base,
}); err != nil {
t.Fatal(err)
}
if got := verifier.EvaluateProperties()["reachable"]; got != ltl.VerdictPending {
t.Fatalf("step %d: got %v, want pending", index+1, got)
}
}
residual, err := json.Marshal(verifier.Residuals()["reachable"])
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(residual), `"op":"and"`) {
t.Errorf("residual accumulated obligations (%d bytes): %s", len(residual), residual)
}
if !strings.Contains(string(residual), `"op":"eventually"`) {
t.Errorf("residual lost the eventually: %s", residual)
}
if !strings.Contains(string(residual), `"unit":"milliseconds"`) {
t.Errorf("residual lost the bound: %s", residual)
}
}
+114
View File
@@ -0,0 +1,114 @@
package verifier
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"slices"
"testing"
)
// hostParityScreen is the canonical screen both hosts are driven over: one row
// per fact combination that any verb distinguishes. Each host builds it in its
// own model (this file as a hierarchy tree, pkg/spec/test/host-parity.test.ts as
// a DOM), enumerates every verb, and asserts the SAME committed golden.
var hostParityScreen = []struct {
name string
clickable bool
enabled bool
editable bool
scrollable bool
positiveBounds bool
}{
{name: "root", enabled: true, scrollable: true, positiveBounds: true},
{name: "save", clickable: true, enabled: true, positiveBounds: true},
{name: "cancel", clickable: true, positiveBounds: true},
{name: "amount", enabled: true, editable: true, positiveBounds: true},
{name: "list", enabled: true, scrollable: true, positiveBounds: true},
{name: "row", enabled: true, positiveBounds: true},
{name: "collapsed", clickable: true, enabled: true},
}
// hostParityTreeJSON is hostParityScreen as the native host sees it. Tree order
// is pre-order, so it matches hostParityScreen index for index.
const hostParityTreeJSON = `{
"attributes": {"resource-id": "root", "scrollable": "true", "bounds": "[0,0,400,800]"},
"enabled": true,
"children": [
{"attributes": {"resource-id": "save", "bounds": "[0,0,200,60]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "cancel", "bounds": "[200,0,400,60]"}, "clickable": true, "enabled": false, "children": []},
{"attributes": {"resource-id": "amount", "bounds": "[0,100,400,160]"}, "editable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "list", "scrollable": "true", "bounds": "[0,200,400,600]"}, "enabled": true, "children": []},
{"attributes": {"resource-id": "row", "bounds": "[0,600,400,680]"}, "enabled": true, "children": []},
{"attributes": {"resource-id": "collapsed", "bounds": "[0,0,0,0]"}, "clickable": true, "enabled": true, "children": []}
]
}`
// TestHostsAgreeOnTargetEligibility is the guard on the claim that one
// specification induces one action space on every platform. The native host and
// the web host used to route verbs themselves and had drifted: web sent
// `swipes` to scrollable containers only, so a swipe on a list row was
// reachable on Android and unreachable on web for the same spec.
//
// Per-verb eligibility now has ONE definition (pkg/spec/src/targets.ts); a host
// reports facts and never filters. This test is what notices if a second
// definition grows back on either side: both hosts enumerate the same canonical
// screen and must name the same targets, verb for verb, in the same order.
// pkg/spec/test/host-parity.test.ts asserts the SAME golden from the web host,
// so a match on both sides proves the two hosts agree without either invoking
// the other.
func TestHostsAgreeOnTargetEligibility(t *testing.T) {
golden := loadHostParityGolden(t)
for _, verb := range policyVerbs {
t.Run(verb, func(t *testing.T) {
want, ok := golden[verb]
if !ok {
t.Fatalf("golden has no entry for %s", verb)
}
verifier := newVerifier(t, WithSeed(0x5eed))
loadActionSpec(t, verifier, fmt.Sprintf(
"import { %s } from \"@sanderling/spec\";\nglobalThis.actions = %s;", verb, verb))
pushTree(t, verifier, hostParityTreeJSON)
entries, err := verifier.enumerateBuiltin(verb)
if err != nil {
t.Fatalf("enumerate %s: %v", verb, err)
}
if len(entries) == 0 {
t.Fatalf("%s enumerated nothing at all", verb)
}
got := []string{}
for _, entry := range entries {
if entry.targetIndex < 0 {
continue
}
if entry.targetIndex >= len(hostParityScreen) {
t.Fatalf("%s produced target index %d, off the %d-row screen",
verb, entry.targetIndex, len(hostParityScreen))
}
got = append(got, hostParityScreen[entry.targetIndex].name)
}
if !slices.Equal(got, want) {
t.Errorf("native host targets for %s\n got=%v\nwant=%v", verb, got, want)
}
})
}
}
func loadHostParityGolden(t *testing.T) map[string][]string {
t.Helper()
path, err := filepath.Abs("../../pkg/spec/test/fixtures/host-parity-golden.json")
if err != nil {
t.Fatal(err)
}
body, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read golden: %v", err)
}
golden := map[string][]string{}
if err := json.Unmarshal(body, &golden); err != nil {
t.Fatalf("decode golden: %v", err)
}
return golden
}
+100 -121
View File
@@ -1,6 +1,7 @@
package verifier
import (
"encoding/json"
"errors"
"fmt"
"math"
@@ -130,54 +131,21 @@ type ActionCandidate struct {
prob float64
}
// verbActionKind maps a picker verb to the action kind it dispatches.
func verbActionKind(verb string) ActionKind {
switch verb {
case "taps":
return ActionKindTap
case "doubleTaps":
return ActionKindDoubleTap
case "longPresses":
return ActionKindLongPress
case "typing":
return ActionKindInputText
case "scrolls":
return ActionKindScroll
case "swipes":
return ActionKindSwipe
default:
return ""
}
}
// maxLabelRunes caps a visible-text label so joined descendant text stays short
// enough to render on one numbered line.
const maxLabelRunes = 40
// gestureDirections are the directional scrolls emitted per scrollable
// container. Vertical only: most mobile lists scroll up/down, and keeping the
// set tiny is the whole point of folding per-element swipes away.
var gestureDirections = []string{"down", "up"}
// Candidates enumerates every action the spec's weighted actionsRoot yields at
// the current step, each tagged with a plainly-worded description and its
// effective weight, for the LLM generator to pick one number from. It walks the
// SAME tree the seeded picker draws: weighted branches recurse (accumulating the
// selection probability), authored actions()/whenRoute leaves are called once
// for their concrete actions, and builtin verbs enumerate per applicable
// element. Disabled controls are dropped, per-element gestures fold into a few
// directional scrolls over scrollable containers, and identical descriptions
// dedup (summing weight).
// for their concrete actions, and builtin verbs come straight from the picker's
// own enumeration. Identical descriptions dedup, summing weight.
func (v *Verifier) Candidates() []ActionCandidate {
if v.lastTree == nil {
return nil
}
// A cross-fade frame's layout is mid-animation, often in a collapsed
// coordinate space, so acting on it taps garbage (e.g. the soft keyboard).
// Skip it so the LLM re-observes a settled frame next step.
if v.lastTree.Transitional() {
return nil
}
root := v.runtime.GlobalObject().Get("actions")
if root == nil || goja.IsUndefined(root) || goja.IsNull(root) {
return nil
@@ -333,6 +301,18 @@ func (v *Verifier) candidateFromDescriptor(value goja.Value, nodeIndex map[*hier
Direction: direction,
Action: Action{Kind: kind, On: target.selector, Direction: direction},
}, true
case ActionKindSwipe:
from := v.resolveTarget(object.Get("from"), nodeIndex)
to := v.resolveTarget(object.Get("to"), nodeIndex)
return ActionCandidate{
Kind: kind,
Action: Action{
Kind: kind,
FromX: from.x, FromY: from.y,
ToX: to.x, ToY: to.y,
DurationMillis: intField(object, "durationMillis"),
},
}, true
case ActionKindPressKey:
return ActionCandidate{
Kind: kind,
@@ -410,103 +390,80 @@ func (v *Verifier) findBySelector(selector string) *hierarchy.Element {
return v.lastTree.Find(selector)
}
// collectBuiltin enumerates a builtin verb over the current tree: tap-family and
// typing emit one candidate per applicable element; scrolls/swipes fold into
// directional gestures over scrollable containers.
// collectBuiltin turns the picker's own enumeration of a builtin verb into
// candidates. The list comes from the bundle's __sanderlingEnumerateBuiltin__
// (pick.ts builtinCandidates), which is what the seeded policy draws from, so
// the two policies select over one action space and cannot drift apart. Each
// entry's action arrives on the wire contract DecodeAction already reads, so a
// chosen candidate executes the action the seeded draw would have executed.
func (v *Verifier) collectBuiltin(verb string, prob float64, weighted bool, nodeIndex map[*hierarchy.Element]*hierarchy.Node, out *[]ActionCandidate) {
switch verb {
case "taps", "doubleTaps", "longPresses":
kind := verbActionKind(verb)
for _, element := range v.elementsForVerb(verb) {
x, y := element.Bounds.Center()
*out = append(*out, ActionCandidate{
Kind: kind,
Label: visibleLabel(element, nodeIndex),
Action: Action{Kind: kind, On: selectorForElement(v.lastTree, element), X: x, Y: y},
prob: prob,
Weighted: weighted,
})
entries, err := v.enumerateBuiltin(verb)
if err != nil {
return
}
targets := v.targets()
for _, entry := range entries {
candidate := ActionCandidate{
Kind: entry.action.Kind,
Direction: entry.action.Direction,
// Builtin typing enumerates the field, not the value: the seeded
// policy draws its text from the corpus and the model writes its own.
LLMText: entry.action.Kind == ActionKindInputText,
Action: entry.action,
prob: prob,
Weighted: weighted,
}
case "typing":
for _, element := range v.elementsForVerb(verb) {
x, y := element.Bounds.Center()
*out = append(*out, ActionCandidate{
Kind: ActionKindInputText,
Label: visibleLabel(element, nodeIndex),
InputType: inputTypeHint(element),
LLMText: true,
Action: Action{Kind: ActionKindInputText, On: selectorForElement(v.lastTree, element), X: x, Y: y},
prob: prob,
Weighted: weighted,
})
if entry.targetIndex >= 0 && entry.targetIndex < len(targets) {
element := targets[entry.targetIndex].element
candidate.Label = visibleLabel(element, nodeIndex)
candidate.InputType = inputTypeHint(element)
}
case "scrolls", "swipes":
v.collectGestures(prob, weighted, out)
*out = append(*out, candidate)
}
}
// collectGestures emits directional scrolls scoped to each scrollable container,
// never per element and never element-labeled. Folding both scrolls and swipes
// here is what removes the flood of mislabeled `Swipe "X"` gestures.
func (v *Verifier) collectGestures(prob float64, weighted bool, out *[]ActionCandidate) {
scope := v.scopedElements()
for _, element := range v.lastTree.Elements {
if !scope[element] {
continue
}
if element.Attributes["scrollable"] != "true" {
continue
}
if element.Bounds.Width() <= 0 || element.Bounds.Height() <= 0 {
continue
}
selector := selectorForElement(v.lastTree, element)
for _, direction := range gestureDirections {
action := Action{Kind: ActionKindScroll, On: selector, Direction: direction}
if selector == "" {
action.FromX, action.FromY, action.ToX, action.ToY = scrollGeometry(element.Bounds, direction)
}
*out = append(*out, ActionCandidate{
Kind: ActionKindScroll,
Direction: direction,
Action: action,
prob: prob,
Weighted: weighted,
})
}
}
// builtinCandidate is one entry of the shared builtin enumeration: the concrete
// action, plus the index of the host candidate it targets (-1 when the verb has
// no target, as for a key press or a wait).
type builtinCandidate struct {
action Action
targetIndex int
}
// scrollGeometry lowers a directional scroll to swipe endpoints over the given
// container bounds, matching the runner's own derivation, used only when the
// container has no resolving selector.
func scrollGeometry(bounds hierarchy.Bounds, direction string) (fromX, fromY, toX, toY int) {
cx, cy := bounds.Center()
fromX, fromY, toX, toY = cx, cy, cx, cy
switch direction {
case "down":
toY = cy - 4*bounds.Height()/10
case "up":
toY = cy + 4*bounds.Height()/10
case "left":
toX = cx + 4*bounds.Width()/10
case "right":
toX = cx - 4*bounds.Width()/10
// enumerateBuiltin invokes the bundle's shared enumeration for one verb. A spec
// loaded without the runtime entry (a raw-JS unit fixture) has no enumerator, so
// the verb contributes nothing rather than falling back to a second enumeration.
func (v *Verifier) enumerateBuiltin(verb string) ([]builtinCandidate, error) {
if v.enumerateBuiltinFn == nil {
return nil, errors.New("verifier: builtin enumeration not available")
}
return fromX, fromY, max(0, toX), max(0, toY)
}
// elementsForVerb returns the in-scope elements a builtin verb applies to, in
// tree order (the seeded picker's enumeration order), reusing verbAccepts.
func (v *Verifier) elementsForVerb(verb string) []*hierarchy.Element {
scope := v.scopedElements()
var elements []*hierarchy.Element
for _, element := range v.lastTree.Elements {
if scope[element] && verbAccepts(verb, element) {
elements = append(elements, element)
value, err := v.enumerateBuiltinFn(goja.Undefined(), v.runtime.ToValue(verb))
if err != nil {
return nil, fmt.Errorf("enumerate %s: %w", verb, err)
}
if value == nil || goja.IsUndefined(value) || goja.IsNull(value) {
return nil, nil
}
raw, err := json.Marshal(value.Export())
if err != nil {
return nil, fmt.Errorf("marshal %s enumeration: %w", verb, err)
}
var wire []struct {
Action json.RawMessage `json:"action"`
TargetIndex int `json:"targetIndex"`
}
if err := json.Unmarshal(raw, &wire); err != nil {
return nil, fmt.Errorf("decode %s enumeration: %w", verb, err)
}
entries := make([]builtinCandidate, 0, len(wire))
for _, item := range wire {
action, err := DecodeAction(item.Action)
if err != nil {
continue
}
entries = append(entries, builtinCandidate{action: action, targetIndex: item.TargetIndex})
}
return elements
return entries, nil
}
// finalizeCandidates renders each candidate's description, dedups identical
@@ -555,6 +512,18 @@ func describeCandidate(candidate ActionCandidate) string {
return fmt.Sprintf("Type %q into %q", candidate.Action.Text, candidate.Label)
case ActionKindScroll:
return "Scroll " + candidate.Direction
case ActionKindSwipe:
// A swipe carries endpoints and no selector, so the coordinates are what
// keep two swipes distinct. The label is prepended when the origin
// element has one, because "swipe that row" is the interaction a model
// reaches for and a bare pair of points does not say which row.
where := fmt.Sprintf("from (%d,%d) to (%d,%d)",
candidate.Action.FromX, candidate.Action.FromY,
candidate.Action.ToX, candidate.Action.ToY)
if candidate.Label == "" {
return "Swipe " + where
}
return fmt.Sprintf("Swipe %q %s", candidate.Label, where)
case ActionKindPressKey:
return "Press " + candidate.Action.Key
case ActionKindWait:
@@ -682,3 +651,13 @@ func stringField(object *goja.Object, key string) string {
}
return value.String()
}
// intField reads a numeric property off a goja object, returning 0 when absent,
// null, or undefined.
func intField(object *goja.Object, key string) int {
value := object.Get(key)
if value == nil || goja.IsUndefined(value) || goja.IsNull(value) {
return 0
}
return int(value.ToInteger())
}
+51 -16
View File
@@ -10,7 +10,7 @@ import (
// enumTreeJSON exercises every labeling path: a clickable wrapper whose own text
// is empty but whose child Text reads "Add credit" (descendant borrowing), an
// editable field labeled by its hint, a text-labeled button, a DISABLED button,
// and a scrollable list (the only valid gesture origin).
// and a scrollable list (the only valid scroll origin).
const enumTreeJSON = `{
"attributes": {"bounds": "[0,0,1080,2400]"},
"children": [
@@ -25,10 +25,13 @@ const enumTreeJSON = `{
}`
// enumVerifier loads a spec whose actions root is the given plain-object graph
// and stages the given tree, so Candidates walks a controlled action tree.
// and stages the given tree, so Candidates walks a controlled action tree. The
// spec is bundled with the goja runtime entry because the model arm reads the
// picker's own builtin enumeration out of that bundle.
func enumVerifier(t *testing.T, actionsJS, treeJSON string) *Verifier {
t.Helper()
v := newLoadedVerifier(t, "globalThis.actions = "+actionsJS+";")
v := newVerifier(t)
loadActionSpec(t, v, "globalThis.actions = "+actionsJS+";")
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatalf("parse tree: %v", err)
@@ -119,26 +122,38 @@ func TestCandidatesLabelsEditableFieldByHintNotTypedValue(t *testing.T) {
}
}
func TestCandidatesFoldsGesturesIntoDirectionalScrolls(t *testing.T) {
func TestCandidatesKeepsGestureVerbsDistinct(t *testing.T) {
v := enumVerifier(t,
"{kind:'weighted', branches:[[1,{kind:'builtin',verb:'scrolls'}],[1,{kind:'builtin',verb:'swipes'}]]}",
enumTreeJSON)
candidates := v.Candidates()
// Gestures are directional and scoped to the one scrollable container: no
// per-element, element-labeled Swipe entries.
for _, candidate := range candidates {
if strings.HasPrefix(candidate.Description, "Swipe") {
t.Errorf("gesture kept as element-labeled swipe: %q", candidate.Description)
}
}
// `scrolls` folds to one directional pair over the single scrollable
// container, which is what keeps the list short.
if !hasCandidate(candidates, "Scroll down") || !hasCandidate(candidates, "Scroll up") {
t.Errorf("want directional scrolls, got %v", descriptions(candidates))
}
// scrolls and swipes fold into the SAME directional entries: one each.
if got := count(candidates, "Scroll down"); got != 1 {
t.Errorf("Scroll down appears %d times, want 1 (folded)", got)
t.Errorf("Scroll down appears %d times, want 1 over the one container", got)
}
// `swipes` is a different verb, not a second name for the scroll: a
// free-form drag from any element, named by the control it starts on. That
// is what puts swipe-to-dismiss on a row within the model's reach.
if !hasCandidatePrefix(candidates, `Swipe "Sign in"`) {
t.Errorf("want a swipe naming the non-scrollable row, got %v", descriptions(candidates))
}
if hasCandidatePrefix(candidates, "Scroll \"") {
t.Errorf("scroll candidates must stay container-scoped: %v", descriptions(candidates))
}
}
func hasCandidatePrefix(candidates []ActionCandidate, prefix string) bool {
for _, candidate := range candidates {
if strings.HasPrefix(candidate.Description, prefix) {
return true
}
}
return false
}
func TestCandidatesWeightsCombineAcrossPaths(t *testing.T) {
@@ -228,6 +243,23 @@ func TestCandidatesCallsAuthoredLeafOnce(t *testing.T) {
}
}
func TestCandidatesSurfaceAuthoredUntargetedActions(t *testing.T) {
// A spec that authors a swipe, a key press, or a wait must reach the model
// with all three: the seeded picker executes whatever the leaf returns, so a
// kind the enumeration drops is an action only one policy can take.
actions := `{kind:'actions', generate: () => [
{kind:'Swipe', from:{x:10,y:600}, to:{x:10,y:100}, durationMillis: 250},
{kind:'PressKey', key:'back'},
{kind:'Wait'}
]}`
candidates := enumVerifier(t, actions, enumTreeJSON).Candidates()
for _, want := range []string{"Swipe from (10,600) to (10,100)", "Press back", "Wait"} {
if !hasCandidate(candidates, want) {
t.Errorf("authored %q missing: %v", want, descriptions(candidates))
}
}
}
func TestCandidatesOffRouteLeafYieldsNothing(t *testing.T) {
v := enumVerifier(t, "{kind:'actions', generate: () => []}", enumTreeJSON)
if got := v.Candidates(); len(got) != 0 {
@@ -248,9 +280,14 @@ func TestCandidatesSkipsCrossFadeFrames(t *testing.T) {
]
}`
v := enumVerifier(t, "{kind:'builtin', verb:'taps'}", crossFade)
if got := v.Candidates(); got != nil {
if got := v.Candidates(); len(got) != 0 {
t.Errorf("cross-fade frame should yield no candidates, got %v", descriptions(got))
}
// The seeded policy is skipped by the SAME guard, in the shared producer,
// so neither arm acts on a mid-animation layout.
if got := v.targets(); len(got) != 0 {
t.Errorf("cross-fade frame should yield no host targets, got %d", len(got))
}
}
func TestCandidatesNilWithoutTreeOrActions(t *testing.T) {
@@ -345,5 +382,3 @@ func newLoadedVerifier(t *testing.T, source string) *Verifier {
}
return v
}
+11 -8
View File
@@ -48,22 +48,25 @@ func TestCrossRuntimeParity(t *testing.T) {
}
// installStubHost replaces the verifier's hierarchy-backed __sanderlingHost__
// with one returning a FIXED candidate list for every verb, keeping the seed the
// verifier was constructed with. It must run BEFORE Load, because the bundled
// goja runtime entry captures the host when the spec evaluates.
// with one returning a FIXED target list, keeping the seed the verifier was
// constructed with. Every fact is set so the shared eligibility rule admits all
// three targets for every verb, leaving the draw order as the only variable. It
// must run BEFORE Load, because the bundled goja runtime entry captures the host
// when the spec evaluates.
func installStubHost(t *testing.T, verifier *Verifier) {
t.Helper()
const stub = `
const candidates = [
{ x: 50, y: 60, selector: "id:alpha", width: 100, height: 40 },
{ x: 150, y: 160, selector: "id:beta", width: 120, height: 48 },
{ x: 250, y: 260, selector: "id:gamma", width: 80, height: 32 },
const facts = { clickable: true, enabled: true, editable: true, scrollable: true };
const targets = [
{ x: 50, y: 60, selector: "id:alpha", width: 100, height: 40, ...facts },
{ x: 150, y: 160, selector: "id:beta", width: 120, height: 48, ...facts },
{ x: 250, y: 260, selector: "id:gamma", width: 80, height: 32, ...facts },
];
const seedHi = globalThis.__sanderlingHost__.seedHi;
const seedLo = globalThis.__sanderlingHost__.seedLo;
globalThis.__sanderlingHost__ = {
platform: () => "android",
queryCandidates: () => candidates,
queryTargets: () => targets,
reportUnsupported: () => {},
seedHi,
seedLo,
+161
View File
@@ -0,0 +1,161 @@
package verifier
import (
"errors"
"fmt"
"maps"
"slices"
"testing"
)
// policyTreeJSON gives every builtin verb something to act on, with every label
// distinct so no two candidates render the same way.
const policyTreeJSON = `{
"attributes": {"bounds": "[0,0,400,800]"},
"children": [
{"attributes": {"resource-id": "Save", "text": "Save", "bounds": "[0,0,200,60]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "Cancel", "text": "Cancel", "bounds": "[200,0,400,60]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "Amount", "class": "EditText", "hintText": "Amount", "bounds": "[0,100,400,160]"}, "enabled": true, "children": []},
{"attributes": {"resource-id": "Note", "class": "EditText", "hintText": "Note", "bounds": "[0,200,400,260]"}, "enabled": true, "children": []},
{"attributes": {"resource-id": "List", "scrollable": "true", "bounds": "[0,300,400,700]"}, "children": []}
]
}`
// policyVerbs is every builtin verb a spec can put in its action tree.
var policyVerbs = []string{
"taps", "doubleTaps", "longPresses", "typing", "scrolls", "swipes", "pressKeys", "waitOnce",
}
// seededDrawBudget is how many times the seeded picker is driven per verb. The
// candidate sets here are a handful of entries wide, so this exhausts them many
// times over; a miss would mean the picker cannot reach one of its own
// candidates, which is itself the bug worth failing on.
const seededDrawBudget = 300
// TestPoliciesEnumerateTheSameActions is the guard on the claim the paper makes
// about the two action policies: they differ in the pick and in nothing else.
// For every builtin verb, the actions the seeded picker can draw and the
// candidates the model policy is offered must be the same set. Enumeration lives
// in one place (pick.ts builtinCandidates) precisely so this cannot drift, and
// this test is what notices if a second one ever grows back.
func TestPoliciesEnumerateTheSameActions(t *testing.T) {
for _, verb := range policyVerbs {
t.Run(verb, func(t *testing.T) {
seeded := seededReachableActions(t, verb)
model := modelOfferedActions(t, verb)
if len(model) == 0 {
t.Fatalf("%s offered the model no candidates at all", verb)
}
if !slices.Equal(slices.Sorted(maps.Keys(seeded)), slices.Sorted(maps.Keys(model))) {
t.Errorf("action spaces differ for %s\n seeded=%v\n model=%v",
verb, slices.Sorted(maps.Keys(seeded)), slices.Sorted(maps.Keys(model)))
}
})
}
}
// TestModelCandidateDescriptionsAreUniqueAndNamed checks the rendering half of
// the contract: every enumerated action reaches the model as its own distinctly
// named line, so the number it picks and the action that executes agree.
func TestModelCandidateDescriptionsAreUniqueAndNamed(t *testing.T) {
for _, verb := range policyVerbs {
t.Run(verb, func(t *testing.T) {
verifier := loadVerbSpec(t, verb)
seen := map[string]bool{}
for _, candidate := range verifier.Candidates() {
if candidate.Description == "" {
t.Fatalf("%s produced a candidate with no description: %+v", verb, candidate.Action)
}
if seen[candidate.Description] {
t.Errorf("%s rendered %q twice, so the model cannot address both",
verb, candidate.Description)
}
seen[candidate.Description] = true
}
})
}
}
// TestModelIsOfferedTheUntargetedVerbs pins the two verbs the model arm used to
// be blind to: with no element to enumerate over, a key press and a wait were
// dropped, so the model could never navigate back or let the app settle.
func TestModelIsOfferedTheUntargetedVerbs(t *testing.T) {
verifier := loadVerbSpec(t, "pressKeys")
if !hasCandidate(verifier.Candidates(), "Press back") {
t.Errorf("pressKeys missing from the model's candidates: %v",
descriptions(verifier.Candidates()))
}
verifier = loadVerbSpec(t, "waitOnce")
if !hasCandidate(verifier.Candidates(), "Wait") {
t.Errorf("waitOnce missing from the model's candidates: %v",
descriptions(verifier.Candidates()))
}
}
// loadVerbSpec builds a verifier whose whole action tree is one builtin verb,
// with policyTreeJSON pushed as the current state.
func loadVerbSpec(t *testing.T, verb string) *Verifier {
t.Helper()
verifier := newVerifier(t, WithSeed(0x5eed))
loadActionSpec(t, verifier, fmt.Sprintf(
"import { %s } from \"@sanderling/spec\";\nglobalThis.actions = %s;", verb, verb))
pushTree(t, verifier, policyTreeJSON)
return verifier
}
// seededReachableActions drives the seeded picker over its draw budget and
// returns every distinct action it produced.
func seededReachableActions(t *testing.T, verb string) map[string]Action {
t.Helper()
verifier := loadVerbSpec(t, verb)
reachable := map[string]Action{}
for range seededDrawBudget {
action, err := verifier.NextAction()
if errors.Is(err, ErrNoAction) {
continue
}
if err != nil {
t.Fatalf("%s next action: %v", verb, err)
}
reachable[actionIdentity(action)] = action
}
return reachable
}
// modelOfferedActions returns the actions behind the numbered list the model
// policy picks from.
func modelOfferedActions(t *testing.T, verb string) map[string]Action {
t.Helper()
verifier := loadVerbSpec(t, verb)
offered := map[string]Action{}
for _, candidate := range verifier.Candidates() {
offered[actionIdentity(candidate.Action)] = candidate.Action
}
return offered
}
// actionIdentity keys an action by everything except the values the policy owns
// rather than the candidate set: the typed text, which the seeded arm draws from
// the edge-case corpus and the model writes itself, and a swipe's drag distance,
// which the seeded arm draws and the enumeration lists at a nominal length.
// Comparing those would compare policies instead of action spaces. A swipe's
// direction is NOT policy-owned, so it survives as the sign of the drag.
func actionIdentity(action Action) string {
action.Text = ""
if action.Kind == ActionKindSwipe {
action.ToX = sign(action.ToX - action.FromX)
action.ToY = sign(action.ToY - action.FromY)
}
return fmt.Sprintf("%+v", action)
}
func sign(value int) int {
switch {
case value > 0:
return 1
case value < 0:
return -1
default:
return 0
}
}
+101 -23
View File
@@ -105,30 +105,108 @@ func TestTyping_ExcludeOffAppPackage(t *testing.T) {
}
}
// TestSwipes_ExcludeOffAppPackage proves swipes anchor on app nodes only, so
// exploration never scrolls the keyboard's emoji list instead of the app.
func TestSwipes_ExcludeOffAppPackage(t *testing.T) {
verifier := newVerifier(t, WithAppPackage("com.folio"))
loadActionSpec(t, verifier, `
import { swipes } from "@sanderling/spec";
globalThis.actions = swipes;
`)
pushTree(t, verifier, scopedTreeJSON)
// gestureTreeJSON gives each gesture verb a legal and an illegal anchor: an app
// list holding a plain row, against the soft keyboard's own scrollable strip
// holding one of its keys.
const gestureTreeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,500]", "package": "com.folio"},
"children": [
{"attributes": {"testTag": "AppList", "scrollable": "true", "bounds": "[0,0,100,300]", "package": "com.folio"}, "children": [
{"attributes": {"testTag": "Row", "bounds": "[0,0,100,60]", "package": "com.folio"}, "children": []}
]},
{"attributes": {"testTag": "EmojiStrip", "scrollable": "true", "bounds": "[0,400,100,440]", "package": "com.google.android.inputmethod.latin"}, "children": [
{"attributes": {"testTag": "EmojiKey", "bounds": "[0,400,100,420]", "package": "com.google.android.inputmethod.latin"}, "children": []}
]}
]
}`
// Both the root and SubmitButton (com.folio) are valid anchors; only the
// keyboard key at center (50,420) must be excluded. Draw many times so the
// invariant is not satisfied by a lucky seed.
for i := range 200 {
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindSwipe {
t.Fatalf("kind = %v, want Swipe", action.Kind)
}
if action.FromX == 50 && action.FromY == 420 {
t.Fatalf("draw %d anchored on the keyboard key (50,420); off-app node leaked into swipe targets", i)
}
// TestGestures_ExcludeOffAppPackage proves both gesture verbs anchor on app
// nodes only, so exploration never drags the keyboard instead of the app, and
// pins what each verb accepts within the app and which way it drags. Scrolls
// anchor on the scrollable list alone and stay vertical, because every
// scrollable container earns a candidate and scrolling a list means up and down.
// Swipes anchor on any app element, the row and the root included, and drag in
// all four directions, which is what puts swipe-to-dismiss on a list row inside
// the action space.
func TestGestures_ExcludeOffAppPackage(t *testing.T) {
tests := []struct {
verb string
kind ActionKind
anchors map[[2]int]bool
directions map[string]bool
}{
{
"scrolls",
ActionKindScroll,
map[[2]int]bool{{50, 150}: true},
map[string]bool{"down": true, "up": true},
},
{
"swipes",
ActionKindSwipe,
map[[2]int]bool{{50, 250}: true, {50, 150}: true, {50, 30}: true},
map[string]bool{"down": true, "up": true, "left": true, "right": true},
},
}
for _, test := range tests {
t.Run(test.verb, func(t *testing.T) {
verifier := newVerifier(t, WithAppPackage("com.folio"))
loadActionSpec(t, verifier, `
import { `+test.verb+` } from "@sanderling/spec";
globalThis.actions = `+test.verb+`;
`)
pushTree(t, verifier, gestureTreeJSON)
// Draw many times so neither the exclusion nor the coverage below is
// satisfied by a lucky seed. The keyboard strip (50,420) and its key
// (50,410) are the anchors that must never appear.
seen := map[[2]int]bool{}
seenDirections := map[string]bool{}
for i := range 400 {
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != test.kind {
t.Fatalf("kind = %v, want %v", action.Kind, test.kind)
}
anchor := [2]int{action.FromX, action.FromY}
if !test.anchors[anchor] {
t.Fatalf("draw %d anchored at %v, outside %v", i, anchor, test.anchors)
}
direction := gestureDirection(action)
if !test.directions[direction] {
t.Fatalf("draw %d dragged %s, outside %v", i, direction, test.directions)
}
seen[anchor] = true
seenDirections[direction] = true
}
if len(seen) != len(test.anchors) {
t.Errorf("reached anchors %v, want all of %v", seen, test.anchors)
}
if len(seenDirections) != len(test.directions) {
t.Errorf("reached directions %v, want all of %v", seenDirections, test.directions)
}
})
}
}
// gestureDirection names which way a drawn gesture drags. A scroll carries the
// name it was enumerated under; a swipe carries only its endpoints, so the sign
// of the drag is what says where the finger went.
func gestureDirection(action Action) string {
if action.Kind == ActionKindScroll {
return action.Direction
}
switch {
case action.ToX > action.FromX:
return "right"
case action.ToX < action.FromX:
return "left"
case action.ToY > action.FromY:
return "down"
default:
return "up"
}
}
+40
View File
@@ -1276,6 +1276,46 @@ func TestOverrideExtractorValues_PropagatesNestedObjectFields(t *testing.T) {
}
}
// TestOverrideExtractorValues_RecordedStateMatchesEvaluatedState pins the
// reported state to the state predicates read. The web path evaluates
// extractor bodies in V8 and injects the results here, so a diff or a witness
// built from the goja value would describe a state no property ever saw.
func TestOverrideExtractorValues_RecordedStateMatchesEvaluatedState(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
if err := verifier.PushSnapshot(SnapshotInput{
Snapshots: Snapshots{"ledger.balance": json.RawMessage(`100`)},
}); err != nil {
t.Fatal(err)
}
if _, err := verifier.OverrideExtractorValues(map[int]json.RawMessage{
1: json.RawMessage(`-7`),
}); err != nil {
t.Fatal(err)
}
verifier.EvaluateProperties()
balance := verifier.runtime.GlobalObject().Get("balance").ToObject(verifier.runtime)
evaluated := balance.Get("current").String()
change, ok := verifier.ChangedExtractors()["extractor_1"]
if !ok {
t.Fatal("ChangedExtractors reported no change for the overridden extractor")
}
if string(change.Curr) != evaluated {
t.Errorf("ChangedExtractors curr = %s, want %s (the value predicates read)",
change.Curr, evaluated)
}
witness := verifier.Witness("balanceNonNegative")
if witness == nil {
t.Fatal("balanceNonNegative did not violate on the overridden value")
}
if got := string(witness.Extractors["extractor_1"]); got != evaluated {
t.Errorf("witness extractor = %s, want %s (the value predicates read)",
got, evaluated)
}
}
// TestUnsupportedVerbs_CollectedDedupedInOrder drives the real host binding the
// shared picker invokes (__sanderlingHost__.reportUnsupported) and asserts the
// verifier collects each verb once, in first-seen order, for the run report.
+106 -73
View File
@@ -39,22 +39,30 @@ type Verifier struct {
// reimplementing the corpus on the Go side.
sampleInputFn goja.Callable
// enumerateBuiltinFn is the bundle-installed __sanderlingEnumerateBuiltin__,
// which lists every action a builtin verb can yield right now. It is the same
// enumeration the seeded picker draws from, so the LLM action backend selects
// over the picker's action space rather than one of its own.
enumerateBuiltinFn goja.Callable
evaluators map[string]*ltl.Evaluator
priorVerdicts map[string]ltl.Verdict
newlyViolated []string
witnesses map[string]Witness
lastTree *hierarchy.Tree
lastScreenshot []byte
scopeCache map[*hierarchy.Element]bool
scopeCacheTree *hierarchy.Tree
lastAction *Action
lastLogs []LogEntry
lastExceptions []Exception
stepTime time.Time
stepIndex int
runStart time.Time
lastTree *hierarchy.Tree
lastScreenshot []byte
scopeCache map[*hierarchy.Element]bool
scopeCacheTree *hierarchy.Tree
targetCache []targetElement
targetCacheTree *hierarchy.Tree
lastAction *Action
lastLogs []LogEntry
lastExceptions []Exception
stepTime time.Time
stepIndex int
runStart time.Time
appPackage string
platform string
@@ -178,18 +186,30 @@ func (v *Verifier) Load(source string) error {
}
}
if fn := v.runtime.GlobalObject().Get("__sanderlingEnumerateBuiltin__"); fn != nil {
if callable, ok := goja.AssertFunction(fn); ok {
v.enumerateBuiltinFn = callable
}
}
return nil
}
// buildFormula walks the formula-spec registry and produces a Go ltl.Formula
// tree rooted at the given spec index. Specs built at the top level are
// always wrapped in Always unless the top-level spec is already an Always.
// tree rooted at the given spec index.
//
// A top-level spec that is not already a temporal obligation is wrapped in
// Always, which is what an author writing a bare predicate or a combinator
// means. An always is left alone, and so is an eventually: wrapping
// `eventually(p).within(5, "minutes")` would turn one reachability goal into
// "within five minutes of every step", a different and far stronger property.
func (v *Verifier) buildFormula(rootIndex int) (ltl.Formula, error) {
inner, err := v.buildFormulaNode(rootIndex)
if err != nil {
return nil, err
}
if _, ok := inner.(ltl.AlwaysFormula); ok {
switch inner.(type) {
case ltl.AlwaysFormula, ltl.EventuallyFormula:
return inner, nil
}
return ltl.Always(inner), nil
@@ -386,6 +406,12 @@ func (v *Verifier) ChangedExtractors() map[string]ExtractorChange {
// call this unconditionally. The override must run *after* PushSnapshot
// (which advanced `previous`) and *before* EvaluateProperties.
//
// The JSON snapshot `curr` is replaced alongside the value, so the diffs in
// ChangedExtractors and the witness recorded by captureWitness describe the
// state the verdict was computed from. Recording the goja value while a
// predicate read the V8 one makes a witness explain a violation with a state
// that never reached the property.
//
// Out-of-range indices are tolerated (skipped) rather than fatal: V8 and goja
// register extractors from the same spec bundle so counts should always
// match, but a stale or partial override map should not block valid overrides
@@ -405,6 +431,7 @@ func (v *Verifier) OverrideExtractorValues(overrides map[int]json.RawMessage) (s
return skipped, fmt.Errorf("extractor override %d: %w", index, conversionErr)
}
v.extractors[index].currentValue = value
v.extractors[index].curr = encodeExtractorValue(value)
}
return skipped, nil
}
@@ -420,7 +447,7 @@ type SnapshotInput struct {
// callers may leave it nil.
ScreenshotPNG []byte
LastAction *Action
StepTime time.Time
StepTime time.Time
// StepIndex is the runner's step number for this snapshot. Evaluators label
// observations with it so violation witnesses carry runner step numbers even
// when transitional steps were skipped. Zero means unlabeled; evaluators
@@ -470,21 +497,27 @@ func (v *Verifier) EvaluateProperties() map[string]ltl.Verdict {
}
// Witness is the verifier-level record of a property violation: the LTL reason
// (a predicate's thrown-error text, "predicate false", or a liveness failure),
// the step it fired at, and a snapshot of every extractor's current value at
// that step. The snapshot lets a reader see the state that produced the
// violation without replaying the run.
// (a predicate's thrown-error text, "predicate false", or a liveness failure)
// and the two step indices a deferred obligation spans.
//
// Step is the origin: the step whose observation armed the obligation that
// failed. DetectedStep is the observation whose reduction produced the
// violation, which for a next or an eventually is later. Extractors is that
// observation's state, so it belongs to DetectedStep and not to Step; the two
// were previously conflated under one index.
type Witness struct {
Property string
Reason string
Step int
IsError bool
Extractors map[string]json.RawMessage
Property string
Reason string
Step int
DetectedStep int
IsError bool
Extractors map[string]json.RawMessage
}
// captureWitness records the witness for a property that just transitioned to
// violated, snapshotting the current extractor values so the cause is visible
// after the run.
// after the run. The snapshot is the state of the observation being reduced,
// which the witness records as its detection step.
func (v *Verifier) captureWitness(name string) {
evaluator, ok := v.evaluators[name]
if !ok {
@@ -495,11 +528,12 @@ func (v *Verifier) captureWitness(name string) {
return
}
v.witnesses[name] = Witness{
Property: name,
Reason: violation.Reason,
Step: violation.Step,
IsError: violation.IsError,
Extractors: v.extractorSnapshot(),
Property: name,
Reason: violation.Reason,
Step: violation.Step,
DetectedStep: v.stepIndex,
IsError: violation.IsError,
Extractors: v.extractorSnapshot(),
}
}
@@ -730,66 +764,65 @@ func selectorForElement(tree *hierarchy.Tree, element *hierarchy.Element) string
return ""
}
// candidatesForVerb enumerates the host-side targets a builtin verb may draw
// from, in v.lastTree.Elements ORDER (the order is part of the picker's parity
// contract). The filters are LIFTED from the old Go picker:
// targets enumerates every element this host can offer, in v.lastTree.Elements
// ORDER (the order is part of the picker's parity contract). It is the native
// half of the single candidate producer: the picker reads it through
// __sanderlingHost__.queryTargets, applies the SHARED per-verb eligibility rule
// (pkg/spec/src/targets.ts), and expands what survives into concrete actions for
// both policies. Which verb may act on which element is decided there, once, so
// this host and the web host cannot mean different things by the same verb.
//
// taps/doubleTaps/longPresses: clickable + enabled + positive bounds
// typing: editable + enabled + positive bounds
// scrolls: scrollable attribute + positive bounds
// swipes: any in-scope element with positive bounds
// This host's job is the facts: clickable/enabled/editable come off the
// accessibility node, scrollable off its attribute, and the geometry off its
// bounds. Every target carries the resolving selector so the runner can re-route
// by id/text. Out-of-scope nodes (the soft keyboard, system UI, the launcher)
// are dropped by scopedElements.
//
// Every candidate carries the resolving selector so the runner can re-route by
// id/text. Out-of-scope nodes (the soft keyboard, system UI, the launcher) are
// dropped by scopedElements.
func (v *Verifier) candidatesForVerb(verb string) []candidate {
if v.lastTree == nil {
// A cross-fade frame yields nothing at all. Its layout is mid-animation, often
// in a collapsed coordinate space, so acting on it lands on garbage; skipping it
// here rather than in one policy means both policies re-observe a settled frame
// instead of one of them acting on the animation.
func (v *Verifier) targets() []targetElement {
if v.lastTree == nil || v.lastTree.Transitional() {
return nil
}
if v.targetCacheTree == v.lastTree {
return v.targetCache
}
scope := v.scopedElements()
var result []candidate
result := make([]targetElement, 0, len(v.lastTree.Elements))
for _, element := range v.lastTree.Elements {
if !scope[element] {
continue
}
if !verbAccepts(verb, element) {
continue
}
x, y := element.Bounds.Center()
result = append(result, candidate{
x: x,
y: y,
width: element.Bounds.Width(),
height: element.Bounds.Height(),
selector: selectorForElement(v.lastTree, element),
result = append(result, targetElement{
element: element,
x: x,
y: y,
width: element.Bounds.Width(),
height: element.Bounds.Height(),
selector: selectorForElement(v.lastTree, element),
clickable: element.Clickable,
enabled: element.Enabled,
editable: element.Editable,
scrollable: element.Attributes["scrollable"] == "true",
})
}
v.targetCache = result
v.targetCacheTree = v.lastTree
return result
}
type candidate struct {
// targetElement is one host-enumerated element with the facts the shared
// eligibility rule reads. The host reports facts; it does not filter by verb.
type targetElement struct {
element *hierarchy.Element
x, y int
width, height int
selector string
}
// verbAccepts applies the per-verb element filter.
func verbAccepts(verb string, element *hierarchy.Element) bool {
positiveBounds := element.Bounds.Width() > 0 && element.Bounds.Height() > 0
switch verb {
case "taps", "doubleTaps", "longPresses":
return element.Clickable && element.Enabled && positiveBounds
case "typing":
return element.Editable && element.Enabled && positiveBounds
case "scrolls":
return element.Attributes["scrollable"] == "true" && positiveBounds
case "swipes":
// Any visible element is a valid swipe origin, but it must have real
// bounds: a zero-bounds node centers at (0,0), and a downward swipe from
// the top-left corner is the system gesture that pulls down the
// notification shade, dragging the fuzzer out of the app.
return positiveBounds
default:
return false
}
clickable bool
enabled bool
editable bool
scrollable bool
}
+33 -12
View File
@@ -1,24 +1,45 @@
package verifier
import (
"slices"
"testing"
"github.com/priyanshujain/sanderling/internal/hierarchy"
)
// TestVerbAcceptsSwipeRequiresPositiveBounds locks the fix for the notification
// shade: a zero-bounds element centers at (0,0), and a downward swipe from the
// top-left corner is the system gesture that pulls the shade over the app. The
// swipe verb must reject zero-bounds nodes like every other verb does.
func TestVerbAcceptsSwipeRequiresPositiveBounds(t *testing.T) {
zeroBounds := &hierarchy.Element{Bounds: hierarchy.Bounds{}}
if verbAccepts("swipes", zeroBounds) {
t.Error("swipes must reject a zero-bounds element (it centers at (0,0) and pulls the notification shade)")
// TestTargetsReportFactsWithoutFiltering pins the native host's half of the
// split introduced to stop the two hosts drifting: it reports what an element
// IS and never decides which verb may act on it. The verb decision is one shared
// rule (pkg/spec/src/targets.ts) both hosts consume, asserted across engines by
// TestHostsAgreeOnTargetEligibility.
func TestTargetsReportFactsWithoutFiltering(t *testing.T) {
tree, err := hierarchy.Parse(hostParityTreeJSON)
if err != nil {
t.Fatal(err)
}
realBounds := &hierarchy.Element{Bounds: hierarchy.Bounds{Left: 100, Top: 400, Right: 980, Bottom: 600}}
if !verbAccepts("swipes", realBounds) {
t.Error("swipes must accept an element with positive bounds")
v := &Verifier{lastTree: tree}
targets := v.targets()
if len(targets) != len(hostParityScreen) {
t.Fatalf("targets() returned %d elements, want all %d in the tree",
len(targets), len(hostParityScreen))
}
for index, target := range targets {
want := hostParityScreen[index]
got := []bool{
target.clickable, target.enabled, target.editable, target.scrollable,
}
expected := []bool{
want.clickable, want.enabled, want.editable, want.scrollable,
}
if !slices.Equal(got, expected) {
t.Errorf("%s clickable/enabled/editable/scrollable = %v, want %v",
want.name, got, expected)
}
positiveBounds := target.width > 0 && target.height > 0
if positiveBounds != want.positiveBounds {
t.Errorf("%s positive bounds = %v, want %v",
want.name, positiveBounds, want.positiveBounds)
}
}
}
+19 -7
View File
@@ -65,9 +65,9 @@ export type GeneratorNode =
| { kind: "builtin"; verb: BuiltinVerb }
| { kind: "llm"; config: { model: string; instructions?: string } };
// Candidate is one host-enumerated target for a builtin verb. The host
// resolves geometry (and a native selector) so no element handle crosses into
// the picker. width/height let swipe/scroll size a gesture off the element.
// Candidate is the resolved geometry of one target. The host resolves it (and a
// native selector) so no element handle crosses into the picker. width/height
// let swipe/scroll size a gesture off the element.
export interface Candidate {
x: number;
y: number;
@@ -76,15 +76,27 @@ export interface Candidate {
height?: number;
}
// TargetElement is one host-enumerated element, offered to EVERY verb, carrying
// the facts per-verb eligibility is decided from. The host reports the facts; it
// does not apply them. targets.ts acceptsTarget owns that decision for both
// hosts, so a verb cannot mean one thing on native and another on web.
export interface TargetElement extends Candidate {
clickable: boolean;
enabled: boolean;
editable: boolean;
scrollable: boolean;
}
// Host is the platform backing the picker draws against. In this foundation
// workflow only the interface is defined and exercised against a stub; the
// goja and DOM implementations land in the rewire workflow.
export interface Host {
platform(): "android" | "ios" | "web";
// queryCandidates returns the host-enumerated targets for a verb, in a
// deterministic order. The picker indexes into this list with the PCG, so
// the order is part of the parity contract.
queryCandidates(verb: BuiltinVerb): Candidate[];
// queryTargets returns every element the host can offer, in a deterministic
// order, with no per-verb filtering: the picker applies acceptsTarget. The
// picker indexes into this list with the PCG, so the order is part of the
// parity contract.
queryTargets(): TargetElement[];
// reportUnsupported is invoked at most once per verb@platform (see verbs.ts)
// when a verb has no support on this platform.
reportUnsupported(verb: BuiltinVerb): void;
+1 -1
View File
@@ -1,6 +1,6 @@
// Goja-side runtime entry for the native verifier (internal/verifier).
//
// Go installs globalThis.__sanderlingHost__ (platform/seed/queryCandidates/
// Go installs globalThis.__sanderlingHost__ (platform/seed/queryTargets/
// reportUnsupported, implemented over the hierarchy tree in bindings.go) before
// the spec evaluates. This module bundles AFTER the spec, reads that host, and
// wires the shared picker via installRuntime so the goja verifier and the V8 web
+161 -119
View File
@@ -1,29 +1,44 @@
// The shared deterministic action picker for W2 approach B.
// The shared deterministic action picker.
//
// walk() traverses a GeneratorNode tree, and nextAction() wraps it with the
// 16-attempt retry that matches worker.go NextAction. Both engines (the goja
// verifier and the V8 web runtime) run THIS code, drawing through the shared
// Pcg, so a given seed yields an identical action stream on every platform.
//
// builtinCandidates() is the ONE enumeration of what a builtin verb can do at
// the current step. The seeded policy draws a single entry from it below; the
// model policy (Go, internal/verifier/llm.go) reads the same list through
// __sanderlingEnumerateBuiltin__. Neither policy can reach an action the other
// cannot, because neither owns an enumeration of its own.
//
// PARITY CONTRACT - draw order.
// Every random decision goes through the Pcg in a FIXED, pinned order. Changing
// this order shifts the stream for a seed and breaks cross-engine
// reproducibility, so treat it as load-bearing:
//
// pickUniform(list): NO draw for an empty or single-entry list, otherwise ONE
// intN(list.length) draw. Both the actions node and the
// builtin node select through it.
// weighted node: ONE float64() draw, then an ASCENDING cumulative scan over
// max(0, weight). (matches worker.go pickWeighted.)
// actions node: the generator runs FIRST (any from(...).generate() inside
// draws intN(itemCount) for >1 items, nothing otherwise),
// THEN if the returned list has >1 entry, ONE intN(len) draw;
// a 0- or 1-element list draws nothing. (pickFromResult.)
// builtin node, per verb, in this exact sequence:
// taps/doubleTaps/longPresses: intN(candidateCount) [1 draw]
// typing: intN(candidateCount), intN(corpusLength)
// swipes: intN(candidateCount),
// 200 + intN(401) magnitude, intN(4) direction
// scrolls: intN(candidateCount), intN(4) direction
// pressKeys: intN(keyCount)
// waitOnce: no draw
// THEN pickUniform over the returned list.
// builtin node: pickUniform over builtinCandidates(verb), THEN the one
// value that verb's enumeration leaves to the policy:
// `typing` ONE intN(corpusLength) draw for the text,
// `swipes` ONE 200 + intN(401) draw for the drag distance.
// Every other verb draws nothing further.
//
// The enumerated candidate count per verb, over the host targets the verb
// accepts (targets.ts acceptsTarget), which is what pickUniform draws over:
//
// taps/doubleTaps/longPresses: one per accepted target
// typing: one per accepted target
// scrolls: two per scrollable container (down, up)
// swipes: four per accepted target (down, up, left, right)
// pressKeys: one per key in the platform's pool
// waitOnce: exactly one, so it never draws
//
// Builtin targets are resolved to a {x, y} Point by the host BEFORE the picker
// sees them, so no element handle crosses into this module.
@@ -32,24 +47,114 @@ import type { Pcg } from "./pcg.ts";
import type {
ActionDescriptor,
BuiltinVerb,
Candidate,
GeneratorNode,
Host,
} from "./action-tree.ts";
import type { Direction, Point } from "./types.ts";
import { INPUT_CORPUS, NATIVE_PRESS_KEYS, WEB_PRESS_KEYS } from "./corpus.ts";
import { setSamplerRng } from "./sampler-rng.ts";
import { acceptsTarget } from "./targets.ts";
import { supports, warnUnsupportedOnce } from "./verbs.ts";
// SWIPE_MIN_MAGNITUDE / SWIPE_MAGNITUDE_SPAN reproduce worker.go's
// `200 + rng.IntN(401)` swipe distance in pixels (200..600 inclusive).
const WAIT_MILLIS = 500;
// SCROLL_DIRECTIONS and SWIPE_DIRECTIONS are the directions each gesture verb
// enumerates, one candidate per (target, direction). Scrolls stay vertical:
// they target every scrollable container, so they are what makes the enumerated
// list long, and scrolling a mobile list means up and down. Swipes take all
// four, because swipe-to-dismiss and swipe-to-delete are horizontal gestures on
// list rows, and folding them away puts that defect class out of reach.
const SCROLL_DIRECTIONS: readonly Direction[] = ["down", "up"];
const SWIPE_DIRECTIONS: readonly Direction[] = ["down", "up", "left", "right"];
// SWIPE_MIN_MAGNITUDE / SWIPE_MAGNITUDE_SPAN are the free-form swipe distance in
// pixels the seeded policy draws, 200..600 inclusive. SWIPE_NOMINAL_MAGNITUDE is
// the distance the enumeration lists, so every enumerated candidate is already a
// runnable gesture before any policy has drawn anything.
const SWIPE_MIN_MAGNITUDE = 200;
const SWIPE_MAGNITUDE_SPAN = 401;
const SWIPE_NOMINAL_MAGNITUDE = 400;
const SWIPE_DURATION_MILLIS = 250;
const DIRECTIONS: readonly Direction[] = ["up", "down", "left", "right"];
const MAX_RETRIES = 16;
// BuiltinCandidate is one enumerated action for a builtin verb, paired with the
// index of the host target it acts on, in host.queryTargets() order (NOT the
// verb-filtered order, so a host can resolve it without repeating the filter).
// targetIndex is -1 for the untargeted verbs (a key press, a wait); the model
// policy uses it to name the control the action lands on.
export interface BuiltinCandidate {
action: ActionDescriptor;
targetIndex: number;
// swipe is set on a `swipes` candidate only. The enumeration fixes where the
// drag starts and which way it travels and lists a nominal distance; the
// seeded policy rebuilds the gesture from these at a drawn distance, the way a
// typing candidate names the field and leaves the text to the policy.
swipe?: { origin: Point; direction: Direction };
}
// builtinCandidates enumerates EVERY action a builtin verb can yield against the
// host's current state. It is the single candidate producer both policies read,
// over the single eligibility rule both hosts consume. An unsupported verb
// reports once and enumerates nothing, so a platform that cannot dispatch a verb
// never offers it to either policy.
export function builtinCandidates(verb: BuiltinVerb, host: Host): BuiltinCandidate[] {
if (!supports(verb, host.platform())) {
warnUnsupportedOnce(host, verb);
return [];
}
if (verb === "waitOnce") {
return [{ action: { kind: "Wait", durationMillis: WAIT_MILLIS }, targetIndex: -1 }];
}
if (verb === "pressKeys") {
const keys = host.platform() === "web" ? WEB_PRESS_KEYS : NATIVE_PRESS_KEYS;
return keys.map((key) => ({
action: { kind: "PressKey", key } as ActionDescriptor,
targetIndex: -1,
}));
}
const candidates: BuiltinCandidate[] = [];
host.queryTargets().forEach((target, targetIndex) => {
if (!acceptsTarget(verb, target)) return;
const point = withSelector({ x: target.x, y: target.y }, target.selector);
const add = (action: ActionDescriptor) => candidates.push({ action, targetIndex });
switch (verb) {
case "taps":
add({ kind: "Tap", on: point });
break;
case "doubleTaps":
add({ kind: "DoubleTap", on: point });
break;
case "longPresses":
add({ kind: "LongPress", on: point });
break;
case "typing":
// text is left empty: it is the one value the policy supplies, drawn
// from the corpus by the seeded arm and written by the model.
add({ kind: "InputText", into: point, text: "" });
break;
case "scrolls":
for (const direction of SCROLL_DIRECTIONS) {
add(scrollDescriptor({ x: target.x, y: target.y }, direction, target));
}
break;
case "swipes":
for (const direction of SWIPE_DIRECTIONS) {
const origin = { x: target.x, y: target.y };
candidates.push({
action: swipeDescriptor(origin, direction, SWIPE_NOMINAL_MAGNITUDE),
targetIndex,
swipe: { origin, direction },
});
}
break;
}
});
return candidates;
}
// nextAction resolves an action for the current step, retrying walk() up to 16
// times when it yields null (matches worker.go NextAction). Returns null when
// every attempt comes up empty.
@@ -79,7 +184,7 @@ export function walk(
// generator so author sampling shares this single deterministic stream.
setSamplerRng(rng);
try {
return walkActions(node.generate(), rng);
return pickUniform(node.generate(), rng);
} finally {
setSamplerRng(null);
}
@@ -115,13 +220,13 @@ function walkWeighted(
return last ? walk(last[1], rng, host) : null;
}
function walkActions(
generated: ActionDescriptor[],
rng: Pcg,
): ActionDescriptor | null {
if (generated.length === 0) return null;
if (generated.length === 1) return generated[0] ?? null;
return generated[rng.intN(generated.length)] ?? null;
// pickUniform selects one entry from a list, drawing nothing when there is no
// choice to make. It is the only selection rule in this module: the seeded
// policy is exactly "enumerate, then pickUniform".
function pickUniform<T>(list: readonly T[], rng: Pcg): T | null {
if (list.length === 0) return null;
if (list.length === 1) return list[0] ?? null;
return list[rng.intN(list.length)] ?? null;
}
function walkBuiltin(
@@ -129,41 +234,17 @@ function walkBuiltin(
rng: Pcg,
host: Host,
): ActionDescriptor | null {
if (!supports(verb, host.platform())) {
warnUnsupportedOnce(host, verb);
return null;
}
if (verb === "waitOnce") {
return { kind: "Wait", durationMillis: 500 };
}
if (verb === "pressKeys") {
return walkPressKey(rng, host);
}
const candidates = host.queryCandidates(verb);
if (candidates.length === 0) return null;
const picked = candidates[rng.intN(candidates.length)];
const picked = pickUniform(builtinCandidates(verb, host), rng);
if (!picked) return null;
const point: Point = { x: picked.x, y: picked.y };
switch (verb) {
case "taps":
return tapDescriptor("Tap", point, picked.selector);
case "doubleTaps":
return tapDescriptor("DoubleTap", point, picked.selector);
case "longPresses":
return tapDescriptor("LongPress", point, picked.selector);
case "typing": {
const text = INPUT_CORPUS[rng.intN(INPUT_CORPUS.length)] ?? "";
return { kind: "InputText", into: withSelector(point, picked.selector), text };
}
case "swipes":
return buildSwipe(point, rng);
case "scrolls": {
const direction = DIRECTIONS[rng.intN(DIRECTIONS.length)] ?? "down";
return buildScroll(point, direction, picked, rng);
}
if (picked.action.kind === "InputText") {
return { ...picked.action, text: INPUT_CORPUS[rng.intN(INPUT_CORPUS.length)] ?? "" };
}
if (picked.swipe) {
const magnitude = SWIPE_MIN_MAGNITUDE + rng.intN(SWIPE_MAGNITUDE_SPAN);
const { origin, direction } = picked.swipe;
return swipeDescriptor(origin, direction, magnitude);
}
return picked.action;
}
// withSelector attaches a native selector to a resolved Point so the runner can
@@ -173,82 +254,43 @@ function withSelector(point: Point, selector?: string): Point {
return { ...point, selector } as Point;
}
function tapDescriptor(
kind: "Tap" | "DoubleTap" | "LongPress",
point: Point,
selector?: string,
): ActionDescriptor {
return { kind, on: withSelector(point, selector) } as ActionDescriptor;
}
// buildScroll lowers a scroll to a swipe over the container, matching
// worker.go's geometry: the gesture drags opposite the named content motion,
// scrollDescriptor lowers a scroll to a drag over the container, matching
// runner.go's geometry: the gesture drags opposite the named content motion,
// magnitude 40% of the container extent. Missing width/height (web root) yields
// a zero-length endpoint, which the runner re-derives from container bounds.
function buildScroll(
function scrollDescriptor(
from: Point,
direction: Direction,
candidate: { width?: number; height?: number },
_rng: Pcg,
candidate: Candidate,
): ActionDescriptor {
const width = candidate.width ?? 0;
const height = candidate.height ?? 0;
let toX = from.x;
let toY = from.y;
switch (direction) {
case "down":
toY = from.y - Math.trunc((4 * height) / 10);
break;
case "up":
toY = from.y + Math.trunc((4 * height) / 10);
break;
case "left":
toX = from.x + Math.trunc((4 * width) / 10);
break;
case "right":
toX = from.x - Math.trunc((4 * width) / 10);
break;
}
if (direction === "down") toY = from.y - Math.trunc((4 * height) / 10);
if (direction === "up") toY = from.y + Math.trunc((4 * height) / 10);
return {
kind: "Scroll",
direction,
in: from,
from,
to: { x: Math.max(0, toX), y: Math.max(0, toY) },
to: { x: from.x, y: Math.max(0, toY) },
} as ActionDescriptor;
}
function walkPressKey(rng: Pcg, host: Host): ActionDescriptor | null {
const keys = host.platform() === "web" ? WEB_PRESS_KEYS : NATIVE_PRESS_KEYS;
if (keys.length === 0) return null;
const key = keys[rng.intN(keys.length)] ?? keys[0];
if (key === undefined) return null;
return { kind: "PressKey", key };
// swipeDescriptor builds a free-form drag from a point over a raw pixel
// distance, rather than a fraction of a container extent: `swipes` targets any
// element with real bounds, and most of those have no scroll extent to size a
// gesture against. `direction` names where the finger travels, because a swipe
// IS the gesture; a scroll names content motion and drags the other way.
function swipeDescriptor(
from: Point,
direction: Direction,
magnitude: number,
): ActionDescriptor {
const horizontal = direction === "left" || direction === "right";
const forward = direction === "down" || direction === "right";
const travel = forward ? magnitude : -magnitude;
const to = horizontal
? { x: Math.max(0, from.x + travel), y: from.y }
: { x: from.x, y: Math.max(0, from.y + travel) };
return { kind: "Swipe", from, to, durationMillis: SWIPE_DURATION_MILLIS };
}
function buildSwipe(from: Point, rng: Pcg): ActionDescriptor {
const magnitude = SWIPE_MIN_MAGNITUDE + rng.intN(SWIPE_MAGNITUDE_SPAN);
let toX = from.x;
let toY = from.y;
switch (rng.intN(4)) {
case 0:
toY = from.y - magnitude;
break;
case 1:
toY = from.y + magnitude;
break;
case 2:
toX = from.x - magnitude;
break;
case 3:
toX = from.x + magnitude;
break;
}
return {
kind: "Swipe",
from,
to: { x: Math.max(0, toX), y: Math.max(0, toY) },
durationMillis: SWIPE_DURATION_MILLIS,
};
}
+12 -2
View File
@@ -8,9 +8,9 @@
// identical action stream by construction.
import { Pcg } from "./pcg.ts";
import { nextAction, walk } from "./pick.ts";
import { builtinCandidates, nextAction, walk } from "./pick.ts";
import { INPUT_CORPUS } from "./corpus.ts";
import type { ActionDescriptor, GeneratorNode, Host } from "./action-tree.ts";
import type { ActionDescriptor, BuiltinVerb, GeneratorNode, Host } from "./action-tree.ts";
import type { Point } from "./types.ts";
// SerializedAction is the flat, camelCase wire shape JS emits and Go decodes
@@ -138,6 +138,16 @@ export function installRuntime(
"__sanderlingSampleInput__",
() => INPUT_CORPUS[rng.intN(INPUT_CORPUS.length)] ?? "",
);
// The model policy (Go) selects from the SAME enumeration the seeded picker
// draws from, reached through here rather than reimplemented on the Go side.
// Each entry is serialized with the wire contract Go already decodes, so the
// two policies also agree on the action a chosen candidate executes.
defineLockedGlobal("__sanderlingEnumerateBuiltin__", (verb: BuiltinVerb) =>
builtinCandidates(verb, host).map((candidate) => ({
action: serializeAction(candidate.action),
targetIndex: candidate.targetIndex,
})),
);
defineLockedGlobal("__sanderlingExtractors__", () => evaluateExtractors());
// __sanderlingSetupAction__ walks ONLY the setup generator once, for the LLM
// action generator (Go), which drives selection itself and must not run the
+69
View File
@@ -0,0 +1,69 @@
// Per-verb target eligibility: the ONE definition both hosts consume.
//
// A host enumerates every element it can offer (the goja verifier over the
// hierarchy tree, the V8 web runtime over the DOM) and reports a fixed set of
// facts about each. It does NOT decide which verb may act on which element:
// acceptsTarget does, here, once. When each host routed verbs itself the two
// drifted, and the same spec induced a different action space per platform --
// web sent `swipes` to scrollable containers only, so swipe-to-dismiss on a
// list row was reachable on native and unreachable on web.
//
// What stays platform-specific is how a fact is COMPUTED, because the source
// models genuinely differ: `clickable` is an accessibility attribute on
// Android/iOS and a CSS selector match on web. The fact vocabulary below is the
// contract; the mapping onto it is each host's business, and it is the only
// place the platforms are allowed to disagree.
import type { BuiltinVerb, TargetElement } from "./action-tree.ts";
// TargetFact names one property of a target a verb can require.
export type TargetFact =
| "clickable"
| "enabled"
| "editable"
| "scrollable"
| "positiveBounds";
// VERB_REQUIRED_FACTS lists the facts a target must have for a verb to act on
// it. `null` marks a verb with no target at all: a key press and a wait are
// enumerated by the picker without consulting the host.
const VERB_REQUIRED_FACTS: Record<BuiltinVerb, readonly TargetFact[] | null> = {
taps: ["clickable", "enabled", "positiveBounds"],
doubleTaps: ["clickable", "enabled", "positiveBounds"],
longPresses: ["clickable", "enabled", "positiveBounds"],
typing: ["editable", "enabled", "positiveBounds"],
// Scrolls target containers that can actually scroll, and enumerate down/up.
scrolls: ["scrollable", "positiveBounds"],
// Any visible element is a valid swipe origin. Swipe-to-dismiss and
// swipe-to-delete live on list rows and cards, which are not scrollable
// containers, so scoping swipes to containers would put that whole class of
// interaction out of reach.
swipes: ["positiveBounds"],
pressKeys: null,
waitOnce: null,
};
// acceptsTarget is the per-verb element filter. Every targeted verb demands
// real bounds: a zero-bounds node centers at (0,0), and a downward gesture from
// the top-left corner is the system gesture that pulls down the notification
// shade, dragging the fuzzer out of the app.
export function acceptsTarget(verb: BuiltinVerb, target: TargetElement): boolean {
const required = VERB_REQUIRED_FACTS[verb];
if (required === null) return false;
return required.every((fact) => hasFact(target, fact));
}
function hasFact(target: TargetElement, fact: TargetFact): boolean {
switch (fact) {
case "clickable":
return target.clickable;
case "enabled":
return target.enabled;
case "editable":
return target.editable;
case "scrollable":
return target.scrollable;
case "positiveBounds":
return (target.width ?? 0) > 0 && (target.height ?? 0) > 0;
}
}
+56 -73
View File
@@ -4,7 +4,7 @@
//
// This file is the WEB Host. It installs globalThis.__sanderling__ (extract +
// LTL formula binds) before the spec evaluates, implements the Host interface
// (platform/seed/queryCandidates/reportUnsupported) over the live DOM, and then
// (platform/seed/queryTargets/reportUnsupported) over the live DOM, and then
// delegates ALL action generation to the shared picker via installRuntime
// (runtime-entry.ts -> pick.ts). The goja verifier runs the SAME picker over the
// SAME Pcg, so a given seed yields an identical action stream by construction.
@@ -13,11 +13,11 @@
// window.__sanderlingNextAction__() over CDP each tick. LTL predicates are
// stubbed: properties run host-side in goja, which loads its own bundle.
//
// Element references never cross V8/host. queryCandidates resolves each element
// to a {x, y} Point via getBoundingClientRect before the picker sees it.
// Element references never cross V8/host. queryTargets resolves each element to
// a {x, y} Point via getBoundingClientRect before the picker sees it.
import { installRuntime } from "./runtime-entry.ts";
import type { BuiltinVerb, Candidate, Host } from "./action-tree.ts";
import type { BuiltinVerb, Candidate, Host, TargetElement } from "./action-tree.ts";
interface Handle {
readonly current: unknown;
@@ -85,7 +85,12 @@ function noopFormula(): unknown {
const KNOWN_KEY_TO_CSS: Record<string, (value: string) => string> = {
id: (v) => `[id="${cssEscape(v)}"]`,
"resource-id": (v) => `[id="${cssEscape(v)}"]`,
testTag: (v) => `[data-testid="${cssEscape(v)}"]`,
// The native table aliases testTag onto resource-id, which the host DOM walk
// fills from el.id, so the native path already accepts a testTag emitted as
// an id (what Compose Multiplatform does on web). Accept both here so the
// tables agree. `:is()` keeps this one compound, since a multi-key selector
// concatenates the parts.
testTag: (v) => `:is([data-testid="${cssEscape(v)}"], [id="${cssEscape(v)}"])`,
testID: (v) => `[data-testid="${cssEscape(v)}"]`,
"data-testid": (v) => `[data-testid="${cssEscape(v)}"]`,
className: (v) => `[class~="${cssEscape(v)}"]`,
@@ -467,9 +472,12 @@ function sanitizeAt(value: unknown, depth: number, seen: WeakSet<object>): unkno
return out;
}
// Per-verb DOM selector sets. The tappable set backs taps/doubleTaps/longPresses
// (every tappable element is also a valid long-press target); the editable set
// backs typing; swipes/scrolls target the scrollable element.
// The DOM half of the fact mapping. Which verb may act on which target is NOT
// decided here: queryTargets reports facts and targets.ts acceptsTarget applies
// them, the same rule the native host's targets run through. These selectors are
// only how the DOM answers "is this clickable" / "is this editable", the two
// facts with no direct DOM equivalent of the accessibility attributes native
// platforms expose.
const TAPPABLE_SELECTOR = 'a, button, input, select, textarea, [role="button"], [onclick]';
const EDITABLE_SELECTOR = "input, textarea, [contenteditable]";
@@ -477,12 +485,6 @@ const NON_TEXT_INPUT_TYPES = [
"button", "submit", "checkbox", "radio", "range", "color", "file", "image", "reset",
];
function isVisible(element: HTMLElement): boolean {
if ((element as HTMLButtonElement).disabled) return false;
const rect = element.getBoundingClientRect();
return rect.width > 0 && rect.height > 0;
}
function isEditableElement(element: HTMLElement): boolean {
if (element.isContentEditable) return true;
const tag = element.tagName.toLowerCase();
@@ -494,6 +496,14 @@ function isEditableElement(element: HTMLElement): boolean {
return false;
}
// isScrollable mirrors the native `scrollable` accessibility attribute: the
// container can actually scroll, i.e. its content overflows its box. The
// document scrolling root is not special-cased in: when the page does not
// overflow there is no scroll to perform, and native would offer none either.
function isScrollable(element: HTMLElement): boolean {
return element.scrollHeight > element.clientHeight || element.scrollWidth > element.clientWidth;
}
function pointOf(element: Element): Candidate {
const rect = element.getBoundingClientRect();
return {
@@ -504,41 +514,33 @@ function pointOf(element: Element): Candidate {
};
}
function tappableCandidates(): Candidate[] {
return Array.from(document.querySelectorAll<HTMLElement>(TAPPABLE_SELECTOR))
.filter(isVisible)
.map(pointOf);
// collectTargets walks the document ONCE and reports every element with the facts
// the shared eligibility rule reads. The tappable/editable membership sets are
// resolved by selector first so the DOM's answer to "clickable" and "editable"
// stays expressed in CSS, as it always was.
function collectTargets(): TargetElement[] {
const clickable = new Set<Element>(Array.from(document.querySelectorAll(TAPPABLE_SELECTOR)));
const editable = new Set<Element>(
Array.from(document.querySelectorAll<HTMLElement>(EDITABLE_SELECTOR)).filter(
isEditableElement,
),
);
return Array.from(document.querySelectorAll<HTMLElement>("*")).map((element) => ({
...pointOf(element),
clickable: clickable.has(element),
enabled: !(element as HTMLButtonElement).disabled,
editable: editable.has(element),
scrollable: isScrollable(element),
}));
}
function editableCandidates(): Candidate[] {
return Array.from(document.querySelectorAll<HTMLElement>(EDITABLE_SELECTOR))
.filter((element) => isEditableElement(element) && isVisible(element))
.map(pointOf);
}
// Scrollable candidates back swipe/scroll: elements that overflow their box,
// plus the scrolling root so a page-level scroll always has a target.
function scrollableCandidates(): Candidate[] {
const root = document.scrollingElement ?? document.documentElement;
const candidates: Candidate[] = root ? [pointOf(root)] : [];
for (const element of Array.from(document.querySelectorAll<HTMLElement>("*"))) {
if (element === root) continue;
if (element.scrollHeight <= element.clientHeight && element.scrollWidth <= element.clientWidth) {
continue;
}
if (!isVisible(element)) continue;
candidates.push(pointOf(element));
}
return candidates;
}
// Per-tick candidate cache: the picker's 16-attempt retry re-queries the same
// verb, so we avoid re-walking the DOM and re-flushing layout within one tick.
// Per-tick target cache: the picker's 16-attempt retry re-queries every tick, so
// we avoid re-walking the DOM and re-flushing layout within one tick.
// installRuntime resets it before each __sanderlingNextAction__ invocation.
const candidateCache = new Map<BuiltinVerb, Candidate[]>();
let cachedTargets: TargetElement[] | null = null;
function resetCandidateCache(): void {
candidateCache.clear();
function resetTargetCache(): void {
cachedTargets = null;
}
const host: Host = {
@@ -546,28 +548,9 @@ const host: Host = {
seedHi: () => SEED_HI,
// lo = 0 matches the goja side's rand.NewPCG(seed, 0).
seedLo: () => 0n,
queryCandidates(verb: BuiltinVerb): Candidate[] {
const cached = candidateCache.get(verb);
if (cached) return cached;
let candidates: Candidate[];
switch (verb) {
case "taps":
case "doubleTaps":
case "longPresses":
candidates = tappableCandidates();
break;
case "typing":
candidates = editableCandidates();
break;
case "swipes":
case "scrolls":
candidates = scrollableCandidates();
break;
default:
candidates = [];
}
candidateCache.set(verb, candidates);
return candidates;
queryTargets(): TargetElement[] {
if (!cachedTargets) cachedTargets = collectTargets();
return cachedTargets;
},
reportUnsupported(verb: BuiltinVerb): void {
console.warn(`[sanderling] verb ${verb} is unsupported on web`);
@@ -578,11 +561,11 @@ const host: Host = {
// this module (the web bundle imports the runtime first). Resolve the root
// lazily so installRuntime captures it once the spec has evaluated. The root
// resolver runs once per __sanderlingNextAction__ tick (before the retry loop),
// so it is also where we reset the per-tick candidate cache.
// so it is also where we reset the per-tick target cache.
installRuntime(
host,
() => {
resetCandidateCache();
resetTargetCache();
return (globalThis as { actions?: import("./action-tree.ts").GeneratorNode }).actions ?? null;
},
evaluateExtractors,
@@ -593,10 +576,10 @@ installRuntime(
export const __testing__ = {
host,
seedBigInt,
tappableCandidates,
editableCandidates,
scrollableCandidates,
resetCandidateCache,
collectTargets,
TAPPABLE_SELECTOR,
EDITABLE_SELECTOR,
resetTargetCache,
runtime,
extractors,
evaluateExtractors,
+6 -1
View File
@@ -112,7 +112,12 @@ function installFakeRuntime(): RecordedRuntime {
extract: <T>(getter: (state: State) => T, name?: string): Extracted<T> => {
calls.extracts.push(getter as (state: State) => unknown);
calls.extractNames.push(name);
return { current: undefined as unknown as T, previous: undefined };
const handle: Extracted<T> = {
current: undefined as unknown as T,
previous: undefined,
named: () => handle,
};
return handle;
},
always: (predicateOrFormula: (() => boolean) | Formula): Formula => {
calls.alwaysArgs.push(predicateOrFormula);
+8 -4
View File
@@ -15,10 +15,14 @@ import type {
function installRuntime(initialState: State): void {
const state = { current: initialState };
const runtime: SanderlingRuntime = {
extract: <T>(getter: (s: State) => T): Extracted<T> => ({
current: getter(state.current),
previous: undefined,
}),
extract: <T>(getter: (s: State) => T): Extracted<T> => {
const handle: Extracted<T> = {
current: getter(state.current),
previous: undefined,
named: () => handle,
};
return handle;
},
always: () => ({ __sanderlingFormula: true } as Formula),
now: () => ({ __sanderlingFormula: true } as Formula),
next: () => ({ __sanderlingFormula: true } as Formula),
+48
View File
@@ -0,0 +1,48 @@
{
"taps": [
"save"
],
"doubleTaps": [
"save"
],
"longPresses": [
"save"
],
"typing": [
"amount"
],
"scrolls": [
"root",
"root",
"list",
"list"
],
"swipes": [
"root",
"root",
"root",
"root",
"save",
"save",
"save",
"save",
"cancel",
"cancel",
"cancel",
"cancel",
"amount",
"amount",
"amount",
"amount",
"list",
"list",
"list",
"list",
"row",
"row",
"row",
"row"
],
"pressKeys": [],
"waitOnce": []
}
+73
View File
@@ -0,0 +1,73 @@
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { test } from "node:test";
import { builtinCandidates } from "../src/pick.ts";
import { resetWarnings } from "../src/verbs.ts";
import type { BuiltinVerb } from "../src/action-tree.ts";
import { fakeElement, withFakeDocument, type FakeElementSpec } from "./web-dom-harness.ts";
import { __testing__ } from "../src/web-runtime.ts";
const { host } = __testing__;
const goldenPath = fileURLToPath(new URL("./fixtures/host-parity-golden.json", import.meta.url));
const golden: Record<string, string[]> = JSON.parse(readFileSync(goldenPath, "utf8"));
const VERBS: BuiltinVerb[] = [
"taps",
"doubleTaps",
"longPresses",
"typing",
"scrolls",
"swipes",
"pressKeys",
"waitOnce",
];
// SCREEN is the canonical screen both hosts are driven over: one row per fact
// combination that any verb distinguishes. The native host builds the same rows,
// in the same order, as a hierarchy tree in
// internal/verifier/host_parity_test.go.
const SCREEN: (FakeElementSpec & { name: string })[] = [
{ name: "root", tag: "html", x: 0, y: 0, width: 400, height: 800, overflows: true },
{ name: "save", tag: "button", x: 0, y: 0, width: 200, height: 60, clickable: true },
{
name: "cancel",
tag: "button",
x: 200,
y: 0,
width: 200,
height: 60,
clickable: true,
disabled: true,
},
{ name: "amount", tag: "input", x: 0, y: 100, width: 400, height: 60, editable: true },
{ name: "list", tag: "div", x: 0, y: 200, width: 400, height: 400, overflows: true },
{ name: "row", tag: "div", x: 0, y: 600, width: 400, height: 80 },
{ name: "collapsed", tag: "button", x: 0, y: 0, width: 0, height: 0, clickable: true },
];
// The web host and the native host used to route verbs themselves and had
// drifted: web sent `swipes` to scrollable containers only, so a swipe on a list
// row was reachable on Android and unreachable on web for the same spec.
//
// Per-verb eligibility now has ONE definition (src/targets.ts); a host reports
// facts and never filters. This test is what notices if a second definition
// grows back on either side. internal/verifier/host_parity_test.go asserts the
// SAME golden from the native host, so a match on both sides proves the two
// hosts agree without either invoking the other.
test("web host targets match the cross-host golden, verb for verb", () => {
const elements = SCREEN.map(fakeElement);
withFakeDocument(elements, () => {
for (const verb of VERBS) {
resetWarnings();
const candidates = builtinCandidates(verb, host);
assert.notEqual(candidates.length, 0, `${verb} enumerated nothing at all`);
const named = candidates
.filter((candidate) => candidate.targetIndex >= 0)
.map((candidate) => SCREEN[candidate.targetIndex]!.name);
assert.deepEqual(named, golden[verb], `web host targets for ${verb}`);
}
});
});
+15 -11
View File
@@ -1,11 +1,11 @@
// Shared cross-runtime parity scenario: the FIXED seed, FIXED candidate list,
// Shared cross-runtime parity scenario: the FIXED seed, FIXED target list,
// and FIXED action root that both the node test (parity.test.ts) and the goja
// test (internal/verifier/parity_test.go) drive. Each side runs the SAME
// pick.ts over the SAME Pcg and asserts the SAME committed golden
// (fixtures/parity-golden.json); matching one golden on both sides proves the
// two engines agree without either invoking the other.
//
// The candidate ORDER and the per-tick PCG draw order are the parity contract.
// The target ORDER and the per-tick PCG draw order are the parity contract.
// The weighted root mixes a tap branch (1 candidate draw) with a typing branch
// (1 candidate draw + 1 corpus draw), so a tick exercises weighted selection, a
// builtin, and the input corpus together; reordering candidates or adding or
@@ -15,25 +15,29 @@ import { Pcg } from "../src/pcg.ts";
import { nextAction } from "../src/pick.ts";
import { serializeAction, type SerializedAction } from "../src/runtime-entry.ts";
import { taps, typing, weighted } from "../src/actions.ts";
import type { BuiltinVerb, Candidate, GeneratorNode, Host } from "../src/action-tree.ts";
import type { GeneratorNode, Host, TargetElement } from "../src/action-tree.ts";
export const PARITY_SEED_HI = 0x9e3779b97f4a7c15n;
export const PARITY_STEPS = 20;
export const PARITY_CANDIDATES: Candidate[] = [
{ x: 50, y: 60, selector: "id:alpha", width: 100, height: 40 },
{ x: 150, y: 160, selector: "id:beta", width: 120, height: 48 },
{ x: 250, y: 260, selector: "id:gamma", width: 80, height: 32 },
// Every fact is set so the shared eligibility rule admits all three targets for
// every verb, leaving the draw order as the only variable.
const EVERY_FACT = { clickable: true, enabled: true, editable: true, scrollable: true };
export const PARITY_TARGETS: TargetElement[] = [
{ x: 50, y: 60, selector: "id:alpha", width: 100, height: 40, ...EVERY_FACT },
{ x: 150, y: 160, selector: "id:beta", width: 120, height: 48, ...EVERY_FACT },
{ x: 250, y: 260, selector: "id:gamma", width: 80, height: 32, ...EVERY_FACT },
];
// A 3:1 weighted split over taps and typing. The stub host returns the same
// candidate list for every verb so the only variables are the draw order and
// the JS engine's number/bigint behavior.
// A 3:1 weighted split over taps and typing. The stub host offers the same
// target list to every verb so the only variables are the draw order and the JS
// engine's number/bigint behavior.
export const PARITY_ROOT: GeneratorNode = weighted([3, taps], [1, typing]);
const HOST: Host = {
platform: () => "android",
queryCandidates: (_verb: BuiltinVerb) => PARITY_CANDIDATES,
queryTargets: () => PARITY_TARGETS,
reportUnsupported: () => {},
seedHi: () => PARITY_SEED_HI,
seedLo: () => 0n,
+241 -60
View File
@@ -1,30 +1,31 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { Pcg } from "../src/pcg.ts";
import { nextAction, walk } from "../src/pick.ts";
import { builtinCandidates, nextAction, walk } from "../src/pick.ts";
import { INPUT_CORPUS, NATIVE_PRESS_KEYS, WEB_PRESS_KEYS } from "../src/corpus.ts";
import { resetWarnings } from "../src/verbs.ts";
import type {
ActionDescriptor,
BuiltinVerb,
Candidate,
GeneratorNode,
Host,
TargetElement,
} from "../src/action-tree.ts";
import type { Direction, Point } from "../src/types.ts";
type Platform = "android" | "ios" | "web";
// stubHost returns a fixed candidate list for every verb and records each
// reportUnsupported call so warn-once semantics are observable.
// stubHost returns a fixed target list, eligible for every verb, and records
// each reportUnsupported call so warn-once semantics are observable.
function stubHost(
platform: Platform,
candidates: Candidate[],
targets: TargetElement[],
): Host & { unsupported: BuiltinVerb[] } {
const unsupported: BuiltinVerb[] = [];
return {
unsupported,
platform: () => platform,
queryCandidates: () => candidates,
queryTargets: () => targets,
reportUnsupported: (verb) => {
unsupported.push(verb);
},
@@ -33,10 +34,12 @@ function stubHost(
};
}
const POINTS: Candidate[] = [
{ x: 10, y: 20, selector: "id:a", width: 100, height: 200 },
{ x: 30, y: 40, selector: "id:b", width: 100, height: 200 },
{ x: 50, y: 60, selector: "id:c", width: 100, height: 200 },
const EVERY_FACT = { clickable: true, enabled: true, editable: true, scrollable: true };
const POINTS: TargetElement[] = [
{ x: 10, y: 20, selector: "id:a", width: 100, height: 200, ...EVERY_FACT },
{ x: 30, y: 40, selector: "id:b", width: 100, height: 200, ...EVERY_FACT },
{ x: 50, y: 60, selector: "id:c", width: 100, height: 200, ...EVERY_FACT },
];
function builtin(verb: BuiltinVerb): GeneratorNode {
@@ -81,59 +84,132 @@ test("typing draws candidate index then corpus index, in that order", () => {
assert.equal(action.text, INPUT_CORPUS[corpusIndex]);
});
test("swipes draw candidate, magnitude (200+intN(401)), then direction", () => {
// scrollCandidate mirrors what the picker builds for one (container, direction)
// pair: a drag opposite the named content motion, 40% of the container extent.
function scrollCandidate(target: TargetElement, direction: "down" | "up") {
const from = { x: target.x, y: target.y };
const extent = Math.trunc((4 * (target.height ?? 0)) / 10);
const toY = direction === "down" ? from.y - extent : from.y + extent;
return {
kind: "Scroll",
direction,
in: from,
from,
to: { x: from.x, y: Math.max(0, toY) },
};
}
// swipeCandidate mirrors the free-form drag: a raw pixel distance, with the
// direction naming where the finger travels.
function swipeCandidate(
target: Point,
direction: Direction,
magnitude: number,
) {
const from = { x: target.x, y: target.y };
const horizontal = direction === "left" || direction === "right";
const forward = direction === "down" || direction === "right";
const travel = forward ? magnitude : -magnitude;
return {
kind: "Swipe",
from,
to: horizontal
? { x: Math.max(0, from.x + travel), y: from.y }
: { x: from.x, y: Math.max(0, from.y + travel) },
durationMillis: 250,
};
}
const NOMINAL_SWIPE_MAGNITUDE = 400;
const SCROLL_DIRECTIONS = ["down", "up"] as const;
const SWIPE_DIRECTIONS = ["down", "up", "left", "right"] as const;
// swipeDirection recovers which way a drawn swipe travelled from its endpoints.
function swipeDirection(from: Point, to: Point): Direction {
if (to.x !== from.x) return to.x > from.x ? "right" : "left";
return to.y > from.y ? "down" : "up";
}
test("scrolls enumerate every container up and down only", () => {
resetWarnings();
const host = stubHost("android", POINTS);
const rng = new Pcg(7n, 0n);
const oracle = new Pcg(7n, 0n);
const candidateIndex = oracle.intN(POINTS.length);
const magnitude = 200 + oracle.intN(401);
const direction = oracle.intN(4);
const from = { x: POINTS[candidateIndex]!.x, y: POINTS[candidateIndex]!.y };
const expectedTo = { x: from.x, y: from.y };
switch (direction) {
case 0:
expectedTo.y = Math.max(0, from.y - magnitude);
break;
case 1:
expectedTo.y = Math.max(0, from.y + magnitude);
break;
case 2:
expectedTo.x = Math.max(0, from.x - magnitude);
break;
case 3:
expectedTo.x = Math.max(0, from.x + magnitude);
break;
}
const action = walk(builtin("swipes"), rng, host) as ActionDescriptor & {
kind: "Swipe";
};
assert.equal(action.kind, "Swipe");
assert.deepEqual(action.from, from);
assert.deepEqual(action.to, expectedTo);
assert.equal(action.durationMillis, 250);
assert.deepEqual(
builtinCandidates("scrolls", host),
POINTS.flatMap((target, targetIndex) =>
SCROLL_DIRECTIONS.map((direction) => ({
action: scrollCandidate(target, direction),
targetIndex,
})),
),
);
});
test("scrolls draw candidate index then direction", () => {
test("swipes enumerate a free-form drag per target in all four directions", () => {
resetWarnings();
const host = stubHost("android", POINTS);
const rng = new Pcg(99n, 0n);
const oracle = new Pcg(99n, 0n);
const candidateIndex = oracle.intN(POINTS.length);
const directionIndex = oracle.intN(4);
const directions = ["up", "down", "left", "right"] as const;
// The swipe candidate is its own action shape, sized in raw pixels rather than
// off the target's extent, and it carries what the policy needs to redraw the
// distance. It is not the scroll gesture under a second name.
assert.deepEqual(
builtinCandidates("swipes", host),
POINTS.flatMap((target, targetIndex) =>
SWIPE_DIRECTIONS.map((direction) => ({
action: swipeCandidate(target, direction, NOMINAL_SWIPE_MAGNITUDE),
targetIndex,
swipe: { origin: { x: target.x, y: target.y }, direction },
})),
),
);
});
const action = walk(builtin("scrolls"), rng, host) as ActionDescriptor & {
kind: "Scroll";
};
assert.equal(action.kind, "Scroll");
assert.equal(action.direction, directions[directionIndex]);
assert.deepEqual(action.in, {
x: POINTS[candidateIndex]!.x,
y: POINTS[candidateIndex]!.y,
});
test("the gesture verbs differ in target filter and in direction set", () => {
resetWarnings();
// Same host, same targets: what separates the two verbs here is the direction
// set alone. Scrolls stay vertical because every scrollable container gets a
// candidate; swipes reach sideways because swipe-to-dismiss does.
const host = stubHost("android", POINTS);
const scrolls = builtinCandidates("scrolls", host);
const swipes = builtinCandidates("swipes", host);
const scrollDirections = new Set(
scrolls.map(
(entry) => (entry.action as ActionDescriptor & { kind: "Scroll" }).direction,
),
);
const swipeDirections = new Set(swipes.map((entry) => entry.swipe!.direction));
assert.deepEqual([...scrollDirections].sort(), ["down", "up"]);
assert.deepEqual([...swipeDirections].sort(), ["down", "left", "right", "up"]);
assert.equal(scrolls.length, POINTS.length * 2);
assert.equal(swipes.length, POINTS.length * 4);
});
test("scrolls draw one index over the enumerated candidates", () => {
resetWarnings();
const host = stubHost("android", POINTS);
const enumerated = builtinCandidates("scrolls", host);
const oracle = new Pcg(99n, 0n);
const index = oracle.intN(enumerated.length);
const action = walk(builtin("scrolls"), new Pcg(99n, 0n), host);
assert.deepEqual(action, enumerated[index]!.action);
});
test("swipes draw candidate index then magnitude, in that order", () => {
resetWarnings();
const host = stubHost("android", POINTS);
const enumerated = builtinCandidates("swipes", host);
const oracle = new Pcg(7n, 0n);
const index = oracle.intN(enumerated.length);
const magnitude = 200 + oracle.intN(401);
const picked = enumerated[index]!.swipe!;
const action = walk(builtin("swipes"), new Pcg(7n, 0n), host);
assert.deepEqual(
action,
swipeCandidate(picked.origin, picked.direction, magnitude),
);
});
test("doubleTaps and longPresses draw exactly one candidate index", () => {
@@ -174,14 +250,28 @@ test("waitOnce emits a 500ms wait and draws nothing", () => {
assert.notEqual(before, after);
});
test("pressKeys on native draws from NATIVE_PRESS_KEYS", () => {
test("pressKeys enumerates the platform's whole key pool", () => {
resetWarnings();
for (const [platform, keys] of [
["android", NATIVE_PRESS_KEYS],
["web", WEB_PRESS_KEYS],
] as const) {
const enumerated = builtinCandidates("pressKeys", stubHost(platform, POINTS));
assert.deepEqual(
enumerated,
keys.map((key) => ({ action: { kind: "PressKey", key }, targetIndex: -1 })),
);
}
});
test("pressKeys on native emits the only key without drawing", () => {
resetWarnings();
const host = stubHost("android", POINTS);
const rng = new Pcg(42n, 0n);
const oracle = new Pcg(42n, 0n);
const index = oracle.intN(NATIVE_PRESS_KEYS.length);
const action = walk(builtin("pressKeys"), rng, host);
assert.deepEqual(action, { kind: "PressKey", key: NATIVE_PRESS_KEYS[index] });
assert.deepEqual(action, { kind: "PressKey", key: NATIVE_PRESS_KEYS[0] });
// One key is no choice, so the pool consumed no draw.
assert.equal(rng.float64(), new Pcg(42n, 0n).float64());
});
test("pressKeys on web draws from WEB_PRESS_KEYS", () => {
@@ -194,6 +284,97 @@ test("pressKeys on web draws from WEB_PRESS_KEYS", () => {
assert.deepEqual(action, { kind: "PressKey", key: WEB_PRESS_KEYS[index] });
});
test("an unsupported verb enumerates nothing and reports once", () => {
resetWarnings();
// No platform in the matrix declines a verb today, so the branch is reached
// through a platform the matrix has never heard of.
const host = stubHost("desktop" as Platform, POINTS);
assert.deepEqual(builtinCandidates("taps", host), []);
assert.deepEqual(builtinCandidates("taps", host), []);
assert.deepEqual(host.unsupported, ["taps"]);
});
test("the seeded pick is an index into the shared enumeration", () => {
resetWarnings();
// Whatever the verb, the drawn action is one of the enumerated entries: the
// seeded policy adds a choice, never an action the model policy cannot see.
// `typing` and `swipes` are covered separately, being the two verbs whose
// enumeration leaves one value for the policy to fill in.
const host = stubHost("android", POINTS);
for (const verb of [
"taps",
"doubleTaps",
"longPresses",
"scrolls",
"pressKeys",
"waitOnce",
] as const) {
const enumerated = builtinCandidates(verb, host).map((entry) =>
JSON.stringify(entry.action),
);
for (let seed = 1; seed <= 50; seed++) {
const action = walk(builtin(verb), new Pcg(BigInt(seed), 0n), host);
assert.ok(
enumerated.includes(JSON.stringify(action)),
`${verb} drew ${JSON.stringify(action)}, which is not an enumerated candidate`,
);
}
}
});
test("typing enumerates the field and the policy supplies the text", () => {
resetWarnings();
const host = stubHost("android", POINTS);
const enumerated = builtinCandidates("typing", host);
// The candidate set names fields only: an empty text is the slot the seeded
// corpus draw and the model's own value both fill.
for (const entry of enumerated) {
assert.equal((entry.action as ActionDescriptor & { kind: "InputText" }).text, "");
}
const oracle = new Pcg(2024n, 0n);
const index = oracle.intN(enumerated.length);
const text = INPUT_CORPUS[oracle.intN(INPUT_CORPUS.length)];
assert.deepEqual(walk(builtin("typing"), new Pcg(2024n, 0n), host), {
...enumerated[index]!.action,
text,
});
});
test("swipes enumerate origin and direction, the policy adds distance", () => {
resetWarnings();
const host = stubHost("android", POINTS);
const origins = new Set(
builtinCandidates("swipes", host).map((entry) =>
JSON.stringify([entry.swipe!.origin, entry.swipe!.direction]),
),
);
const drawn = new Set<Direction>();
for (let seed = 1; seed <= 200; seed++) {
const action = walk(builtin("swipes"), new Pcg(BigInt(seed), 0n), host) as
ActionDescriptor & { kind: "Swipe"; from: Point; to: Point };
assert.equal(action.kind, "Swipe");
const direction = swipeDirection(action.from, action.to);
drawn.add(direction);
assert.ok(
origins.has(JSON.stringify([action.from, direction])),
`swipe from ${JSON.stringify(action.from)} going ${direction} is not enumerated`,
);
// The drawn distance stays inside 200..600, clamped at the screen edge.
const horizontal = direction === "left" || direction === "right";
const distance = horizontal
? Math.abs(action.to.x - action.from.x)
: Math.abs(action.to.y - action.from.y);
const clamped = horizontal ? action.to.x === 0 : action.to.y === 0;
assert.ok(distance <= 600, `drag of ${distance}px exceeds the drawn range`);
assert.ok(
distance >= 200 || clamped,
`drag of ${distance}px is under the drawn range and not clamped`,
);
}
// Every enumerated direction is reachable by a draw, sideways included.
assert.deepEqual([...drawn].sort(), ["down", "left", "right", "up"]);
});
test("empty candidate list yields null without drawing", () => {
resetWarnings();
const host = stubHost("android", []);
+1 -1
View File
@@ -19,7 +19,7 @@ function countingHost(platform: "android" | "ios" | "web"): Host & { calls: Buil
return {
calls,
platform: () => platform,
queryCandidates: () => [],
queryTargets: () => [],
reportUnsupported: (verb) => {
calls.push(verb);
},
+88
View File
@@ -0,0 +1,88 @@
// A minimal stand-in for the DOM surface the web host reads, shared by the web
// runtime's own tests and the cross-host eligibility test. The host asks the
// document for three things -- every element, the tappable set, the editable set
// -- and reads geometry, `disabled` and the scroll extents off each element, so
// that is all a fake has to answer.
import { __testing__ } from "../src/web-runtime.ts";
const { TAPPABLE_SELECTOR, EDITABLE_SELECTOR } = __testing__;
export interface FakeElementSpec {
tag: string;
x: number;
y: number;
width: number;
height: number;
// clickable/editable place the element in the selector sets the host queries;
// the fake answers those queries directly rather than matching CSS.
clickable?: boolean;
editable?: boolean;
disabled?: boolean;
// overflows makes the element's content taller than its box, which is how the
// host decides an element is scrollable.
overflows?: boolean;
}
export interface FakeElement extends FakeElementSpec {
tagName: string;
type: string;
isContentEditable: boolean;
scrollHeight: number;
clientHeight: number;
scrollWidth: number;
clientWidth: number;
getBoundingClientRect(): {
left: number;
top: number;
width: number;
height: number;
right: number;
bottom: number;
};
}
export function fakeElement(spec: FakeElementSpec): FakeElement {
const editable = spec.editable ?? false;
return {
...spec,
tagName: spec.tag.toUpperCase(),
type: spec.tag === "input" ? "text" : "",
isContentEditable: editable && spec.tag !== "input" && spec.tag !== "textarea",
scrollHeight: spec.overflows ? spec.height * 2 : spec.height,
clientHeight: spec.height,
scrollWidth: spec.width,
clientWidth: spec.width,
getBoundingClientRect: () => ({
left: spec.x,
top: spec.y,
width: spec.width,
height: spec.height,
right: spec.x + spec.width,
bottom: spec.y + spec.height,
}),
};
}
// withFakeDocument installs a document answering the host's three queries over
// `elements`, resets the host's per-tick cache, and restores the real document
// afterwards.
export function withFakeDocument(elements: FakeElement[], run: () => void): void {
const global = globalThis as Record<string, unknown>;
const original = global.document;
const answers: Record<string, FakeElement[]> = {
"*": elements,
[TAPPABLE_SELECTOR]: elements.filter((element) => element.clickable),
[EDITABLE_SELECTOR]: elements.filter((element) => element.editable),
};
global.document = {
querySelectorAll: (selector: string) => answers[selector] ?? [],
};
__testing__.resetTargetCache();
try {
run();
} finally {
__testing__.resetTargetCache();
global.document = original;
}
}
+67 -68
View File
@@ -83,78 +83,60 @@ test("installRuntime defined the host-invoked globals", () => {
assert.equal(typeof g.__sanderling__, "object");
});
// A button and a text input, each with a deterministic bounding box, exercise
// the per-verb selector routing without a full DOM.
function fakeElement(tag: string, rect: { x: number; y: number; w: number; h: number }) {
return {
tagName: tag.toUpperCase(),
disabled: false,
isContentEditable: false,
type: tag === "input" ? "text" : "",
scrollHeight: 0,
clientHeight: 0,
scrollWidth: 0,
clientWidth: 0,
getBoundingClientRect: () => ({
left: rect.x,
top: rect.y,
width: rect.w,
height: rect.h,
right: rect.x + rect.w,
bottom: rect.y + rect.h,
}),
};
}
const { fakeElement, withFakeDocument } = await import("./web-dom-harness.ts");
function withFakeDocument(map: Record<string, unknown[]>, run: () => void) {
const g = globalThis as Record<string, unknown>;
const original = g.document;
g.document = {
querySelectorAll: (selector: string) => map[selector] ?? [],
scrollingElement: null,
documentElement: null,
};
try {
run();
} finally {
g.document = original;
}
}
test("queryCandidates routes taps to the tappable selector set", () => {
const button = fakeElement("button", { x: 10, y: 20, w: 40, h: 8 });
withFakeDocument(
{ 'a, button, input, select, textarea, [role="button"], [onclick]': [button] },
() => {
__testing__.resetCandidateCache();
const candidates = host.queryCandidates("taps");
assert.equal(candidates.length, 1);
assert.deepEqual({ x: candidates[0]!.x, y: candidates[0]!.y }, { x: 30, y: 24 });
},
);
});
test("queryCandidates routes typing to editable inputs only", () => {
const input = fakeElement("input", { x: 0, y: 0, w: 100, h: 20 });
withFakeDocument({ "input, textarea, [contenteditable]": [input] }, () => {
__testing__.resetCandidateCache();
const candidates = host.queryCandidates("typing");
assert.equal(candidates.length, 1);
assert.deepEqual({ x: candidates[0]!.x, y: candidates[0]!.y }, { x: 50, y: 10 });
// The host reports facts and never routes verbs: which of these a verb may act
// on is decided by the shared rule in src/targets.ts, exercised across both
// engines by host-parity.test.ts.
test("queryTargets reports the tappable selector set as clickable", () => {
const button = fakeElement({ tag: "button", x: 10, y: 20, width: 40, height: 8, clickable: true });
const plain = fakeElement({ tag: "div", x: 0, y: 0, width: 100, height: 100 });
withFakeDocument([button, plain], () => {
const targets = host.queryTargets();
assert.equal(targets.length, 2);
assert.equal(targets[0]!.clickable, true);
assert.deepEqual({ x: targets[0]!.x, y: targets[0]!.y }, { x: 30, y: 24 });
assert.equal(targets[1]!.clickable, false);
});
});
test("queryCandidates caches within a tick until reset", () => {
const first = fakeElement("button", { x: 0, y: 0, w: 10, h: 10 });
withFakeDocument(
{ 'a, button, input, select, textarea, [role="button"], [onclick]': [first] },
() => {
__testing__.resetCandidateCache();
const a = host.queryCandidates("taps");
const b = host.queryCandidates("taps");
assert.equal(a, b);
},
);
test("queryTargets reports only real text inputs as editable", () => {
const input = fakeElement({ tag: "input", x: 0, y: 0, width: 100, height: 20, editable: true });
const checkbox = fakeElement({ tag: "input", x: 0, y: 40, width: 20, height: 20, editable: true });
checkbox.type = "checkbox";
withFakeDocument([input, checkbox], () => {
const targets = host.queryTargets();
assert.equal(targets[0]!.editable, true);
assert.deepEqual({ x: targets[0]!.x, y: targets[0]!.y }, { x: 50, y: 10 });
assert.equal(targets[1]!.editable, false);
});
});
// A disabled control used to be dropped from every verb's candidates, because
// the web host folded `disabled` into its visibility check. It is a fact of its
// own now, so `taps` still skips it while `swipes` can still start on it, which
// is what the native host has always done.
test("queryTargets reports a disabled control rather than dropping it", () => {
const disabled = fakeElement({
tag: "button", x: 0, y: 0, width: 40, height: 20, clickable: true, disabled: true,
});
withFakeDocument([disabled], () => {
const targets = host.queryTargets();
assert.equal(targets.length, 1);
assert.equal(targets[0]!.clickable, true);
assert.equal(targets[0]!.enabled, false);
});
});
test("queryTargets caches within a tick until reset", () => {
const button = fakeElement({ tag: "button", x: 0, y: 0, width: 10, height: 10, clickable: true });
withFakeDocument([button], () => {
const first = host.queryTargets();
const second = host.queryTargets();
assert.equal(first, second);
__testing__.resetTargetCache();
assert.notEqual(host.queryTargets(), first);
});
});
// evaluateExtractors builds State, which references document and window.
@@ -328,6 +310,23 @@ test("selectorFromObject maps known keys to their canonical attribute", () => {
});
});
// Compose Multiplatform emits its testTag into `id`, which the native table
// already accepts via the resource-id alias. The web table must not be the one
// place that rejects it.
test("selectorFromObject resolves testTag through data-testid or id", () => {
assert.deepEqual(selectorFromObject({ testTag: "LoginSubmit" }), {
css: `:is([data-testid="LoginSubmit"], [id="LoginSubmit"])`,
});
});
// Multi-key selectors concatenate their parts into one compound, so the
// two-attribute testTag match has to stay a single compound piece.
test("selectorFromObject composes testTag with a second key", () => {
assert.deepEqual(selectorFromObject({ testTag: "Row", "aria-label": "first" }), {
css: `:is([data-testid="Row"], [id="Row"])[aria-label="first"]`,
});
});
test("selectorFromObject falls back to a literal attribute for unknown keys", () => {
assert.deepEqual(selectorFromObject({ "data-foo": "bar" }), {
css: `[data-foo="bar"]`,
+22 -1
View File
@@ -59,9 +59,30 @@ function WitnessView({
)}
</div>
) : null}
{witness.detected_step && witness.detected_step !== witness.step ? (
<div className="violations-panel-witness-line">
<span className="violations-panel-witness-key">detected at</span>
{onJumpToStep ? (
<button
type="button"
className="violations-panel-witness-step"
onClick={() => onJumpToStep(witness.detected_step as number)}
>
step {witness.detected_step}
</button>
) : (
<span className="violations-panel-witness-value">
step {witness.detected_step}
</span>
)}
</div>
) : null}
{evidence.length > 0 ? (
<details className="violations-panel-residual" open={open}>
<summary>witness</summary>
<summary>
witness
{witness.detected_step ? ` (state at step ${witness.detected_step})` : ""}
</summary>
<dl className="violations-panel-witness-evidence">
{evidence.map(([name, value]) => (
<div key={name} className="violations-panel-witness-line">
+3
View File
@@ -128,6 +128,9 @@ export interface Witness {
// which for deferred obligations (next, eventually) is earlier than the
// step whose record carries the witness.
step?: number;
// detected_step is the step whose evaluation produced the violation. The
// extractors below are that step's state, not step's.
detected_step?: number;
extractors?: Record<string, unknown>;
}