fix ltl semantics and unify action enumeration (#71)

* fix(ltl): give every thunk a construction identity

Two distinct unnamed predicates both described as "Thunk(...)", so obligation
collapse merged their residuals and could drop a live violation. Identity is
assigned at construction and the fields are unexported, so a thunk cannot be
built without one.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(ltl): reduce a thrown-predicate residual instead of panicking

The verifier substitutes an ErrorFormula for the residual of a property whose
predicate threw, and that residual is fed back in on the next step. reduce had
no case for it, so the run crashed. It re-reports the same failure now.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(ltl): make a bounded always the dual of a bounded eventually

G<=n(f) and not F<=n(not f) disagreed on traces where the inner was still
pending when the window closed, so nnf's negation normal form was not semantics
preserving. Both sides now range over the observations at which their inner can
definitely resolve: the eventually keeps a pending inner as a disjunct, and the
always discharges vacuously at window close.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(ltl): arm a one-shot root once per run

A root that carries its own horizon is one obligation for the whole run, not one
per observation. Re-instantiating a top-level eventually monitored G F<=n(p)
instead of F<=n(p) and left one live obligation per step behind; a bounded
always restarted its window every step and never closed.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(verifier): stop wrapping a top-level eventually in always

`eventually(p).within(300, "seconds")` as a property meant "within 300 seconds
of every step", which spawned an obligation per step with its own resolved
deadline. A 553-step run carried 553 of them and serialized a 75 KB residual.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(ltl): serialize the resolved deadline of a bounded window

Two obligations spawned at different steps from one duration-bounded formula
differ only in the deadline the evaluator resolved for them, so they serialized
identically and the trace erased a distinction the evaluator makes. The authored
window stays in amount/unit; the resolved deadline rides alongside.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(verifier): split a witness's origin step from its detection step

A deferred obligation spans two steps: the one that armed it and the one whose
reduction failed. They were conflated under one index, so the extractor snapshot
(which is the detecting step's state) was reported against the origin step.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(runner): record a witness's detection step in the trace

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* feat(replay-ui): show the step a violation was detected at

The witness evidence is the detecting step's state, so say which step that is
and let a reader jump to it.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(verifier): record the extractor state the predicates actually read

On the web path extractor bodies are evaluated in V8 and injected here, but only
the goja value was replaced. The trace diff and the violation witness therefore
described a state no property ever saw.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* refactor(spec): one candidate producer over one target-eligibility rule

Both hosts routed verbs themselves and both policies enumerated their own
actions, and all four drifted. Web sent `swipes` to scrollable containers only,
so swipe-to-dismiss on a list row was reachable on native and unreachable on
web; the model policy folded gestures its own way and could not reach what the
seeded picker drew.

A host now reports facts about every element and never decides which verb may
act on it: targets.ts acceptsTarget owns that for both. pick.ts builtinCandidates
is the single enumeration, and the model policy reads it through
__sanderlingEnumerateBuiltin__ instead of reimplementing it in Go.

Gesture verbs change with it: scrolls stay vertical over scrollable containers,
swipes go free-form in all four directions from any element with real bounds.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(runner): name a builtin scroll by its drag origin

A builtin gesture carries endpoints and no selector, so every scroll rendered as
"Scroll down " in the prompt's recent-action memory and two scrollable regions
were indistinguishable.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(chrome): clear storage over cdp instead of scripting an opaque origin

Launch runs while the tab is still on about:blank, whose opaque origin denies
storage access, so localStorage.clear() threw SecurityError and every web run
died at launch. Storage.clearDataForOrigin needs no navigation. The exception
helper lands here because "Uncaught" is what hid this for so long.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(chrome): enable the swiftshader webgl fallback

Headless Chrome runs with --disable-gpu, and without this flag it refuses the
software WebGL backend: getContext returns null, so a canvas-rendered app paints
nothing and every screenshot is identical black.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* fix(web): resolve testTag through data-testid or id

Compose Multiplatform emits its testTag into the element id, which the native
table already accepts via the resource-id alias. The two web selector tables
were the only place that rejected it.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* test(spec): type-check the spec api as part of make test

The fake runtime in api.test.ts did not return a chainable handle from extract,
so the file had not type-checked since named() was added. Wiring the check into
make test stops it drifting again.

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J

* docs(manual): one-shot eventually and the gesture verbs

Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J
This commit is contained in:
pj authored and GitHub committed 2026-08-12 18:06:04 +05:30
1 parent 7343085614
commit 26b49b379a
48 files changed
+2556 -664

No files matched your search

+5 -2
View File
@@ -29,7 +29,7 @@ WEB_DIST := replay-ui/dist
GOLINES := $(shell $(GO) env GOPATH)/bin/golines GOLINES := $(shell $(GO) env GOPATH)/bin/golines
.PHONY: bootstrap proto sidecar sanderling install test test-go test-browser test-companion test-kotlin test-spec-api web-test web-typecheck web-build web-dev replay-dev docs clean release-cli release-npm-dry fmt fmt-go fmt-kotlin fmt-ts fmt-swift .PHONY: bootstrap proto sidecar sanderling install test test-go test-browser test-companion test-kotlin test-spec-api spec-typecheck web-test web-typecheck web-build web-dev replay-dev docs clean release-cli release-npm-dry fmt fmt-go fmt-kotlin fmt-ts fmt-swift
bootstrap: bootstrap:
$(GO) mod download $(GO) mod download
@@ -101,7 +101,7 @@ fmt-ts:
fmt-swift: fmt-swift:
xcrun swift-format format -i -r companion/Sources xcrun swift-format format -i -r companion/Sources
test: test-go test-spec-api web-typecheck web-test test: test-go spec-typecheck test-spec-api web-typecheck web-test
test-go: test-go:
$(GO) test $(GO_PACKAGES) $(GO) test $(GO_PACKAGES)
@@ -126,6 +126,9 @@ test-kotlin:
test-spec-api: test-spec-api:
cd pkg/spec && npm test --silent cd pkg/spec && npm test --silent
spec-typecheck:
cd pkg/spec && npm run check --silent
docs: $(DOCS_OUT) build/site/_assets docs: $(DOCS_OUT) build/site/_assets
@echo "built $(words $(DOCS_OUT)) pages to build/site" @echo "built $(words $(DOCS_OUT)) pages to build/site"
+13 -3
View File
@@ -171,6 +171,8 @@ Extractors are evaluated before properties and action generators. Use `.previous
| `now(f)` | `f` evaluated at the current step (for use inside `always`/`next` bodies) | | `now(f)` | `f` evaluated at the current step (for use inside `always`/`next` bodies) |
| `next(f)` | `f` evaluated at the step immediately after the current one | | `next(f)` | `f` evaluated at the step immediately after the current one |
A property that is an `eventually` at the top level is one goal for the whole run: it is armed at the first step and discharged for good the first time it holds. Written inside `always(...)` the same formula is armed again at every step, which asks for the window to be met from every step in the run.
**Formula combinators** - available on every `Formula`: **Formula combinators** - available on every `Formula`:
| Method | Meaning | | Method | Meaning |
@@ -207,11 +209,19 @@ On web, `"back"` maps to Backspace and `"home"` is not supported. All other keys
| `doubleTaps` | Random double tap on a clickable element | | `doubleTaps` | Random double tap on a clickable element |
| `longPresses` | Random long press on a clickable element | | `longPresses` | Random long press on a clickable element |
| `typing` | Types a value from the edge-case corpus into a random editable field | | `typing` | Types a value from the edge-case corpus into a random editable field |
| `scrolls` | Random scroll gesture | | `scrolls` | Scrolls a random scrollable container up or down |
| `swipes` | Random swipe gesture | | `swipes` | Random up, down, left, or right swipe from any visible element |
| `waitOnce` | Idles one step | | `waitOnce` | Idles one step |
| `pressKeys` | Presses a random supported key | | `pressKeys` | Presses a random supported key |
`scrolls` anchors on a scrollable container and moves its content up or down. It never
scrolls sideways, because every scrollable container on screen gets a candidate and a
sideways pair doubles that list for little return; write `Scroll({ in, direction })` when
you need one.
`swipes` is a free drag of 200 to 600 px from any element with real bounds, in any of the
four directions. The sideways ones are what reach swipe-to-dismiss and swipe-to-delete on
a list row.
### `actions(generator)` ### `actions(generator)`
Wraps a callback that returns `Action[]`. The callback runs each step the generator is eligible. Wraps a callback that returns `Action[]`. The callback runs each step the generator is eligible.
@@ -280,7 +290,7 @@ export const generator = llm({
Set `OPENROUTER_API_KEY` or `OPENAI_API_KEY` (OpenRouter wins if both are set). With a plain OpenAI key, drop the vendor prefix from the model id. The model needs image input and strict `json_schema` structured output. Set `OPENROUTER_API_KEY` or `OPENAI_API_KEY` (OpenRouter wins if both are set). With a plain OpenAI key, drop the vendor prefix from the model id. The model needs image input and strict `json_schema` structured output.
Each step it gets a screenshot plus a numbered list of the concrete actions your tree yields right now, each tagged with its weight, and picks one number. `instructions` are appended to the prompt: say what the app is, not how to test it — the model works that part out. Everything else is unchanged. Setup actions still run first, typing still falls back to the edge-case corpus when the model supplies no text, and the trace records the reasoning, the chosen number, and `source: "llm"` so the replay UI can show why each pick happened. Each step it gets a screenshot plus a numbered list of the concrete actions your tree yields right now, each tagged with its weight, and picks one number. That list is the seeded picker's own candidate enumeration, so both modes explore the same action space and only the choice differs. `instructions` are appended to the prompt: say what the app is, not how to test it; the model works that part out. Everything else is unchanged. Setup actions still run first, typing still falls back to the edge-case corpus when the model supplies no text, and the trace records the reasoning, the chosen number, and `source: "llm"` so the replay UI can show why each pick happened.
It is one model call per step, so keep `--duration` modest. It is one model call per step, so keep `--duration` modest.
+69 -11
View File
@@ -5,6 +5,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"net/url"
"strconv" "strconv"
"strings" "strings"
"sync" "sync"
@@ -14,6 +15,7 @@ import (
"github.com/chromedp/cdproto/network" "github.com/chromedp/cdproto/network"
"github.com/chromedp/cdproto/page" "github.com/chromedp/cdproto/page"
"github.com/chromedp/cdproto/runtime" "github.com/chromedp/cdproto/runtime"
"github.com/chromedp/cdproto/storage"
"github.com/chromedp/chromedp" "github.com/chromedp/chromedp"
"github.com/chromedp/chromedp/kb" "github.com/chromedp/chromedp/kb"
@@ -37,6 +39,11 @@ func New() *Driver {
append(chromedp.DefaultExecAllocatorOptions[:], append(chromedp.DefaultExecAllocatorOptions[:],
chromedp.Flag("headless", true), chromedp.Flag("headless", true),
chromedp.Flag("disable-gpu", true), chromedp.Flag("disable-gpu", true),
// Chrome refuses to fall back to the SwiftShader WebGL backend
// without this flag, so with --disable-gpu a canvas app (Compose
// for Web, Flutter web, anything on WebGL) gets a null context and
// paints nothing: black screenshots and an empty accessibility DOM.
chromedp.Flag("enable-unsafe-swiftshader", true),
chromedp.NoSandbox, chromedp.NoSandbox,
// CI runners give Chrome a tiny /dev/shm; without this the browser // CI runners give Chrome a tiny /dev/shm; without this the browser
// process hangs on startup and never reports its DevTools socket. // process hangs on startup and never reports its DevTools socket.
@@ -91,17 +98,8 @@ func New() *Driver {
func (d *Driver) Launch(ctx context.Context, bundleID string, clearState bool, _ map[string]string) error { func (d *Driver) Launch(ctx context.Context, bundleID string, clearState bool, _ map[string]string) error {
if clearState { if clearState {
if err := chromedp.Run(d.tabCtx, network.ClearBrowserCookies()); err != nil { if err := d.clearState(bundleID); err != nil {
return fmt.Errorf("clear cookies: %w", err)
}
if err := chromedp.Run(d.tabCtx, chromedp.ActionFunc(func(ctx context.Context) error {
_, exp, err := runtime.Evaluate(`localStorage.clear(); sessionStorage.clear();`).Do(ctx)
if exp != nil {
return fmt.Errorf("clear storage: %s", exp.Text)
}
return err return err
})); err != nil {
return fmt.Errorf("clear storage: %w", err)
} }
} }
if err := chromedp.Run(d.tabCtx, chromedp.Navigate(bundleID)); err != nil { if err := chromedp.Run(d.tabCtx, chromedp.Navigate(bundleID)); err != nil {
@@ -125,6 +123,66 @@ func (d *Driver) Launch(ctx context.Context, bundleID string, clearState bool, _
return nil return nil
} }
// clearState wipes the target's stored data before the application loads.
// Script cannot do it: the tab still sits on about:blank, whose opaque origin
// denies storage access, so `localStorage.clear()` throws SecurityError and
// every web run dies at launch. The Storage domain clears by origin instead,
// which needs no navigation. sessionStorage is per-tab and outside that
// domain's reach; it only survives when a relaunch reuses a tab already on
// the target origin, which is the one case where script can reach it.
func (d *Driver) clearState(bundleID string) error {
if err := chromedp.Run(d.tabCtx, network.ClearBrowserCookies()); err != nil {
return fmt.Errorf("clear cookies: %w", err)
}
origin := securityOrigin(bundleID)
if origin == "" {
return nil
}
clearForOrigin := storage.ClearDataForOrigin(origin, string(storage.TypeAll))
if err := chromedp.Run(d.tabCtx, clearForOrigin); err != nil {
return fmt.Errorf("clear storage for %s: %w", origin, err)
}
script := fmt.Sprintf(
`location.origin === %q && (sessionStorage.clear(), true)`, origin)
return chromedp.Run(d.tabCtx, chromedp.ActionFunc(func(ctx context.Context) error {
_, exception, err := runtime.Evaluate(script).Do(ctx)
if err != nil {
return fmt.Errorf("clear session storage: %w", err)
}
if exception != nil {
return fmt.Errorf("clear session storage: %s", exceptionMessage(exception))
}
return nil
}))
}
// securityOrigin returns the scheme://host[:port] the Storage domain keys data
// by, or "" for a target that has no such origin (data:, file:, about:blank),
// where there is no per-origin storage to clear.
func securityOrigin(bundleID string) string {
parsed, err := url.Parse(bundleID)
if err != nil || parsed.Host == "" {
return ""
}
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return ""
}
return parsed.Scheme + "://" + parsed.Host
}
// exceptionMessage renders a page exception for an error string. The
// description carries the actual message ("SecurityError: Failed to read the
// 'localStorage' property..."); Text alone is the useless "Uncaught".
func exceptionMessage(exception *runtime.ExceptionDetails) string {
if exception == nil {
return ""
}
if exception.Exception != nil && exception.Exception.Description != "" {
return exception.Exception.Description
}
return exception.Text
}
func (d *Driver) Terminate(_ context.Context) error { func (d *Driver) Terminate(_ context.Context) error {
d.tabCancel() d.tabCancel()
d.allocCancel() d.allocCancel()
@@ -483,7 +541,7 @@ func (d *Driver) InstallBundle(ctx context.Context, source []byte) error {
return fmt.Errorf("evaluate bundle: %w", err) return fmt.Errorf("evaluate bundle: %w", err)
} }
if exception != nil { if exception != nil {
return fmt.Errorf("bundle threw: %s", exception.Text) return fmt.Errorf("bundle threw: %s", exceptionMessage(exception))
} }
return nil return nil
}), }),
+74
View File
@@ -6,10 +6,84 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
"net/http"
"net/http/httptest"
"testing" "testing"
"time" "time"
"github.com/chromedp/chromedp"
) )
// TestLaunch_ClearStateWipesStorageForTheTargetOrigin covers the CLI's default
// path (--clear-data). The tab sits on about:blank when Launch runs, an opaque
// origin that denies storage access, so clearing by script there throws
// SecurityError and kills every web run before the app loads.
func TestLaunch_ClearStateWipesStorageForTheTargetOrigin(t *testing.T) {
const page = `<body><script>
const visits = Number(localStorage.getItem("visits") ?? "0") + 1;
localStorage.setItem("visits", String(visits));
sessionStorage.setItem("tab", "dirty");
</script></body>`
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "text/html")
_, _ = w.Write([]byte(page))
}))
defer server.Close()
d := New()
defer d.Terminate(context.Background())
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
if err := d.Launch(ctx, server.URL, true, nil); err != nil {
t.Fatalf("Launch with clearState on a fresh tab: %v", err)
}
if err := d.Launch(ctx, server.URL, false, nil); err != nil {
t.Fatalf("Launch: %v", err)
}
var visits string
if err := chromedp.Run(d.tabCtx,
chromedp.Evaluate(`localStorage.getItem("visits")`, &visits)); err != nil {
t.Fatalf("read localStorage: %v", err)
}
if visits != "2" {
t.Fatalf("visits = %q, want 2 (two loads, storage kept)", visits)
}
if err := d.Launch(ctx, server.URL, true, nil); err != nil {
t.Fatalf("Launch with clearState on the target origin: %v", err)
}
if err := chromedp.Run(d.tabCtx,
chromedp.Evaluate(`localStorage.getItem("visits")`, &visits)); err != nil {
t.Fatalf("read localStorage: %v", err)
}
if visits != "1" {
t.Errorf("visits = %q, want 1 (storage cleared before the app loaded)", visits)
}
}
// TestLaunch_WebGLContextIsAvailable pins the SwiftShader fallback. Headless
// Chrome runs with --disable-gpu, and without --enable-unsafe-swiftshader it
// refuses the software WebGL backend: getContext returns null, so a
// canvas-rendered app paints nothing and every screenshot is identical black.
func TestLaunch_WebGLContextIsAvailable(t *testing.T) {
d := New()
defer d.Terminate(context.Background())
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
if err := d.Launch(ctx, "data:text/html,<body></body>", false, nil); err != nil {
t.Fatalf("Launch: %v", err)
}
var hasContext bool
if err := chromedp.Run(d.tabCtx, chromedp.Evaluate(
`!!document.createElement("canvas").getContext("webgl2")`, &hasContext)); err != nil {
t.Fatalf("evaluate: %v", err)
}
if !hasContext {
t.Error("webgl2 context is null; a canvas-rendered app would render nothing")
}
}
// TestActionMethods_HonorCallerCancellation confirms the DeviceDriver action // TestActionMethods_HonorCallerCancellation confirms the DeviceDriver action
// methods route through runCtx so a cancelled caller context aborts the CDP // methods route through runCtx so a cancelled caller context aborts the CDP
// round-trip instead of blocking on d.tabCtx. Without this a hung browser would // round-trip instead of blocking on d.tabCtx. Without this a hung browser would
+47
View File
@@ -0,0 +1,47 @@
package chrome
import (
"testing"
"github.com/chromedp/cdproto/runtime"
)
func TestSecurityOrigin(t *testing.T) {
cases := map[string]string{
"http://localhost:8088": "http://localhost:8088",
"http://localhost:5173/": "http://localhost:5173",
"https://app.example.com/a/b?c=d": "https://app.example.com",
"data:text/html,<body>hi</body>": "",
"file:///tmp/index.html": "",
"about:blank": "",
"": "",
"://not a url": "",
}
for target, want := range cases {
if got := securityOrigin(target); got != want {
t.Errorf("securityOrigin(%q) = %q, want %q", target, got, want)
}
}
}
// TestExceptionMessage_PrefersDescription pins the useful half of a page
// exception. ExceptionDetails.Text is "Uncaught" for every throw; the
// description is where the error a caller can act on lives.
func TestExceptionMessage_PrefersDescription(t *testing.T) {
withDescription := &runtime.ExceptionDetails{
Text: "Uncaught",
Exception: &runtime.RemoteObject{
Description: "SecurityError: Access is denied for this document.",
},
}
if got := exceptionMessage(withDescription); got != withDescription.Exception.Description {
t.Errorf("exceptionMessage = %q, want the description", got)
}
textOnly := &runtime.ExceptionDetails{Text: "Uncaught SyntaxError"}
if got := exceptionMessage(textOnly); got != "Uncaught SyntaxError" {
t.Errorf("exceptionMessage = %q, want the text fallback", got)
}
if got := exceptionMessage(nil); got != "" {
t.Errorf("exceptionMessage(nil) = %q, want empty", got)
}
}
+7 -1
View File
@@ -42,7 +42,13 @@ func TranslateStringSelector(selector string) (string, bool, error) {
return `[aria-label="` + cssEscape(value) + `"]`, false, nil return `[aria-label="` + cssEscape(value) + `"]`, false, nil
case "descPrefix": case "descPrefix":
return `[aria-label^="` + cssEscape(value) + `"]`, false, nil return `[aria-label^="` + cssEscape(value) + `"]`, false, nil
case "testTag", "testID", "testid", "data-testid": case "testTag":
// Mirrors the in-page table and the native resource-id alias: a
// testTag reaches the DOM as data-testid or as an id, depending on
// the toolkit. `:is()` keeps this one compound selector.
escaped := cssEscape(value)
return `:is([data-testid="` + escaped + `"], [id="` + escaped + `"])`, false, nil
case "testID", "testid", "data-testid":
return `[data-testid="` + cssEscape(value) + `"]`, false, nil return `[data-testid="` + cssEscape(value) + `"]`, false, nil
case "placeholder", "placeholderValue", "hintText": case "placeholder", "placeholderValue", "hintText":
return `[placeholder="` + cssEscape(value) + `"]`, false, nil return `[placeholder="` + cssEscape(value) + `"]`, false, nil
+1 -1
View File
@@ -21,7 +21,7 @@ func TestTranslateStringSelector_KnownKeys(t *testing.T) {
{"accessibilityLabel:logout", `[aria-label="logout"]`, false}, {"accessibilityLabel:logout", `[aria-label="logout"]`, false},
{"aria-label:Sign in", `[aria-label="Sign in"]`, false}, {"aria-label:Sign in", `[aria-label="Sign in"]`, false},
{"descPrefix:account:", `[aria-label^="account:"]`, false}, {"descPrefix:account:", `[aria-label^="account:"]`, false},
{"testTag:submit", `[data-testid="submit"]`, false}, {"testTag:submit", `:is([data-testid="submit"], [id="submit"])`, false},
{"testID:submit", `[data-testid="submit"]`, false}, {"testID:submit", `[data-testid="submit"]`, false},
{"placeholder:Email", `[placeholder="Email"]`, false}, {"placeholder:Email", `[placeholder="Email"]`, false},
} }
+170
View File
@@ -0,0 +1,170 @@
package ltl
import (
"math/rand/v2"
"testing"
"testing/quick"
"time"
)
// dualityTrace holds the per-step truth of each predicate. The thunks read the
// current step out of the trace rather than counting their own invocations, so
// a formula and its negation see the same values no matter how many times each
// predicate is reduced.
type dualityTrace struct {
step int
values [2][]bool
}
func newDualityTrace(random *rand.Rand, steps int) *dualityTrace {
trace := &dualityTrace{}
for index := range trace.values {
trace.values[index] = make([]bool, steps)
for step := range trace.values[index] {
trace.values[index][step] = random.IntN(2) == 0
}
}
return trace
}
func (t *dualityTrace) atoms() []Formula {
return []Formula{
ThunkNamed("p0", func() (bool, error) { return t.values[0][t.step], nil }),
ThunkNamed("p1", func() (bool, error) { return t.values[1][t.step], nil }),
Pure(true),
Pure(false),
}
}
// randomFormula draws a formula of at most the given depth over the atoms.
// Every operator the evaluator can reduce is reachable, including the bounded
// Always that only nnf produces.
func randomFormula(random *rand.Rand, depth int, atoms []Formula) Formula {
if depth == 0 {
return atoms[random.IntN(len(atoms))]
}
inner := func() Formula { return randomFormula(random, depth-1, atoms) }
bound := random.IntN(3) + 1
switch random.IntN(12) {
case 0, 1:
return atoms[random.IntN(len(atoms))]
case 2:
return Not(inner())
case 3:
return Next(inner())
case 4:
return Now(inner())
case 5:
return And(inner(), inner())
case 6:
return Or(inner(), inner())
case 7:
return Implies(inner(), inner())
case 8:
return Always(inner())
case 9:
return Eventually(inner())
case 10:
if random.IntN(2) == 0 {
return EventuallyWithinSteps(inner(), bound)
}
return EventuallyWithin(inner(), time.Duration(bound)*time.Second)
default:
if random.IntN(2) == 0 {
return AlwaysFormula{Inner: inner(), StepBound: bound, HasStepBound: true}
}
return AlwaysFormula{Inner: inner(), Duration: time.Duration(bound) * time.Second}
}
}
// TestNNF_ExcludedMiddleHoldsOnRandomTraces is the semantic counterpart to the
// describe()-comparing laws in nnf_test.go: those check that nnf produces the
// syntax the dual laws predict, this checks that the syntax it produces means
// the same thing. If any operator pair reduces non-dually then some trace makes
// a formula and its own nnf-negation both violate, so their disjunction
// violates and their conjunction holds.
func TestNNF_ExcludedMiddleHoldsOnRandomTraces(t *testing.T) {
const steps = 8
law := func(seed uint64) bool {
random := rand.New(rand.NewPCG(seed, 0x9e3779b97f4a7c15))
trace := newDualityTrace(random, steps)
formula := randomFormula(random, 3, trace.atoms())
tautology := NewEvaluator(Or(formula, nnf(Not(formula))))
contradiction := NewEvaluator(And(formula, nnf(Not(formula))))
for index := range steps {
trace.step = index
now := time.Unix(int64(index), 0)
if tautology.ObserveAtStep(now, index+1) == VerdictViolated {
t.Logf("seed %d: %s violated at step %d", seed, Describe(formula), index+1)
return false
}
if contradiction.ObserveAtStep(now, index+1) == VerdictHolds {
t.Logf("seed %d: %s held at step %d", seed, Describe(formula), index+1)
return false
}
}
return tautology.Finalize() != VerdictViolated
}
if err := quick.Check(law, &quick.Config{MaxCount: 2000}); err != nil {
t.Error(err)
}
}
// TestNNF_BoundedAlwaysOverNextIsDual is the counterexample that showed nnf was
// not semantics preserving: phi = G<=1(X p) with p true only at the first step.
// Before the bounded operators were made dual, phi violated at step 2 and
// nnf(not phi) violated at step 1, so both a formula and its negation failed on
// one trace. Exactly one of the pair may violate.
func TestNNF_BoundedAlwaysOverNextIsDual(t *testing.T) {
step := 0
predicate := ThunkNamed("p", func() (bool, error) { return step == 0, nil })
formula := AlwaysFormula{Inner: Next(predicate), StepBound: 1, HasStepBound: true}
negated := nnf(Not(formula))
run := func(root Formula) Verdict {
step = 0
evaluator := NewEvaluator(root)
for index := range 3 {
step = index
if evaluator.ObserveAtStep(time.Unix(int64(index), 0), index+1) == VerdictViolated {
return VerdictViolated
}
}
return evaluator.Finalize()
}
if got := run(formula); got == VerdictViolated {
t.Errorf("G<=1(X p) = %v; the window closes on a deferred check, which is not a breach", got)
}
if got := run(negated); got != VerdictViolated {
t.Errorf("nnf(not G<=1(X p)) = %v, want violated", got)
}
if got := run(Or(formula, negated)); got == VerdictViolated {
t.Errorf("excluded middle violated: %v", got)
}
if got := run(And(formula, negated)); got != VerdictViolated {
t.Errorf("phi and not phi = %v, want violated", got)
}
}
// TestEventually_PendingInnerSurvivesAsDisjunct locks the conjunct/disjunct
// mirror the duality rests on: Always keeps a pending inner as a conjunct of
// its residual, so Eventually must keep one as a disjunct. Dropping it made an
// inner that can only discharge on a later step unsatisfiable.
func TestEventually_PendingInnerSurvivesAsDisjunct(t *testing.T) {
values := []bool{false, true, false}
step := 0
formula := EventuallyWithinSteps(Next(ThunkNamed("p", func() (bool, error) {
return values[step], nil
})), 2)
evaluator := NewEvaluator(formula)
if got := evaluator.ObserveAtStep(time.Unix(0, 0), 1); got != VerdictPending {
t.Fatalf("step 1: got %v, want pending", got)
}
step = 1
if got := evaluator.ObserveAtStep(time.Unix(1, 0), 2); got != VerdictHolds {
t.Errorf("step 2: got %v, want holds (X p armed at step 1 discharged here)", got)
}
}
+83 -22
View File
@@ -37,6 +37,10 @@ type Evaluator struct {
violated bool violated bool
steps int steps int
violation *Violation violation *Violation
// oneShot marks a root that is armed once at the first observation rather
// than re-asserted at every one; armed records that it has been.
oneShot bool
armed bool
} }
// obligation pairs a residual formula with the step that spawned it, so a // obligation pairs a residual formula with the step that spawned it, so a
@@ -62,7 +66,8 @@ type Violation struct {
} }
func NewEvaluator(formula Formula) *Evaluator { func NewEvaluator(formula Formula) *Evaluator {
return &Evaluator{root: nnf(formula)} normalized := nnf(formula)
return &Evaluator{root: normalized, oneShot: isOneShotRoot(normalized)}
} }
// Observe evaluates the formula against the current state and returns the // Observe evaluates the formula against the current state and returns the
@@ -89,8 +94,11 @@ func (e *Evaluator) ObserveAtStep(now time.Time, step int) Verdict {
} }
e.steps = step e.steps = step
fresh := obligation{formula: rootObligation(e.root), origin: step} obligations := make([]obligation, 0, len(e.pending)+1)
obligations := append(e.pending, fresh) obligations = append(obligations, e.pending...)
if formula, ok := e.instantiateRoot(); ok {
obligations = append(obligations, obligation{formula: formula, origin: step})
}
e.pending = e.pending[:0] e.pending = e.pending[:0]
for _, entry := range obligations { for _, entry := range obligations {
@@ -121,8 +129,11 @@ func (e *Evaluator) ObserveAtStep(now time.Time, step int) Verdict {
// collapse removes structurally-identical obligations, keeping the first // collapse removes structurally-identical obligations, keeping the first
// occurrence in order so the surviving entry carries the earliest origin step. // occurrence in order so the surviving entry carries the earliest origin step.
// Distinct predicates never merge because ThunkFormula's name participates in // Equal describe() keys mean the same operators over the same predicates with
// its describe() key, so deduping cannot hide a violation. // the same remaining bounds, so the merged obligations reduce identically on
// every future and dropping one cannot hide a violation. Distinct predicates
// never merge because every thunk's construction-time identity is part of its
// key, whether or not the caller named it.
func collapse(obligations []obligation) []obligation { func collapse(obligations []obligation) []obligation {
if len(obligations) < 2 { if len(obligations) < 2 {
return obligations return obligations
@@ -264,10 +275,48 @@ func (e *Evaluator) Residual() Formula {
return combined return combined
} }
// rootObligation returns the formula to instantiate at each step. An outer // instantiateRoot returns the obligation to register for this observation, and
// Always is stripped so its inner is re-evaluated every step; any other root // whether there is one at all. A one-shot root is armed only at the first
// formula is itself re-instantiated each step (matching the v0.1 semantics // observation; a recurring root is re-asserted at every one.
// where a bare Thunk is re-observed on every call). func (e *Evaluator) instantiateRoot() (Formula, bool) {
if !e.oneShot {
return rootObligation(e.root), true
}
if e.armed {
return nil, false
}
e.armed = true
return e.root, true
}
// isOneShotRoot reports whether a root formula is a single obligation for the
// whole run rather than one instance per observation.
//
// A root that carries its own horizon is one-shot: an eventually is a
// reachability goal ("this happens at some point"), and a bounded always is a
// single window. Re-instantiating either at every step would monitor a
// different property -- G F<=n(p) instead of F<=n(p), and G(p) instead of
// G<=n(p), the latter because a re-instantiated window restarts and never
// closes -- and would leave one live obligation per step behind.
//
// Every other root keeps the implicit-always reading: an unbounded always
// re-instantiates its inner (which is what gives each instance its own origin
// step), and a bare predicate or connective is re-asserted each observation.
func isOneShotRoot(root Formula) bool {
switch concrete := root.(type) {
case EventuallyFormula:
return true
case AlwaysFormula:
return concrete.HasStepBound || concrete.HasDeadline || concrete.Duration > 0
default:
return false
}
}
// rootObligation returns the formula a recurring root instantiates at each
// step. An outer Always is stripped so its inner is re-evaluated every step;
// any other root formula is itself re-instantiated each step (matching the
// v0.1 semantics where a bare Thunk is re-observed on every call).
func rootObligation(root Formula) Formula { func rootObligation(root Formula) Formula {
if always, ok := root.(AlwaysFormula); ok { if always, ok := root.(AlwaysFormula); ok {
return always.Inner return always.Inner
@@ -333,8 +382,14 @@ func reduce(formula Formula, now time.Time) reduceResult {
} }
return violatedWith(concrete, "pure false") return violatedWith(concrete, "pure false")
case ErrorFormula:
// A thrown predicate substituted into a residual at the trace
// boundary. Reducing it re-reports the same failure rather than
// crashing the run.
return violatedByError(concrete, concrete.Message)
case ThunkFormula: case ThunkFormula:
result, err := concrete.Func() result, err := concrete.predicate()
if err != nil { if err != nil {
return violatedByError(concrete, err.Error()) return violatedByError(concrete, err.Error())
} }
@@ -363,6 +418,9 @@ func reduce(formula Formula, now time.Time) reduceResult {
if innerResult.status == statusHolds { if innerResult.status == statusHolds {
return holds() return holds()
} }
// The window is measured in observations at which the inner could have
// discharged, so an inner that is merely pending has not discharged and
// the window closing on it is a violation.
if concrete.HasStepBound && concrete.StepBound <= 1 { if concrete.HasStepBound && concrete.StepBound <= 1 {
return violatedFrom(innerResult, concrete, "eventually bound exhausted") return violatedFrom(innerResult, concrete, "eventually bound exhausted")
} }
@@ -373,6 +431,14 @@ func reduce(formula Formula, now time.Time) reduceResult {
if concrete.HasStepBound { if concrete.HasStepBound {
next.StepBound = concrete.StepBound - 1 next.StepBound = concrete.StepBound - 1
} }
// F(inner) unrolls to inner or X F(inner). A pending inner is a
// deferred way of satisfying the promise, so it is kept as a disjunct
// rather than dropped; dropping it is what made an inner that only
// resolves on a later step unsatisfiable, and it is the mirror of the
// conjunct Always keeps below.
if innerResult.status == statusPending {
return pending(OrFormula{Left: innerResult.formula, Right: next})
}
return pending(next) return pending(next)
case ImpliesFormula: case ImpliesFormula:
@@ -444,25 +510,20 @@ func reduce(formula Formula, now time.Time) reduceResult {
if innerResult.status == statusViolated { if innerResult.status == statusViolated {
return violatedFrom(innerResult, concrete, "always inner violated") return violatedFrom(innerResult, concrete, "always inner violated")
} }
// A bounded Always is the dual of a bounded Eventually: once the window // A bounded Always must reduce exactly as its dual does, so that
// closes without a breach it is vacuously satisfied. A pending inner at // G<=n(f) and not F<=n(not f) agree on every trace. The dual of "the
// the closing step is a deferred obligation (a strong next, or an inner // window closed on an inner that never definitely held, so violate" is
// liveness that has not discharged); it must be carried so a later step // "the window closed on an inner that was never definitely breached, so
// or Finalize resolves it, never dropped to holds. // hold". A pending inner has not been breached inside the window, so it
// discharges vacuously here exactly as its negation violates on the
// Eventually side.
if concrete.HasStepBound && concrete.StepBound <= 1 { if concrete.HasStepBound && concrete.StepBound <= 1 {
if innerResult.status == statusHolds {
return holds() return holds()
} }
return pending(innerResult.formula)
}
if concrete.HasDeadline && !now.Before(concrete.Deadline) { if concrete.HasDeadline && !now.Before(concrete.Deadline) {
if innerResult.status == statusHolds {
return holds() return holds()
} }
return pending(innerResult.formula)
}
next := concrete next := concrete
next.Inner = concrete.Inner
if concrete.HasStepBound { if concrete.HasStepBound {
next.StepBound = concrete.StepBound - 1 next.StepBound = concrete.StepBound - 1
} }
+24 -6
View File
@@ -70,14 +70,32 @@ func TestImplies_TemporalAntecedent_FalseAntecedentHolds(t *testing.T) {
} }
} }
// A bounded Always whose inner is a still-pending deferred Next obligation must // A bounded Always whose inner is definitely false inside the window violates.
// carry that obligation past the window, not drop it to holds. The pre-fix // The pre-fix window-close branch returned holds() unconditionally and lost
// window-close branch returned holds() unconditionally and lost the violation. // this; the breach check runs before the window check and must stay there.
func TestBoundedAlways_PendingInnerCarried(t *testing.T) { func TestBoundedAlways_ViolatedInnerInsideWindow(t *testing.T) {
inner := AlwaysFormula{Inner: Next(Thunk(thunkSeq(false))), StepBound: 1, HasStepBound: true} inner := AlwaysFormula{Inner: Thunk(thunkSeq(false)), StepBound: 1, HasStepBound: true}
formula := Always(inner) formula := Always(inner)
if verdict, _ := runAndFinalize(formula, 3); verdict != VerdictViolated { if verdict, _ := runAndFinalize(formula, 3); verdict != VerdictViolated {
t.Fatalf("Always(boundedAlways(Next(false),1)): got %v, want violated", verdict) t.Fatalf("Always(boundedAlways(false,1)): got %v, want violated", verdict)
}
}
// A bounded Always whose inner is still a deferred Next obligation when the
// window closes discharges vacuously: nothing was breached inside the window.
// This is the exact dual of the bounded Eventually violating when its inner has
// not held by the time the window closes
// (TestEventuallyWithinSteps_NextInnerHitsBoundFirstStep), and the pair is what
// makes nnf's G/F dualisation semantics preserving. It costs the deferred check
// the window closed on: G<=n and F<=n both range over the observations at which
// their inner can definitely resolve, never past them.
func TestBoundedAlways_PendingInnerDischargesAtWindowClose(t *testing.T) {
inner := AlwaysFormula{Inner: Next(Thunk(thunkSeq(false))), StepBound: 1, HasStepBound: true}
if verdict, _ := runAndFinalize(Always(inner), 3); verdict != VerdictHolds {
t.Fatalf("Always(boundedAlways(Next(false),1)): got %v, want holds", verdict)
}
if verdict, _ := runAndFinalize(Always(nnf(Not(inner))), 3); verdict != VerdictViolated {
t.Fatalf("its negation: got %v, want violated", verdict)
} }
} }
+59 -1
View File
@@ -152,7 +152,7 @@ func TestViolationLatchIsMonotonic(t *testing.T) {
// holds/violated at run end, making sanderling lie about pass/fail. // holds/violated at run end, making sanderling lie about pass/fail.
func TestFinalize_KleeneConnectives(t *testing.T) { func TestFinalize_KleeneConnectives(t *testing.T) {
pure := func(v bool) Formula { return PureFormula{Value: v} } pure := func(v bool) Formula { return PureFormula{Value: v} }
pendingThunk := ThunkFormula{Name: "t", Func: func() (bool, error) { return true, nil }} pendingThunk := ThunkNamed("t", func() (bool, error) { return true, nil })
eventuallyViolated := EventuallyFormula{Inner: PureFormula{Value: false}} eventuallyViolated := EventuallyFormula{Inner: PureFormula{Value: false}}
nextPending := NextFormula{Inner: PureFormula{Value: true}} nextPending := NextFormula{Inner: PureFormula{Value: true}}
alwaysHolds := AlwaysFormula{Inner: PureFormula{Value: true}} alwaysHolds := AlwaysFormula{Inner: PureFormula{Value: true}}
@@ -224,3 +224,61 @@ func TestCollapse_NamedThunkLeakBoundsPendingSet(t *testing.T) {
t.Errorf("pending set leaked to %d obligations", len(evaluator.pending)) t.Errorf("pending set leaked to %d obligations", len(evaluator.pending))
} }
} }
// TestCollapse_UnnamedPredicatesDoNotMerge is the lost-violation counterexample
// from the attribution analysis, run with unnamed thunks. Every unnamed thunk
// used to print "Thunk(...)", so the four Eventually residuals below shared one
// collapse key and the obligation spawned at step 2 was dropped: the run
// reported holds while a genuine violation was outstanding.
//
// root = And(Or(F a, c), Or(F b, d)), d = not c
// a never true, b true from step 6, c true except at steps 2 and 4
//
// At steps 1, 3 and 5 the left disjunct discharges via c and the right spawns
// F b; at steps 2 and 4 the right discharges via d and the left spawns F a.
// F a can never discharge, so the run violates with origin 2.
func TestCollapse_UnnamedPredicatesDoNotMerge(t *testing.T) {
step := 0
a := Thunk(func() (bool, error) { return false, nil })
b := Thunk(func() (bool, error) { return step >= 6, nil })
c := Thunk(func() (bool, error) { return step != 2 && step != 4, nil })
d := Thunk(func() (bool, error) { return step == 2 || step == 4, nil })
evaluator := NewEvaluator(And(Or(Eventually(a), c), Or(Eventually(b), d)))
for index := 1; index <= 10; index++ {
step = index
if got := evaluator.ObserveAtStep(time.Unix(int64(index), 0), index); got == VerdictViolated {
t.Fatalf("step %d violated early", index)
}
}
if got := evaluator.Finalize(); got != VerdictViolated {
t.Fatalf("Finalize = %v, want violated (F a can never discharge)", got)
}
witness := evaluator.Violation()
if witness == nil {
t.Fatal("Violation = nil, want non-nil")
}
if witness.Step != 2 {
t.Errorf("origin = %d, want 2", witness.Step)
}
}
// TestReduce_ErrorFormulaViolates: the verifier substitutes an ErrorFormula for
// the residual of a property whose predicate threw, and that residual is fed
// back into the evaluator on the next step. Reducing one used to panic.
func TestReduce_ErrorFormulaViolates(t *testing.T) {
evaluator := NewEvaluator(Always(ErrorFormula{Message: "boom"}))
if got := evaluator.Observe(); got != VerdictViolated {
t.Fatalf("got %v, want violated", got)
}
witness := evaluator.Violation()
if witness == nil {
t.Fatal("Violation = nil, want non-nil")
}
if witness.Reason != "boom" {
t.Errorf("Reason = %q, want %q", witness.Reason, "boom")
}
if !witness.IsError {
t.Error("IsError = false, want true")
}
}
+70 -34
View File
@@ -4,6 +4,7 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"strings" "strings"
"sync/atomic"
"time" "time"
) )
@@ -13,9 +14,9 @@ type Formula interface {
describe() string describe() string
} }
// PredicateLabel lets a ThunkFormula expose the identity of the closure it // PredicateLabel lets a ThunkFormula expose the caller's label for the closure
// wraps. ThunkFormula satisfies it through its Name field; an empty name // it wraps. ThunkFormula satisfies it through the name passed to ThunkNamed;
// serializes without a name. // an unnamed thunk serializes without a name.
type PredicateLabel interface { type PredicateLabel interface {
PredicateName() string PredicateName() string
} }
@@ -50,17 +51,26 @@ type PureFormula struct {
Value bool Value bool
} }
// ThunkFormula wraps an opaque predicate closure. Func returns the predicate's // ThunkFormula wraps an opaque predicate closure. The closure returns the
// boolean result and a non-nil error when the predicate threw; a thrown // predicate's boolean result and a non-nil error when the predicate threw; a
// predicate is a witnessed violation distinct from a plain false. Name carries // thrown predicate is a witnessed violation distinct from a plain false.
// the predicate's identity so two distinct predicates produce distinct //
// describe() keys and are never merged during obligation collapse. // Every thunk carries an identity assigned at construction, and the identity
// is part of its describe() key. Two thunks are therefore equal keys only when
// they are copies of the same constructed value, which is what lets obligation
// collapse merge residuals without ever merging distinct predicates. The
// fields are unexported so a thunk cannot be built without one.
type ThunkFormula struct { type ThunkFormula struct {
Func func() (bool, error) predicate func() (bool, error)
Name string name string
identity uint64
} }
func (t ThunkFormula) PredicateName() string { return t.Name } func (t ThunkFormula) PredicateName() string { return t.name }
// thunkIdentities hands out the per-thunk identity. It only has to separate
// thunks within one process, so a counter is enough.
var thunkIdentities atomic.Uint64
// NowFormula marks its inner formula for evaluation at the current step only. // NowFormula marks its inner formula for evaluation at the current step only.
// Primarily used so that now(...).implies(...) parses unambiguously. // Primarily used so that now(...).implies(...) parses unambiguously.
@@ -113,10 +123,16 @@ func Always(inner Formula) Formula { return AlwaysFormula{Inner: inner} }
func Pure(value bool) Formula { return PureFormula{Value: value} } func Pure(value bool) Formula { return PureFormula{Value: value} }
func Thunk(function func() (bool, error)) Formula { return ThunkFormula{Func: function} } func Thunk(function func() (bool, error)) Formula {
return ThunkFormula{predicate: function, identity: thunkIdentities.Add(1)}
}
func ThunkNamed(name string, function func() (bool, error)) Formula { func ThunkNamed(name string, function func() (bool, error)) Formula {
return ThunkFormula{Func: function, Name: name} return ThunkFormula{
predicate: function,
name: name,
identity: thunkIdentities.Add(1),
}
} }
func Now(inner Formula) Formula { return NowFormula{Inner: inner} } func Now(inner Formula) Formula { return NowFormula{Inner: inner} }
@@ -172,10 +188,7 @@ func (a AlwaysFormula) describe() string {
} }
func (p PureFormula) describe() string { return fmt.Sprintf("Pure(%t)", p.Value) } func (p PureFormula) describe() string { return fmt.Sprintf("Pure(%t)", p.Value) }
func (t ThunkFormula) describe() string { func (t ThunkFormula) describe() string {
if t.Name != "" { return fmt.Sprintf("Thunk(%s#%d)", t.name, t.identity)
return "Thunk(" + t.Name + ")"
}
return "Thunk(...)"
} }
func (n NowFormula) describe() string { return "Now(" + n.Inner.describe() + ")" } func (n NowFormula) describe() string { return "Now(" + n.Inner.describe() + ")" }
func (n NextFormula) describe() string { return "Next(" + n.Inner.describe() + ")" } func (n NextFormula) describe() string { return "Next(" + n.Inner.describe() + ")" }
@@ -206,10 +219,43 @@ func (n NotFormula) describe() string { return "Not(" + n.Inner.describe() + ")"
func Describe(formula Formula) string { return formula.describe() } func Describe(formula Formula) string { return formula.describe() }
// withinNode mirrors the optional `within` clause attached to bounded // withinNode mirrors the optional `within` clause attached to bounded
// Eventually nodes in the JSON AST. // Always/Eventually nodes in the JSON AST.
type withinNode struct { type withinNode struct {
Amount int64 `json:"amount"` Amount int64 `json:"amount"`
Unit string `json:"unit"` Unit string `json:"unit"`
// Deadline is the absolute instant the window closes, in unix
// milliseconds, present once the evaluator resolved a relative duration
// against an observation. Two obligations spawned at different steps from
// the same duration differ only here, so without it they serialize
// identically and the trace erases the distinction the evaluator makes.
Deadline int64 `json:"deadline,omitempty"`
}
// withinFor renders the bound clause of a bounded Always or Eventually. The
// authored window (steps or duration) stays in amount/unit so readers keep
// seeing what the spec asked for; the resolved deadline rides alongside.
func withinFor(
hasStepBound bool,
stepBound int,
duration time.Duration,
hasDeadline bool,
deadline time.Time,
) *withinNode {
var node *withinNode
switch {
case hasStepBound:
node = &withinNode{Amount: int64(stepBound), Unit: "steps"}
case duration > 0:
node = &withinNode{Amount: duration.Milliseconds(), Unit: "milliseconds"}
case hasDeadline:
return &withinNode{Amount: deadline.UnixMilli(), Unit: "deadline"}
default:
return nil
}
if hasDeadline {
node.Deadline = deadline.UnixMilli()
}
return node
} }
func (a AlwaysFormula) MarshalJSON() ([]byte, error) { func (a AlwaysFormula) MarshalJSON() ([]byte, error) {
@@ -218,14 +264,9 @@ func (a AlwaysFormula) MarshalJSON() ([]byte, error) {
Arg Formula `json:"arg"` Arg Formula `json:"arg"`
Within *withinNode `json:"within,omitempty"` Within *withinNode `json:"within,omitempty"`
}{Op: "always", Arg: a.Inner} }{Op: "always", Arg: a.Inner}
switch { payload.Within = withinFor(
case a.HasStepBound: a.HasStepBound, a.StepBound, a.Duration, a.HasDeadline, a.Deadline,
payload.Within = &withinNode{Amount: int64(a.StepBound), Unit: "steps"} )
case a.Duration > 0:
payload.Within = &withinNode{Amount: a.Duration.Milliseconds(), Unit: "milliseconds"}
case a.HasDeadline:
payload.Within = &withinNode{Amount: a.Deadline.UnixMilli(), Unit: "deadline"}
}
return json.Marshal(payload) return json.Marshal(payload)
} }
@@ -256,14 +297,9 @@ func (e EventuallyFormula) MarshalJSON() ([]byte, error) {
Arg Formula `json:"arg"` Arg Formula `json:"arg"`
Within *withinNode `json:"within,omitempty"` Within *withinNode `json:"within,omitempty"`
}{Op: "eventually", Arg: e.Inner} }{Op: "eventually", Arg: e.Inner}
switch { payload.Within = withinFor(
case e.HasStepBound: e.HasStepBound, e.StepBound, e.Duration, e.HasDeadline, e.Deadline,
payload.Within = &withinNode{Amount: int64(e.StepBound), Unit: "steps"} )
case e.Duration > 0:
payload.Within = &withinNode{Amount: e.Duration.Milliseconds(), Unit: "milliseconds"}
case e.HasDeadline:
payload.Within = &withinNode{Amount: e.Deadline.UnixMilli(), Unit: "deadline"}
}
return json.Marshal(payload) return json.Marshal(payload)
} }
+29
View File
@@ -288,3 +288,32 @@ func TestResidual_HoldsViolatedPending(t *testing.T) {
t.Errorf("pending residual:\n got: %s\nwant: %s", body, want) t.Errorf("pending residual:\n got: %s\nwant: %s", body, want)
} }
} }
// TestMarshalJSON_ResolvedDeadlineDistinguishesObligations: obligations spawned
// at different steps from one duration-bounded eventually differ only in the
// deadline the evaluator resolved for them, and collapse keeps them apart on
// exactly that. The serialized AST has to keep them apart too, or the trace
// shows N copies of one node where the evaluator has N different obligations.
func TestMarshalJSON_ResolvedDeadlineDistinguishesObligations(t *testing.T) {
base := time.UnixMilli(1700000000000)
first := EventuallyFormula{
Inner: PureFormula{Value: false},
Duration: 300 * time.Second,
Deadline: base.Add(300 * time.Second),
HasDeadline: true,
}
second := first
second.Deadline = base.Add(301 * time.Second)
firstBody, _ := json.Marshal(first)
secondBody, _ := json.Marshal(second)
if string(firstBody) == string(secondBody) {
t.Errorf("obligations with different deadlines serialize identically: %s", firstBody)
}
if !strings.Contains(string(firstBody), `"unit":"milliseconds"`) {
t.Errorf("authored window lost: %s", firstBody)
}
if !strings.Contains(string(firstBody), `"deadline":1700000300000`) {
t.Errorf("resolved deadline missing: %s", firstBody)
}
}
+106
View File
@@ -0,0 +1,106 @@
package ltl
import (
"testing"
"time"
)
// TestRoot_BoundedEventuallyStaysOneObligation locks the cost of the one-shot
// root. A duration-bounded top-level eventually is one reachability goal, so
// the pending set holds one obligation for the whole run. Re-instantiating it
// every step monitored G F<=n(p) instead and left one live obligation per step
// behind: a 553-step run carried 553 of them and re-ran the predicate once per
// obligation per step.
func TestRoot_BoundedEventuallyStaysOneObligation(t *testing.T) {
const steps = 600
calls := 0
formula := EventuallyWithin(ThunkNamed("p", func() (bool, error) {
calls++
return false, nil
}), 300*time.Second)
evaluator := NewEvaluator(formula)
base := time.Unix(0, 0)
for index := range steps {
if got := evaluator.ObserveAtStep(base.Add(time.Duration(index)*110*time.Millisecond), index+1); got != VerdictPending {
t.Fatalf("step %d: got %v, want pending", index+1, got)
}
if len(evaluator.pending) != 1 {
t.Fatalf("step %d: %d pending obligations, want 1", index+1, len(evaluator.pending))
}
}
if calls != steps {
t.Errorf("predicate ran %d times over %d steps, want one call per step", calls, steps)
}
}
// TestRoot_TopLevelEventuallyIsSatisfiedOnce pins the semantics behind that
// bound: a top-level eventually is discharged for good the first time it is
// satisfied. Under the old implicit-always reading it was re-armed at every
// step, so a property that had already been reached could still violate later.
func TestRoot_TopLevelEventuallyIsSatisfiedOnce(t *testing.T) {
reached := false
evaluator := NewEvaluator(EventuallyWithinSteps(ThunkNamed("p", func() (bool, error) {
return reached, nil
}), 2))
if got := evaluator.ObserveAtStep(time.Unix(0, 0), 1); got != VerdictPending {
t.Fatalf("step 1: got %v, want pending", got)
}
reached = true
if got := evaluator.ObserveAtStep(time.Unix(1, 0), 2); got != VerdictHolds {
t.Fatalf("step 2: got %v, want holds", got)
}
reached = false
for index := 3; index <= 6; index++ {
if got := evaluator.ObserveAtStep(time.Unix(int64(index), 0), index); got != VerdictHolds {
t.Fatalf("step %d: got %v, want holds (the goal was already reached)", index, got)
}
}
if got := evaluator.Finalize(); got != VerdictHolds {
t.Errorf("Finalize = %v, want holds", got)
}
}
// TestRoot_BoundedAlwaysKeepsItsBound: a bounded root Always is a single
// window, not a recurrence. Stripping it and re-instantiating its inner every
// step dropped the bound, so G<=1(p) behaved as G(p) and a false p after the
// window closed still violated.
func TestRoot_BoundedAlwaysKeepsItsBound(t *testing.T) {
values := []bool{true, true, false}
step := 0
formula := AlwaysFormula{
Inner: ThunkNamed("p", func() (bool, error) { return values[step], nil }),
StepBound: 1,
HasStepBound: true,
}
evaluator := NewEvaluator(formula)
for index := range values {
step = index
if got := evaluator.ObserveAtStep(time.Unix(int64(index), 0), index+1); got == VerdictViolated {
t.Fatalf("step %d: violated outside the 1-step window", index+1)
}
}
}
// TestRoot_UnboundedAlwaysStillReInstantiates: the one-shot rule must not touch
// the recurrence root every spec property is built on. Each step gets its own
// instance of the inner, which is what gives a deferred failure the origin step
// that armed it.
func TestRoot_UnboundedAlwaysStillReInstantiates(t *testing.T) {
values := []bool{true, true, false}
step := 0
evaluator := NewEvaluator(Always(ThunkNamed("p", func() (bool, error) {
return values[step], nil
})))
for index := range 2 {
step = index
if got := evaluator.ObserveAtStep(time.Unix(int64(index), 0), index+1); got != VerdictHolds {
t.Fatalf("step %d: got %v, want holds", index+1, got)
}
}
step = 2
if got := evaluator.ObserveAtStep(time.Unix(2, 0), 3); got != VerdictViolated {
t.Errorf("step 3: got %v, want violated", got)
}
}
+8 -1
View File
@@ -302,7 +302,14 @@ func describeAction(action verifier.Action) string {
case verifier.ActionKindInputText: case verifier.ActionKindInputText:
return fmt.Sprintf("InputText %s = %q", actionTarget(action), action.Text) return fmt.Sprintf("InputText %s = %q", actionTarget(action), action.Text)
case verifier.ActionKindScroll: case verifier.ActionKindScroll:
return fmt.Sprintf("Scroll %s %s", action.Direction, action.On) // A builtin gesture carries endpoints rather than a selector, so name the
// container by where the drag starts; that is what tells two scrollable
// regions apart in the recent-action memory.
target := action.On
if target == "" {
target = fmt.Sprintf("(%d,%d)", action.FromX, action.FromY)
}
return fmt.Sprintf("Scroll %s %s", action.Direction, target)
case verifier.ActionKindSwipe: case verifier.ActionKindSwipe:
// Coordinates make a repeated identical swipe recognizable in the // Coordinates make a repeated identical swipe recognizable in the
// prompt's recent-action memory. // prompt's recent-action memory.
+14
View File
@@ -62,6 +62,20 @@ func TestActionForCandidatePassesNonTypingThrough(t *testing.T) {
} }
} }
func TestDescribeActionNamesGesturesByOrigin(t *testing.T) {
builtin := verifier.Action{
Kind: verifier.ActionKindScroll, Direction: "down",
FromX: 200, FromY: 500, ToX: 200, ToY: 340,
}
if got := describeAction(builtin); got != "Scroll down (200,500)" {
t.Errorf("builtin gesture described as %q, want the drag origin", got)
}
authored := verifier.Action{Kind: verifier.ActionKindScroll, Direction: "up", On: "id:List"}
if got := describeAction(authored); got != "Scroll up id:List" {
t.Errorf("authored scroll described as %q, want its selector", got)
}
}
func TestActionForCandidateUsesModelText(t *testing.T) { func TestActionForCandidateUsesModelText(t *testing.T) {
source := &llmSource{} source := &llmSource{}
candidate := verifier.ActionCandidate{ candidate := verifier.ActionCandidate{
+9 -2
View File
@@ -1006,7 +1006,9 @@ func violationRecords(properties []string, witnesses map[string]trace.Witness, d
// collectWitnesses gathers the violation witness for each newly-violated // collectWitnesses gathers the violation witness for each newly-violated
// property, logs its cause, and returns them keyed by property name for the // property, logs its cause, and returns them keyed by property name for the
// trace. Properties without a captured witness are skipped. // trace. Properties without a captured witness are skipped. stepIndex is the
// trace line the witness lands on, and stands in as the detection step for a
// verifier that observed no labeled step (a run-end finalize).
func collectWitnesses(verifierInstance *verifier.Verifier, properties []string, logger *slog.Logger, stepIndex int) map[string]trace.Witness { func collectWitnesses(verifierInstance *verifier.Verifier, properties []string, logger *slog.Logger, stepIndex int) map[string]trace.Witness {
if len(properties) == 0 { if len(properties) == 0 {
return nil return nil
@@ -1017,13 +1019,18 @@ func collectWitnesses(verifierInstance *verifier.Verifier, properties []string,
if witness == nil { if witness == nil {
continue continue
} }
detectedStep := witness.DetectedStep
if detectedStep == 0 {
detectedStep = stepIndex
}
logger.Warn("property violated", logger.Warn("property violated",
"step", witness.Step, "detected_step", stepIndex, "step", witness.Step, "detected_step", detectedStep,
"property", name, "reason", witness.Reason, "error", witness.IsError) "property", name, "reason", witness.Reason, "error", witness.IsError)
witnesses[name] = trace.Witness{ witnesses[name] = trace.Witness{
Reason: witness.Reason, Reason: witness.Reason,
IsError: witness.IsError, IsError: witness.IsError,
Step: witness.Step, Step: witness.Step,
DetectedStep: detectedStep,
Extractors: witness.Extractors, Extractors: witness.Extractors,
} }
} }
+5
View File
@@ -363,6 +363,11 @@ globalThis.actions = actions(() => []);
if witness.Step != 2 { if witness.Step != 2 {
t.Errorf("witness step: got %d, want 2 (causing step)", witness.Step) t.Errorf("witness step: got %d, want 2 (causing step)", witness.Step)
} }
// The two indices the witness spans are recorded separately: the
// extractor snapshot it carries is step 3's state, not step 2's.
if witness.DetectedStep != 3 {
t.Errorf("witness detected step: got %d, want 3", witness.DetectedStep)
}
} }
if err := scanner.Err(); err != nil { if err := scanner.Err(); err != nil {
t.Fatalf("scan trace: %v", err) t.Fatalf("scan trace: %v", err)
+118
View File
@@ -0,0 +1,118 @@
package runner
import (
"bufio"
"context"
"encoding/json"
"os"
"path/filepath"
"testing"
"time"
mockdriver "github.com/priyanshujain/sanderling/internal/driver/mock"
"github.com/priyanshujain/sanderling/internal/trace"
)
// engineDisagreementSpec makes the two runtimes disagree on purpose: the
// extractor body returns "goja" when it runs in-process, and the web driver
// below reports "v8" for the same extractor. The property is true of one value
// and false of the other, so the verdict names the engine the verdict used.
const engineDisagreementSpec = `
import { actions, always, extract } from "@sanderling/spec";
const engine = extract("engine", () => "goja");
globalThis.properties = {
ranInGoja: always(() => engine.current === "goja"),
};
globalThis.actions = actions(() => []);
`
// webMockDriver presents the mock device driver as a web target so the runner
// takes the V8 path, where extractor values come from the page rather than
// from goja.
type webMockDriver struct {
*mockdriver.Driver
overrides map[int]json.RawMessage
}
func (d *webMockDriver) InstallBundle(context.Context, []byte) error { return nil }
func (d *webMockDriver) EvaluateExtractors(context.Context) (map[int]json.RawMessage, error) {
return d.overrides, nil
}
func (d *webMockDriver) NextActionFromV8(context.Context) (json.RawMessage, error) {
return nil, nil
}
// TestRunner_TraceRecordsTheValueTheVerdictUsed fails if the trace and the
// verdict disagree about an extractor. A witness is only an explanation of a
// violation if it holds the state the violated property was evaluated against.
func TestRunner_TraceRecordsTheValueTheVerdictUsed(t *testing.T) {
state := newHarnessWithSpec(t, engineDisagreementSpec)
const pageValue = `"v8"`
web := &webMockDriver{
Driver: state.mock,
overrides: map[int]json.RawMessage{0: json.RawMessage(pageValue)},
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
summary, err := Run(ctx, Options{
Duration: 100 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
Driver: web,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
if !containsProperty(summary.Violations, "ranInGoja") {
t.Fatalf("ranInGoja did not violate, so the page value never reached the verdict: %v",
summary.Violations)
}
file, err := os.Open(filepath.Join(state.writer.Directory(), "trace.jsonl"))
if err != nil {
t.Fatal(err)
}
defer file.Close()
type traceLine struct {
Step int `json:"step"`
ExtractorChanges map[string]trace.ExtractorChange `json:"extractor_changes"`
Witnesses map[string]trace.Witness `json:"witnesses"`
}
changes, witnesses := 0, 0
scanner := bufio.NewScanner(file)
scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024)
for scanner.Scan() {
var line traceLine
if err := json.Unmarshal(scanner.Bytes(), &line); err != nil {
t.Fatalf("trace line decode: %v", err)
}
if change, ok := line.ExtractorChanges["engine"]; ok {
changes++
if got := string(change.Curr); got != pageValue {
t.Errorf("step %d: trace records engine=%s, verdict used %s",
line.Step, got, pageValue)
}
}
for name, witness := range line.Witnesses {
witnesses++
if got := string(witness.Extractors["engine"]); got != pageValue {
t.Errorf("step %d: %s witness records engine=%s, verdict used %s",
line.Step, name, got, pageValue)
}
}
}
if err := scanner.Err(); err != nil {
t.Fatalf("scan trace: %v", err)
}
if changes == 0 {
t.Error("no extractor change reached the trace; nothing was compared")
}
if witnesses == 0 {
t.Error("no witness reached the trace; nothing was compared")
}
}
+7 -5
View File
@@ -40,16 +40,18 @@ type Step struct {
Witnesses map[string]Witness `json:"witnesses,omitempty"` Witnesses map[string]Witness `json:"witnesses,omitempty"`
} }
// Witness is the trace-side record of a property violation: why it fired and a // Witness is the trace-side record of a property violation: why it fired, the
// snapshot of every extractor's value at the violating step. // two steps a deferred obligation spans, and the extractor values behind it.
type Witness struct { type Witness struct {
Reason string `json:"reason,omitempty"` Reason string `json:"reason,omitempty"`
IsError bool `json:"is_error,omitempty"` IsError bool `json:"is_error,omitempty"`
// Step is the step the failed obligation originated at: the step that // Step is the step the failed obligation originated at: the step that
// caused the violation. For a deferred obligation (a next, an eventually) // armed it. For a deferred obligation (a next, an eventually) this is
// this is earlier than the step whose record carries the witness, which is // earlier than the step at which the failure was detected.
// where the failure was detected.
Step int `json:"step,omitempty"` Step int `json:"step,omitempty"`
// DetectedStep is the observation whose evaluation produced the violation.
// Extractors is that step's state, not Step's.
DetectedStep int `json:"detected_step,omitempty"`
Extractors map[string]json.RawMessage `json:"extractors,omitempty"` Extractors map[string]json.RawMessage `json:"extractors,omitempty"`
} }
+19 -14
View File
@@ -91,7 +91,7 @@ func (v *Verifier) installRuntimeBindings() error {
// installHost exposes globalThis.__sanderlingHost__ for the goja runtime entry. // installHost exposes globalThis.__sanderlingHost__ for the goja runtime entry.
// The shared picker (pick.ts) draws against it: platform() drives the verb // The shared picker (pick.ts) draws against it: platform() drives the verb
// matrix and press-key pool; seedHi/seedLo construct its Pcg; queryCandidates // matrix and press-key pool; seedHi/seedLo construct its Pcg; queryTargets
// enumerates targets over the hierarchy tree; reportUnsupported records the // enumerates targets over the hierarchy tree; reportUnsupported records the
// verb for the run report. // verb for the run report.
func (v *Verifier) installHost() error { func (v *Verifier) installHost() error {
@@ -111,7 +111,7 @@ func (v *Verifier) installHost() error {
}); err != nil { }); err != nil {
return err return err
} }
if err := host.Set("queryCandidates", v.bindQueryCandidates); err != nil { if err := host.Set("queryTargets", v.bindQueryTargets); err != nil {
return err return err
} }
if err := host.Set("reportUnsupported", func(call goja.FunctionCall) goja.Value { if err := host.Set("reportUnsupported", func(call goja.FunctionCall) goja.Value {
@@ -133,20 +133,25 @@ func (v *Verifier) recordUnsupported(verb string) {
v.unsupported = append(v.unsupported, verb) v.unsupported = append(v.unsupported, verb)
} }
// bindQueryCandidates returns the host-enumerated targets for a verb as an // bindQueryTargets returns every host-enumerated target as an array of
// array of {x, y, selector, width, height}, in tree order. // {x, y, selector, width, height, clickable, enabled, editable, scrollable}, in
func (v *Verifier) bindQueryCandidates(call goja.FunctionCall) goja.Value { // tree order. It takes no verb: the shared rule in targets.ts decides which of
verb := call.Argument(0).String() // these a verb may act on.
candidates := v.candidatesForVerb(verb) func (v *Verifier) bindQueryTargets(goja.FunctionCall) goja.Value {
targets := v.targets()
array := v.runtime.NewArray() array := v.runtime.NewArray()
for index, candidate := range candidates { for index, target := range targets {
item := v.runtime.NewObject() item := v.runtime.NewObject()
_ = item.Set("x", candidate.x) _ = item.Set("x", target.x)
_ = item.Set("y", candidate.y) _ = item.Set("y", target.y)
_ = item.Set("width", candidate.width) _ = item.Set("width", target.width)
_ = item.Set("height", candidate.height) _ = item.Set("height", target.height)
if candidate.selector != "" { _ = item.Set("clickable", target.clickable)
_ = item.Set("selector", candidate.selector) _ = item.Set("enabled", target.enabled)
_ = item.Set("editable", target.editable)
_ = item.Set("scrollable", target.scrollable)
if target.selector != "" {
_ = item.Set("selector", target.selector)
} }
_ = array.Set(fmt.Sprintf("%d", index), item) _ = array.Set(fmt.Sprintf("%d", index), item)
} }
+52
View File
@@ -4,6 +4,7 @@ import (
"encoding/json" "encoding/json"
"strings" "strings"
"testing" "testing"
"time"
"github.com/priyanshujain/sanderling/internal/ltl" "github.com/priyanshujain/sanderling/internal/ltl"
) )
@@ -116,3 +117,54 @@ func TestWithin_InvalidUnitPanics(t *testing.T) {
} }
} }
} }
// TestTopLevelEventually_IsOneReachabilityObligation drives the folio-web shape
// `eventually(p).within(300, "seconds")` as a top-level property over a run of
// the same length and cadence as the 553-step run that exposed this: 60 seconds
// of steps, a predicate that never fires, a window far longer than the run.
//
// The property is one reachability goal, so the run leaves one obligation and
// one residual node behind. Wrapping the root in Always made it "within 300
// seconds of EVERY step" instead, which spawned an obligation per step (each
// with its own resolved deadline, so none of them collapsed), re-ran the
// predicate once per obligation per step, and serialized a 75 KB residual.
func TestTopLevelEventually_IsOneReachabilityObligation(t *testing.T) {
const source = `
globalThis.seen = __sanderling__.extract(state => state.snapshots["seen"] ?? false, "seen");
globalThis.properties = {
reachable: __sanderling__.eventually(() => seen.current).within(300, 'seconds'),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, source)
base := time.Unix(1780000000, 0)
const steps = 553
for index := range steps {
if err := verifier.PushSnapshot(SnapshotInput{
Snapshots: Snapshots{"seen": json.RawMessage(`false`)},
StepIndex: index + 1,
StepTime: base.Add(time.Duration(index) * 108 * time.Millisecond),
RunStart: base,
}); err != nil {
t.Fatal(err)
}
if got := verifier.EvaluateProperties()["reachable"]; got != ltl.VerdictPending {
t.Fatalf("step %d: got %v, want pending", index+1, got)
}
}
residual, err := json.Marshal(verifier.Residuals()["reachable"])
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(residual), `"op":"and"`) {
t.Errorf("residual accumulated obligations (%d bytes): %s", len(residual), residual)
}
if !strings.Contains(string(residual), `"op":"eventually"`) {
t.Errorf("residual lost the eventually: %s", residual)
}
if !strings.Contains(string(residual), `"unit":"milliseconds"`) {
t.Errorf("residual lost the bound: %s", residual)
}
}
+114
View File
@@ -0,0 +1,114 @@
package verifier
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"slices"
"testing"
)
// hostParityScreen is the canonical screen both hosts are driven over: one row
// per fact combination that any verb distinguishes. Each host builds it in its
// own model (this file as a hierarchy tree, pkg/spec/test/host-parity.test.ts as
// a DOM), enumerates every verb, and asserts the SAME committed golden.
var hostParityScreen = []struct {
name string
clickable bool
enabled bool
editable bool
scrollable bool
positiveBounds bool
}{
{name: "root", enabled: true, scrollable: true, positiveBounds: true},
{name: "save", clickable: true, enabled: true, positiveBounds: true},
{name: "cancel", clickable: true, positiveBounds: true},
{name: "amount", enabled: true, editable: true, positiveBounds: true},
{name: "list", enabled: true, scrollable: true, positiveBounds: true},
{name: "row", enabled: true, positiveBounds: true},
{name: "collapsed", clickable: true, enabled: true},
}
// hostParityTreeJSON is hostParityScreen as the native host sees it. Tree order
// is pre-order, so it matches hostParityScreen index for index.
const hostParityTreeJSON = `{
"attributes": {"resource-id": "root", "scrollable": "true", "bounds": "[0,0,400,800]"},
"enabled": true,
"children": [
{"attributes": {"resource-id": "save", "bounds": "[0,0,200,60]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "cancel", "bounds": "[200,0,400,60]"}, "clickable": true, "enabled": false, "children": []},
{"attributes": {"resource-id": "amount", "bounds": "[0,100,400,160]"}, "editable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "list", "scrollable": "true", "bounds": "[0,200,400,600]"}, "enabled": true, "children": []},
{"attributes": {"resource-id": "row", "bounds": "[0,600,400,680]"}, "enabled": true, "children": []},
{"attributes": {"resource-id": "collapsed", "bounds": "[0,0,0,0]"}, "clickable": true, "enabled": true, "children": []}
]
}`
// TestHostsAgreeOnTargetEligibility is the guard on the claim that one
// specification induces one action space on every platform. The native host and
// the web host used to route verbs themselves and had drifted: web sent
// `swipes` to scrollable containers only, so a swipe on a list row was
// reachable on Android and unreachable on web for the same spec.
//
// Per-verb eligibility now has ONE definition (pkg/spec/src/targets.ts); a host
// reports facts and never filters. This test is what notices if a second
// definition grows back on either side: both hosts enumerate the same canonical
// screen and must name the same targets, verb for verb, in the same order.
// pkg/spec/test/host-parity.test.ts asserts the SAME golden from the web host,
// so a match on both sides proves the two hosts agree without either invoking
// the other.
func TestHostsAgreeOnTargetEligibility(t *testing.T) {
golden := loadHostParityGolden(t)
for _, verb := range policyVerbs {
t.Run(verb, func(t *testing.T) {
want, ok := golden[verb]
if !ok {
t.Fatalf("golden has no entry for %s", verb)
}
verifier := newVerifier(t, WithSeed(0x5eed))
loadActionSpec(t, verifier, fmt.Sprintf(
"import { %s } from \"@sanderling/spec\";\nglobalThis.actions = %s;", verb, verb))
pushTree(t, verifier, hostParityTreeJSON)
entries, err := verifier.enumerateBuiltin(verb)
if err != nil {
t.Fatalf("enumerate %s: %v", verb, err)
}
if len(entries) == 0 {
t.Fatalf("%s enumerated nothing at all", verb)
}
got := []string{}
for _, entry := range entries {
if entry.targetIndex < 0 {
continue
}
if entry.targetIndex >= len(hostParityScreen) {
t.Fatalf("%s produced target index %d, off the %d-row screen",
verb, entry.targetIndex, len(hostParityScreen))
}
got = append(got, hostParityScreen[entry.targetIndex].name)
}
if !slices.Equal(got, want) {
t.Errorf("native host targets for %s\n got=%v\nwant=%v", verb, got, want)
}
})
}
}
func loadHostParityGolden(t *testing.T) map[string][]string {
t.Helper()
path, err := filepath.Abs("../../pkg/spec/test/fixtures/host-parity-golden.json")
if err != nil {
t.Fatal(err)
}
body, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read golden: %v", err)
}
golden := map[string][]string{}
if err := json.Unmarshal(body, &golden); err != nil {
t.Fatalf("decode golden: %v", err)
}
return golden
}
+98 -119
View File
@@ -1,6 +1,7 @@
package verifier package verifier
import ( import (
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"math" "math"
@@ -130,54 +131,21 @@ type ActionCandidate struct {
prob float64 prob float64
} }
// verbActionKind maps a picker verb to the action kind it dispatches.
func verbActionKind(verb string) ActionKind {
switch verb {
case "taps":
return ActionKindTap
case "doubleTaps":
return ActionKindDoubleTap
case "longPresses":
return ActionKindLongPress
case "typing":
return ActionKindInputText
case "scrolls":
return ActionKindScroll
case "swipes":
return ActionKindSwipe
default:
return ""
}
}
// maxLabelRunes caps a visible-text label so joined descendant text stays short // maxLabelRunes caps a visible-text label so joined descendant text stays short
// enough to render on one numbered line. // enough to render on one numbered line.
const maxLabelRunes = 40 const maxLabelRunes = 40
// gestureDirections are the directional scrolls emitted per scrollable
// container. Vertical only: most mobile lists scroll up/down, and keeping the
// set tiny is the whole point of folding per-element swipes away.
var gestureDirections = []string{"down", "up"}
// Candidates enumerates every action the spec's weighted actionsRoot yields at // Candidates enumerates every action the spec's weighted actionsRoot yields at
// the current step, each tagged with a plainly-worded description and its // the current step, each tagged with a plainly-worded description and its
// effective weight, for the LLM generator to pick one number from. It walks the // effective weight, for the LLM generator to pick one number from. It walks the
// SAME tree the seeded picker draws: weighted branches recurse (accumulating the // SAME tree the seeded picker draws: weighted branches recurse (accumulating the
// selection probability), authored actions()/whenRoute leaves are called once // selection probability), authored actions()/whenRoute leaves are called once
// for their concrete actions, and builtin verbs enumerate per applicable // for their concrete actions, and builtin verbs come straight from the picker's
// element. Disabled controls are dropped, per-element gestures fold into a few // own enumeration. Identical descriptions dedup, summing weight.
// directional scrolls over scrollable containers, and identical descriptions
// dedup (summing weight).
func (v *Verifier) Candidates() []ActionCandidate { func (v *Verifier) Candidates() []ActionCandidate {
if v.lastTree == nil { if v.lastTree == nil {
return nil return nil
} }
// A cross-fade frame's layout is mid-animation, often in a collapsed
// coordinate space, so acting on it taps garbage (e.g. the soft keyboard).
// Skip it so the LLM re-observes a settled frame next step.
if v.lastTree.Transitional() {
return nil
}
root := v.runtime.GlobalObject().Get("actions") root := v.runtime.GlobalObject().Get("actions")
if root == nil || goja.IsUndefined(root) || goja.IsNull(root) { if root == nil || goja.IsUndefined(root) || goja.IsNull(root) {
return nil return nil
@@ -333,6 +301,18 @@ func (v *Verifier) candidateFromDescriptor(value goja.Value, nodeIndex map[*hier
Direction: direction, Direction: direction,
Action: Action{Kind: kind, On: target.selector, Direction: direction}, Action: Action{Kind: kind, On: target.selector, Direction: direction},
}, true }, true
case ActionKindSwipe:
from := v.resolveTarget(object.Get("from"), nodeIndex)
to := v.resolveTarget(object.Get("to"), nodeIndex)
return ActionCandidate{
Kind: kind,
Action: Action{
Kind: kind,
FromX: from.x, FromY: from.y,
ToX: to.x, ToY: to.y,
DurationMillis: intField(object, "durationMillis"),
},
}, true
case ActionKindPressKey: case ActionKindPressKey:
return ActionCandidate{ return ActionCandidate{
Kind: kind, Kind: kind,
@@ -410,103 +390,80 @@ func (v *Verifier) findBySelector(selector string) *hierarchy.Element {
return v.lastTree.Find(selector) return v.lastTree.Find(selector)
} }
// collectBuiltin enumerates a builtin verb over the current tree: tap-family and // collectBuiltin turns the picker's own enumeration of a builtin verb into
// typing emit one candidate per applicable element; scrolls/swipes fold into // candidates. The list comes from the bundle's __sanderlingEnumerateBuiltin__
// directional gestures over scrollable containers. // (pick.ts builtinCandidates), which is what the seeded policy draws from, so
// the two policies select over one action space and cannot drift apart. Each
// entry's action arrives on the wire contract DecodeAction already reads, so a
// chosen candidate executes the action the seeded draw would have executed.
func (v *Verifier) collectBuiltin(verb string, prob float64, weighted bool, nodeIndex map[*hierarchy.Element]*hierarchy.Node, out *[]ActionCandidate) { func (v *Verifier) collectBuiltin(verb string, prob float64, weighted bool, nodeIndex map[*hierarchy.Element]*hierarchy.Node, out *[]ActionCandidate) {
switch verb { entries, err := v.enumerateBuiltin(verb)
case "taps", "doubleTaps", "longPresses": if err != nil {
kind := verbActionKind(verb) return
for _, element := range v.elementsForVerb(verb) { }
x, y := element.Bounds.Center() targets := v.targets()
*out = append(*out, ActionCandidate{ for _, entry := range entries {
Kind: kind, candidate := ActionCandidate{
Label: visibleLabel(element, nodeIndex), Kind: entry.action.Kind,
Action: Action{Kind: kind, On: selectorForElement(v.lastTree, element), X: x, Y: y}, Direction: entry.action.Direction,
// Builtin typing enumerates the field, not the value: the seeded
// policy draws its text from the corpus and the model writes its own.
LLMText: entry.action.Kind == ActionKindInputText,
Action: entry.action,
prob: prob, prob: prob,
Weighted: weighted, Weighted: weighted,
})
} }
case "typing": if entry.targetIndex >= 0 && entry.targetIndex < len(targets) {
for _, element := range v.elementsForVerb(verb) { element := targets[entry.targetIndex].element
x, y := element.Bounds.Center() candidate.Label = visibleLabel(element, nodeIndex)
*out = append(*out, ActionCandidate{ candidate.InputType = inputTypeHint(element)
Kind: ActionKindInputText,
Label: visibleLabel(element, nodeIndex),
InputType: inputTypeHint(element),
LLMText: true,
Action: Action{Kind: ActionKindInputText, On: selectorForElement(v.lastTree, element), X: x, Y: y},
prob: prob,
Weighted: weighted,
})
} }
case "scrolls", "swipes": *out = append(*out, candidate)
v.collectGestures(prob, weighted, out)
} }
} }
// collectGestures emits directional scrolls scoped to each scrollable container, // builtinCandidate is one entry of the shared builtin enumeration: the concrete
// never per element and never element-labeled. Folding both scrolls and swipes // action, plus the index of the host candidate it targets (-1 when the verb has
// here is what removes the flood of mislabeled `Swipe "X"` gestures. // no target, as for a key press or a wait).
func (v *Verifier) collectGestures(prob float64, weighted bool, out *[]ActionCandidate) { type builtinCandidate struct {
scope := v.scopedElements() action Action
for _, element := range v.lastTree.Elements { targetIndex int
if !scope[element] {
continue
}
if element.Attributes["scrollable"] != "true" {
continue
}
if element.Bounds.Width() <= 0 || element.Bounds.Height() <= 0 {
continue
}
selector := selectorForElement(v.lastTree, element)
for _, direction := range gestureDirections {
action := Action{Kind: ActionKindScroll, On: selector, Direction: direction}
if selector == "" {
action.FromX, action.FromY, action.ToX, action.ToY = scrollGeometry(element.Bounds, direction)
}
*out = append(*out, ActionCandidate{
Kind: ActionKindScroll,
Direction: direction,
Action: action,
prob: prob,
Weighted: weighted,
})
}
}
} }
// scrollGeometry lowers a directional scroll to swipe endpoints over the given // enumerateBuiltin invokes the bundle's shared enumeration for one verb. A spec
// container bounds, matching the runner's own derivation, used only when the // loaded without the runtime entry (a raw-JS unit fixture) has no enumerator, so
// container has no resolving selector. // the verb contributes nothing rather than falling back to a second enumeration.
func scrollGeometry(bounds hierarchy.Bounds, direction string) (fromX, fromY, toX, toY int) { func (v *Verifier) enumerateBuiltin(verb string) ([]builtinCandidate, error) {
cx, cy := bounds.Center() if v.enumerateBuiltinFn == nil {
fromX, fromY, toX, toY = cx, cy, cx, cy return nil, errors.New("verifier: builtin enumeration not available")
switch direction {
case "down":
toY = cy - 4*bounds.Height()/10
case "up":
toY = cy + 4*bounds.Height()/10
case "left":
toX = cx + 4*bounds.Width()/10
case "right":
toX = cx - 4*bounds.Width()/10
} }
return fromX, fromY, max(0, toX), max(0, toY) value, err := v.enumerateBuiltinFn(goja.Undefined(), v.runtime.ToValue(verb))
} if err != nil {
return nil, fmt.Errorf("enumerate %s: %w", verb, err)
// elementsForVerb returns the in-scope elements a builtin verb applies to, in
// tree order (the seeded picker's enumeration order), reusing verbAccepts.
func (v *Verifier) elementsForVerb(verb string) []*hierarchy.Element {
scope := v.scopedElements()
var elements []*hierarchy.Element
for _, element := range v.lastTree.Elements {
if scope[element] && verbAccepts(verb, element) {
elements = append(elements, element)
} }
if value == nil || goja.IsUndefined(value) || goja.IsNull(value) {
return nil, nil
} }
return elements raw, err := json.Marshal(value.Export())
if err != nil {
return nil, fmt.Errorf("marshal %s enumeration: %w", verb, err)
}
var wire []struct {
Action json.RawMessage `json:"action"`
TargetIndex int `json:"targetIndex"`
}
if err := json.Unmarshal(raw, &wire); err != nil {
return nil, fmt.Errorf("decode %s enumeration: %w", verb, err)
}
entries := make([]builtinCandidate, 0, len(wire))
for _, item := range wire {
action, err := DecodeAction(item.Action)
if err != nil {
continue
}
entries = append(entries, builtinCandidate{action: action, targetIndex: item.TargetIndex})
}
return entries, nil
} }
// finalizeCandidates renders each candidate's description, dedups identical // finalizeCandidates renders each candidate's description, dedups identical
@@ -555,6 +512,18 @@ func describeCandidate(candidate ActionCandidate) string {
return fmt.Sprintf("Type %q into %q", candidate.Action.Text, candidate.Label) return fmt.Sprintf("Type %q into %q", candidate.Action.Text, candidate.Label)
case ActionKindScroll: case ActionKindScroll:
return "Scroll " + candidate.Direction return "Scroll " + candidate.Direction
case ActionKindSwipe:
// A swipe carries endpoints and no selector, so the coordinates are what
// keep two swipes distinct. The label is prepended when the origin
// element has one, because "swipe that row" is the interaction a model
// reaches for and a bare pair of points does not say which row.
where := fmt.Sprintf("from (%d,%d) to (%d,%d)",
candidate.Action.FromX, candidate.Action.FromY,
candidate.Action.ToX, candidate.Action.ToY)
if candidate.Label == "" {
return "Swipe " + where
}
return fmt.Sprintf("Swipe %q %s", candidate.Label, where)
case ActionKindPressKey: case ActionKindPressKey:
return "Press " + candidate.Action.Key return "Press " + candidate.Action.Key
case ActionKindWait: case ActionKindWait:
@@ -682,3 +651,13 @@ func stringField(object *goja.Object, key string) string {
} }
return value.String() return value.String()
} }
// intField reads a numeric property off a goja object, returning 0 when absent,
// null, or undefined.
func intField(object *goja.Object, key string) int {
value := object.Get(key)
if value == nil || goja.IsUndefined(value) || goja.IsNull(value) {
return 0
}
return int(value.ToInteger())
}
+51 -16
View File
@@ -10,7 +10,7 @@ import (
// enumTreeJSON exercises every labeling path: a clickable wrapper whose own text // enumTreeJSON exercises every labeling path: a clickable wrapper whose own text
// is empty but whose child Text reads "Add credit" (descendant borrowing), an // is empty but whose child Text reads "Add credit" (descendant borrowing), an
// editable field labeled by its hint, a text-labeled button, a DISABLED button, // editable field labeled by its hint, a text-labeled button, a DISABLED button,
// and a scrollable list (the only valid gesture origin). // and a scrollable list (the only valid scroll origin).
const enumTreeJSON = `{ const enumTreeJSON = `{
"attributes": {"bounds": "[0,0,1080,2400]"}, "attributes": {"bounds": "[0,0,1080,2400]"},
"children": [ "children": [
@@ -25,10 +25,13 @@ const enumTreeJSON = `{
}` }`
// enumVerifier loads a spec whose actions root is the given plain-object graph // enumVerifier loads a spec whose actions root is the given plain-object graph
// and stages the given tree, so Candidates walks a controlled action tree. // and stages the given tree, so Candidates walks a controlled action tree. The
// spec is bundled with the goja runtime entry because the model arm reads the
// picker's own builtin enumeration out of that bundle.
func enumVerifier(t *testing.T, actionsJS, treeJSON string) *Verifier { func enumVerifier(t *testing.T, actionsJS, treeJSON string) *Verifier {
t.Helper() t.Helper()
v := newLoadedVerifier(t, "globalThis.actions = "+actionsJS+";") v := newVerifier(t)
loadActionSpec(t, v, "globalThis.actions = "+actionsJS+";")
tree, err := hierarchy.Parse(treeJSON) tree, err := hierarchy.Parse(treeJSON)
if err != nil { if err != nil {
t.Fatalf("parse tree: %v", err) t.Fatalf("parse tree: %v", err)
@@ -119,26 +122,38 @@ func TestCandidatesLabelsEditableFieldByHintNotTypedValue(t *testing.T) {
} }
} }
func TestCandidatesFoldsGesturesIntoDirectionalScrolls(t *testing.T) { func TestCandidatesKeepsGestureVerbsDistinct(t *testing.T) {
v := enumVerifier(t, v := enumVerifier(t,
"{kind:'weighted', branches:[[1,{kind:'builtin',verb:'scrolls'}],[1,{kind:'builtin',verb:'swipes'}]]}", "{kind:'weighted', branches:[[1,{kind:'builtin',verb:'scrolls'}],[1,{kind:'builtin',verb:'swipes'}]]}",
enumTreeJSON) enumTreeJSON)
candidates := v.Candidates() candidates := v.Candidates()
// Gestures are directional and scoped to the one scrollable container: no // `scrolls` folds to one directional pair over the single scrollable
// per-element, element-labeled Swipe entries. // container, which is what keeps the list short.
for _, candidate := range candidates {
if strings.HasPrefix(candidate.Description, "Swipe") {
t.Errorf("gesture kept as element-labeled swipe: %q", candidate.Description)
}
}
if !hasCandidate(candidates, "Scroll down") || !hasCandidate(candidates, "Scroll up") { if !hasCandidate(candidates, "Scroll down") || !hasCandidate(candidates, "Scroll up") {
t.Errorf("want directional scrolls, got %v", descriptions(candidates)) t.Errorf("want directional scrolls, got %v", descriptions(candidates))
} }
// scrolls and swipes fold into the SAME directional entries: one each.
if got := count(candidates, "Scroll down"); got != 1 { if got := count(candidates, "Scroll down"); got != 1 {
t.Errorf("Scroll down appears %d times, want 1 (folded)", got) t.Errorf("Scroll down appears %d times, want 1 over the one container", got)
} }
// `swipes` is a different verb, not a second name for the scroll: a
// free-form drag from any element, named by the control it starts on. That
// is what puts swipe-to-dismiss on a row within the model's reach.
if !hasCandidatePrefix(candidates, `Swipe "Sign in"`) {
t.Errorf("want a swipe naming the non-scrollable row, got %v", descriptions(candidates))
}
if hasCandidatePrefix(candidates, "Scroll \"") {
t.Errorf("scroll candidates must stay container-scoped: %v", descriptions(candidates))
}
}
func hasCandidatePrefix(candidates []ActionCandidate, prefix string) bool {
for _, candidate := range candidates {
if strings.HasPrefix(candidate.Description, prefix) {
return true
}
}
return false
} }
func TestCandidatesWeightsCombineAcrossPaths(t *testing.T) { func TestCandidatesWeightsCombineAcrossPaths(t *testing.T) {
@@ -228,6 +243,23 @@ func TestCandidatesCallsAuthoredLeafOnce(t *testing.T) {
} }
} }
func TestCandidatesSurfaceAuthoredUntargetedActions(t *testing.T) {
// A spec that authors a swipe, a key press, or a wait must reach the model
// with all three: the seeded picker executes whatever the leaf returns, so a
// kind the enumeration drops is an action only one policy can take.
actions := `{kind:'actions', generate: () => [
{kind:'Swipe', from:{x:10,y:600}, to:{x:10,y:100}, durationMillis: 250},
{kind:'PressKey', key:'back'},
{kind:'Wait'}
]}`
candidates := enumVerifier(t, actions, enumTreeJSON).Candidates()
for _, want := range []string{"Swipe from (10,600) to (10,100)", "Press back", "Wait"} {
if !hasCandidate(candidates, want) {
t.Errorf("authored %q missing: %v", want, descriptions(candidates))
}
}
}
func TestCandidatesOffRouteLeafYieldsNothing(t *testing.T) { func TestCandidatesOffRouteLeafYieldsNothing(t *testing.T) {
v := enumVerifier(t, "{kind:'actions', generate: () => []}", enumTreeJSON) v := enumVerifier(t, "{kind:'actions', generate: () => []}", enumTreeJSON)
if got := v.Candidates(); len(got) != 0 { if got := v.Candidates(); len(got) != 0 {
@@ -248,9 +280,14 @@ func TestCandidatesSkipsCrossFadeFrames(t *testing.T) {
] ]
}` }`
v := enumVerifier(t, "{kind:'builtin', verb:'taps'}", crossFade) v := enumVerifier(t, "{kind:'builtin', verb:'taps'}", crossFade)
if got := v.Candidates(); got != nil { if got := v.Candidates(); len(got) != 0 {
t.Errorf("cross-fade frame should yield no candidates, got %v", descriptions(got)) t.Errorf("cross-fade frame should yield no candidates, got %v", descriptions(got))
} }
// The seeded policy is skipped by the SAME guard, in the shared producer,
// so neither arm acts on a mid-animation layout.
if got := v.targets(); len(got) != 0 {
t.Errorf("cross-fade frame should yield no host targets, got %d", len(got))
}
} }
func TestCandidatesNilWithoutTreeOrActions(t *testing.T) { func TestCandidatesNilWithoutTreeOrActions(t *testing.T) {
@@ -345,5 +382,3 @@ func newLoadedVerifier(t *testing.T, source string) *Verifier {
} }
return v return v
} }
+11 -8
View File
@@ -48,22 +48,25 @@ func TestCrossRuntimeParity(t *testing.T) {
} }
// installStubHost replaces the verifier's hierarchy-backed __sanderlingHost__ // installStubHost replaces the verifier's hierarchy-backed __sanderlingHost__
// with one returning a FIXED candidate list for every verb, keeping the seed the // with one returning a FIXED target list, keeping the seed the verifier was
// verifier was constructed with. It must run BEFORE Load, because the bundled // constructed with. Every fact is set so the shared eligibility rule admits all
// goja runtime entry captures the host when the spec evaluates. // three targets for every verb, leaving the draw order as the only variable. It
// must run BEFORE Load, because the bundled goja runtime entry captures the host
// when the spec evaluates.
func installStubHost(t *testing.T, verifier *Verifier) { func installStubHost(t *testing.T, verifier *Verifier) {
t.Helper() t.Helper()
const stub = ` const stub = `
const candidates = [ const facts = { clickable: true, enabled: true, editable: true, scrollable: true };
{ x: 50, y: 60, selector: "id:alpha", width: 100, height: 40 }, const targets = [
{ x: 150, y: 160, selector: "id:beta", width: 120, height: 48 }, { x: 50, y: 60, selector: "id:alpha", width: 100, height: 40, ...facts },
{ x: 250, y: 260, selector: "id:gamma", width: 80, height: 32 }, { x: 150, y: 160, selector: "id:beta", width: 120, height: 48, ...facts },
{ x: 250, y: 260, selector: "id:gamma", width: 80, height: 32, ...facts },
]; ];
const seedHi = globalThis.__sanderlingHost__.seedHi; const seedHi = globalThis.__sanderlingHost__.seedHi;
const seedLo = globalThis.__sanderlingHost__.seedLo; const seedLo = globalThis.__sanderlingHost__.seedLo;
globalThis.__sanderlingHost__ = { globalThis.__sanderlingHost__ = {
platform: () => "android", platform: () => "android",
queryCandidates: () => candidates, queryTargets: () => targets,
reportUnsupported: () => {}, reportUnsupported: () => {},
seedHi, seedHi,
seedLo, seedLo,
+161
View File
@@ -0,0 +1,161 @@
package verifier
import (
"errors"
"fmt"
"maps"
"slices"
"testing"
)
// policyTreeJSON gives every builtin verb something to act on, with every label
// distinct so no two candidates render the same way.
const policyTreeJSON = `{
"attributes": {"bounds": "[0,0,400,800]"},
"children": [
{"attributes": {"resource-id": "Save", "text": "Save", "bounds": "[0,0,200,60]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "Cancel", "text": "Cancel", "bounds": "[200,0,400,60]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "Amount", "class": "EditText", "hintText": "Amount", "bounds": "[0,100,400,160]"}, "enabled": true, "children": []},
{"attributes": {"resource-id": "Note", "class": "EditText", "hintText": "Note", "bounds": "[0,200,400,260]"}, "enabled": true, "children": []},
{"attributes": {"resource-id": "List", "scrollable": "true", "bounds": "[0,300,400,700]"}, "children": []}
]
}`
// policyVerbs is every builtin verb a spec can put in its action tree.
var policyVerbs = []string{
"taps", "doubleTaps", "longPresses", "typing", "scrolls", "swipes", "pressKeys", "waitOnce",
}
// seededDrawBudget is how many times the seeded picker is driven per verb. The
// candidate sets here are a handful of entries wide, so this exhausts them many
// times over; a miss would mean the picker cannot reach one of its own
// candidates, which is itself the bug worth failing on.
const seededDrawBudget = 300
// TestPoliciesEnumerateTheSameActions is the guard on the claim the paper makes
// about the two action policies: they differ in the pick and in nothing else.
// For every builtin verb, the actions the seeded picker can draw and the
// candidates the model policy is offered must be the same set. Enumeration lives
// in one place (pick.ts builtinCandidates) precisely so this cannot drift, and
// this test is what notices if a second one ever grows back.
func TestPoliciesEnumerateTheSameActions(t *testing.T) {
for _, verb := range policyVerbs {
t.Run(verb, func(t *testing.T) {
seeded := seededReachableActions(t, verb)
model := modelOfferedActions(t, verb)
if len(model) == 0 {
t.Fatalf("%s offered the model no candidates at all", verb)
}
if !slices.Equal(slices.Sorted(maps.Keys(seeded)), slices.Sorted(maps.Keys(model))) {
t.Errorf("action spaces differ for %s\n seeded=%v\n model=%v",
verb, slices.Sorted(maps.Keys(seeded)), slices.Sorted(maps.Keys(model)))
}
})
}
}
// TestModelCandidateDescriptionsAreUniqueAndNamed checks the rendering half of
// the contract: every enumerated action reaches the model as its own distinctly
// named line, so the number it picks and the action that executes agree.
func TestModelCandidateDescriptionsAreUniqueAndNamed(t *testing.T) {
for _, verb := range policyVerbs {
t.Run(verb, func(t *testing.T) {
verifier := loadVerbSpec(t, verb)
seen := map[string]bool{}
for _, candidate := range verifier.Candidates() {
if candidate.Description == "" {
t.Fatalf("%s produced a candidate with no description: %+v", verb, candidate.Action)
}
if seen[candidate.Description] {
t.Errorf("%s rendered %q twice, so the model cannot address both",
verb, candidate.Description)
}
seen[candidate.Description] = true
}
})
}
}
// TestModelIsOfferedTheUntargetedVerbs pins the two verbs the model arm used to
// be blind to: with no element to enumerate over, a key press and a wait were
// dropped, so the model could never navigate back or let the app settle.
func TestModelIsOfferedTheUntargetedVerbs(t *testing.T) {
verifier := loadVerbSpec(t, "pressKeys")
if !hasCandidate(verifier.Candidates(), "Press back") {
t.Errorf("pressKeys missing from the model's candidates: %v",
descriptions(verifier.Candidates()))
}
verifier = loadVerbSpec(t, "waitOnce")
if !hasCandidate(verifier.Candidates(), "Wait") {
t.Errorf("waitOnce missing from the model's candidates: %v",
descriptions(verifier.Candidates()))
}
}
// loadVerbSpec builds a verifier whose whole action tree is one builtin verb,
// with policyTreeJSON pushed as the current state.
func loadVerbSpec(t *testing.T, verb string) *Verifier {
t.Helper()
verifier := newVerifier(t, WithSeed(0x5eed))
loadActionSpec(t, verifier, fmt.Sprintf(
"import { %s } from \"@sanderling/spec\";\nglobalThis.actions = %s;", verb, verb))
pushTree(t, verifier, policyTreeJSON)
return verifier
}
// seededReachableActions drives the seeded picker over its draw budget and
// returns every distinct action it produced.
func seededReachableActions(t *testing.T, verb string) map[string]Action {
t.Helper()
verifier := loadVerbSpec(t, verb)
reachable := map[string]Action{}
for range seededDrawBudget {
action, err := verifier.NextAction()
if errors.Is(err, ErrNoAction) {
continue
}
if err != nil {
t.Fatalf("%s next action: %v", verb, err)
}
reachable[actionIdentity(action)] = action
}
return reachable
}
// modelOfferedActions returns the actions behind the numbered list the model
// policy picks from.
func modelOfferedActions(t *testing.T, verb string) map[string]Action {
t.Helper()
verifier := loadVerbSpec(t, verb)
offered := map[string]Action{}
for _, candidate := range verifier.Candidates() {
offered[actionIdentity(candidate.Action)] = candidate.Action
}
return offered
}
// actionIdentity keys an action by everything except the values the policy owns
// rather than the candidate set: the typed text, which the seeded arm draws from
// the edge-case corpus and the model writes itself, and a swipe's drag distance,
// which the seeded arm draws and the enumeration lists at a nominal length.
// Comparing those would compare policies instead of action spaces. A swipe's
// direction is NOT policy-owned, so it survives as the sign of the drag.
func actionIdentity(action Action) string {
action.Text = ""
if action.Kind == ActionKindSwipe {
action.ToX = sign(action.ToX - action.FromX)
action.ToY = sign(action.ToY - action.FromY)
}
return fmt.Sprintf("%+v", action)
}
func sign(value int) int {
switch {
case value > 0:
return 1
case value < 0:
return -1
default:
return 0
}
}
+92 -14
View File
@@ -105,30 +105,108 @@ func TestTyping_ExcludeOffAppPackage(t *testing.T) {
} }
} }
// TestSwipes_ExcludeOffAppPackage proves swipes anchor on app nodes only, so // gestureTreeJSON gives each gesture verb a legal and an illegal anchor: an app
// exploration never scrolls the keyboard's emoji list instead of the app. // list holding a plain row, against the soft keyboard's own scrollable strip
func TestSwipes_ExcludeOffAppPackage(t *testing.T) { // holding one of its keys.
const gestureTreeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,500]", "package": "com.folio"},
"children": [
{"attributes": {"testTag": "AppList", "scrollable": "true", "bounds": "[0,0,100,300]", "package": "com.folio"}, "children": [
{"attributes": {"testTag": "Row", "bounds": "[0,0,100,60]", "package": "com.folio"}, "children": []}
]},
{"attributes": {"testTag": "EmojiStrip", "scrollable": "true", "bounds": "[0,400,100,440]", "package": "com.google.android.inputmethod.latin"}, "children": [
{"attributes": {"testTag": "EmojiKey", "bounds": "[0,400,100,420]", "package": "com.google.android.inputmethod.latin"}, "children": []}
]}
]
}`
// TestGestures_ExcludeOffAppPackage proves both gesture verbs anchor on app
// nodes only, so exploration never drags the keyboard instead of the app, and
// pins what each verb accepts within the app and which way it drags. Scrolls
// anchor on the scrollable list alone and stay vertical, because every
// scrollable container earns a candidate and scrolling a list means up and down.
// Swipes anchor on any app element, the row and the root included, and drag in
// all four directions, which is what puts swipe-to-dismiss on a list row inside
// the action space.
func TestGestures_ExcludeOffAppPackage(t *testing.T) {
tests := []struct {
verb string
kind ActionKind
anchors map[[2]int]bool
directions map[string]bool
}{
{
"scrolls",
ActionKindScroll,
map[[2]int]bool{{50, 150}: true},
map[string]bool{"down": true, "up": true},
},
{
"swipes",
ActionKindSwipe,
map[[2]int]bool{{50, 250}: true, {50, 150}: true, {50, 30}: true},
map[string]bool{"down": true, "up": true, "left": true, "right": true},
},
}
for _, test := range tests {
t.Run(test.verb, func(t *testing.T) {
verifier := newVerifier(t, WithAppPackage("com.folio")) verifier := newVerifier(t, WithAppPackage("com.folio"))
loadActionSpec(t, verifier, ` loadActionSpec(t, verifier, `
import { swipes } from "@sanderling/spec"; import { `+test.verb+` } from "@sanderling/spec";
globalThis.actions = swipes; globalThis.actions = `+test.verb+`;
`) `)
pushTree(t, verifier, scopedTreeJSON) pushTree(t, verifier, gestureTreeJSON)
// Both the root and SubmitButton (com.folio) are valid anchors; only the // Draw many times so neither the exclusion nor the coverage below is
// keyboard key at center (50,420) must be excluded. Draw many times so the // satisfied by a lucky seed. The keyboard strip (50,420) and its key
// invariant is not satisfied by a lucky seed. // (50,410) are the anchors that must never appear.
for i := range 200 { seen := map[[2]int]bool{}
seenDirections := map[string]bool{}
for i := range 400 {
action, err := verifier.NextAction() action, err := verifier.NextAction()
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if action.Kind != ActionKindSwipe { if action.Kind != test.kind {
t.Fatalf("kind = %v, want Swipe", action.Kind) t.Fatalf("kind = %v, want %v", action.Kind, test.kind)
} }
if action.FromX == 50 && action.FromY == 420 { anchor := [2]int{action.FromX, action.FromY}
t.Fatalf("draw %d anchored on the keyboard key (50,420); off-app node leaked into swipe targets", i) if !test.anchors[anchor] {
t.Fatalf("draw %d anchored at %v, outside %v", i, anchor, test.anchors)
} }
direction := gestureDirection(action)
if !test.directions[direction] {
t.Fatalf("draw %d dragged %s, outside %v", i, direction, test.directions)
}
seen[anchor] = true
seenDirections[direction] = true
}
if len(seen) != len(test.anchors) {
t.Errorf("reached anchors %v, want all of %v", seen, test.anchors)
}
if len(seenDirections) != len(test.directions) {
t.Errorf("reached directions %v, want all of %v", seenDirections, test.directions)
}
})
}
}
// gestureDirection names which way a drawn gesture drags. A scroll carries the
// name it was enumerated under; a swipe carries only its endpoints, so the sign
// of the drag is what says where the finger went.
func gestureDirection(action Action) string {
if action.Kind == ActionKindScroll {
return action.Direction
}
switch {
case action.ToX > action.FromX:
return "right"
case action.ToX < action.FromX:
return "left"
case action.ToY > action.FromY:
return "down"
default:
return "up"
} }
} }
+40
View File
@@ -1276,6 +1276,46 @@ func TestOverrideExtractorValues_PropagatesNestedObjectFields(t *testing.T) {
} }
} }
// TestOverrideExtractorValues_RecordedStateMatchesEvaluatedState pins the
// reported state to the state predicates read. The web path evaluates
// extractor bodies in V8 and injects the results here, so a diff or a witness
// built from the goja value would describe a state no property ever saw.
func TestOverrideExtractorValues_RecordedStateMatchesEvaluatedState(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
if err := verifier.PushSnapshot(SnapshotInput{
Snapshots: Snapshots{"ledger.balance": json.RawMessage(`100`)},
}); err != nil {
t.Fatal(err)
}
if _, err := verifier.OverrideExtractorValues(map[int]json.RawMessage{
1: json.RawMessage(`-7`),
}); err != nil {
t.Fatal(err)
}
verifier.EvaluateProperties()
balance := verifier.runtime.GlobalObject().Get("balance").ToObject(verifier.runtime)
evaluated := balance.Get("current").String()
change, ok := verifier.ChangedExtractors()["extractor_1"]
if !ok {
t.Fatal("ChangedExtractors reported no change for the overridden extractor")
}
if string(change.Curr) != evaluated {
t.Errorf("ChangedExtractors curr = %s, want %s (the value predicates read)",
change.Curr, evaluated)
}
witness := verifier.Witness("balanceNonNegative")
if witness == nil {
t.Fatal("balanceNonNegative did not violate on the overridden value")
}
if got := string(witness.Extractors["extractor_1"]); got != evaluated {
t.Errorf("witness extractor = %s, want %s (the value predicates read)",
got, evaluated)
}
}
// TestUnsupportedVerbs_CollectedDedupedInOrder drives the real host binding the // TestUnsupportedVerbs_CollectedDedupedInOrder drives the real host binding the
// shared picker invokes (__sanderlingHost__.reportUnsupported) and asserts the // shared picker invokes (__sanderlingHost__.reportUnsupported) and asserts the
// verifier collects each verb once, in first-seen order, for the run report. // verifier collects each verb once, in first-seen order, for the run report.
+80 -47
View File
@@ -39,6 +39,12 @@ type Verifier struct {
// reimplementing the corpus on the Go side. // reimplementing the corpus on the Go side.
sampleInputFn goja.Callable sampleInputFn goja.Callable
// enumerateBuiltinFn is the bundle-installed __sanderlingEnumerateBuiltin__,
// which lists every action a builtin verb can yield right now. It is the same
// enumeration the seeded picker draws from, so the LLM action backend selects
// over the picker's action space rather than one of its own.
enumerateBuiltinFn goja.Callable
evaluators map[string]*ltl.Evaluator evaluators map[string]*ltl.Evaluator
priorVerdicts map[string]ltl.Verdict priorVerdicts map[string]ltl.Verdict
@@ -49,6 +55,8 @@ type Verifier struct {
lastScreenshot []byte lastScreenshot []byte
scopeCache map[*hierarchy.Element]bool scopeCache map[*hierarchy.Element]bool
scopeCacheTree *hierarchy.Tree scopeCacheTree *hierarchy.Tree
targetCache []targetElement
targetCacheTree *hierarchy.Tree
lastAction *Action lastAction *Action
lastLogs []LogEntry lastLogs []LogEntry
lastExceptions []Exception lastExceptions []Exception
@@ -178,18 +186,30 @@ func (v *Verifier) Load(source string) error {
} }
} }
if fn := v.runtime.GlobalObject().Get("__sanderlingEnumerateBuiltin__"); fn != nil {
if callable, ok := goja.AssertFunction(fn); ok {
v.enumerateBuiltinFn = callable
}
}
return nil return nil
} }
// buildFormula walks the formula-spec registry and produces a Go ltl.Formula // buildFormula walks the formula-spec registry and produces a Go ltl.Formula
// tree rooted at the given spec index. Specs built at the top level are // tree rooted at the given spec index.
// always wrapped in Always unless the top-level spec is already an Always. //
// A top-level spec that is not already a temporal obligation is wrapped in
// Always, which is what an author writing a bare predicate or a combinator
// means. An always is left alone, and so is an eventually: wrapping
// `eventually(p).within(5, "minutes")` would turn one reachability goal into
// "within five minutes of every step", a different and far stronger property.
func (v *Verifier) buildFormula(rootIndex int) (ltl.Formula, error) { func (v *Verifier) buildFormula(rootIndex int) (ltl.Formula, error) {
inner, err := v.buildFormulaNode(rootIndex) inner, err := v.buildFormulaNode(rootIndex)
if err != nil { if err != nil {
return nil, err return nil, err
} }
if _, ok := inner.(ltl.AlwaysFormula); ok { switch inner.(type) {
case ltl.AlwaysFormula, ltl.EventuallyFormula:
return inner, nil return inner, nil
} }
return ltl.Always(inner), nil return ltl.Always(inner), nil
@@ -386,6 +406,12 @@ func (v *Verifier) ChangedExtractors() map[string]ExtractorChange {
// call this unconditionally. The override must run *after* PushSnapshot // call this unconditionally. The override must run *after* PushSnapshot
// (which advanced `previous`) and *before* EvaluateProperties. // (which advanced `previous`) and *before* EvaluateProperties.
// //
// The JSON snapshot `curr` is replaced alongside the value, so the diffs in
// ChangedExtractors and the witness recorded by captureWitness describe the
// state the verdict was computed from. Recording the goja value while a
// predicate read the V8 one makes a witness explain a violation with a state
// that never reached the property.
//
// Out-of-range indices are tolerated (skipped) rather than fatal: V8 and goja // Out-of-range indices are tolerated (skipped) rather than fatal: V8 and goja
// register extractors from the same spec bundle so counts should always // register extractors from the same spec bundle so counts should always
// match, but a stale or partial override map should not block valid overrides // match, but a stale or partial override map should not block valid overrides
@@ -405,6 +431,7 @@ func (v *Verifier) OverrideExtractorValues(overrides map[int]json.RawMessage) (s
return skipped, fmt.Errorf("extractor override %d: %w", index, conversionErr) return skipped, fmt.Errorf("extractor override %d: %w", index, conversionErr)
} }
v.extractors[index].currentValue = value v.extractors[index].currentValue = value
v.extractors[index].curr = encodeExtractorValue(value)
} }
return skipped, nil return skipped, nil
} }
@@ -470,21 +497,27 @@ func (v *Verifier) EvaluateProperties() map[string]ltl.Verdict {
} }
// Witness is the verifier-level record of a property violation: the LTL reason // Witness is the verifier-level record of a property violation: the LTL reason
// (a predicate's thrown-error text, "predicate false", or a liveness failure), // (a predicate's thrown-error text, "predicate false", or a liveness failure)
// the step it fired at, and a snapshot of every extractor's current value at // and the two step indices a deferred obligation spans.
// that step. The snapshot lets a reader see the state that produced the //
// violation without replaying the run. // Step is the origin: the step whose observation armed the obligation that
// failed. DetectedStep is the observation whose reduction produced the
// violation, which for a next or an eventually is later. Extractors is that
// observation's state, so it belongs to DetectedStep and not to Step; the two
// were previously conflated under one index.
type Witness struct { type Witness struct {
Property string Property string
Reason string Reason string
Step int Step int
DetectedStep int
IsError bool IsError bool
Extractors map[string]json.RawMessage Extractors map[string]json.RawMessage
} }
// captureWitness records the witness for a property that just transitioned to // captureWitness records the witness for a property that just transitioned to
// violated, snapshotting the current extractor values so the cause is visible // violated, snapshotting the current extractor values so the cause is visible
// after the run. // after the run. The snapshot is the state of the observation being reduced,
// which the witness records as its detection step.
func (v *Verifier) captureWitness(name string) { func (v *Verifier) captureWitness(name string) {
evaluator, ok := v.evaluators[name] evaluator, ok := v.evaluators[name]
if !ok { if !ok {
@@ -498,6 +531,7 @@ func (v *Verifier) captureWitness(name string) {
Property: name, Property: name,
Reason: violation.Reason, Reason: violation.Reason,
Step: violation.Step, Step: violation.Step,
DetectedStep: v.stepIndex,
IsError: violation.IsError, IsError: violation.IsError,
Extractors: v.extractorSnapshot(), Extractors: v.extractorSnapshot(),
} }
@@ -730,66 +764,65 @@ func selectorForElement(tree *hierarchy.Tree, element *hierarchy.Element) string
return "" return ""
} }
// candidatesForVerb enumerates the host-side targets a builtin verb may draw // targets enumerates every element this host can offer, in v.lastTree.Elements
// from, in v.lastTree.Elements ORDER (the order is part of the picker's parity // ORDER (the order is part of the picker's parity contract). It is the native
// contract). The filters are LIFTED from the old Go picker: // half of the single candidate producer: the picker reads it through
// __sanderlingHost__.queryTargets, applies the SHARED per-verb eligibility rule
// (pkg/spec/src/targets.ts), and expands what survives into concrete actions for
// both policies. Which verb may act on which element is decided there, once, so
// this host and the web host cannot mean different things by the same verb.
// //
// taps/doubleTaps/longPresses: clickable + enabled + positive bounds // This host's job is the facts: clickable/enabled/editable come off the
// typing: editable + enabled + positive bounds // accessibility node, scrollable off its attribute, and the geometry off its
// scrolls: scrollable attribute + positive bounds // bounds. Every target carries the resolving selector so the runner can re-route
// swipes: any in-scope element with positive bounds // by id/text. Out-of-scope nodes (the soft keyboard, system UI, the launcher)
// are dropped by scopedElements.
// //
// Every candidate carries the resolving selector so the runner can re-route by // A cross-fade frame yields nothing at all. Its layout is mid-animation, often
// id/text. Out-of-scope nodes (the soft keyboard, system UI, the launcher) are // in a collapsed coordinate space, so acting on it lands on garbage; skipping it
// dropped by scopedElements. // here rather than in one policy means both policies re-observe a settled frame
func (v *Verifier) candidatesForVerb(verb string) []candidate { // instead of one of them acting on the animation.
if v.lastTree == nil { func (v *Verifier) targets() []targetElement {
if v.lastTree == nil || v.lastTree.Transitional() {
return nil return nil
} }
if v.targetCacheTree == v.lastTree {
return v.targetCache
}
scope := v.scopedElements() scope := v.scopedElements()
var result []candidate result := make([]targetElement, 0, len(v.lastTree.Elements))
for _, element := range v.lastTree.Elements { for _, element := range v.lastTree.Elements {
if !scope[element] { if !scope[element] {
continue continue
} }
if !verbAccepts(verb, element) {
continue
}
x, y := element.Bounds.Center() x, y := element.Bounds.Center()
result = append(result, candidate{ result = append(result, targetElement{
element: element,
x: x, x: x,
y: y, y: y,
width: element.Bounds.Width(), width: element.Bounds.Width(),
height: element.Bounds.Height(), height: element.Bounds.Height(),
selector: selectorForElement(v.lastTree, element), selector: selectorForElement(v.lastTree, element),
clickable: element.Clickable,
enabled: element.Enabled,
editable: element.Editable,
scrollable: element.Attributes["scrollable"] == "true",
}) })
} }
v.targetCache = result
v.targetCacheTree = v.lastTree
return result return result
} }
type candidate struct { // targetElement is one host-enumerated element with the facts the shared
// eligibility rule reads. The host reports facts; it does not filter by verb.
type targetElement struct {
element *hierarchy.Element
x, y int x, y int
width, height int width, height int
selector string selector string
} clickable bool
enabled bool
// verbAccepts applies the per-verb element filter. editable bool
func verbAccepts(verb string, element *hierarchy.Element) bool { scrollable bool
positiveBounds := element.Bounds.Width() > 0 && element.Bounds.Height() > 0
switch verb {
case "taps", "doubleTaps", "longPresses":
return element.Clickable && element.Enabled && positiveBounds
case "typing":
return element.Editable && element.Enabled && positiveBounds
case "scrolls":
return element.Attributes["scrollable"] == "true" && positiveBounds
case "swipes":
// Any visible element is a valid swipe origin, but it must have real
// bounds: a zero-bounds node centers at (0,0), and a downward swipe from
// the top-left corner is the system gesture that pulls down the
// notification shade, dragging the fuzzer out of the app.
return positiveBounds
default:
return false
}
} }
+33 -12
View File
@@ -1,24 +1,45 @@
package verifier package verifier
import ( import (
"slices"
"testing" "testing"
"github.com/priyanshujain/sanderling/internal/hierarchy" "github.com/priyanshujain/sanderling/internal/hierarchy"
) )
// TestVerbAcceptsSwipeRequiresPositiveBounds locks the fix for the notification // TestTargetsReportFactsWithoutFiltering pins the native host's half of the
// shade: a zero-bounds element centers at (0,0), and a downward swipe from the // split introduced to stop the two hosts drifting: it reports what an element
// top-left corner is the system gesture that pulls the shade over the app. The // IS and never decides which verb may act on it. The verb decision is one shared
// swipe verb must reject zero-bounds nodes like every other verb does. // rule (pkg/spec/src/targets.ts) both hosts consume, asserted across engines by
func TestVerbAcceptsSwipeRequiresPositiveBounds(t *testing.T) { // TestHostsAgreeOnTargetEligibility.
zeroBounds := &hierarchy.Element{Bounds: hierarchy.Bounds{}} func TestTargetsReportFactsWithoutFiltering(t *testing.T) {
if verbAccepts("swipes", zeroBounds) { tree, err := hierarchy.Parse(hostParityTreeJSON)
t.Error("swipes must reject a zero-bounds element (it centers at (0,0) and pulls the notification shade)") if err != nil {
t.Fatal(err)
}
v := &Verifier{lastTree: tree}
targets := v.targets()
if len(targets) != len(hostParityScreen) {
t.Fatalf("targets() returned %d elements, want all %d in the tree",
len(targets), len(hostParityScreen))
}
for index, target := range targets {
want := hostParityScreen[index]
got := []bool{
target.clickable, target.enabled, target.editable, target.scrollable,
}
expected := []bool{
want.clickable, want.enabled, want.editable, want.scrollable,
}
if !slices.Equal(got, expected) {
t.Errorf("%s clickable/enabled/editable/scrollable = %v, want %v",
want.name, got, expected)
}
positiveBounds := target.width > 0 && target.height > 0
if positiveBounds != want.positiveBounds {
t.Errorf("%s positive bounds = %v, want %v",
want.name, positiveBounds, want.positiveBounds)
} }
realBounds := &hierarchy.Element{Bounds: hierarchy.Bounds{Left: 100, Top: 400, Right: 980, Bottom: 600}}
if !verbAccepts("swipes", realBounds) {
t.Error("swipes must accept an element with positive bounds")
} }
} }
+19 -7
View File
@@ -65,9 +65,9 @@ export type GeneratorNode =
| { kind: "builtin"; verb: BuiltinVerb } | { kind: "builtin"; verb: BuiltinVerb }
| { kind: "llm"; config: { model: string; instructions?: string } }; | { kind: "llm"; config: { model: string; instructions?: string } };
// Candidate is one host-enumerated target for a builtin verb. The host // Candidate is the resolved geometry of one target. The host resolves it (and a
// resolves geometry (and a native selector) so no element handle crosses into // native selector) so no element handle crosses into the picker. width/height
// the picker. width/height let swipe/scroll size a gesture off the element. // let swipe/scroll size a gesture off the element.
export interface Candidate { export interface Candidate {
x: number; x: number;
y: number; y: number;
@@ -76,15 +76,27 @@ export interface Candidate {
height?: number; height?: number;
} }
// TargetElement is one host-enumerated element, offered to EVERY verb, carrying
// the facts per-verb eligibility is decided from. The host reports the facts; it
// does not apply them. targets.ts acceptsTarget owns that decision for both
// hosts, so a verb cannot mean one thing on native and another on web.
export interface TargetElement extends Candidate {
clickable: boolean;
enabled: boolean;
editable: boolean;
scrollable: boolean;
}
// Host is the platform backing the picker draws against. In this foundation // Host is the platform backing the picker draws against. In this foundation
// workflow only the interface is defined and exercised against a stub; the // workflow only the interface is defined and exercised against a stub; the
// goja and DOM implementations land in the rewire workflow. // goja and DOM implementations land in the rewire workflow.
export interface Host { export interface Host {
platform(): "android" | "ios" | "web"; platform(): "android" | "ios" | "web";
// queryCandidates returns the host-enumerated targets for a verb, in a // queryTargets returns every element the host can offer, in a deterministic
// deterministic order. The picker indexes into this list with the PCG, so // order, with no per-verb filtering: the picker applies acceptsTarget. The
// the order is part of the parity contract. // picker indexes into this list with the PCG, so the order is part of the
queryCandidates(verb: BuiltinVerb): Candidate[]; // parity contract.
queryTargets(): TargetElement[];
// reportUnsupported is invoked at most once per verb@platform (see verbs.ts) // reportUnsupported is invoked at most once per verb@platform (see verbs.ts)
// when a verb has no support on this platform. // when a verb has no support on this platform.
reportUnsupported(verb: BuiltinVerb): void; reportUnsupported(verb: BuiltinVerb): void;
+1 -1
View File
@@ -1,6 +1,6 @@
// Goja-side runtime entry for the native verifier (internal/verifier). // Goja-side runtime entry for the native verifier (internal/verifier).
// //
// Go installs globalThis.__sanderlingHost__ (platform/seed/queryCandidates/ // Go installs globalThis.__sanderlingHost__ (platform/seed/queryTargets/
// reportUnsupported, implemented over the hierarchy tree in bindings.go) before // reportUnsupported, implemented over the hierarchy tree in bindings.go) before
// the spec evaluates. This module bundles AFTER the spec, reads that host, and // the spec evaluates. This module bundles AFTER the spec, reads that host, and
// wires the shared picker via installRuntime so the goja verifier and the V8 web // wires the shared picker via installRuntime so the goja verifier and the V8 web
+160 -118
View File
@@ -1,29 +1,44 @@
// The shared deterministic action picker for W2 approach B. // The shared deterministic action picker.
// //
// walk() traverses a GeneratorNode tree, and nextAction() wraps it with the // walk() traverses a GeneratorNode tree, and nextAction() wraps it with the
// 16-attempt retry that matches worker.go NextAction. Both engines (the goja // 16-attempt retry that matches worker.go NextAction. Both engines (the goja
// verifier and the V8 web runtime) run THIS code, drawing through the shared // verifier and the V8 web runtime) run THIS code, drawing through the shared
// Pcg, so a given seed yields an identical action stream on every platform. // Pcg, so a given seed yields an identical action stream on every platform.
// //
// builtinCandidates() is the ONE enumeration of what a builtin verb can do at
// the current step. The seeded policy draws a single entry from it below; the
// model policy (Go, internal/verifier/llm.go) reads the same list through
// __sanderlingEnumerateBuiltin__. Neither policy can reach an action the other
// cannot, because neither owns an enumeration of its own.
//
// PARITY CONTRACT - draw order. // PARITY CONTRACT - draw order.
// Every random decision goes through the Pcg in a FIXED, pinned order. Changing // Every random decision goes through the Pcg in a FIXED, pinned order. Changing
// this order shifts the stream for a seed and breaks cross-engine // this order shifts the stream for a seed and breaks cross-engine
// reproducibility, so treat it as load-bearing: // reproducibility, so treat it as load-bearing:
// //
// pickUniform(list): NO draw for an empty or single-entry list, otherwise ONE
// intN(list.length) draw. Both the actions node and the
// builtin node select through it.
// weighted node: ONE float64() draw, then an ASCENDING cumulative scan over // weighted node: ONE float64() draw, then an ASCENDING cumulative scan over
// max(0, weight). (matches worker.go pickWeighted.) // max(0, weight). (matches worker.go pickWeighted.)
// actions node: the generator runs FIRST (any from(...).generate() inside // actions node: the generator runs FIRST (any from(...).generate() inside
// draws intN(itemCount) for >1 items, nothing otherwise), // draws intN(itemCount) for >1 items, nothing otherwise),
// THEN if the returned list has >1 entry, ONE intN(len) draw; // THEN pickUniform over the returned list.
// a 0- or 1-element list draws nothing. (pickFromResult.) // builtin node: pickUniform over builtinCandidates(verb), THEN the one
// builtin node, per verb, in this exact sequence: // value that verb's enumeration leaves to the policy:
// taps/doubleTaps/longPresses: intN(candidateCount) [1 draw] // `typing` ONE intN(corpusLength) draw for the text,
// typing: intN(candidateCount), intN(corpusLength) // `swipes` ONE 200 + intN(401) draw for the drag distance.
// swipes: intN(candidateCount), // Every other verb draws nothing further.
// 200 + intN(401) magnitude, intN(4) direction //
// scrolls: intN(candidateCount), intN(4) direction // The enumerated candidate count per verb, over the host targets the verb
// pressKeys: intN(keyCount) // accepts (targets.ts acceptsTarget), which is what pickUniform draws over:
// waitOnce: no draw //
// taps/doubleTaps/longPresses: one per accepted target
// typing: one per accepted target
// scrolls: two per scrollable container (down, up)
// swipes: four per accepted target (down, up, left, right)
// pressKeys: one per key in the platform's pool
// waitOnce: exactly one, so it never draws
// //
// Builtin targets are resolved to a {x, y} Point by the host BEFORE the picker // Builtin targets are resolved to a {x, y} Point by the host BEFORE the picker
// sees them, so no element handle crosses into this module. // sees them, so no element handle crosses into this module.
@@ -32,24 +47,114 @@ import type { Pcg } from "./pcg.ts";
import type { import type {
ActionDescriptor, ActionDescriptor,
BuiltinVerb, BuiltinVerb,
Candidate,
GeneratorNode, GeneratorNode,
Host, Host,
} from "./action-tree.ts"; } from "./action-tree.ts";
import type { Direction, Point } from "./types.ts"; import type { Direction, Point } from "./types.ts";
import { INPUT_CORPUS, NATIVE_PRESS_KEYS, WEB_PRESS_KEYS } from "./corpus.ts"; import { INPUT_CORPUS, NATIVE_PRESS_KEYS, WEB_PRESS_KEYS } from "./corpus.ts";
import { setSamplerRng } from "./sampler-rng.ts"; import { setSamplerRng } from "./sampler-rng.ts";
import { acceptsTarget } from "./targets.ts";
import { supports, warnUnsupportedOnce } from "./verbs.ts"; import { supports, warnUnsupportedOnce } from "./verbs.ts";
// SWIPE_MIN_MAGNITUDE / SWIPE_MAGNITUDE_SPAN reproduce worker.go's const WAIT_MILLIS = 500;
// `200 + rng.IntN(401)` swipe distance in pixels (200..600 inclusive).
// SCROLL_DIRECTIONS and SWIPE_DIRECTIONS are the directions each gesture verb
// enumerates, one candidate per (target, direction). Scrolls stay vertical:
// they target every scrollable container, so they are what makes the enumerated
// list long, and scrolling a mobile list means up and down. Swipes take all
// four, because swipe-to-dismiss and swipe-to-delete are horizontal gestures on
// list rows, and folding them away puts that defect class out of reach.
const SCROLL_DIRECTIONS: readonly Direction[] = ["down", "up"];
const SWIPE_DIRECTIONS: readonly Direction[] = ["down", "up", "left", "right"];
// SWIPE_MIN_MAGNITUDE / SWIPE_MAGNITUDE_SPAN are the free-form swipe distance in
// pixels the seeded policy draws, 200..600 inclusive. SWIPE_NOMINAL_MAGNITUDE is
// the distance the enumeration lists, so every enumerated candidate is already a
// runnable gesture before any policy has drawn anything.
const SWIPE_MIN_MAGNITUDE = 200; const SWIPE_MIN_MAGNITUDE = 200;
const SWIPE_MAGNITUDE_SPAN = 401; const SWIPE_MAGNITUDE_SPAN = 401;
const SWIPE_NOMINAL_MAGNITUDE = 400;
const SWIPE_DURATION_MILLIS = 250; const SWIPE_DURATION_MILLIS = 250;
const DIRECTIONS: readonly Direction[] = ["up", "down", "left", "right"];
const MAX_RETRIES = 16; const MAX_RETRIES = 16;
// BuiltinCandidate is one enumerated action for a builtin verb, paired with the
// index of the host target it acts on, in host.queryTargets() order (NOT the
// verb-filtered order, so a host can resolve it without repeating the filter).
// targetIndex is -1 for the untargeted verbs (a key press, a wait); the model
// policy uses it to name the control the action lands on.
export interface BuiltinCandidate {
action: ActionDescriptor;
targetIndex: number;
// swipe is set on a `swipes` candidate only. The enumeration fixes where the
// drag starts and which way it travels and lists a nominal distance; the
// seeded policy rebuilds the gesture from these at a drawn distance, the way a
// typing candidate names the field and leaves the text to the policy.
swipe?: { origin: Point; direction: Direction };
}
// builtinCandidates enumerates EVERY action a builtin verb can yield against the
// host's current state. It is the single candidate producer both policies read,
// over the single eligibility rule both hosts consume. An unsupported verb
// reports once and enumerates nothing, so a platform that cannot dispatch a verb
// never offers it to either policy.
export function builtinCandidates(verb: BuiltinVerb, host: Host): BuiltinCandidate[] {
if (!supports(verb, host.platform())) {
warnUnsupportedOnce(host, verb);
return [];
}
if (verb === "waitOnce") {
return [{ action: { kind: "Wait", durationMillis: WAIT_MILLIS }, targetIndex: -1 }];
}
if (verb === "pressKeys") {
const keys = host.platform() === "web" ? WEB_PRESS_KEYS : NATIVE_PRESS_KEYS;
return keys.map((key) => ({
action: { kind: "PressKey", key } as ActionDescriptor,
targetIndex: -1,
}));
}
const candidates: BuiltinCandidate[] = [];
host.queryTargets().forEach((target, targetIndex) => {
if (!acceptsTarget(verb, target)) return;
const point = withSelector({ x: target.x, y: target.y }, target.selector);
const add = (action: ActionDescriptor) => candidates.push({ action, targetIndex });
switch (verb) {
case "taps":
add({ kind: "Tap", on: point });
break;
case "doubleTaps":
add({ kind: "DoubleTap", on: point });
break;
case "longPresses":
add({ kind: "LongPress", on: point });
break;
case "typing":
// text is left empty: it is the one value the policy supplies, drawn
// from the corpus by the seeded arm and written by the model.
add({ kind: "InputText", into: point, text: "" });
break;
case "scrolls":
for (const direction of SCROLL_DIRECTIONS) {
add(scrollDescriptor({ x: target.x, y: target.y }, direction, target));
}
break;
case "swipes":
for (const direction of SWIPE_DIRECTIONS) {
const origin = { x: target.x, y: target.y };
candidates.push({
action: swipeDescriptor(origin, direction, SWIPE_NOMINAL_MAGNITUDE),
targetIndex,
swipe: { origin, direction },
});
}
break;
}
});
return candidates;
}
// nextAction resolves an action for the current step, retrying walk() up to 16 // nextAction resolves an action for the current step, retrying walk() up to 16
// times when it yields null (matches worker.go NextAction). Returns null when // times when it yields null (matches worker.go NextAction). Returns null when
// every attempt comes up empty. // every attempt comes up empty.
@@ -79,7 +184,7 @@ export function walk(
// generator so author sampling shares this single deterministic stream. // generator so author sampling shares this single deterministic stream.
setSamplerRng(rng); setSamplerRng(rng);
try { try {
return walkActions(node.generate(), rng); return pickUniform(node.generate(), rng);
} finally { } finally {
setSamplerRng(null); setSamplerRng(null);
} }
@@ -115,13 +220,13 @@ function walkWeighted(
return last ? walk(last[1], rng, host) : null; return last ? walk(last[1], rng, host) : null;
} }
function walkActions( // pickUniform selects one entry from a list, drawing nothing when there is no
generated: ActionDescriptor[], // choice to make. It is the only selection rule in this module: the seeded
rng: Pcg, // policy is exactly "enumerate, then pickUniform".
): ActionDescriptor | null { function pickUniform<T>(list: readonly T[], rng: Pcg): T | null {
if (generated.length === 0) return null; if (list.length === 0) return null;
if (generated.length === 1) return generated[0] ?? null; if (list.length === 1) return list[0] ?? null;
return generated[rng.intN(generated.length)] ?? null; return list[rng.intN(list.length)] ?? null;
} }
function walkBuiltin( function walkBuiltin(
@@ -129,41 +234,17 @@ function walkBuiltin(
rng: Pcg, rng: Pcg,
host: Host, host: Host,
): ActionDescriptor | null { ): ActionDescriptor | null {
if (!supports(verb, host.platform())) { const picked = pickUniform(builtinCandidates(verb, host), rng);
warnUnsupportedOnce(host, verb);
return null;
}
if (verb === "waitOnce") {
return { kind: "Wait", durationMillis: 500 };
}
if (verb === "pressKeys") {
return walkPressKey(rng, host);
}
const candidates = host.queryCandidates(verb);
if (candidates.length === 0) return null;
const picked = candidates[rng.intN(candidates.length)];
if (!picked) return null; if (!picked) return null;
const point: Point = { x: picked.x, y: picked.y }; if (picked.action.kind === "InputText") {
return { ...picked.action, text: INPUT_CORPUS[rng.intN(INPUT_CORPUS.length)] ?? "" };
switch (verb) {
case "taps":
return tapDescriptor("Tap", point, picked.selector);
case "doubleTaps":
return tapDescriptor("DoubleTap", point, picked.selector);
case "longPresses":
return tapDescriptor("LongPress", point, picked.selector);
case "typing": {
const text = INPUT_CORPUS[rng.intN(INPUT_CORPUS.length)] ?? "";
return { kind: "InputText", into: withSelector(point, picked.selector), text };
}
case "swipes":
return buildSwipe(point, rng);
case "scrolls": {
const direction = DIRECTIONS[rng.intN(DIRECTIONS.length)] ?? "down";
return buildScroll(point, direction, picked, rng);
} }
if (picked.swipe) {
const magnitude = SWIPE_MIN_MAGNITUDE + rng.intN(SWIPE_MAGNITUDE_SPAN);
const { origin, direction } = picked.swipe;
return swipeDescriptor(origin, direction, magnitude);
} }
return picked.action;
} }
// withSelector attaches a native selector to a resolved Point so the runner can // withSelector attaches a native selector to a resolved Point so the runner can
@@ -173,82 +254,43 @@ function withSelector(point: Point, selector?: string): Point {
return { ...point, selector } as Point; return { ...point, selector } as Point;
} }
function tapDescriptor( // scrollDescriptor lowers a scroll to a drag over the container, matching
kind: "Tap" | "DoubleTap" | "LongPress", // runner.go's geometry: the gesture drags opposite the named content motion,
point: Point,
selector?: string,
): ActionDescriptor {
return { kind, on: withSelector(point, selector) } as ActionDescriptor;
}
// buildScroll lowers a scroll to a swipe over the container, matching
// worker.go's geometry: the gesture drags opposite the named content motion,
// magnitude 40% of the container extent. Missing width/height (web root) yields // magnitude 40% of the container extent. Missing width/height (web root) yields
// a zero-length endpoint, which the runner re-derives from container bounds. // a zero-length endpoint, which the runner re-derives from container bounds.
function buildScroll( function scrollDescriptor(
from: Point, from: Point,
direction: Direction, direction: Direction,
candidate: { width?: number; height?: number }, candidate: Candidate,
_rng: Pcg,
): ActionDescriptor { ): ActionDescriptor {
const width = candidate.width ?? 0;
const height = candidate.height ?? 0; const height = candidate.height ?? 0;
let toX = from.x;
let toY = from.y; let toY = from.y;
switch (direction) { if (direction === "down") toY = from.y - Math.trunc((4 * height) / 10);
case "down": if (direction === "up") toY = from.y + Math.trunc((4 * height) / 10);
toY = from.y - Math.trunc((4 * height) / 10);
break;
case "up":
toY = from.y + Math.trunc((4 * height) / 10);
break;
case "left":
toX = from.x + Math.trunc((4 * width) / 10);
break;
case "right":
toX = from.x - Math.trunc((4 * width) / 10);
break;
}
return { return {
kind: "Scroll", kind: "Scroll",
direction, direction,
in: from, in: from,
from, from,
to: { x: Math.max(0, toX), y: Math.max(0, toY) }, to: { x: from.x, y: Math.max(0, toY) },
} as ActionDescriptor; } as ActionDescriptor;
} }
function walkPressKey(rng: Pcg, host: Host): ActionDescriptor | null { // swipeDescriptor builds a free-form drag from a point over a raw pixel
const keys = host.platform() === "web" ? WEB_PRESS_KEYS : NATIVE_PRESS_KEYS; // distance, rather than a fraction of a container extent: `swipes` targets any
if (keys.length === 0) return null; // element with real bounds, and most of those have no scroll extent to size a
const key = keys[rng.intN(keys.length)] ?? keys[0]; // gesture against. `direction` names where the finger travels, because a swipe
if (key === undefined) return null; // IS the gesture; a scroll names content motion and drags the other way.
return { kind: "PressKey", key }; function swipeDescriptor(
from: Point,
direction: Direction,
magnitude: number,
): ActionDescriptor {
const horizontal = direction === "left" || direction === "right";
const forward = direction === "down" || direction === "right";
const travel = forward ? magnitude : -magnitude;
const to = horizontal
? { x: Math.max(0, from.x + travel), y: from.y }
: { x: from.x, y: Math.max(0, from.y + travel) };
return { kind: "Swipe", from, to, durationMillis: SWIPE_DURATION_MILLIS };
} }
function buildSwipe(from: Point, rng: Pcg): ActionDescriptor {
const magnitude = SWIPE_MIN_MAGNITUDE + rng.intN(SWIPE_MAGNITUDE_SPAN);
let toX = from.x;
let toY = from.y;
switch (rng.intN(4)) {
case 0:
toY = from.y - magnitude;
break;
case 1:
toY = from.y + magnitude;
break;
case 2:
toX = from.x - magnitude;
break;
case 3:
toX = from.x + magnitude;
break;
}
return {
kind: "Swipe",
from,
to: { x: Math.max(0, toX), y: Math.max(0, toY) },
durationMillis: SWIPE_DURATION_MILLIS,
};
}
+12 -2
View File
@@ -8,9 +8,9 @@
// identical action stream by construction. // identical action stream by construction.
import { Pcg } from "./pcg.ts"; import { Pcg } from "./pcg.ts";
import { nextAction, walk } from "./pick.ts"; import { builtinCandidates, nextAction, walk } from "./pick.ts";
import { INPUT_CORPUS } from "./corpus.ts"; import { INPUT_CORPUS } from "./corpus.ts";
import type { ActionDescriptor, GeneratorNode, Host } from "./action-tree.ts"; import type { ActionDescriptor, BuiltinVerb, GeneratorNode, Host } from "./action-tree.ts";
import type { Point } from "./types.ts"; import type { Point } from "./types.ts";
// SerializedAction is the flat, camelCase wire shape JS emits and Go decodes // SerializedAction is the flat, camelCase wire shape JS emits and Go decodes
@@ -138,6 +138,16 @@ export function installRuntime(
"__sanderlingSampleInput__", "__sanderlingSampleInput__",
() => INPUT_CORPUS[rng.intN(INPUT_CORPUS.length)] ?? "", () => INPUT_CORPUS[rng.intN(INPUT_CORPUS.length)] ?? "",
); );
// The model policy (Go) selects from the SAME enumeration the seeded picker
// draws from, reached through here rather than reimplemented on the Go side.
// Each entry is serialized with the wire contract Go already decodes, so the
// two policies also agree on the action a chosen candidate executes.
defineLockedGlobal("__sanderlingEnumerateBuiltin__", (verb: BuiltinVerb) =>
builtinCandidates(verb, host).map((candidate) => ({
action: serializeAction(candidate.action),
targetIndex: candidate.targetIndex,
})),
);
defineLockedGlobal("__sanderlingExtractors__", () => evaluateExtractors()); defineLockedGlobal("__sanderlingExtractors__", () => evaluateExtractors());
// __sanderlingSetupAction__ walks ONLY the setup generator once, for the LLM // __sanderlingSetupAction__ walks ONLY the setup generator once, for the LLM
// action generator (Go), which drives selection itself and must not run the // action generator (Go), which drives selection itself and must not run the
+69
View File
@@ -0,0 +1,69 @@
// Per-verb target eligibility: the ONE definition both hosts consume.
//
// A host enumerates every element it can offer (the goja verifier over the
// hierarchy tree, the V8 web runtime over the DOM) and reports a fixed set of
// facts about each. It does NOT decide which verb may act on which element:
// acceptsTarget does, here, once. When each host routed verbs itself the two
// drifted, and the same spec induced a different action space per platform --
// web sent `swipes` to scrollable containers only, so swipe-to-dismiss on a
// list row was reachable on native and unreachable on web.
//
// What stays platform-specific is how a fact is COMPUTED, because the source
// models genuinely differ: `clickable` is an accessibility attribute on
// Android/iOS and a CSS selector match on web. The fact vocabulary below is the
// contract; the mapping onto it is each host's business, and it is the only
// place the platforms are allowed to disagree.
import type { BuiltinVerb, TargetElement } from "./action-tree.ts";
// TargetFact names one property of a target a verb can require.
export type TargetFact =
| "clickable"
| "enabled"
| "editable"
| "scrollable"
| "positiveBounds";
// VERB_REQUIRED_FACTS lists the facts a target must have for a verb to act on
// it. `null` marks a verb with no target at all: a key press and a wait are
// enumerated by the picker without consulting the host.
const VERB_REQUIRED_FACTS: Record<BuiltinVerb, readonly TargetFact[] | null> = {
taps: ["clickable", "enabled", "positiveBounds"],
doubleTaps: ["clickable", "enabled", "positiveBounds"],
longPresses: ["clickable", "enabled", "positiveBounds"],
typing: ["editable", "enabled", "positiveBounds"],
// Scrolls target containers that can actually scroll, and enumerate down/up.
scrolls: ["scrollable", "positiveBounds"],
// Any visible element is a valid swipe origin. Swipe-to-dismiss and
// swipe-to-delete live on list rows and cards, which are not scrollable
// containers, so scoping swipes to containers would put that whole class of
// interaction out of reach.
swipes: ["positiveBounds"],
pressKeys: null,
waitOnce: null,
};
// acceptsTarget is the per-verb element filter. Every targeted verb demands
// real bounds: a zero-bounds node centers at (0,0), and a downward gesture from
// the top-left corner is the system gesture that pulls down the notification
// shade, dragging the fuzzer out of the app.
export function acceptsTarget(verb: BuiltinVerb, target: TargetElement): boolean {
const required = VERB_REQUIRED_FACTS[verb];
if (required === null) return false;
return required.every((fact) => hasFact(target, fact));
}
function hasFact(target: TargetElement, fact: TargetFact): boolean {
switch (fact) {
case "clickable":
return target.clickable;
case "enabled":
return target.enabled;
case "editable":
return target.editable;
case "scrollable":
return target.scrollable;
case "positiveBounds":
return (target.width ?? 0) > 0 && (target.height ?? 0) > 0;
}
}
+56 -73
View File
@@ -4,7 +4,7 @@
// //
// This file is the WEB Host. It installs globalThis.__sanderling__ (extract + // This file is the WEB Host. It installs globalThis.__sanderling__ (extract +
// LTL formula binds) before the spec evaluates, implements the Host interface // LTL formula binds) before the spec evaluates, implements the Host interface
// (platform/seed/queryCandidates/reportUnsupported) over the live DOM, and then // (platform/seed/queryTargets/reportUnsupported) over the live DOM, and then
// delegates ALL action generation to the shared picker via installRuntime // delegates ALL action generation to the shared picker via installRuntime
// (runtime-entry.ts -> pick.ts). The goja verifier runs the SAME picker over the // (runtime-entry.ts -> pick.ts). The goja verifier runs the SAME picker over the
// SAME Pcg, so a given seed yields an identical action stream by construction. // SAME Pcg, so a given seed yields an identical action stream by construction.
@@ -13,11 +13,11 @@
// window.__sanderlingNextAction__() over CDP each tick. LTL predicates are // window.__sanderlingNextAction__() over CDP each tick. LTL predicates are
// stubbed: properties run host-side in goja, which loads its own bundle. // stubbed: properties run host-side in goja, which loads its own bundle.
// //
// Element references never cross V8/host. queryCandidates resolves each element // Element references never cross V8/host. queryTargets resolves each element to
// to a {x, y} Point via getBoundingClientRect before the picker sees it. // a {x, y} Point via getBoundingClientRect before the picker sees it.
import { installRuntime } from "./runtime-entry.ts"; import { installRuntime } from "./runtime-entry.ts";
import type { BuiltinVerb, Candidate, Host } from "./action-tree.ts"; import type { BuiltinVerb, Candidate, Host, TargetElement } from "./action-tree.ts";
interface Handle { interface Handle {
readonly current: unknown; readonly current: unknown;
@@ -85,7 +85,12 @@ function noopFormula(): unknown {
const KNOWN_KEY_TO_CSS: Record<string, (value: string) => string> = { const KNOWN_KEY_TO_CSS: Record<string, (value: string) => string> = {
id: (v) => `[id="${cssEscape(v)}"]`, id: (v) => `[id="${cssEscape(v)}"]`,
"resource-id": (v) => `[id="${cssEscape(v)}"]`, "resource-id": (v) => `[id="${cssEscape(v)}"]`,
testTag: (v) => `[data-testid="${cssEscape(v)}"]`, // The native table aliases testTag onto resource-id, which the host DOM walk
// fills from el.id, so the native path already accepts a testTag emitted as
// an id (what Compose Multiplatform does on web). Accept both here so the
// tables agree. `:is()` keeps this one compound, since a multi-key selector
// concatenates the parts.
testTag: (v) => `:is([data-testid="${cssEscape(v)}"], [id="${cssEscape(v)}"])`,
testID: (v) => `[data-testid="${cssEscape(v)}"]`, testID: (v) => `[data-testid="${cssEscape(v)}"]`,
"data-testid": (v) => `[data-testid="${cssEscape(v)}"]`, "data-testid": (v) => `[data-testid="${cssEscape(v)}"]`,
className: (v) => `[class~="${cssEscape(v)}"]`, className: (v) => `[class~="${cssEscape(v)}"]`,
@@ -467,9 +472,12 @@ function sanitizeAt(value: unknown, depth: number, seen: WeakSet<object>): unkno
return out; return out;
} }
// Per-verb DOM selector sets. The tappable set backs taps/doubleTaps/longPresses // The DOM half of the fact mapping. Which verb may act on which target is NOT
// (every tappable element is also a valid long-press target); the editable set // decided here: queryTargets reports facts and targets.ts acceptsTarget applies
// backs typing; swipes/scrolls target the scrollable element. // them, the same rule the native host's targets run through. These selectors are
// only how the DOM answers "is this clickable" / "is this editable", the two
// facts with no direct DOM equivalent of the accessibility attributes native
// platforms expose.
const TAPPABLE_SELECTOR = 'a, button, input, select, textarea, [role="button"], [onclick]'; const TAPPABLE_SELECTOR = 'a, button, input, select, textarea, [role="button"], [onclick]';
const EDITABLE_SELECTOR = "input, textarea, [contenteditable]"; const EDITABLE_SELECTOR = "input, textarea, [contenteditable]";
@@ -477,12 +485,6 @@ const NON_TEXT_INPUT_TYPES = [
"button", "submit", "checkbox", "radio", "range", "color", "file", "image", "reset", "button", "submit", "checkbox", "radio", "range", "color", "file", "image", "reset",
]; ];
function isVisible(element: HTMLElement): boolean {
if ((element as HTMLButtonElement).disabled) return false;
const rect = element.getBoundingClientRect();
return rect.width > 0 && rect.height > 0;
}
function isEditableElement(element: HTMLElement): boolean { function isEditableElement(element: HTMLElement): boolean {
if (element.isContentEditable) return true; if (element.isContentEditable) return true;
const tag = element.tagName.toLowerCase(); const tag = element.tagName.toLowerCase();
@@ -494,6 +496,14 @@ function isEditableElement(element: HTMLElement): boolean {
return false; return false;
} }
// isScrollable mirrors the native `scrollable` accessibility attribute: the
// container can actually scroll, i.e. its content overflows its box. The
// document scrolling root is not special-cased in: when the page does not
// overflow there is no scroll to perform, and native would offer none either.
function isScrollable(element: HTMLElement): boolean {
return element.scrollHeight > element.clientHeight || element.scrollWidth > element.clientWidth;
}
function pointOf(element: Element): Candidate { function pointOf(element: Element): Candidate {
const rect = element.getBoundingClientRect(); const rect = element.getBoundingClientRect();
return { return {
@@ -504,41 +514,33 @@ function pointOf(element: Element): Candidate {
}; };
} }
function tappableCandidates(): Candidate[] { // collectTargets walks the document ONCE and reports every element with the facts
return Array.from(document.querySelectorAll<HTMLElement>(TAPPABLE_SELECTOR)) // the shared eligibility rule reads. The tappable/editable membership sets are
.filter(isVisible) // resolved by selector first so the DOM's answer to "clickable" and "editable"
.map(pointOf); // stays expressed in CSS, as it always was.
function collectTargets(): TargetElement[] {
const clickable = new Set<Element>(Array.from(document.querySelectorAll(TAPPABLE_SELECTOR)));
const editable = new Set<Element>(
Array.from(document.querySelectorAll<HTMLElement>(EDITABLE_SELECTOR)).filter(
isEditableElement,
),
);
return Array.from(document.querySelectorAll<HTMLElement>("*")).map((element) => ({
...pointOf(element),
clickable: clickable.has(element),
enabled: !(element as HTMLButtonElement).disabled,
editable: editable.has(element),
scrollable: isScrollable(element),
}));
} }
function editableCandidates(): Candidate[] { // Per-tick target cache: the picker's 16-attempt retry re-queries every tick, so
return Array.from(document.querySelectorAll<HTMLElement>(EDITABLE_SELECTOR)) // we avoid re-walking the DOM and re-flushing layout within one tick.
.filter((element) => isEditableElement(element) && isVisible(element))
.map(pointOf);
}
// Scrollable candidates back swipe/scroll: elements that overflow their box,
// plus the scrolling root so a page-level scroll always has a target.
function scrollableCandidates(): Candidate[] {
const root = document.scrollingElement ?? document.documentElement;
const candidates: Candidate[] = root ? [pointOf(root)] : [];
for (const element of Array.from(document.querySelectorAll<HTMLElement>("*"))) {
if (element === root) continue;
if (element.scrollHeight <= element.clientHeight && element.scrollWidth <= element.clientWidth) {
continue;
}
if (!isVisible(element)) continue;
candidates.push(pointOf(element));
}
return candidates;
}
// Per-tick candidate cache: the picker's 16-attempt retry re-queries the same
// verb, so we avoid re-walking the DOM and re-flushing layout within one tick.
// installRuntime resets it before each __sanderlingNextAction__ invocation. // installRuntime resets it before each __sanderlingNextAction__ invocation.
const candidateCache = new Map<BuiltinVerb, Candidate[]>(); let cachedTargets: TargetElement[] | null = null;
function resetCandidateCache(): void { function resetTargetCache(): void {
candidateCache.clear(); cachedTargets = null;
} }
const host: Host = { const host: Host = {
@@ -546,28 +548,9 @@ const host: Host = {
seedHi: () => SEED_HI, seedHi: () => SEED_HI,
// lo = 0 matches the goja side's rand.NewPCG(seed, 0). // lo = 0 matches the goja side's rand.NewPCG(seed, 0).
seedLo: () => 0n, seedLo: () => 0n,
queryCandidates(verb: BuiltinVerb): Candidate[] { queryTargets(): TargetElement[] {
const cached = candidateCache.get(verb); if (!cachedTargets) cachedTargets = collectTargets();
if (cached) return cached; return cachedTargets;
let candidates: Candidate[];
switch (verb) {
case "taps":
case "doubleTaps":
case "longPresses":
candidates = tappableCandidates();
break;
case "typing":
candidates = editableCandidates();
break;
case "swipes":
case "scrolls":
candidates = scrollableCandidates();
break;
default:
candidates = [];
}
candidateCache.set(verb, candidates);
return candidates;
}, },
reportUnsupported(verb: BuiltinVerb): void { reportUnsupported(verb: BuiltinVerb): void {
console.warn(`[sanderling] verb ${verb} is unsupported on web`); console.warn(`[sanderling] verb ${verb} is unsupported on web`);
@@ -578,11 +561,11 @@ const host: Host = {
// this module (the web bundle imports the runtime first). Resolve the root // this module (the web bundle imports the runtime first). Resolve the root
// lazily so installRuntime captures it once the spec has evaluated. The root // lazily so installRuntime captures it once the spec has evaluated. The root
// resolver runs once per __sanderlingNextAction__ tick (before the retry loop), // resolver runs once per __sanderlingNextAction__ tick (before the retry loop),
// so it is also where we reset the per-tick candidate cache. // so it is also where we reset the per-tick target cache.
installRuntime( installRuntime(
host, host,
() => { () => {
resetCandidateCache(); resetTargetCache();
return (globalThis as { actions?: import("./action-tree.ts").GeneratorNode }).actions ?? null; return (globalThis as { actions?: import("./action-tree.ts").GeneratorNode }).actions ?? null;
}, },
evaluateExtractors, evaluateExtractors,
@@ -593,10 +576,10 @@ installRuntime(
export const __testing__ = { export const __testing__ = {
host, host,
seedBigInt, seedBigInt,
tappableCandidates, collectTargets,
editableCandidates, TAPPABLE_SELECTOR,
scrollableCandidates, EDITABLE_SELECTOR,
resetCandidateCache, resetTargetCache,
runtime, runtime,
extractors, extractors,
evaluateExtractors, evaluateExtractors,
+6 -1
View File
@@ -112,7 +112,12 @@ function installFakeRuntime(): RecordedRuntime {
extract: <T>(getter: (state: State) => T, name?: string): Extracted<T> => { extract: <T>(getter: (state: State) => T, name?: string): Extracted<T> => {
calls.extracts.push(getter as (state: State) => unknown); calls.extracts.push(getter as (state: State) => unknown);
calls.extractNames.push(name); calls.extractNames.push(name);
return { current: undefined as unknown as T, previous: undefined }; const handle: Extracted<T> = {
current: undefined as unknown as T,
previous: undefined,
named: () => handle,
};
return handle;
}, },
always: (predicateOrFormula: (() => boolean) | Formula): Formula => { always: (predicateOrFormula: (() => boolean) | Formula): Formula => {
calls.alwaysArgs.push(predicateOrFormula); calls.alwaysArgs.push(predicateOrFormula);
+6 -2
View File
@@ -15,10 +15,14 @@ import type {
function installRuntime(initialState: State): void { function installRuntime(initialState: State): void {
const state = { current: initialState }; const state = { current: initialState };
const runtime: SanderlingRuntime = { const runtime: SanderlingRuntime = {
extract: <T>(getter: (s: State) => T): Extracted<T> => ({ extract: <T>(getter: (s: State) => T): Extracted<T> => {
const handle: Extracted<T> = {
current: getter(state.current), current: getter(state.current),
previous: undefined, previous: undefined,
}), named: () => handle,
};
return handle;
},
always: () => ({ __sanderlingFormula: true } as Formula), always: () => ({ __sanderlingFormula: true } as Formula),
now: () => ({ __sanderlingFormula: true } as Formula), now: () => ({ __sanderlingFormula: true } as Formula),
next: () => ({ __sanderlingFormula: true } as Formula), next: () => ({ __sanderlingFormula: true } as Formula),
+48
View File
@@ -0,0 +1,48 @@
{
"taps": [
"save"
],
"doubleTaps": [
"save"
],
"longPresses": [
"save"
],
"typing": [
"amount"
],
"scrolls": [
"root",
"root",
"list",
"list"
],
"swipes": [
"root",
"root",
"root",
"root",
"save",
"save",
"save",
"save",
"cancel",
"cancel",
"cancel",
"cancel",
"amount",
"amount",
"amount",
"amount",
"list",
"list",
"list",
"list",
"row",
"row",
"row",
"row"
],
"pressKeys": [],
"waitOnce": []
}
+73
View File
@@ -0,0 +1,73 @@
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { test } from "node:test";
import { builtinCandidates } from "../src/pick.ts";
import { resetWarnings } from "../src/verbs.ts";
import type { BuiltinVerb } from "../src/action-tree.ts";
import { fakeElement, withFakeDocument, type FakeElementSpec } from "./web-dom-harness.ts";
import { __testing__ } from "../src/web-runtime.ts";
const { host } = __testing__;
const goldenPath = fileURLToPath(new URL("./fixtures/host-parity-golden.json", import.meta.url));
const golden: Record<string, string[]> = JSON.parse(readFileSync(goldenPath, "utf8"));
const VERBS: BuiltinVerb[] = [
"taps",
"doubleTaps",
"longPresses",
"typing",
"scrolls",
"swipes",
"pressKeys",
"waitOnce",
];
// SCREEN is the canonical screen both hosts are driven over: one row per fact
// combination that any verb distinguishes. The native host builds the same rows,
// in the same order, as a hierarchy tree in
// internal/verifier/host_parity_test.go.
const SCREEN: (FakeElementSpec & { name: string })[] = [
{ name: "root", tag: "html", x: 0, y: 0, width: 400, height: 800, overflows: true },
{ name: "save", tag: "button", x: 0, y: 0, width: 200, height: 60, clickable: true },
{
name: "cancel",
tag: "button",
x: 200,
y: 0,
width: 200,
height: 60,
clickable: true,
disabled: true,
},
{ name: "amount", tag: "input", x: 0, y: 100, width: 400, height: 60, editable: true },
{ name: "list", tag: "div", x: 0, y: 200, width: 400, height: 400, overflows: true },
{ name: "row", tag: "div", x: 0, y: 600, width: 400, height: 80 },
{ name: "collapsed", tag: "button", x: 0, y: 0, width: 0, height: 0, clickable: true },
];
// The web host and the native host used to route verbs themselves and had
// drifted: web sent `swipes` to scrollable containers only, so a swipe on a list
// row was reachable on Android and unreachable on web for the same spec.
//
// Per-verb eligibility now has ONE definition (src/targets.ts); a host reports
// facts and never filters. This test is what notices if a second definition
// grows back on either side. internal/verifier/host_parity_test.go asserts the
// SAME golden from the native host, so a match on both sides proves the two
// hosts agree without either invoking the other.
test("web host targets match the cross-host golden, verb for verb", () => {
const elements = SCREEN.map(fakeElement);
withFakeDocument(elements, () => {
for (const verb of VERBS) {
resetWarnings();
const candidates = builtinCandidates(verb, host);
assert.notEqual(candidates.length, 0, `${verb} enumerated nothing at all`);
const named = candidates
.filter((candidate) => candidate.targetIndex >= 0)
.map((candidate) => SCREEN[candidate.targetIndex]!.name);
assert.deepEqual(named, golden[verb], `web host targets for ${verb}`);
}
});
});
+15 -11
View File
@@ -1,11 +1,11 @@
// Shared cross-runtime parity scenario: the FIXED seed, FIXED candidate list, // Shared cross-runtime parity scenario: the FIXED seed, FIXED target list,
// and FIXED action root that both the node test (parity.test.ts) and the goja // and FIXED action root that both the node test (parity.test.ts) and the goja
// test (internal/verifier/parity_test.go) drive. Each side runs the SAME // test (internal/verifier/parity_test.go) drive. Each side runs the SAME
// pick.ts over the SAME Pcg and asserts the SAME committed golden // pick.ts over the SAME Pcg and asserts the SAME committed golden
// (fixtures/parity-golden.json); matching one golden on both sides proves the // (fixtures/parity-golden.json); matching one golden on both sides proves the
// two engines agree without either invoking the other. // two engines agree without either invoking the other.
// //
// The candidate ORDER and the per-tick PCG draw order are the parity contract. // The target ORDER and the per-tick PCG draw order are the parity contract.
// The weighted root mixes a tap branch (1 candidate draw) with a typing branch // The weighted root mixes a tap branch (1 candidate draw) with a typing branch
// (1 candidate draw + 1 corpus draw), so a tick exercises weighted selection, a // (1 candidate draw + 1 corpus draw), so a tick exercises weighted selection, a
// builtin, and the input corpus together; reordering candidates or adding or // builtin, and the input corpus together; reordering candidates or adding or
@@ -15,25 +15,29 @@ import { Pcg } from "../src/pcg.ts";
import { nextAction } from "../src/pick.ts"; import { nextAction } from "../src/pick.ts";
import { serializeAction, type SerializedAction } from "../src/runtime-entry.ts"; import { serializeAction, type SerializedAction } from "../src/runtime-entry.ts";
import { taps, typing, weighted } from "../src/actions.ts"; import { taps, typing, weighted } from "../src/actions.ts";
import type { BuiltinVerb, Candidate, GeneratorNode, Host } from "../src/action-tree.ts"; import type { GeneratorNode, Host, TargetElement } from "../src/action-tree.ts";
export const PARITY_SEED_HI = 0x9e3779b97f4a7c15n; export const PARITY_SEED_HI = 0x9e3779b97f4a7c15n;
export const PARITY_STEPS = 20; export const PARITY_STEPS = 20;
export const PARITY_CANDIDATES: Candidate[] = [ // Every fact is set so the shared eligibility rule admits all three targets for
{ x: 50, y: 60, selector: "id:alpha", width: 100, height: 40 }, // every verb, leaving the draw order as the only variable.
{ x: 150, y: 160, selector: "id:beta", width: 120, height: 48 }, const EVERY_FACT = { clickable: true, enabled: true, editable: true, scrollable: true };
{ x: 250, y: 260, selector: "id:gamma", width: 80, height: 32 },
export const PARITY_TARGETS: TargetElement[] = [
{ x: 50, y: 60, selector: "id:alpha", width: 100, height: 40, ...EVERY_FACT },
{ x: 150, y: 160, selector: "id:beta", width: 120, height: 48, ...EVERY_FACT },
{ x: 250, y: 260, selector: "id:gamma", width: 80, height: 32, ...EVERY_FACT },
]; ];
// A 3:1 weighted split over taps and typing. The stub host returns the same // A 3:1 weighted split over taps and typing. The stub host offers the same
// candidate list for every verb so the only variables are the draw order and // target list to every verb so the only variables are the draw order and the JS
// the JS engine's number/bigint behavior. // engine's number/bigint behavior.
export const PARITY_ROOT: GeneratorNode = weighted([3, taps], [1, typing]); export const PARITY_ROOT: GeneratorNode = weighted([3, taps], [1, typing]);
const HOST: Host = { const HOST: Host = {
platform: () => "android", platform: () => "android",
queryCandidates: (_verb: BuiltinVerb) => PARITY_CANDIDATES, queryTargets: () => PARITY_TARGETS,
reportUnsupported: () => {}, reportUnsupported: () => {},
seedHi: () => PARITY_SEED_HI, seedHi: () => PARITY_SEED_HI,
seedLo: () => 0n, seedLo: () => 0n,
+241 -60
View File
@@ -1,30 +1,31 @@
import { test } from "node:test"; import { test } from "node:test";
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { Pcg } from "../src/pcg.ts"; import { Pcg } from "../src/pcg.ts";
import { nextAction, walk } from "../src/pick.ts"; import { builtinCandidates, nextAction, walk } from "../src/pick.ts";
import { INPUT_CORPUS, NATIVE_PRESS_KEYS, WEB_PRESS_KEYS } from "../src/corpus.ts"; import { INPUT_CORPUS, NATIVE_PRESS_KEYS, WEB_PRESS_KEYS } from "../src/corpus.ts";
import { resetWarnings } from "../src/verbs.ts"; import { resetWarnings } from "../src/verbs.ts";
import type { import type {
ActionDescriptor, ActionDescriptor,
BuiltinVerb, BuiltinVerb,
Candidate,
GeneratorNode, GeneratorNode,
Host, Host,
TargetElement,
} from "../src/action-tree.ts"; } from "../src/action-tree.ts";
import type { Direction, Point } from "../src/types.ts";
type Platform = "android" | "ios" | "web"; type Platform = "android" | "ios" | "web";
// stubHost returns a fixed candidate list for every verb and records each // stubHost returns a fixed target list, eligible for every verb, and records
// reportUnsupported call so warn-once semantics are observable. // each reportUnsupported call so warn-once semantics are observable.
function stubHost( function stubHost(
platform: Platform, platform: Platform,
candidates: Candidate[], targets: TargetElement[],
): Host & { unsupported: BuiltinVerb[] } { ): Host & { unsupported: BuiltinVerb[] } {
const unsupported: BuiltinVerb[] = []; const unsupported: BuiltinVerb[] = [];
return { return {
unsupported, unsupported,
platform: () => platform, platform: () => platform,
queryCandidates: () => candidates, queryTargets: () => targets,
reportUnsupported: (verb) => { reportUnsupported: (verb) => {
unsupported.push(verb); unsupported.push(verb);
}, },
@@ -33,10 +34,12 @@ function stubHost(
}; };
} }
const POINTS: Candidate[] = [ const EVERY_FACT = { clickable: true, enabled: true, editable: true, scrollable: true };
{ x: 10, y: 20, selector: "id:a", width: 100, height: 200 },
{ x: 30, y: 40, selector: "id:b", width: 100, height: 200 }, const POINTS: TargetElement[] = [
{ x: 50, y: 60, selector: "id:c", width: 100, height: 200 }, { x: 10, y: 20, selector: "id:a", width: 100, height: 200, ...EVERY_FACT },
{ x: 30, y: 40, selector: "id:b", width: 100, height: 200, ...EVERY_FACT },
{ x: 50, y: 60, selector: "id:c", width: 100, height: 200, ...EVERY_FACT },
]; ];
function builtin(verb: BuiltinVerb): GeneratorNode { function builtin(verb: BuiltinVerb): GeneratorNode {
@@ -81,59 +84,132 @@ test("typing draws candidate index then corpus index, in that order", () => {
assert.equal(action.text, INPUT_CORPUS[corpusIndex]); assert.equal(action.text, INPUT_CORPUS[corpusIndex]);
}); });
test("swipes draw candidate, magnitude (200+intN(401)), then direction", () => { // scrollCandidate mirrors what the picker builds for one (container, direction)
// pair: a drag opposite the named content motion, 40% of the container extent.
function scrollCandidate(target: TargetElement, direction: "down" | "up") {
const from = { x: target.x, y: target.y };
const extent = Math.trunc((4 * (target.height ?? 0)) / 10);
const toY = direction === "down" ? from.y - extent : from.y + extent;
return {
kind: "Scroll",
direction,
in: from,
from,
to: { x: from.x, y: Math.max(0, toY) },
};
}
// swipeCandidate mirrors the free-form drag: a raw pixel distance, with the
// direction naming where the finger travels.
function swipeCandidate(
target: Point,
direction: Direction,
magnitude: number,
) {
const from = { x: target.x, y: target.y };
const horizontal = direction === "left" || direction === "right";
const forward = direction === "down" || direction === "right";
const travel = forward ? magnitude : -magnitude;
return {
kind: "Swipe",
from,
to: horizontal
? { x: Math.max(0, from.x + travel), y: from.y }
: { x: from.x, y: Math.max(0, from.y + travel) },
durationMillis: 250,
};
}
const NOMINAL_SWIPE_MAGNITUDE = 400;
const SCROLL_DIRECTIONS = ["down", "up"] as const;
const SWIPE_DIRECTIONS = ["down", "up", "left", "right"] as const;
// swipeDirection recovers which way a drawn swipe travelled from its endpoints.
function swipeDirection(from: Point, to: Point): Direction {
if (to.x !== from.x) return to.x > from.x ? "right" : "left";
return to.y > from.y ? "down" : "up";
}
test("scrolls enumerate every container up and down only", () => {
resetWarnings(); resetWarnings();
const host = stubHost("android", POINTS); const host = stubHost("android", POINTS);
const rng = new Pcg(7n, 0n); assert.deepEqual(
const oracle = new Pcg(7n, 0n); builtinCandidates("scrolls", host),
const candidateIndex = oracle.intN(POINTS.length); POINTS.flatMap((target, targetIndex) =>
const magnitude = 200 + oracle.intN(401); SCROLL_DIRECTIONS.map((direction) => ({
const direction = oracle.intN(4); action: scrollCandidate(target, direction),
targetIndex,
const from = { x: POINTS[candidateIndex]!.x, y: POINTS[candidateIndex]!.y }; })),
const expectedTo = { x: from.x, y: from.y }; ),
switch (direction) { );
case 0:
expectedTo.y = Math.max(0, from.y - magnitude);
break;
case 1:
expectedTo.y = Math.max(0, from.y + magnitude);
break;
case 2:
expectedTo.x = Math.max(0, from.x - magnitude);
break;
case 3:
expectedTo.x = Math.max(0, from.x + magnitude);
break;
}
const action = walk(builtin("swipes"), rng, host) as ActionDescriptor & {
kind: "Swipe";
};
assert.equal(action.kind, "Swipe");
assert.deepEqual(action.from, from);
assert.deepEqual(action.to, expectedTo);
assert.equal(action.durationMillis, 250);
}); });
test("scrolls draw candidate index then direction", () => { test("swipes enumerate a free-form drag per target in all four directions", () => {
resetWarnings(); resetWarnings();
const host = stubHost("android", POINTS); const host = stubHost("android", POINTS);
const rng = new Pcg(99n, 0n); // The swipe candidate is its own action shape, sized in raw pixels rather than
const oracle = new Pcg(99n, 0n); // off the target's extent, and it carries what the policy needs to redraw the
const candidateIndex = oracle.intN(POINTS.length); // distance. It is not the scroll gesture under a second name.
const directionIndex = oracle.intN(4); assert.deepEqual(
const directions = ["up", "down", "left", "right"] as const; builtinCandidates("swipes", host),
POINTS.flatMap((target, targetIndex) =>
SWIPE_DIRECTIONS.map((direction) => ({
action: swipeCandidate(target, direction, NOMINAL_SWIPE_MAGNITUDE),
targetIndex,
swipe: { origin: { x: target.x, y: target.y }, direction },
})),
),
);
});
const action = walk(builtin("scrolls"), rng, host) as ActionDescriptor & { test("the gesture verbs differ in target filter and in direction set", () => {
kind: "Scroll"; resetWarnings();
}; // Same host, same targets: what separates the two verbs here is the direction
assert.equal(action.kind, "Scroll"); // set alone. Scrolls stay vertical because every scrollable container gets a
assert.equal(action.direction, directions[directionIndex]); // candidate; swipes reach sideways because swipe-to-dismiss does.
assert.deepEqual(action.in, { const host = stubHost("android", POINTS);
x: POINTS[candidateIndex]!.x, const scrolls = builtinCandidates("scrolls", host);
y: POINTS[candidateIndex]!.y, const swipes = builtinCandidates("swipes", host);
});
const scrollDirections = new Set(
scrolls.map(
(entry) => (entry.action as ActionDescriptor & { kind: "Scroll" }).direction,
),
);
const swipeDirections = new Set(swipes.map((entry) => entry.swipe!.direction));
assert.deepEqual([...scrollDirections].sort(), ["down", "up"]);
assert.deepEqual([...swipeDirections].sort(), ["down", "left", "right", "up"]);
assert.equal(scrolls.length, POINTS.length * 2);
assert.equal(swipes.length, POINTS.length * 4);
});
test("scrolls draw one index over the enumerated candidates", () => {
resetWarnings();
const host = stubHost("android", POINTS);
const enumerated = builtinCandidates("scrolls", host);
const oracle = new Pcg(99n, 0n);
const index = oracle.intN(enumerated.length);
const action = walk(builtin("scrolls"), new Pcg(99n, 0n), host);
assert.deepEqual(action, enumerated[index]!.action);
});
test("swipes draw candidate index then magnitude, in that order", () => {
resetWarnings();
const host = stubHost("android", POINTS);
const enumerated = builtinCandidates("swipes", host);
const oracle = new Pcg(7n, 0n);
const index = oracle.intN(enumerated.length);
const magnitude = 200 + oracle.intN(401);
const picked = enumerated[index]!.swipe!;
const action = walk(builtin("swipes"), new Pcg(7n, 0n), host);
assert.deepEqual(
action,
swipeCandidate(picked.origin, picked.direction, magnitude),
);
}); });
test("doubleTaps and longPresses draw exactly one candidate index", () => { test("doubleTaps and longPresses draw exactly one candidate index", () => {
@@ -174,14 +250,28 @@ test("waitOnce emits a 500ms wait and draws nothing", () => {
assert.notEqual(before, after); assert.notEqual(before, after);
}); });
test("pressKeys on native draws from NATIVE_PRESS_KEYS", () => { test("pressKeys enumerates the platform's whole key pool", () => {
resetWarnings();
for (const [platform, keys] of [
["android", NATIVE_PRESS_KEYS],
["web", WEB_PRESS_KEYS],
] as const) {
const enumerated = builtinCandidates("pressKeys", stubHost(platform, POINTS));
assert.deepEqual(
enumerated,
keys.map((key) => ({ action: { kind: "PressKey", key }, targetIndex: -1 })),
);
}
});
test("pressKeys on native emits the only key without drawing", () => {
resetWarnings(); resetWarnings();
const host = stubHost("android", POINTS); const host = stubHost("android", POINTS);
const rng = new Pcg(42n, 0n); const rng = new Pcg(42n, 0n);
const oracle = new Pcg(42n, 0n);
const index = oracle.intN(NATIVE_PRESS_KEYS.length);
const action = walk(builtin("pressKeys"), rng, host); const action = walk(builtin("pressKeys"), rng, host);
assert.deepEqual(action, { kind: "PressKey", key: NATIVE_PRESS_KEYS[index] }); assert.deepEqual(action, { kind: "PressKey", key: NATIVE_PRESS_KEYS[0] });
// One key is no choice, so the pool consumed no draw.
assert.equal(rng.float64(), new Pcg(42n, 0n).float64());
}); });
test("pressKeys on web draws from WEB_PRESS_KEYS", () => { test("pressKeys on web draws from WEB_PRESS_KEYS", () => {
@@ -194,6 +284,97 @@ test("pressKeys on web draws from WEB_PRESS_KEYS", () => {
assert.deepEqual(action, { kind: "PressKey", key: WEB_PRESS_KEYS[index] }); assert.deepEqual(action, { kind: "PressKey", key: WEB_PRESS_KEYS[index] });
}); });
test("an unsupported verb enumerates nothing and reports once", () => {
resetWarnings();
// No platform in the matrix declines a verb today, so the branch is reached
// through a platform the matrix has never heard of.
const host = stubHost("desktop" as Platform, POINTS);
assert.deepEqual(builtinCandidates("taps", host), []);
assert.deepEqual(builtinCandidates("taps", host), []);
assert.deepEqual(host.unsupported, ["taps"]);
});
test("the seeded pick is an index into the shared enumeration", () => {
resetWarnings();
// Whatever the verb, the drawn action is one of the enumerated entries: the
// seeded policy adds a choice, never an action the model policy cannot see.
// `typing` and `swipes` are covered separately, being the two verbs whose
// enumeration leaves one value for the policy to fill in.
const host = stubHost("android", POINTS);
for (const verb of [
"taps",
"doubleTaps",
"longPresses",
"scrolls",
"pressKeys",
"waitOnce",
] as const) {
const enumerated = builtinCandidates(verb, host).map((entry) =>
JSON.stringify(entry.action),
);
for (let seed = 1; seed <= 50; seed++) {
const action = walk(builtin(verb), new Pcg(BigInt(seed), 0n), host);
assert.ok(
enumerated.includes(JSON.stringify(action)),
`${verb} drew ${JSON.stringify(action)}, which is not an enumerated candidate`,
);
}
}
});
test("typing enumerates the field and the policy supplies the text", () => {
resetWarnings();
const host = stubHost("android", POINTS);
const enumerated = builtinCandidates("typing", host);
// The candidate set names fields only: an empty text is the slot the seeded
// corpus draw and the model's own value both fill.
for (const entry of enumerated) {
assert.equal((entry.action as ActionDescriptor & { kind: "InputText" }).text, "");
}
const oracle = new Pcg(2024n, 0n);
const index = oracle.intN(enumerated.length);
const text = INPUT_CORPUS[oracle.intN(INPUT_CORPUS.length)];
assert.deepEqual(walk(builtin("typing"), new Pcg(2024n, 0n), host), {
...enumerated[index]!.action,
text,
});
});
test("swipes enumerate origin and direction, the policy adds distance", () => {
resetWarnings();
const host = stubHost("android", POINTS);
const origins = new Set(
builtinCandidates("swipes", host).map((entry) =>
JSON.stringify([entry.swipe!.origin, entry.swipe!.direction]),
),
);
const drawn = new Set<Direction>();
for (let seed = 1; seed <= 200; seed++) {
const action = walk(builtin("swipes"), new Pcg(BigInt(seed), 0n), host) as
ActionDescriptor & { kind: "Swipe"; from: Point; to: Point };
assert.equal(action.kind, "Swipe");
const direction = swipeDirection(action.from, action.to);
drawn.add(direction);
assert.ok(
origins.has(JSON.stringify([action.from, direction])),
`swipe from ${JSON.stringify(action.from)} going ${direction} is not enumerated`,
);
// The drawn distance stays inside 200..600, clamped at the screen edge.
const horizontal = direction === "left" || direction === "right";
const distance = horizontal
? Math.abs(action.to.x - action.from.x)
: Math.abs(action.to.y - action.from.y);
const clamped = horizontal ? action.to.x === 0 : action.to.y === 0;
assert.ok(distance <= 600, `drag of ${distance}px exceeds the drawn range`);
assert.ok(
distance >= 200 || clamped,
`drag of ${distance}px is under the drawn range and not clamped`,
);
}
// Every enumerated direction is reachable by a draw, sideways included.
assert.deepEqual([...drawn].sort(), ["down", "left", "right", "up"]);
});
test("empty candidate list yields null without drawing", () => { test("empty candidate list yields null without drawing", () => {
resetWarnings(); resetWarnings();
const host = stubHost("android", []); const host = stubHost("android", []);
+1 -1
View File
@@ -19,7 +19,7 @@ function countingHost(platform: "android" | "ios" | "web"): Host & { calls: Buil
return { return {
calls, calls,
platform: () => platform, platform: () => platform,
queryCandidates: () => [], queryTargets: () => [],
reportUnsupported: (verb) => { reportUnsupported: (verb) => {
calls.push(verb); calls.push(verb);
}, },
+88
View File
@@ -0,0 +1,88 @@
// A minimal stand-in for the DOM surface the web host reads, shared by the web
// runtime's own tests and the cross-host eligibility test. The host asks the
// document for three things -- every element, the tappable set, the editable set
// -- and reads geometry, `disabled` and the scroll extents off each element, so
// that is all a fake has to answer.
import { __testing__ } from "../src/web-runtime.ts";
const { TAPPABLE_SELECTOR, EDITABLE_SELECTOR } = __testing__;
export interface FakeElementSpec {
tag: string;
x: number;
y: number;
width: number;
height: number;
// clickable/editable place the element in the selector sets the host queries;
// the fake answers those queries directly rather than matching CSS.
clickable?: boolean;
editable?: boolean;
disabled?: boolean;
// overflows makes the element's content taller than its box, which is how the
// host decides an element is scrollable.
overflows?: boolean;
}
export interface FakeElement extends FakeElementSpec {
tagName: string;
type: string;
isContentEditable: boolean;
scrollHeight: number;
clientHeight: number;
scrollWidth: number;
clientWidth: number;
getBoundingClientRect(): {
left: number;
top: number;
width: number;
height: number;
right: number;
bottom: number;
};
}
export function fakeElement(spec: FakeElementSpec): FakeElement {
const editable = spec.editable ?? false;
return {
...spec,
tagName: spec.tag.toUpperCase(),
type: spec.tag === "input" ? "text" : "",
isContentEditable: editable && spec.tag !== "input" && spec.tag !== "textarea",
scrollHeight: spec.overflows ? spec.height * 2 : spec.height,
clientHeight: spec.height,
scrollWidth: spec.width,
clientWidth: spec.width,
getBoundingClientRect: () => ({
left: spec.x,
top: spec.y,
width: spec.width,
height: spec.height,
right: spec.x + spec.width,
bottom: spec.y + spec.height,
}),
};
}
// withFakeDocument installs a document answering the host's three queries over
// `elements`, resets the host's per-tick cache, and restores the real document
// afterwards.
export function withFakeDocument(elements: FakeElement[], run: () => void): void {
const global = globalThis as Record<string, unknown>;
const original = global.document;
const answers: Record<string, FakeElement[]> = {
"*": elements,
[TAPPABLE_SELECTOR]: elements.filter((element) => element.clickable),
[EDITABLE_SELECTOR]: elements.filter((element) => element.editable),
};
global.document = {
querySelectorAll: (selector: string) => answers[selector] ?? [],
};
__testing__.resetTargetCache();
try {
run();
} finally {
__testing__.resetTargetCache();
global.document = original;
}
}
+67 -68
View File
@@ -83,78 +83,60 @@ test("installRuntime defined the host-invoked globals", () => {
assert.equal(typeof g.__sanderling__, "object"); assert.equal(typeof g.__sanderling__, "object");
}); });
// A button and a text input, each with a deterministic bounding box, exercise const { fakeElement, withFakeDocument } = await import("./web-dom-harness.ts");
// the per-verb selector routing without a full DOM.
function fakeElement(tag: string, rect: { x: number; y: number; w: number; h: number }) {
return {
tagName: tag.toUpperCase(),
disabled: false,
isContentEditable: false,
type: tag === "input" ? "text" : "",
scrollHeight: 0,
clientHeight: 0,
scrollWidth: 0,
clientWidth: 0,
getBoundingClientRect: () => ({
left: rect.x,
top: rect.y,
width: rect.w,
height: rect.h,
right: rect.x + rect.w,
bottom: rect.y + rect.h,
}),
};
}
function withFakeDocument(map: Record<string, unknown[]>, run: () => void) { // The host reports facts and never routes verbs: which of these a verb may act
const g = globalThis as Record<string, unknown>; // on is decided by the shared rule in src/targets.ts, exercised across both
const original = g.document; // engines by host-parity.test.ts.
g.document = { test("queryTargets reports the tappable selector set as clickable", () => {
querySelectorAll: (selector: string) => map[selector] ?? [], const button = fakeElement({ tag: "button", x: 10, y: 20, width: 40, height: 8, clickable: true });
scrollingElement: null, const plain = fakeElement({ tag: "div", x: 0, y: 0, width: 100, height: 100 });
documentElement: null, withFakeDocument([button, plain], () => {
}; const targets = host.queryTargets();
try { assert.equal(targets.length, 2);
run(); assert.equal(targets[0]!.clickable, true);
} finally { assert.deepEqual({ x: targets[0]!.x, y: targets[0]!.y }, { x: 30, y: 24 });
g.document = original; assert.equal(targets[1]!.clickable, false);
}
}
test("queryCandidates routes taps to the tappable selector set", () => {
const button = fakeElement("button", { x: 10, y: 20, w: 40, h: 8 });
withFakeDocument(
{ 'a, button, input, select, textarea, [role="button"], [onclick]': [button] },
() => {
__testing__.resetCandidateCache();
const candidates = host.queryCandidates("taps");
assert.equal(candidates.length, 1);
assert.deepEqual({ x: candidates[0]!.x, y: candidates[0]!.y }, { x: 30, y: 24 });
},
);
});
test("queryCandidates routes typing to editable inputs only", () => {
const input = fakeElement("input", { x: 0, y: 0, w: 100, h: 20 });
withFakeDocument({ "input, textarea, [contenteditable]": [input] }, () => {
__testing__.resetCandidateCache();
const candidates = host.queryCandidates("typing");
assert.equal(candidates.length, 1);
assert.deepEqual({ x: candidates[0]!.x, y: candidates[0]!.y }, { x: 50, y: 10 });
}); });
}); });
test("queryCandidates caches within a tick until reset", () => { test("queryTargets reports only real text inputs as editable", () => {
const first = fakeElement("button", { x: 0, y: 0, w: 10, h: 10 }); const input = fakeElement({ tag: "input", x: 0, y: 0, width: 100, height: 20, editable: true });
withFakeDocument( const checkbox = fakeElement({ tag: "input", x: 0, y: 40, width: 20, height: 20, editable: true });
{ 'a, button, input, select, textarea, [role="button"], [onclick]': [first] }, checkbox.type = "checkbox";
() => { withFakeDocument([input, checkbox], () => {
__testing__.resetCandidateCache(); const targets = host.queryTargets();
const a = host.queryCandidates("taps"); assert.equal(targets[0]!.editable, true);
const b = host.queryCandidates("taps"); assert.deepEqual({ x: targets[0]!.x, y: targets[0]!.y }, { x: 50, y: 10 });
assert.equal(a, b); assert.equal(targets[1]!.editable, false);
}, });
); });
// A disabled control used to be dropped from every verb's candidates, because
// the web host folded `disabled` into its visibility check. It is a fact of its
// own now, so `taps` still skips it while `swipes` can still start on it, which
// is what the native host has always done.
test("queryTargets reports a disabled control rather than dropping it", () => {
const disabled = fakeElement({
tag: "button", x: 0, y: 0, width: 40, height: 20, clickable: true, disabled: true,
});
withFakeDocument([disabled], () => {
const targets = host.queryTargets();
assert.equal(targets.length, 1);
assert.equal(targets[0]!.clickable, true);
assert.equal(targets[0]!.enabled, false);
});
});
test("queryTargets caches within a tick until reset", () => {
const button = fakeElement({ tag: "button", x: 0, y: 0, width: 10, height: 10, clickable: true });
withFakeDocument([button], () => {
const first = host.queryTargets();
const second = host.queryTargets();
assert.equal(first, second);
__testing__.resetTargetCache();
assert.notEqual(host.queryTargets(), first);
});
}); });
// evaluateExtractors builds State, which references document and window. // evaluateExtractors builds State, which references document and window.
@@ -328,6 +310,23 @@ test("selectorFromObject maps known keys to their canonical attribute", () => {
}); });
}); });
// Compose Multiplatform emits its testTag into `id`, which the native table
// already accepts via the resource-id alias. The web table must not be the one
// place that rejects it.
test("selectorFromObject resolves testTag through data-testid or id", () => {
assert.deepEqual(selectorFromObject({ testTag: "LoginSubmit" }), {
css: `:is([data-testid="LoginSubmit"], [id="LoginSubmit"])`,
});
});
// Multi-key selectors concatenate their parts into one compound, so the
// two-attribute testTag match has to stay a single compound piece.
test("selectorFromObject composes testTag with a second key", () => {
assert.deepEqual(selectorFromObject({ testTag: "Row", "aria-label": "first" }), {
css: `:is([data-testid="Row"], [id="Row"])[aria-label="first"]`,
});
});
test("selectorFromObject falls back to a literal attribute for unknown keys", () => { test("selectorFromObject falls back to a literal attribute for unknown keys", () => {
assert.deepEqual(selectorFromObject({ "data-foo": "bar" }), { assert.deepEqual(selectorFromObject({ "data-foo": "bar" }), {
css: `[data-foo="bar"]`, css: `[data-foo="bar"]`,
+22 -1
View File
@@ -59,9 +59,30 @@ function WitnessView({
)} )}
</div> </div>
) : null} ) : null}
{witness.detected_step && witness.detected_step !== witness.step ? (
<div className="violations-panel-witness-line">
<span className="violations-panel-witness-key">detected at</span>
{onJumpToStep ? (
<button
type="button"
className="violations-panel-witness-step"
onClick={() => onJumpToStep(witness.detected_step as number)}
>
step {witness.detected_step}
</button>
) : (
<span className="violations-panel-witness-value">
step {witness.detected_step}
</span>
)}
</div>
) : null}
{evidence.length > 0 ? ( {evidence.length > 0 ? (
<details className="violations-panel-residual" open={open}> <details className="violations-panel-residual" open={open}>
<summary>witness</summary> <summary>
witness
{witness.detected_step ? ` (state at step ${witness.detected_step})` : ""}
</summary>
<dl className="violations-panel-witness-evidence"> <dl className="violations-panel-witness-evidence">
{evidence.map(([name, value]) => ( {evidence.map(([name, value]) => (
<div key={name} className="violations-panel-witness-line"> <div key={name} className="violations-panel-witness-line">
+3
View File
@@ -128,6 +128,9 @@ export interface Witness {
// which for deferred obligations (next, eventually) is earlier than the // which for deferred obligations (next, eventually) is earlier than the
// step whose record carries the witness. // step whose record carries the witness.
step?: number; step?: number;
// detected_step is the step whose evaluation produced the violation. The
// extractors below are that step's state, not step's.
detected_step?: number;
extractors?: Record<string, unknown>; extractors?: Record<string, unknown>;
} }