Files
sanderling/internal/verifier/verifier_test.go
T
pj b02e86b2e3 ci: dispatch workflows for folio and the replay ui (#73)
* feat(runner): stop the step loop at the first violation on request

* feat(testrun): report violations as a typed error under exit-on-violation

* feat(cli): add --exit-on-violation and exit 2 when it fires

* docs(cli): document --exit-on-violation, --max-steps, and exit codes

* fix(web): enumerate and query across shadow roots in both producers

* test(chrome): compare both producers on a shadow-dom parity page

* test(browser): drive a canvas-under-shadow-root fixture end to end

* fix(web): select the focused field inside a shadow root before typing

* fix(web): report the pathname as the screen when there is no hash route

* fix(web): settle on dom quiescence instead of returning at body ready

* feat(replay-ui): add data-testid hooks the dogfood spec drives

* feat(replay-ui): add the dogfood spec sanderling runs against the replay ui

* fix(replay-ui): scope the screenshot property to the named state panel

* chore(make): add per-platform sanderling build targets

* ci: add dispatch workflows for folio and the replay ui

* docs: describe the dispatch workflows and how to read a failure

* ci(folio): give the ios leg its jdk, android sdk, just, and a clean app start

* refactor(web): use max for the settle budget

* ci: pin calibrated seeds, skip the flaky ios reinstall, bound every job

* ci: authenticate and pin the buf setup step

the anonymous release download hit the shared runner ip rate limit and
failed the job with 'socket hang up' after three retries.

* docs: record that canvas apps need a dom proxy to be text-fuzzable

* fix(ios): bound lifecycle rpcs and claim the target device

a launch the simulator rejects sent the xctest session into a recovery
chain that answered minutes late or never, and the rpc had no deadline,
so the run hung with no trace and no error. also take a per-udid flock:
a second run's reinstall lands under the first's live automation session
and wedges it.

* docs(ci): correct the ios hang wording and note the device lock

* refactor(verifier): derive the lastAction shape from one field list

both hosts must show a spec the same lastAction. one ordered list now
feeds the goja object and the json the web host installs, so they
cannot drift.

* fix(web): install lastAction in the page before extractors read it

state.lastAction was hardcoded null on web, so every property reading it
was silently vacuous: a correct property passed without ever firing.

* fix(web): carry element identity on actions and fix findAll on paths

an action's target was coordinates only, so a property matching on which
element was acted upon could never fire. ax.findAll([a,b]) also returned
nothing on web.

* fix(chrome): wait out a route transition before sampling facts

the tree stays byte-identical and quiet across a cross-fade, so both the
quiet timer and the unchanged-tree escape called it settled mid-flight
and extractors read two screens at once.

* fix: bound the pre-run app launch

launch happens before the runner starts, so --duration never covered it
and a wedged driver hung with no trace and no error.

* fix(folio): read balances from merged cards and treat unreadable as unknown

compose for web merges the whole accountcard subtree, so the balance
child never exists there and every card parsed as 0. the property then
compared 0 to 0 and fired on any submit, which is a false positive
generator. unknown is now null and null is vacuously true.

* test(folio): cover merged-card parsing and unknown balances

* ci(folio): make web an expect-the-bug leg

the web runtime can observe the double submit now, so the health gate
understates it. seed 1 finds it at step 109, 3 runs out of 3.

* docs(ci): explain why a submit tap landing on home is the bug

* fix(ios): read a StaticText's label as its text

AXValue was the only source for text, but a StaticText carries its
string in AXLabel, so nothing on screen had .text on ios: a spec reading
it saw everything on android and nothing here.

* docs(ci): correct the calibrated step ranges

* fix(folio): stop convicting on arithmetic float64 cannot hold

past 2^53 cents the gap between representable values is 128, so a real
1600-cent move reads back as something else and the equality is false
for a healthy submit as readily as a double one. also match parseCents:
a sign or an oversized amount is rejected, not read as an amount.

* test(folio): pin the safe-integer guard and its boundary

* docs: stop teaching the zero-default that caused a false alarm

* docs: write down the silent-vacuity failure modes

* feat(folio): tag the home total and the card transaction count

the total was the only untagged node on the screen, so the spec had to
sum cards and a clipped card broke the sum.

* fix(folio): read the app's own total and refuse contaminated windows

summing cards went null when one was clipped, and the null poisoned the
carrier for the rest of the run. the balance window also spanned every
transaction since the last home visit, so the property convicted on
deltas it could not attribute: the old web witness was 3.16x the typed
amount, not 2x.

* test(folio): pin the window rules and the count invariant

* fix(folio): never read a frame that shows two screens

android dumps a cross-fade with both screens in the tree. the route said
add-transaction while an unscoped find said home, so the oracle took a
half-rendered total as fresh and convicted on a tap that committed
nothing. one function now decides the route and returns null when the
frame is ambiguous.

* test(folio): cover transition frames, card readings and creation

* fix(folio): only disambiguate counts that came from merged text

the equal-length digit rule exists because web merges the card and an
account named -1 makes '12' ambiguous. a dedicated count node has
nothing to disambiguate, so applying it there threw away real evidence.

* ci(folio): pin the recalibrated seeds and drop android to a health gate

web 3 and ios 7 convict 3 runs out of 3 with an exactly 2x witness.
android convicts 2 in 5 because the same seed does not walk the same
trajectory there, so it proves the app runs instead.

* docs(ci): describe the two properties and why android cannot convict

* fix(android): wait out a route cross-fade before snapshotting

the dump could hold two screens at once, and the runner refuses to act
on such a tree, so a quarter of android steps applied no action and the
count varied per run: the same seed never walked the same trajectory.
the ios companion and the chrome driver already do this.

* ci(folio): let the android leg run far enough to see its conviction

* docs: only the repo owner merges

* ci(folio): a thrown predicate is not a conviction

exit 2 means the run recorded a violation, and a predicate that throws
is recorded as one too. so was newAccountBalanceIsZero, an unrelated
property in the same spec. the gate read the exit code and went green
with detection dead.

* ci: install idb-companion from its tap and stop interpolating inputs

idb-companion is not in homebrew-core, so the ios leg died before it
built anything. replay-ui expanded dispatch inputs into the shell.

* docs: correct the snippets and numbers that drifted from the code

* test(sidecar): pin that a slow read counts toward the stability streak

* fix(web): read the page's extractors only on steps that count

the page advances the spec's carriers when it evaluates, but the runner
applied the result only on non-transitional steps. a discarded step
moved the window forward anyway, so the next accepted pair bracketed two
transactions while counting one submit, and convicted a healthy app.
extractor errors now fail the run instead of leaving goja's values in
current against v8's in previous.

* fix(chrome): anchor the transition deadline when the dom goes quiet

it was anchored at script start, so a page that churned past the window
reached the check already expired and returned mid cross-fade. the
driver now publishes the idle timeout it needs, since the caller's 1s
could never spend the 800ms window.

* fix(web): fail on a partial extractor override

same mixed-producer hazard as the install error: some extractors hold
the page's value and the rest hold goja's, and a property comparing
across that split fires on a healthy app.

* docs: six of seven, the seventh is the stock property

* fix(folio): drop a name two cards answer to

homeTxnCountsOf keyed on the account name and let the last card win, so
two accounts the fuzzer named the same collapsed into one entry. a
reading that saw one Travel card and a later one that saw both then
subtracted two different accounts' counts, and
submitCommitsOneTransactionPerAction convicted a healthy app of
double-submitting. it is a gated property in folio-run.sh, so that reads
as "found the submit bug" over a card scrolling into view.

same rule createdAccountHasNonZeroBalance already applies: a name
nothing can attribute is no evidence. counted over every card, since an
unreadable twin spoils the identity too.

* perf(folio): read each frame once

every extractor asked routeOf, and routeOf does five ax.find calls. on
web each find walks the document and every shadow root beneath it, so
the spec cost 110 tree walks a step; homeCards was parsed four times
over. now 5 and once.

keyed on the identity of the state object because both hosts build a new
one per step and hand that one object to every getter, so it cannot
outlive its frame. holding the reference is what keeps that true rather
than likely.

* fix(web): keep an undefined reading's index through JSON

json has no undefined, so an extractor whose getter returned one had its
whole index dropped by JSON.stringify. that index then kept goja's
dump-derived value while its neighbours held the page's, and a property
comparing previous to current across the split fires on a healthy app.
folio has nine on(route, tag) extractors, so this was most extractors on
most steps.

each reading is wrapped in a {value} envelope: the drop now happens
inside the entry, and an absent value means the getter returned
undefined, which is what the goja host records for the same getter. a
json null would instead claim it returned null and x.current ===
undefined would answer differently on the two hosts.

* feat(verifier): report the registered extractor count

the web path needs it to check the page sent one reading per extractor.

* fix(runner): fail when the page reports fewer readings than extractors

the comment here already claimed a partial override was fatal. it was
not: the skipped check only catches indices outside the extractor list,
so a page reporting values for some extractors and not others left the
rest holding goja's reading of the dump with nothing said.

* test(browser): drive an undefined reading through the whole web path

four layers carry it: the page's envelope, the driver's unwrap, the
runner's count check and the verifier's decode. each has a unit test and
only a run proves they compose. goes red both ways, decoding an absent
value as null and dropping the envelope.

* fix(web): offer the aria roles a user activates

only role=button was in the tappable set, so link, checkbox, radio,
switch, tab, option, the menuitems and treeitem were invisible to the
enumeration however plain the control looked. the replay ui builds its
step rows as <li role="option">, and the spec dogfooding it had to
hand-write an action to reach them because no default verb could see a
single row.

both producers build the set from the same role list, since the parity
test compares them element by element.

* test(browser): tap a role-based control end to end

every control on the page is an <li role="option">, the shape the
replay ui gives its step rows, and the spec carries no action of its
own: the property firing is the evidence the default enumeration offered
a tap on one.

* fix(web): read aria-disabled as disabled

the enabled fact came off the disabled property, which only real form
controls have. it reads undefined on the role-based controls the
tappable set now covers, so every one of them looked enabled however
plainly it was marked otherwise, and the fuzzer would spend actions on
inert ones.

both producers answer the same two ways, and the parity fixture carries
a disabled row so the comparison covers it: reverting one side alone
names the element and the fact.

* docs(replay-ui): the enumeration reaches step rows now

the comment said role="option" is not in the tappable selector set,
which stopped being true a few commits ago. selectAStep stays, for the
reason the tab weight below it stays: one row among the page's clickable
elements is a thin chance, and both step-facing properties go vacuous on
a run that never selects one.

* test(runner): bound the last-action test by steps, not wall clock

100ms of wall clock against an assertion that two steps ran fatals under
load with "the web path never installed it", which reads as a
regression. every sibling test in the package uses a long duration and
MaxSteps.

* ci: run the kotlin tests in make test

RouteTransitionTest and the stability poll cover the android settle and
nothing in ci ran them. :sidecar:test needs no android sdk, checked by
running it with ANDROID_HOME pointed at nothing.

* fix(sidecar): measure the stability streak as observed quiet

parameterising pollUntilStable also moved the clock to the start of the
read that opened a run of identical snapshots, so a read's own duration
counted as quiet. the pre-existing caller polls a real uiautomator dump:
at 400ms a read, 750ms of required quiet became 250ms of observed quiet
and the poll settled in two reads instead of four.

the parameters stay, the semantics go back.

* test(sidecar): pin the transition cap by driving it

it asserted 1500 >= 700 + 300, two constants, which can only fail if
someone edits a constant. it now drives awaitSettledTree against a fade
that lands after 700ms and asserts it hands back the settled tree before
the cap. cut the cap to 1000 and it goes red.

* ci: pin buf-setup-action to a commit

it takes a token now, so a floating tag is a token handed to whatever
that tag moves to. note v1 there is a branch, not a tag, so the ref
lookup that resolves it is matching-refs/heads/v1.

* ci: declare least-privilege permissions

none of the three declared any, so each got the repository default.
release.yml and docs.yml already do this. all three only check out,
build, test and upload artifacts.

* ci: fail fast when a server never comes up

the readiness loops fell through silently after 30 tries, so a server
that never started surfaced as an opaque driver failure minutes later.
each now says what did not answer and on which port.

* ci(folio): a missing trace is not a verdict

with no trace the android gate ran its grep against ./trace.jsonl and
reported "never reached AddTransactionScreen, so it never got past
login", which is not what happened. the web and ios branches had the
same misdiagnosis on exit 0.

same class, one line up: the classifier's own failure was swallowed, so
with the evidence reader dead the gate printed a healthy run and exited
0.

* ci(replay-ui): skip a run directory with no trace

the summarise step is if: always(), and under github's bash -eo pipefail
an unmatched glob stays literal, the redirect fails, pipefail carries it
into the assignment and -e kills the step. so a failed fuzz run went red
twice, once for the real reason.
2026-08-15 13:01:27 +05:30

1436 lines
48 KiB
Go

package verifier
import (
"encoding/json"
"errors"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"github.com/dop251/goja"
"github.com/priyanshujain/sanderling/internal/bundler"
"github.com/priyanshujain/sanderling/internal/hierarchy"
"github.com/priyanshujain/sanderling/internal/ltl"
)
func newVerifier(t *testing.T, options ...Option) *Verifier {
t.Helper()
verifier, err := New(options...)
if err != nil {
t.Fatal(err)
}
return verifier
}
func mustLoad(t *testing.T, verifier *Verifier, source string) {
t.Helper()
if err := verifier.Load(source); err != nil {
t.Fatalf("Load: %v", err)
}
}
// bundleActionSpec bundles an inline TS spec authored against @sanderling/spec
// together with the goja runtime entry, so loading it installs
// __sanderlingNextAction__ (the shared picker). Action targets must be resolved
// ax elements (carrying x/y) or builtins; raw selector strings no longer
// resolve to coordinates in the unified contract.
func bundleActionSpec(t *testing.T, specSource string) string {
t.Helper()
dir := t.TempDir()
specPath := filepath.Join(dir, "spec.ts")
if err := os.WriteFile(specPath, []byte(specSource), 0o600); err != nil {
t.Fatal(err)
}
apiPath, err := filepath.Abs("../../pkg/spec/src/index.ts")
if err != nil {
t.Fatal(err)
}
runtimePath, err := filepath.Abs("../../pkg/spec/src/goja-runtime.ts")
if err != nil {
t.Fatal(err)
}
bundle, err := bundler.Bundle(bundler.Options{
EntryFile: specPath,
RuntimeFile: runtimePath,
Aliases: map[string]string{"@sanderling/spec": apiPath},
})
if err != nil {
t.Fatal(err)
}
return string(bundle.JavaScript)
}
// loadActionSpec bundles and loads an inline authored spec into the verifier.
func loadActionSpec(t *testing.T, verifier *Verifier, specSource string) {
t.Helper()
mustLoad(t, verifier, bundleActionSpec(t, specSource))
}
const helloSpec = `
const screen = __sanderling__.extract(state => state.snapshots.screen ?? "");
const balance = __sanderling__.extract(state => state.snapshots["ledger.balance"] ?? 0);
globalThis.screen = screen;
globalThis.balance = balance;
globalThis.properties = {
balanceNonNegative: __sanderling__.always(() => balance.current >= 0),
};
`
func TestLoad_ExposesRuntimeBindings(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
if len(verifier.extractors) != 2 {
t.Errorf("extractors registered: got %d, want 2", len(verifier.extractors))
}
if len(verifier.formulas) != 1 {
t.Errorf("formulas registered: got %d, want 1", len(verifier.formulas))
}
if _, ok := verifier.properties["balanceNonNegative"]; !ok {
t.Errorf("balanceNonNegative property missing: %+v", verifier.properties)
}
}
func TestPushSnapshot_UpdatesExtractorCurrentAndPrevious(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{
"screen": json.RawMessage(`"customer_ledger"`),
"ledger.balance": json.RawMessage(`1500`),
}}); err != nil {
t.Fatal(err)
}
screenValue := verifier.runtime.GlobalObject().Get("screen").ToObject(verifier.runtime)
if screenValue.Get("current").String() != "customer_ledger" {
t.Errorf("screen.current wrong: %v", screenValue.Get("current"))
}
balanceValue := verifier.runtime.GlobalObject().Get("balance").ToObject(verifier.runtime)
if balanceValue.Get("current").ToInteger() != 1500 {
t.Errorf("balance.current wrong: %v", balanceValue.Get("current"))
}
// Push again: previous should mirror the prior current.
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": json.RawMessage(`2000`)}}); err != nil {
t.Fatal(err)
}
balanceValue = verifier.runtime.GlobalObject().Get("balance").ToObject(verifier.runtime)
if balanceValue.Get("previous").ToInteger() != 1500 {
t.Errorf("balance.previous wrong: %v", balanceValue.Get("previous"))
}
if balanceValue.Get("current").ToInteger() != 2000 {
t.Errorf("balance.current wrong: %v", balanceValue.Get("current"))
}
}
func TestEvaluateProperties_HoldsThenViolates(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
cases := []struct {
balance int
want ltl.Verdict
}{
{1500, ltl.VerdictHolds},
{0, ltl.VerdictHolds},
{-1, ltl.VerdictViolated},
{500, ltl.VerdictViolated}, // sticky
}
for index, testCase := range cases {
raw, _ := json.Marshal(testCase.balance)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": raw}}); err != nil {
t.Fatal(err)
}
verdicts := verifier.EvaluateProperties()
if got := verdicts["balanceNonNegative"]; got != testCase.want {
t.Errorf("step %d (balance=%d): got %v, want %v", index, testCase.balance, got, testCase.want)
}
}
}
func TestNewlyViolatedProperties_OnsetOnly(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
// Balance trajectory: holds, holds, violates, stays violated, stays violated.
// Onset must appear only on step 3 even though the residual stays false
// through steps 4 and 5 (LTL `always` sticky semantics).
balances := []int{1500, 1500, -1, 500, 500}
for index, balance := range balances {
raw, _ := json.Marshal(balance)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": raw}}); err != nil {
t.Fatal(err)
}
_ = verifier.EvaluateProperties()
got := verifier.NewlyViolatedProperties()
step := index + 1
if step == 3 {
want := []string{"balanceNonNegative"}
if !slices.Equal(got, want) {
t.Errorf("step %d (onset): got %v, want %v", step, got, want)
}
} else if len(got) != 0 {
t.Errorf("step %d: expected empty onset set, got %v", step, got)
}
}
}
func TestNewlyViolatedProperties_FirstStepViolation(t *testing.T) {
const spec = `
globalThis.properties = {
alwaysFalse: __sanderling__.always(() => false),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, spec)
for step := 1; step <= 3; step++ {
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}}); err != nil {
t.Fatal(err)
}
_ = verifier.EvaluateProperties()
got := verifier.NewlyViolatedProperties()
if step == 1 {
want := []string{"alwaysFalse"}
if !slices.Equal(got, want) {
t.Errorf("step 1 (onset): got %v, want %v", got, want)
}
} else if len(got) != 0 {
t.Errorf("step %d: expected empty onset set after first-step violation, got %v", step, got)
}
}
}
func TestNewlyViolatedProperties_MultipleProperties(t *testing.T) {
const spec = `
const a = __sanderling__.extract(state => state.snapshots["a"] ?? 0);
const b = __sanderling__.extract(state => state.snapshots["b"] ?? 0);
globalThis.properties = {
propA: __sanderling__.always(() => a.current >= 0),
propB: __sanderling__.always(() => b.current >= 0),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, spec)
// propA violates at step 2, propB violates at step 4. Each must surface
// only on its own onset step.
aValues := []int{1, -1, -1, -1}
bValues := []int{1, 1, 1, -1}
expectOnset := map[int][]string{
2: {"propA"},
4: {"propB"},
}
for index := range aValues {
aRaw, _ := json.Marshal(aValues[index])
bRaw, _ := json.Marshal(bValues[index])
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"a": aRaw, "b": bRaw}}); err != nil {
t.Fatal(err)
}
_ = verifier.EvaluateProperties()
got := verifier.NewlyViolatedProperties()
step := index + 1
want := expectOnset[step]
if !slices.Equal(got, want) {
t.Errorf("step %d: got %v, want %v", step, got, want)
}
}
}
func TestNewlyViolatedProperties_DeterministicOrder(t *testing.T) {
const spec = `
globalThis.properties = {
zebra: __sanderling__.always(() => false),
apple: __sanderling__.always(() => false),
mango: __sanderling__.always(() => false),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, spec)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}}); err != nil {
t.Fatal(err)
}
_ = verifier.EvaluateProperties()
got := verifier.NewlyViolatedProperties()
want := []string{"apple", "mango", "zebra"}
if !slices.Equal(got, want) {
t.Errorf("onset order: got %v, want %v (sorted lexicographically)", got, want)
}
}
func TestNextAction_FromActionsGenerator(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"resource-id": "home_button", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
loadActionSpec(t, verifier, `
import { actions, Tap } from "@sanderling/spec";
globalThis.actions = actions(() => {
const home = state.ax.find("id:home_button");
return home ? [Tap({ on: home })] : [];
});
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindTap {
t.Errorf("kind: got %v, want Tap", action.Kind)
}
if action.On != "id:home_button" {
t.Errorf("selector: got %q, want id:home_button", action.On)
}
}
func TestNextAction_WeightedSelectsByWeight(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"resource-id": "home", "bounds": "[0,0,100,40]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "away", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t, WithSeed(42))
loadActionSpec(t, verifier, `
import { actions, weighted, Tap } from "@sanderling/spec";
const tapHome = actions(() => {
const home = state.ax.find("id:home");
return home ? [Tap({ on: home })] : [];
});
const tapAway = actions(() => {
const away = state.ax.find("id:away");
return away ? [Tap({ on: away })] : [];
});
globalThis.actions = weighted([1, tapHome], [99, tapAway]);
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
awayCount := 0
homeCount := 0
for range 200 {
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
switch action.On {
case "id:home":
homeCount++
case "id:away":
awayCount++
}
}
// Weights are 99:1, so away must dominate by a wide margin. Requiring a 5x
// skew (rather than a bare >) keeps the assertion robust to harmless picker
// reshuffles while still failing if the weight is ignored or inverted.
if awayCount <= 5*homeCount {
t.Errorf("expected away to outweigh home by >5x (weights 99:1), got home=%d away=%d", homeCount, awayCount)
}
}
func TestNextAction_EmptyGeneratorReturnsErrNoAction(t *testing.T) {
verifier := newVerifier(t)
loadActionSpec(t, verifier, `
import { actions } from "@sanderling/spec";
globalThis.actions = actions(() => []);
`)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
_, err := verifier.NextAction()
if !errors.Is(err, ErrNoAction) {
t.Errorf("expected ErrNoAction, got %v", err)
}
}
// setupTree carries the three targets the setup-precedence tests resolve.
const setupTree = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,120]"},
"children": [
{"attributes": {"resource-id": "setup", "bounds": "[0,0,100,40]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "main", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "login", "bounds": "[0,80,100,120]"}, "clickable": true, "enabled": true, "children": []}
]
}`
func TestNextAction_SetupTakesPrecedenceWhenYielding(t *testing.T) {
verifier := newVerifier(t)
loadActionSpec(t, verifier, `
import { actions, Tap } from "@sanderling/spec";
globalThis.setup = actions(() => {
const target = state.ax.find("id:setup");
return target ? [Tap({ on: target })] : [];
});
globalThis.actions = actions(() => {
const target = state.ax.find("id:main");
return target ? [Tap({ on: target })] : [];
});
`)
tree, err := hierarchy.Parse(setupTree)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.On != "id:setup" {
t.Errorf("setup precedence: got %q, want id:setup", action.On)
}
}
func TestNextAction_FallsThroughToActionsWhenSetupEmpty(t *testing.T) {
verifier := newVerifier(t)
loadActionSpec(t, verifier, `
import { actions, Tap } from "@sanderling/spec";
globalThis.setup = actions(() => []);
globalThis.actions = actions(() => {
const target = state.ax.find("id:main");
return target ? [Tap({ on: target })] : [];
});
`)
tree, err := hierarchy.Parse(setupTree)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.On != "id:main" {
t.Errorf("fallthrough: got %q, want id:main", action.On)
}
}
func TestNextAction_SetupReengagesAfterRegression(t *testing.T) {
verifier := newVerifier(t)
loadActionSpec(t, verifier, `
import { actions, extract, Tap } from "@sanderling/spec";
const loggedIn = extract(state => state.snapshots["loggedIn"] === true);
globalThis.setup = actions(() => {
if (loggedIn.current) return [];
const target = state.ax.find("id:login");
return target ? [Tap({ on: target })] : [];
});
globalThis.actions = actions(() => {
const target = state.ax.find("id:main");
return target ? [Tap({ on: target })] : [];
});
`)
tree, err := hierarchy.Parse(setupTree)
if err != nil {
t.Fatal(err)
}
push := func(loggedIn bool) {
raw := json.RawMessage(`false`)
if loggedIn {
raw = json.RawMessage(`true`)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"loggedIn": raw}, Tree: tree}); err != nil {
t.Fatal(err)
}
}
push(false)
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.On != "id:login" {
t.Fatalf("step 1 (logged out): got %q, want id:login", action.On)
}
push(true)
action, err = verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.On != "id:main" {
t.Fatalf("step 2 (logged in): got %q, want id:main", action.On)
}
push(false)
action, err = verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.On != "id:login" {
t.Fatalf("step 3 (regressed): got %q, want id:login", action.On)
}
}
func TestNextAction_NoSetupRegistered(t *testing.T) {
verifier := newVerifier(t)
loadActionSpec(t, verifier, `
import { actions, Tap } from "@sanderling/spec";
globalThis.actions = actions(() => {
const target = state.ax.find("id:main");
return target ? [Tap({ on: target })] : [];
});
`)
tree, err := hierarchy.Parse(setupTree)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.On != "id:main" {
t.Errorf("got %q, want id:main", action.On)
}
}
// TestDoubleTapsBuiltin_TargetsClickable verifies the doubleTaps builtin emits
// a DoubleTap action targeting a clickable, enabled element's center.
func TestDoubleTapsBuiltin_TargetsClickable(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"testTag": "SubmitButton", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
loadActionSpec(t, verifier, `
import { doubleTaps } from "@sanderling/spec";
globalThis.actions = doubleTaps;
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindDoubleTap {
t.Fatalf("kind = %v, want DoubleTap", action.Kind)
}
if action.X != 50 || action.Y != 60 {
t.Errorf("coords = (%d,%d), want (50,60) at SubmitButton center", action.X, action.Y)
}
// Action-gated properties read lastAction.on to tell which target the
// chooser hit. An empty On reduces those properties to vacuously-true and
// they never fire on the real tap event.
if action.On == "" {
t.Fatal("On must be populated so action-gated properties can identify the target")
}
if !strings.Contains(action.On, "SubmitButton") {
t.Errorf("On = %q, want a selector containing SubmitButton", action.On)
}
resolved := tree.Find(action.On)
if resolved == nil {
t.Fatalf("On = %q does not resolve in the same tree", action.On)
}
rx, ry := resolved.Bounds.Center()
if rx != action.X || ry != action.Y {
t.Errorf("On %q resolves to (%d,%d), want the picked element's center (%d,%d)",
action.On, rx, ry, action.X, action.Y)
}
}
func TestDoubleTap_RoundTrip(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"resource-id": "save", "bounds": "[0,0,100,40]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
loadActionSpec(t, verifier, `
import { actions, DoubleTap } from "@sanderling/spec";
globalThis.actions = actions(() => {
const save = state.ax.find("id:save");
return save ? [DoubleTap({ on: save })] : [];
});
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindDoubleTap {
t.Errorf("kind: got %v, want DoubleTap", action.Kind)
}
if action.On != "id:save" {
t.Errorf("selector: got %q, want id:save", action.On)
}
}
func TestInputText_RoundTrip(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"resource-id": "phone", "bounds": "[0,0,100,40]"}, "editable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
loadActionSpec(t, verifier, `
import { actions, InputText } from "@sanderling/spec";
globalThis.actions = actions(() => {
const phone = state.ax.find("id:phone");
return phone ? [InputText({ into: phone, text: "+919876543210" })] : [];
});
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindInputText {
t.Errorf("kind: %v", action.Kind)
}
if action.On != "id:phone" || action.Text != "+919876543210" {
t.Errorf("payload wrong: %+v", action)
}
}
// TestTypingBuiltin_TargetsEditableField verifies the typing generator emits an
// InputText action aimed at an editable, enabled element's center and fills it
// with a corpus value, ignoring clickable-but-not-editable elements.
func TestTypingBuiltin_TargetsEditableField(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"testTag": "EmailField", "bounds": "[0,0,100,40]"}, "editable": true, "enabled": true, "children": []},
{"attributes": {"testTag": "SubmitButton", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
loadActionSpec(t, verifier, `
import { typing } from "@sanderling/spec";
globalThis.actions = typing;
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindInputText {
t.Fatalf("kind = %v, want InputText", action.Kind)
}
if action.X != 50 || action.Y != 20 {
t.Errorf("coords = (%d,%d), want (50,20) at EmailField center", action.X, action.Y)
}
// The typing builtin draws from corpus.ts INPUT_CORPUS; the single editable
// field means any draw lands here, so the text must be a corpus member.
if !slices.Contains(testInputCorpus, action.Text) {
t.Errorf("text %q not drawn from the input corpus", action.Text)
}
}
// testInputCorpus mirrors pkg/spec/src/corpus.ts INPUT_CORPUS so the typing
// builtin's emitted text can be asserted to come from the shared pool.
var testInputCorpus = []string{
"", "a", strings.Repeat("a", 4096), "🙂🔥💸", " ", "\t\n", "-1",
"999999999999999999999", "0.0000001", "1e10", "'; DROP TABLE--",
"<script>alert(1)</script>", "../../etc/passwd", "%s%n", "NaN",
}
// TestTypingBuiltin_NoEditableYieldsErrNoAction verifies the typing generator
// declines (ErrNoAction) when no editable element is present, so a weighted
// layer falls through to another generator.
func TestTypingBuiltin_NoEditableYieldsErrNoAction(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"testTag": "SubmitButton", "bounds": "[0,0,100,40]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
loadActionSpec(t, verifier, `
import { typing } from "@sanderling/spec";
globalThis.actions = typing;
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
if _, err := verifier.NextAction(); !errors.Is(err, ErrNoAction) {
t.Fatalf("err = %v, want ErrNoAction", err)
}
}
func TestPushSnapshot_FeedsSnapshotsToExtractorState(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.captured = __sanderling__.extract(state => state.snapshots["k"]);
`)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"k": json.RawMessage(`"hello"`)}}); err != nil {
t.Fatal(err)
}
value := verifier.runtime.GlobalObject().Get("captured").ToObject(verifier.runtime).Get("current")
if value.String() != "hello" {
t.Errorf("snapshot value not propagated: %v", value)
}
}
func TestLoad_PropagatesSyntaxError(t *testing.T) {
verifier := newVerifier(t)
err := verifier.Load(`const x = ;`)
if err == nil || !strings.Contains(err.Error(), "run spec") {
t.Errorf("expected run-spec error, got %v", err)
}
}
func TestEvaluateProperties_ThrowingPredicateDoesNotPanic(t *testing.T) {
const spec = `
globalThis.properties = {
broken: __sanderling__.always(() => { throw new Error("bad predicate"); }),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, spec)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}}); err != nil {
t.Fatal(err)
}
verdicts := verifier.EvaluateProperties()
if got := verdicts["broken"]; got != ltl.VerdictViolated {
t.Errorf("verdict: got %v, want %v", got, ltl.VerdictViolated)
}
witness := verifier.Witness("broken")
if witness == nil {
t.Fatal("Witness: got nil, want non-nil")
}
if !witness.IsError {
t.Errorf("Witness.IsError = false, want true for a thrown predicate")
}
if !strings.Contains(witness.Reason, "bad predicate") {
t.Errorf("Witness.Reason: got %q, want to contain %q", witness.Reason, "bad predicate")
}
}
// An unbounded eventually that never fires stays pending during the run and is
// only reported by Finalize at run end. The witness records why it failed.
func TestFinalize_UnmetEventuallyReportedWithWitness(t *testing.T) {
const spec = `
globalThis.flag = __sanderling__.extract(state => state.snapshots["flag"] ?? false, "flag");
globalThis.properties = {
flagEventuallyTrue: __sanderling__.always(__sanderling__.eventually(() => flag.current === true)),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, spec)
for step := 0; step < 3; step++ {
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"flag": json.RawMessage(`false`)}}); err != nil {
t.Fatal(err)
}
verdicts := verifier.EvaluateProperties()
if got := verdicts["flagEventuallyTrue"]; got != ltl.VerdictPending {
t.Fatalf("step %d: verdict = %v, want pending", step, got)
}
}
ended := verifier.Finalize()
if !slices.Contains(ended, "flagEventuallyTrue") {
t.Fatalf("Finalize ended = %v, want to contain flagEventuallyTrue", ended)
}
witness := verifier.Witness("flagEventuallyTrue")
if witness == nil {
t.Fatal("Witness = nil after Finalize, want non-nil")
}
if !strings.Contains(witness.Reason, "eventually") {
t.Errorf("Witness.Reason = %q, want to mention eventually", witness.Reason)
}
}
// A next obligation spawned at step k and checked against step k+1's state is
// attributed to step k, the step that caused it. StepIndex from SnapshotInput
// labels the evaluator observations so the witness carries runner step numbers.
func TestEvaluateProperties_NextViolationCarriesOriginStepIndex(t *testing.T) {
const spec = `
globalThis.flag = __sanderling__.extract(state => state.snapshots["flag"] ?? false, "flag");
globalThis.properties = {
flagStaysTrue: __sanderling__.always(__sanderling__.next(() => flag.current === true)),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, spec)
push := func(stepIndex int, flag string) map[string]ltl.Verdict {
t.Helper()
if err := verifier.PushSnapshot(SnapshotInput{
Snapshots: Snapshots{"flag": json.RawMessage(flag)},
StepIndex: stepIndex,
}); err != nil {
t.Fatal(err)
}
return verifier.EvaluateProperties()
}
push(7, `true`)
push(8, `true`)
verdicts := push(9, `false`)
if got := verdicts["flagStaysTrue"]; got != ltl.VerdictViolated {
t.Fatalf("step 9: verdict = %v, want violated", got)
}
witness := verifier.Witness("flagStaysTrue")
if witness == nil {
t.Fatal("Witness = nil, want non-nil")
}
if witness.Step != 8 {
t.Errorf("Witness.Step = %d, want 8 (the step that spawned the next obligation)", witness.Step)
}
}
func TestLoad_AcceptsSpecWithoutPropertiesOrActions(t *testing.T) {
verifier := newVerifier(t)
if err := verifier.Load(`const noop = 1;`); err != nil {
t.Fatal(err)
}
if got := verifier.EvaluateProperties(); len(got) != 0 {
t.Errorf("no properties expected, got %v", got)
}
if _, err := verifier.NextAction(); !errors.Is(err, ErrNoAction) {
t.Errorf("expected ErrNoAction, got %v", err)
}
}
// TestSelectorPath_ScopedDescent ensures the JS-side `find([{...}, {...}])`
// shape walks each segment scoped under the previous match.
func TestSelectorPath_ScopedDescent(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "rootView", "bounds": "[0,0,1080,2340]"},
"children": [
{
"attributes": {"testTag": "HomeScreen", "bounds": "[0,0,540,2340]"},
"children": [
{
"attributes": {"testTag": "AccountCard", "bounds": "[0,0,540,200]"},
"children": [
{"attributes": {"testTag": "AccountName", "text": "Checking", "bounds": "[10,10,200,40]"}, "children": []}
]
}
]
},
{
"attributes": {"testTag": "LedgerScreen", "bounds": "[540,0,1080,2340]"},
"children": [
{"attributes": {"testTag": "AccountName", "text": "Other", "bounds": "[600,10,800,40]"}, "children": []}
]
}
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.found = __sanderling__.extract(state =>
state.ax.find([{ testTag: "HomeScreen" }, { testTag: "AccountCard" }, { testTag: "AccountName" }])
);
globalThis.foundUnreachable = __sanderling__.extract(state =>
state.ax.find([{ testTag: "LedgerScreen" }, { testTag: "AccountCard" }])
);
globalThis.allInHome = __sanderling__.extract(state =>
state.ax.findAll([{ testTag: "HomeScreen" }, { testTag: "AccountName" }])
);
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
found := verifier.runtime.GlobalObject().Get("found").ToObject(verifier.runtime).Get("current")
if found == nil || goja.IsUndefined(found) {
t.Fatal("expected path lookup to find AccountName under HomeScreen > AccountCard")
}
text := found.ToObject(verifier.runtime).Get("text")
if text.String() != "Checking" {
t.Fatalf("text = %q, want Checking", text.String())
}
unreachable := verifier.runtime.GlobalObject().Get("foundUnreachable").ToObject(verifier.runtime).Get("current")
if !goja.IsUndefined(unreachable) {
t.Fatalf("AccountCard is not under LedgerScreen, expected undefined, got %v", unreachable)
}
allInHome := verifier.runtime.GlobalObject().Get("allInHome").ToObject(verifier.runtime).Get("current")
allObject := allInHome.ToObject(verifier.runtime)
length := allObject.Get("length").ToInteger()
if length != 1 {
t.Fatalf("findAll path length = %d, want 1 (Checking only, not Other in LedgerScreen)", length)
}
}
// TestSelector_BooleanValue ensures a native JS boolean in the selector
// (e.g. `find({ focused: true })`) matches the "true"/"false" string
// serialization the hierarchy uses for boolean state attributes.
func TestSelector_BooleanValue(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"testTag": "EmailField", "bounds": "[0,0,100,40]"}, "focused": true, "children": []},
{"attributes": {"testTag": "PasswordField", "bounds": "[0,40,100,80]"}, "focused": false, "children": []}
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.focusedTag = __sanderling__.extract(state =>
state.ax.find({ focused: true })?.attrs?.testTag ?? null
);
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
got := verifier.runtime.GlobalObject().Get("focusedTag").ToObject(verifier.runtime).Get("current")
if got == nil || got.String() != "EmailField" {
t.Fatalf("expected EmailField, got %v", got)
}
}
// TestFrom_SeededReplayIsDeterministic guarantees `from()` over a per-step
// dynamic array picks the same element under the same seed across runs. The
// folio spec relies on this to replace Math.random() in account-card taps.
func TestFrom_SeededReplayIsDeterministic(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,200]"},
"children": [
{"attributes": {"resource-id": "card_a", "bounds": "[0,0,100,40]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "card_b", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "card_c", "bounds": "[0,80,100,120]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "card_d", "bounds": "[0,120,100,160]"}, "clickable": true, "enabled": true, "children": []}
]
}`
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
pickedSequence := func(seed uint64) []string {
verifier := newVerifier(t, WithSeed(seed))
loadActionSpec(t, verifier, `
import { actions, from, Tap } from "@sanderling/spec";
const cards = ["id:card_a", "id:card_b", "id:card_c", "id:card_d"];
globalThis.actions = actions(() => {
const target = state.ax.find(from(cards).generate());
return target ? [Tap({ on: target })] : [];
});
`)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
var picks []string
for range 20 {
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
picks = append(picks, action.On)
}
return picks
}
first := pickedSequence(1234)
second := pickedSequence(1234)
for i := range first {
if first[i] != second[i] {
t.Fatalf("step %d: %q != %q (replay not deterministic)", i, first[i], second[i])
}
}
other := pickedSequence(5678)
identical := true
for i := range first {
if first[i] != other[i] {
identical = false
break
}
}
if identical {
t.Fatal("expected different seeds to produce different pick sequences")
}
}
// A thrown predicate is a witnessed violation at the step it first throws, and
// the property latches violated thereafter (sticky always semantics). The
// witness captures the onset step's error text (count=1) and the extractor
// snapshot at that step; it does not keep updating to later counts because the
// verdict has latched.
func TestThrowingPredicate_WitnessCapturesOnsetStep(t *testing.T) {
const spec = `
globalThis.counter = __sanderling__.extract(state => state.snapshots["count"], "counter");
globalThis.properties = {
reportsCounter: __sanderling__.always(() => { throw new Error("count=" + counter.current); }),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, spec)
for step := 1; step <= 3; step++ {
raw := json.RawMessage([]byte{'"', byte('0' + step), '"'})
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"count": raw}}); err != nil {
t.Fatal(err)
}
verdicts := verifier.EvaluateProperties()
if got := verdicts["reportsCounter"]; got != ltl.VerdictViolated {
t.Fatalf("step %d: verdict = %v, want violated", step, got)
}
}
witness := verifier.Witness("reportsCounter")
if witness == nil {
t.Fatal("Witness = nil, want non-nil")
}
if witness.Step != 1 {
t.Errorf("Witness.Step = %d, want 1 (onset)", witness.Step)
}
if !strings.Contains(witness.Reason, "count=1") {
t.Errorf("Witness.Reason = %q, want to contain %q", witness.Reason, "count=1")
}
if got := string(witness.Extractors["counter"]); got != `"1"` {
t.Errorf("Witness.Extractors[counter] = %s, want %q", got, `"1"`)
}
}
func TestOverrideExtractorValues_PreservesPrevious(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": json.RawMessage(`100`)}}); err != nil {
t.Fatal(err)
}
if _, err := verifier.OverrideExtractorValues(map[int]json.RawMessage{1: json.RawMessage(`777`)}); err != nil {
t.Fatal(err)
}
balance := verifier.runtime.GlobalObject().Get("balance").ToObject(verifier.runtime)
if balance.Get("current").ToInteger() != 777 {
t.Errorf("override didn't take: current=%v", balance.Get("current"))
}
// Next push: previous mirrors the *override*, not the snapshot value.
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": json.RawMessage(`200`)}}); err != nil {
t.Fatal(err)
}
balance = verifier.runtime.GlobalObject().Get("balance").ToObject(verifier.runtime)
if balance.Get("previous").ToInteger() != 777 {
t.Errorf("previous should reflect override, got %v", balance.Get("previous"))
}
}
func TestOverrideExtractorValues_NilIsNoop(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": json.RawMessage(`42`)}}); err != nil {
t.Fatal(err)
}
if _, err := verifier.OverrideExtractorValues(nil); err != nil {
t.Fatal(err)
}
if _, err := verifier.OverrideExtractorValues(map[int]json.RawMessage{}); err != nil {
t.Fatal(err)
}
balance := verifier.runtime.GlobalObject().Get("balance").ToObject(verifier.runtime)
if balance.Get("current").ToInteger() != 42 {
t.Errorf("expected snapshot-driven current to remain 42, got %v", balance.Get("current"))
}
}
func TestOverrideExtractorValues_UnknownIndexSkipped(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": json.RawMessage(`42`)}}); err != nil {
t.Fatal(err)
}
skipped, err := verifier.OverrideExtractorValues(map[int]json.RawMessage{
1: json.RawMessage(`777`),
99: json.RawMessage(`1`),
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if skipped != 1 {
t.Errorf("expected 1 skipped entry, got %d", skipped)
}
balance := verifier.runtime.GlobalObject().Get("balance").ToObject(verifier.runtime)
if balance.Get("current").ToInteger() != 777 {
t.Errorf("valid override should still apply alongside skipped one, got current=%v", balance.Get("current"))
}
}
const objectExtractorSpec = `
const card = __sanderling__.extract(state => ({attrs: {testTag: "default"}, balance: 0}));
globalThis.card = card;
globalThis.properties = {
hasTestTag: __sanderling__.always(() => typeof card.current.attrs.testTag === "string"),
};
`
// TestSelectorStringFromJS_CanonicalGrammar guarantees the selector tag stamped
// on returned AX nodes round-trips back to a parseable selector string when the
// node is later used as an action target. Without this, an action emitted from
// `tap({ on: state.ax.find({ testTag: "LoginEmail" }) })` ends up with
// `action.selector = "[object Object]"` in the trace.
func TestSelectorStringFromJS_CanonicalGrammar(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"testTag": "LoginScreen", "bounds": "[0,0,100,40]"},
"children": [
{"attributes": {"testTag": "LoginEmail", "bounds": "[0,0,100,20]"}, "editable": true, "enabled": true, "children": []}
]}
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.objectSelector = __sanderling__.extract(state =>
state.ax.find({ testTag: "LoginScreen" })
);
globalThis.chainSelector = __sanderling__.extract(state =>
state.ax.find([{ testTag: "LoginScreen" }, { testTag: "LoginEmail" }])
);
globalThis.stringSelector = __sanderling__.extract(state =>
state.ax.find("testTag:LoginScreen")
);
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
read := func(name string) string {
handle := verifier.runtime.GlobalObject().Get(name).ToObject(verifier.runtime)
current := handle.Get("current")
if goja.IsUndefined(current) || goja.IsNull(current) {
return ""
}
object := current.ToObject(verifier.runtime)
return object.Get(tagSelector).String()
}
cases := []struct {
name string
want string
}{
{"objectSelector", "testTag:LoginScreen"},
{"chainSelector", "testTag:LoginScreen > testTag:LoginEmail"},
{"stringSelector", "testTag:LoginScreen"},
}
for _, testCase := range cases {
got := read(testCase.name)
if got != testCase.want {
t.Errorf("%s: got %q, want %q", testCase.name, got, testCase.want)
}
if strings.Contains(got, "[object") {
t.Errorf("%s: selector contains garbage %q", testCase.name, got)
}
}
}
// TestChangedExtractors_DiffsBetweenSnapshots verifies the per-step diff
// surfaces only extractors whose value actually changed, keyed by name (or
// extractor_N fallback when unnamed).
func TestChangedExtractors_DiffsBetweenSnapshots(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
__sanderling__.extract(state => state.snapshots["a"] ?? 0, "alpha");
__sanderling__.extract(state => state.snapshots["b"] ?? 0);
`)
push := func(a, b int) {
raw := func(n int) json.RawMessage {
body, _ := json.Marshal(n)
return body
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"a": raw(a), "b": raw(b)}}); err != nil {
t.Fatal(err)
}
}
push(1, 1)
first := verifier.ChangedExtractors()
if _, ok := first["alpha"]; !ok {
t.Errorf("step 1: expected alpha in diff (initial value), got %+v", first)
}
if _, ok := first["extractor_1"]; !ok {
t.Errorf("step 1: expected extractor_1 fallback name in diff, got %+v", first)
}
push(1, 2)
second := verifier.ChangedExtractors()
if _, ok := second["alpha"]; ok {
t.Errorf("step 2: alpha did not change, should not appear: %+v", second)
}
change, ok := second["extractor_1"]
if !ok {
t.Fatalf("step 2: expected extractor_1 in diff, got %+v", second)
}
if string(change.Prev) != "1" || string(change.Curr) != "2" {
t.Errorf("step 2: extractor_1 diff prev=%s curr=%s, want 1 -> 2", change.Prev, change.Curr)
}
push(1, 2)
third := verifier.ChangedExtractors()
if len(third) != 0 {
t.Errorf("step 3: nothing changed, diff should be empty, got %+v", third)
}
}
// TestExtract_DefaultsAndNamedNames verifies bindExtract assigns a fallback
// `extractor_N` name when no name is supplied and respects an explicit one.
func TestExtract_DefaultsAndNamedNames(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
__sanderling__.extract(state => 1);
__sanderling__.extract(state => 2, "ledgerRows");
__sanderling__.extract(state => 3);
`)
if len(verifier.extractors) != 3 {
t.Fatalf("extractors registered: got %d, want 3", len(verifier.extractors))
}
want := []string{"extractor_0", "ledgerRows", "extractor_2"}
for i, name := range want {
if got := verifier.extractors[i].name; got != name {
t.Errorf("extractor %d name: got %q, want %q", i, got, name)
}
}
}
// TestSelectorStringFromJS_NullEmpty verifies that nil/undefined args produce
// an empty string instead of "null"/"undefined" garbage.
func TestSelectorStringFromJS_NullEmpty(t *testing.T) {
verifier := newVerifier(t)
if got := selectorStringFromJS(verifier.runtime, goja.Undefined()); got != "" {
t.Errorf("undefined: got %q, want empty", got)
}
if got := selectorStringFromJS(verifier.runtime, goja.Null()); got != "" {
t.Errorf("null: got %q, want empty", got)
}
}
func TestOverrideExtractorValues_PropagatesNestedObjectFields(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, objectExtractorSpec)
if err := verifier.PushSnapshot(SnapshotInput{}); err != nil {
t.Fatal(err)
}
override := json.RawMessage(`{"attrs": {"testTag": "account-card"}, "balance": 12345}`)
skipped, err := verifier.OverrideExtractorValues(map[int]json.RawMessage{0: override})
if err != nil {
t.Fatal(err)
}
if skipped != 0 {
t.Errorf("unexpected skipped count: %d", skipped)
}
card := verifier.runtime.GlobalObject().Get("card").ToObject(verifier.runtime)
current := card.Get("current").ToObject(verifier.runtime)
attrs := current.Get("attrs").ToObject(verifier.runtime)
if got := attrs.Get("testTag").String(); got != "account-card" {
t.Errorf("nested override missing: card.current.attrs.testTag = %q, want %q", got, "account-card")
}
if got := current.Get("balance").ToInteger(); got != 12345 {
t.Errorf("scalar field missing: card.current.balance = %d, want 12345", got)
}
}
// TestOverrideExtractorValues_RecordedStateMatchesEvaluatedState pins the
// reported state to the state predicates read. The web path evaluates
// extractor bodies in V8 and injects the results here, so a diff or a witness
// built from the goja value would describe a state no property ever saw.
func TestOverrideExtractorValues_RecordedStateMatchesEvaluatedState(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
if err := verifier.PushSnapshot(SnapshotInput{
Snapshots: Snapshots{"ledger.balance": json.RawMessage(`100`)},
}); err != nil {
t.Fatal(err)
}
if _, err := verifier.OverrideExtractorValues(map[int]json.RawMessage{
1: json.RawMessage(`-7`),
}); err != nil {
t.Fatal(err)
}
verifier.EvaluateProperties()
balance := verifier.runtime.GlobalObject().Get("balance").ToObject(verifier.runtime)
evaluated := balance.Get("current").String()
change, ok := verifier.ChangedExtractors()["extractor_1"]
if !ok {
t.Fatal("ChangedExtractors reported no change for the overridden extractor")
}
if string(change.Curr) != evaluated {
t.Errorf("ChangedExtractors curr = %s, want %s (the value predicates read)",
change.Curr, evaluated)
}
witness := verifier.Witness("balanceNonNegative")
if witness == nil {
t.Fatal("balanceNonNegative did not violate on the overridden value")
}
if got := string(witness.Extractors["extractor_1"]); got != evaluated {
t.Errorf("witness extractor = %s, want %s (the value predicates read)",
got, evaluated)
}
}
// TestUnsupportedVerbs_CollectedDedupedInOrder drives the real host binding the
// shared picker invokes (__sanderlingHost__.reportUnsupported) and asserts the
// verifier collects each verb once, in first-seen order, for the run report.
func TestUnsupportedVerbs_CollectedDedupedInOrder(t *testing.T) {
verifier := newVerifier(t)
report, ok := goja.AssertFunction(
verifier.runtime.GlobalObject().Get("__sanderlingHost__").
ToObject(verifier.runtime).Get("reportUnsupported"),
)
if !ok {
t.Fatal("reportUnsupported host binding missing")
}
for _, verb := range []string{"scrolls", "swipes", "scrolls", "longPresses"} {
if _, err := report(goja.Undefined(), verifier.runtime.ToValue(verb)); err != nil {
t.Fatalf("reportUnsupported(%q): %v", verb, err)
}
}
got := verifier.UnsupportedVerbs()
want := []string{"scrolls", "swipes", "longPresses"}
if !slices.Equal(got, want) {
t.Errorf("UnsupportedVerbs = %v, want %v", got, want)
}
}
// TestWithPlatform_IOSReachesPicker asserts WithPlatform("ios") is plumbed all
// the way to the host binding the shared picker reads (host.platform()), and
// that a platform-gated builtin (pressKeys) still resolves on iOS, drawing from
// the native key pool. Bug class: the platform option is dropped before the
// picker, so iOS silently runs the android (default) verb matrix / key pool.
func TestWithPlatform_IOSReachesPicker(t *testing.T) {
verifier := newVerifier(t, WithPlatform("ios"))
platform, ok := goja.AssertFunction(
verifier.runtime.GlobalObject().Get("__sanderlingHost__").
ToObject(verifier.runtime).Get("platform"),
)
if !ok {
t.Fatal("platform host binding missing")
}
value, err := platform(goja.Undefined())
if err != nil {
t.Fatal(err)
}
if value.String() != "ios" {
t.Errorf("host.platform() = %q, want ios", value.String())
}
loadActionSpec(t, verifier, `
import { pressKeys } from "@sanderling/spec";
globalThis.actions = pressKeys;
`)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}}); err != nil {
t.Fatal(err)
}
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindPressKey {
t.Fatalf("kind = %v, want PressKey", action.Kind)
}
// iOS draws from NATIVE_PRESS_KEYS (corpus.ts), which contains only "back".
if action.Key != "back" {
t.Errorf("key = %q, want back (native press-key pool)", action.Key)
}
}
// TestOverrideExtractorValues_EmptyPayloadIsUndefined pins the cross-host
// meaning of a page reading with no value. JSON has no undefined, so the web
// runtime wraps every reading in a {value} envelope and the chrome driver hands
// an absent value through as an empty payload. It has to land here as undefined,
// because that is what PushSnapshot records for a getter that returned undefined
// on native: decoding it as null instead would make `x.current === undefined`
// answer one thing on the native host and another on web, for one spec.
func TestOverrideExtractorValues_EmptyPayloadIsUndefined(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": json.RawMessage(`42`)}}); err != nil {
t.Fatal(err)
}
if _, err := verifier.OverrideExtractorValues(map[int]json.RawMessage{
0: nil,
1: json.RawMessage(`null`),
}); err != nil {
t.Fatal(err)
}
for _, want := range []struct {
expression string
result bool
}{
{"screen.current === undefined", true},
{"screen.current === null", false},
{"balance.current === null", true},
{"balance.current === undefined", false},
} {
value, err := verifier.runtime.RunString(want.expression)
if err != nil {
t.Fatalf("evaluate %s: %v", want.expression, err)
}
if value.ToBoolean() != want.result {
t.Errorf("a spec reading %s gets %v, want %v", want.expression, value, want.result)
}
}
}
// TestExtractorCount_ReportsEveryRegisteredExtractor keeps the web path's
// completeness check honest: it compares the page's reading count against this
// number, so a count that ignored an extractor would let a partial table
// through.
func TestExtractorCount_ReportsEveryRegisteredExtractor(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
if got := verifier.ExtractorCount(); got != 2 {
t.Errorf("ExtractorCount() = %d, want 2 (helloSpec registers screen and balance)", got)
}
}