refactoring default action layer (#51)

* feat(hierarchy): add editable signal with native derivation

* feat(chrome): emit editable flag in hierarchy dump

* feat(verifier): expose editable on ax element objects

* feat(spec): add editable to selector and element types

* feat(verifier): register typing builtin generator

* feat(verifier): typing builtin types edge-case corpus into editable fields

* feat(spec): export typing builtin generator

* feat(spec): add defaultActions bundle

* feat(spec): export @sanderling/spec/defaults subpath

* feat(folio): layer defaultActions breadth over targeted flows

* test(verifier): typing builtin targets editable fields, declines otherwise

* test(hierarchy): editable derivation and selector matching

* test(spec): defaultActions, typing, and defaults barrel resolve

* fix(testrun): alias @sanderling/spec/defaults for the bundler

* test(chrome): editable flag for inputs, textarea, contenteditable

* feat(spec): typing builtin for the web (V8) action path

* chore(folio): auto-boot a bootable AVD in just test/install when none connected

* feat(driver): add ForegroundChecker optional capability

* feat(android): detect foreground package via adb dumpsys

* feat(sidecar): implement ForegroundApp via adb for android

* feat(runner): relaunch app when foreground escapes during exploration

* fix(spec): drop hardware back from defaultActions to stay in-app

* feat(spec): add DoubleTap action type and constructor

* feat(spec): wire DoubleTap through web-runtime serializer

* feat(verifier): bind doubleTap and decode DoubleTap actions

* feat(runner): dispatch DoubleTap as two taps inside one step

* test(doubleTap): cover constructor, verifier round-trip, and runner dispatch

* feat(folio): add noDuplicateTxnPerStep invariant and doubleSubmitTxn action

* fix(folio): track ledger row count across non-ledger steps; pin reproducer seed

* feat(spec): add doubleTaps random-target builtin to defaultActions

* feat(verifier): add doubleTaps random-target generator

* refactor(folio): drop doubleSubmitTxn; fuzzer surfaces double-submit via defaultActions

* fix(folio): make ledgerRowsSeen monotonic to suppress transient-render false positives

* feat(verifier): track newly-violated property set per step

Sticky `always(P)` violations re-surfaced on every step after onset,
flooding traces and summaries with duplicate records. EvaluateProperties
now diffs against the prior verdict map and records the onset set; a new
NewlyViolatedProperties accessor exposes it so callers can emit each
violation exactly once at its onset step. The verdict-map return is
preserved for residual / current-verdict consumers.

* refactor(runner): emit onset-only violations to trace and summary

Switch the per-step violation list from the sticky verdict map to the
verifier's onset set. Each property now appears exactly once across a
run: at the step it first violates, not on every subsequent step where
the residual stays false. Removes the dead violationNames helper.

* style(verifier): use maps.Copy for verdict snapshot

* fix(folio): make login spec content-driven (idempotent across re-entries)

* fix(verifier): canonicalize selector strings

Object/chain JS selectors used to fall through to goja's default
stringification, producing "[object Object]" tags that surfaced as
garbage in trace.action.selector. Emit canonical "k:v" / " > "-joined
strings instead so the tag round-trips back through the hierarchy
selector grammar.

* refactor(folio): replace txn invariants with balanceMatchesAddedTxn

Collapse noDuplicateTxnPerStep and newTxnChangesBalance into a single
per-row property: every newly-appearing ledger row's signed amount must
match the ledger balance delta. A double-submit lands two rows whose
individual amounts cannot both equal the aggregate delta, so each row
fires the property, catching both the row-count and balance-math
classes of bug under one semantic invariant.

* refactor(trace): drop WriteScreenshotAfter

Only one screenshot per step is captured now (concurrently with
hierarchy after settle), so the -after.png variant is unused.

* refactor(runner): one concurrent screenshot per step

Move screenshot capture into the post-action errgroup so it observes
the same UI moment as the hierarchy fetch. Drop the pre-action and
deferred -after captures. Skip WaitForIdle when the action is Wait
since the wait itself provides settling time.

* refactor(inspect-ui): use next step's screenshot for state after

Each step now has one screenshot (the moment of observation). The
"state after" view of step N is the same moment as step (N+1)'s
observation, so reuse that file rather than expecting a separate
-after.png.

* feat(sidecar): structural-hash settle poll

Add pollUntilStable and structuralHash helpers; wire them into the
Stub, Maestro, and iOS backends' waitForIdle. The structural hash
ignores bounds-only flicker (measure passes) but trips on any change
in resource-id/class/content-desc/text, so a Compose cross-fade where
both source and destination composables are momentarily alive no
longer slips through Maestro's waitForAppToSettle and contaminates
the next hierarchy fetch.

* test(sidecar): cover pollUntilStable and structuralHash

Verify the poll returns on two equal snapshots, after transient
churn, and at the cap when never stable; assert the hash ignores
bounds-only flicker and detects content changes.

* feat(spec): accept optional name on extract()

Add an (name, getter) overload so each extractor handle carries a
debuggable label that future trace fields (per-step diffs) can key
off. The web-runtime falls back to extractor_\${index} when none is
supplied so existing call sites keep working unchanged.

* test(spec): cover extract name overload

Verify the runtime receives an undefined name in the legacy shape,
the supplied name in the (name, getter) shape, and that
extract("name") with no getter throws.

* feat(verifier): name extractors for diff surfacing

bindExtract accepts an optional name argument; falls back to
extractor_N when omitted. The name is stored on extractorState
alongside prev/curr value caches that the next change will use to
emit per-step diffs.

* chore(folio): name every extract() call

Give each extractor in the Folio spec a debuggable label so the
inspect UI can render extractor-value diffs at violation steps
keyed by intent (ledgerRows, route, ledgerBalance, ...) rather
than by registration index.

* feat(verifier): track extractor value transitions

Cache each extractor's prior and current JSON-encoded value during
PushSnapshot; expose ChangedExtractors to surface per-step diffs the
runner can emit into the trace. The first observation flushes every
non-null extractor as a change so the inspect UI shows initial state
breadcrumbs alongside later transitions.

* test(verifier): cover ChangedExtractors diffs

Verify initial snapshot reports both named and fallback-named
extractors, a subsequent change surfaces prev/curr, and a no-op
snapshot leaves the diff empty.

* feat(trace): emit extractor_changes per step

Add ExtractorChanges to trace.Step and a runner helper that converts
the verifier's diff map into the trace shape. The inspect UI keys
its violation breadcrumbs off this field.

* feat(inspect-ui): render extractor-change breadcrumbs at violations

Show prev -> curr for each extractor whose value changed on the
selected step, anchored under the violation row in ActionList.
Long values collapse into <details> so the inline diff stays
readable while the full payload is one click away.

* fix(sidecar): cap stability poll independently of settle budget

The previous shape halved durationMillis between waitForAppToSettle
and the structural poll, then hammered hierarchy() at 80ms intervals
- on Maestro this stacked enough RPCs that hierarchy fetches began
timing out under load and the run stalled. Pass the full budget to
waitForAppToSettle and cap the follow-up structural poll at 600ms
with a 120ms interval, so the device sees at most a handful of
extra hierarchy reads per step.

* feat(cli): default --clear-data on so runs start fresh

* feat(sidecar): streak-based settle with route-transition detection

Two changes layered into the stability poll:

1. stabilitySnapshot returns null while the tree carries more than one
   route-level Screen tag (resource-id / testTag / identifier ending
   in "Screen"), so the poll cannot declare a NavHost cross-fade
   stable. Apps following the Compose route convention get this
   detection for free; apps that don't fall through to the generic
   signal below.

2. pollUntilStable now requires an uninterrupted stable streak of at
   least MIN_STABLE_STREAK_MILLIS rather than just N consecutive
   matches. A late transition that fires after a brief calm window
   breaks the streak instead of slipping past. Interval widened to
   250ms so UiAutomation isn't hammered under fuzz load.

* test(sidecar): cover streak reset and route-transition rejection

Verify the poll honors MIN_STABLE_STREAK_MILLIS, that a transient
mid-stream change resets the streak, that null returns block streak
progress through a NavHost cross-fade, and that stabilitySnapshot
counts only route-level attribute keys when summing Screen tags.

* feat(runner): re-fetch on transitional hierarchy capture

Some actions trigger async work (DB write, ViewModel coroutine) whose
navigation transition begins after the sidecar settle poll has already
exited. Without intervention, the next iteration's hierarchy fetch
lands mid cross-fade and the verifier observes a partial extractor
state which then surfaces as a false-positive violation at the step
where the transition completes.

fetchSyncedState pairs hierarchy + screenshot in one goroutine and
retries the pair (up to 4 times, 200ms apart) while the captured tree
contains more than one route-level *Screen tag. Steps that observe
no transition get no added cost; steps that catch a transition pay
up to ~600ms extra wall time but record a tree that matches the
post-transition state the property language expects to compare.

* feat(runner): gate first action on app reaching foreground

* test(runner): cover startup foreground gate and back-press

* feat(verifier): scope random-action targets to app package

Random tap/doubleTap/type/swipe candidates now exclude nodes whose package differs from the app under test, so exploration never fuzzes the soft keyboard, system UI, or permission dialogs. An unset app package or an element with no package stays in scope, preserving behavior on iOS.

* feat(testrun): pass app package into verifier scope filter

* test(verifier): cover package-scoped target selection

* feat(hierarchy): derive package from resource-id prefix

The Android sidecar omits an explicit package attribute, so the verifier's package scope filter was a no-op and the keyboard still leaked into targets. Native nodes carry their package as the resource-id prefix; derive it there when the attribute is absent. Compose testTags are colon-less and stay empty, keeping them in scope.

* test(hierarchy): cover package derivation from resource-id

* chore: stop tracking inspect-ui/dist build artifacts

* feat(android): detect focused-window package via dumpsys window

* feat(driver): add FocusedWindowChecker capability

* fix(runner): gate first observe on the app window being drawn, not just resumed

* test(mock): add FocusedWindowApp with foreground mirroring

* test(runner): cover startup gate waiting for app window to draw

* feat(proto): add Snapshot RPC for atomic hierarchy+screenshot

Pairs hierarchy and screenshot in a single response so the runner can
capture both under a backend mutex, avoiding the cross-fade race where
the two reads describe different frames.

* feat(sidecar): add snapshot default on DriverBackend

Default impl calls hierarchy() then screenshot(). The service layer wraps
the call in a mutex so concurrent runners observe a serialized pair.

* feat(sidecar): wire Snapshot handler with serialization lock

Synchronizes backend.snapshot() so concurrent runners observe a
serialized hierarchy+screenshot pair, eliminating the cross-fade race
where two parallel reads describe different frames.

* test(sidecar): cover Snapshot wire path and serialization lock

SnapshotHandlerTest asserts both fields are populated, concurrent calls
are serialized, and the default impl runs hierarchy then screenshot.

* feat(driver): expose Snapshot on DeviceDriver and sidecar client

Snapshot wraps the new atomic-snapshot gRPC: the runner gets hierarchy
and screenshot from one round-trip whose two reads are serialized on
the sidecar side.

* feat(driver): add Snapshot to chrome and mock drivers

The chrome tab is single-threaded so its Snapshot pairs the two reads
without extra locking. The mock records ActionSnapshot so tests can
assert the runner reaches for the paired RPC.

* refactor(runner): observe each step via the atomic Snapshot RPC

fetchSyncedState now issues one Snapshot per attempt so hierarchy and
screenshot describe the same on-device frame. The transitional retry
stays: that case handles a fully-captured but mid cross-fade frame,
which atomic capture cannot fix.

* test(runner): assert step uses Snapshot, not raw hierarchy/screenshot

TestRunner_UsesAtomicSnapshot catches regressions to the two-goroutine
race, and the existing parallel-fetch test now keys off ActionSnapshot.

* test(driver): cover Snapshot in proto descriptor and sidecar client

Adds Snapshot to the descriptor allowlist and a sidecar-client test that
asserts both fields come back over the wire.

* feat(trace): add Transitional flag to Step

* fix(runner): skip verifier for transitional trees after retry budget

When fetchSyncedState exits its retry loop with a tree that still shows a NavHost cross-fade, the runner now marks the step transitional, writes the step + screenshot to the trace, and skips Verifier.PushSnapshot / EvaluateProperties / ChangedExtractors so the previous-to-current extractor advance is not poisoned by transient state. The next clean step's previous still references the prior clean state. NextAction continues to run so the loop never deadlocks on a never-stabilizing screen.

* test(runner): cover transitional step skips verifier and clean control

* refactor(trace): rename Step.Action to Step.NextAction

The trace step's action field is the action chosen FOR THE NEXT iteration
based on observing this step's hierarchy, not the action that produced
this step. Rename Step.Action to Step.NextAction and the JSON tag to
next_action to make causality explicit at the data level.

* refactor(runner): assign trace action to Step.NextAction field

Follows the rename of trace.Step.Action to Step.NextAction. The runner
already computed the next iteration's action here; only the field name
changes.

* refactor(inspect): decode trace step's next_action JSON field

Mirrors the trace schema rename of action to next_action. The summary
shape exposed to the SPA (action_kind/action_label) keeps its current
JSON tags since these are derived labels, not the raw next-action.

* test(inspect): update fixtures to use next_action trace field

Aligns inspect tests with the trace schema rename. Step constructors
now set NextAction and the JSONL fixtures use the next_action tag.

* refactor(inspect-ui): rename Step.action to Step.next_action

Aligns the SPA type and consumers with the trace schema rename. The
StepSummary.action_kind/action_label labels stay unchanged since they
are derived labels, not the raw next-action.

* fix(folio): extract balanceMatchesAddedSum predicate as testable helper

Move the ledger-balance-vs-added-rows predicate into a pure helper module
so the property's logic is unit-testable in isolation. Marks the sanderling
example as an ES module so cross-package ESM imports resolve under node.

* fix(folio): use sum-of-added-rows in balanceMatchesAddedTxn

The old predicate (every row's signed amount equals delta) silently passed
the double-submit bug because two same-amount rows each match the delta in
isolation. Switching to the sum check (addedSum === delta) catches both the
double-submit case and any future multi-row append whose total drifts from
the balance change.

* test(spec): cover balanceMatchesAddedSum single, sum-match, over, under cases

Pins the sum-based predicate: a single new row matching delta and two new
rows summing to delta both hold; two-row over-sum (double-submit) and
under-sum cases both violate.

* fix(build): rebuild sidecar JAR when Kotlin sources change

Without source-file deps on $(SIDECAR_JAR), make never re-ran shadowJar
after a Kotlin edit, so a stale embedded JAR shipped on every install
and the new sidecar code was silently absent at runtime.

* fix(chrome): launch with no-sandbox so headless Chrome starts in CI

* fix(sidecar): type text at cursor instead of clearing the field

InputText now appends at the focus caret, matching the native driver
and the standard mobile-input contract, instead of deleting existing
content first. Adds an injectable command runner so the behavior is
testable without a device.

* test(sidecar): assert InputText types at cursor without clearing

Captures the adb command stream and verifies a single input-text call
with no preceding delete keyevents, plus the adb escaping cases.

* feat(proto): add LongPress RPC

* chore(proto): regenerate Go stubs for LongPress

* feat(driver): add LongPress to DeviceDriver interface

* feat(sidecar): add LongPress client method

* feat(mock): record LongPress action

* feat(chrome): implement LongPress as press-and-hold

* feat(sidecar): implement longPress across backends

* feat(sidecar): dispatch LongPress RPC to backend

* test(sidecar): cover LongPress dispatch

* test(sidecar): implement longPress in snapshot test backend

* feat(verifier): add LongPress and Scroll action kinds

* feat(folio-spec): predicate that gates balance check on TxnSubmit tap

Replaces the row-sum predicate (which always held by construction since
balance is derived from rows in Folio) with one that compares the typed
amount to the actual balance delta after a tap on TxnSubmit. Catches the
planted double-submit bug.

* feat(folio-spec): wire submitMovesBalanceByTypedAmount property

Adds lastAction and totalBalance extractors and uses them in the new
property. Drops ledgerRows/ledgerBalance extractors since nothing else
referenced them.

* feat(verifier): wire longPresses and scrolls generators

* test(verifier): cover longPresses and scrolls generators

* test(folio-spec): unit tests for submitChangesBalanceByTypedAmount

Covers single vs double submit, the DoubleTap variant, vacuous cases
(null action, wrong kind, wrong target, zero typed), and selector-as-
object coercion.

* feat(spec): add LongPress and Scroll authoring surface

* feat(spec): no-op LongPress and Scroll in web runtime

* feat(spec): re-export longPresses and scrolls as opt-in generators

* test(spec): cover LongPress and Scroll runtime members

* test(proto): expect LongPress in service descriptor

* feat(runner): dispatch LongPress and Scroll actions

* test(runner): cover LongPress and Scroll dispatch

* docs(action-space): move LongPress, Scroll, DoubleTap to current actions

* fix(runner): mark nil/empty hierarchy as transitional

A failed or empty sidecar hierarchy fetch was pushed straight to the
verifier, letting spec extractors crash with "Cannot read property 'map'
of undefined" when findAll returned null. Treat that case like a
transitional capture: skip the verifier push, still record the step, and
keep the loop progressing.

* fix(verifier): populate Action.On when tap chooser picks an element

Coordinate-targeted Taps/DoubleTaps left On empty, so action-gated
properties reading lastAction.on couldn't tell which target was hit and
were vacuously skipped. Resolve the picked element to a stable
key:value selector (resource-id, testTag, text, desc) and validate it
resolves back to the same element so we don't accidentally redirect the
tap to a sibling that shares the identifier.

* fix(folio): add parseTypedAmount helper matching app's parseCents

Raw user input like "50" must become 5000 cents, not 50. The existing
parseDollarCents helper strips non-digits and so reads "50" as 50 cents,
which is correct for formatted balance text but off by 100x for raw
input from the amount field.

* fix(folio): parse raw amount input as cents in submit predicate

txnAmountField holds raw user keystrokes, not formatted balance text.
Route it through parseTypedAmount so "50" reads as $50, matching how
the app commits the transaction.

* fix(folio): carry forward total balance across off-screen transitions

AddTransactionScreen shows neither AccountCard nor LedgerBalance, so the
extractor used to report 0 at the step before submit. That made every
non-zero current balance look like the full delta and tripped the typed
amount property on every honest submit. Remember the last-seen sum and
return it whenever the current snapshot has no balance signal.

* test(folio): cover submit predicate with raw typed-amount inputs

Pipes realistic raw keystrokes through parseTypedAmount + the predicate
so single submits clear and double submits fire as expected.

* feat(folio): add computeHomeTotalBalance helper

Pure helper that tracks Home multi-account total only and carries the last
Home sum across off-Home steps. Ledger's single-account balance is excluded
because mixing it would corrupt cross-screen scale comparisons.

* fix(folio): totalBalance carrier tracks only Home, not Ledger

Home cardSum is a multi-account total; Ledger's LedgerBalance is a single
account on a different scale. Blending them in the carrier produced bogus
cross-screen deltas (prev from Ledger, curr from Home), triggering false
positives in submitMovesBalanceByTypedAmount. Restrict the carrier to
Home AccountCard totals via the computeHomeTotalBalance helper.

* test(spec): cover computeHomeTotalBalance carrier behaviour

Tests Home sums, carrier passthrough on off-Home steps, the Ledger
scale-mismatch case, and a Home > off-Home > Home sequence.

* feat(runner): treat transient apply errors as transitional steps

Sidecar input RPCs occasionally hang with DEADLINE_EXCEEDED or
UNAVAILABLE on long fuzzing runs. The per-step loop previously
propagated any applyAction error and killed the run after a single
flake. Detect transient gRPC failures via status.FromError, mark the
step transitional, skip the post-action idle poll, and continue to the
next step. Fatal errors (outer ctx cancellation, non-transient codes,
verifier crashes) still propagate.

* test(runner): cover transient apply error resilience

TestRunner_TransientApplyErrorMarksTransitional drives the runner
through a wrapper that fails the first TapSelector with a gRPC
DeadlineExceeded then succeeds. Asserts the run does not exit, the
failed step is marked transitional with no violations, and the next
step runs cleanly. TestIsTransientApplyError_Classification covers the
helper's matching rules directly so future code changes don't quietly
drop a transient case.

* fix(folio): gate submit-balance property on Home route landing

totalBalance is only freshly computed when AccountCards are visible on
Home; off-Home landings return the carrier and would false-fire the
property, latching always(next(F)) to false and masking the real
double-submit bug. Skip vacuously when route is not "home".

* test(spec): cover route gate in submit-balance predicate

Adds route arg to existing cases (all use "home") and adds five new
cases: ledger landing with stale carrier, add-transaction with
double-insert delta, null route, plus home-landing positive and
double-insert negative cases anchoring the gate's allow path.
This commit is contained in:
pj authored and GitHub committed 2026-06-01 12:48:51 +05:30
1 parent f572c8ba66
commit 88db9653e5
65 files changed
+4748 -596

No files matched your search

+66
View File
@@ -8,6 +8,7 @@ import (
"os"
"os/exec"
"path/filepath"
"regexp"
"slices"
"strings"
"time"
@@ -236,6 +237,71 @@ func waitForBoot(ctx context.Context, timeout time.Duration) error {
}
}
// ForegroundPackage returns the package of the currently resumed activity on
// the connected device, or "" when it cannot be determined.
func ForegroundPackage(ctx context.Context) (string, error) {
adb, err := AdbBinary()
if err != nil {
return "", err
}
output, err := exec.CommandContext(ctx, adb, "shell", "dumpsys", "activity", "activities").Output()
if err != nil {
return "", err
}
return parseForegroundPackage(string(output)), nil
}
// FocusedWindowPackage returns the package owning the currently focused window,
// or "" when no window is focused (e.g. mid-launch, before the app has drawn).
// Unlike ForegroundPackage, this reflects what is actually on screen:
// ResumedActivity flips to a newly launched app before its first frame renders,
// while mCurrentFocus only names the app once its window is up.
func FocusedWindowPackage(ctx context.Context) (string, error) {
adb, err := AdbBinary()
if err != nil {
return "", err
}
output, err := exec.CommandContext(ctx, adb, "shell", "dumpsys", "window").Output()
if err != nil {
return "", err
}
return parseFocusedWindowPackage(string(output)), nil
}
// resumedActivityPackage matches the "<package>/<activity>" component name that
// dumpsys prints on ResumedActivity and mCurrentFocus lines, capturing the
// package.
var resumedActivityPackage = regexp.MustCompile(`([a-zA-Z][a-zA-Z0-9_.]*)/[a-zA-Z0-9_.$]+`)
// parseForegroundPackage extracts the foreground package from `dumpsys activity
// activities` output by reading the first ResumedActivity component name.
func parseForegroundPackage(dumpsys string) string {
for line := range strings.SplitSeq(dumpsys, "\n") {
if !strings.Contains(line, "ResumedActivity") {
continue
}
if match := resumedActivityPackage.FindStringSubmatch(line); match != nil {
return match[1]
}
}
return ""
}
// parseFocusedWindowPackage extracts the focused-window package from
// `dumpsys window` output by reading the mCurrentFocus component name. A
// "mCurrentFocus=null" line (no focused window) yields "".
func parseFocusedWindowPackage(dumpsys string) string {
for line := range strings.SplitSeq(dumpsys, "\n") {
if !strings.Contains(line, "mCurrentFocus") {
continue
}
if match := resumedActivityPackage.FindStringSubmatch(line); match != nil {
return match[1]
}
}
return ""
}
func bootCompleted(ctx context.Context) (bool, error) {
adb, err := AdbBinary()
if err != nil {
+72
View File
@@ -91,3 +91,75 @@ func TestPathContains(t *testing.T) {
t.Error("did not expect /nope in PATH")
}
}
func TestParseForegroundPackage(t *testing.T) {
cases := []struct {
name string
dumpsys string
want string
}{
{
name: "mResumedActivity folio",
dumpsys: " Stack #0:\n mResumedActivity: ActivityRecord{a1b2c3 u0 app.folio/.MainActivity t42}\n",
want: "app.folio",
},
{
name: "topResumedActivity chrome",
dumpsys: "ResumedActivity: ActivityRecord{ff u0 com.android.chrome/com.google.android.apps.chrome.Main t9}\n topResumedActivity=ActivityRecord{ff u0 com.android.chrome/com.google.android.apps.chrome.Main}",
want: "com.android.chrome",
},
{
name: "launcher",
dumpsys: " mResumedActivity: ActivityRecord{x u0 com.google.android.apps.nexuslauncher/.NexusLauncherActivity t1}",
want: "com.google.android.apps.nexuslauncher",
},
{
name: "no resumed activity",
dumpsys: " some unrelated dumpsys output\n with no resumed line\n",
want: "",
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := parseForegroundPackage(tc.dumpsys); got != tc.want {
t.Errorf("parseForegroundPackage = %q, want %q", got, tc.want)
}
})
}
}
func TestParseFocusedWindowPackage(t *testing.T) {
cases := []struct {
name string
dumpsys string
want string
}{
{
name: "folio focused",
dumpsys: " mCurrentFocus=Window{e00f63a u0 app.folio/app.folio.MainActivity}\n mFocusedApp=ActivityRecord{c0 u0 app.folio/.MainActivity t202}",
want: "app.folio",
},
{
name: "settings focused",
dumpsys: " mCurrentFocus=Window{709 u0 com.android.settings/com.android.settings.SubSettings}",
want: "com.android.settings",
},
{
name: "no focused window mid-launch",
dumpsys: " mCurrentFocus=null\n mFocusedApp=null",
want: "",
},
{
name: "no focus line",
dumpsys: " some unrelated dumpsys window output\n",
want: "",
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := parseFocusedWindowPackage(tc.dumpsys); got != tc.want {
t.Errorf("parseFocusedWindowPackage = %q, want %q", got, tc.want)
}
})
}
}
+39 -1
View File
@@ -36,6 +36,7 @@ func New() *Driver {
append(chromedp.DefaultExecAllocatorOptions[:],
chromedp.Flag("headless", true),
chromedp.Flag("disable-gpu", true),
chromedp.NoSandbox,
)...,
)
tabCtx, tabCancel := chromedp.NewContext(allocCtx)
@@ -190,6 +191,23 @@ func (d *Driver) PressKey(_ context.Context, key string) error {
return chromedp.Run(d.tabCtx, chromedp.KeyEvent(k))
}
func (d *Driver) LongPress(_ context.Context, x, y int) error {
script := fmt.Sprintf(`
(function() {
const el = document.elementFromPoint(%d, %d);
if (!el) return;
el.dispatchEvent(new PointerEvent('pointerdown', {clientX: %d, clientY: %d, bubbles: true}));
setTimeout(function() {
el.dispatchEvent(new PointerEvent('pointerup', {clientX: %d, clientY: %d, bubbles: true}));
}, 600);
})();`,
x, y,
x, y,
x, y,
)
return chromedp.Run(d.tabCtx, chromedp.Evaluate(script, nil))
}
// keyMap covers the keys web specs may emit (enter/tab/escape/arrows).
// "back"/"home" are intentionally absent: backspace/NUL have no navigation
// semantics in a browser, and the V8 action mix already excludes them.
@@ -218,7 +236,8 @@ func (d *Driver) Hierarchy(_ context.Context) (string, error) {
if (el.id) attrs['resource-id'] = el.id;
const label = el.getAttribute('aria-label') || el.getAttribute('alt') || el.getAttribute('title') || '';
if (label) attrs['content-desc'] = label;
if (el.tagName) attrs['tag'] = el.tagName.toLowerCase();
const tag = (el.tagName || '').toLowerCase();
if (tag) attrs['tag'] = tag;
if (el.className && typeof el.className === 'string' && el.className.trim()) {
attrs['class'] = el.className.trim();
}
@@ -226,6 +245,9 @@ func (d *Driver) Hierarchy(_ context.Context) (string, error) {
const isClickable = !!(el.onclick || el.tagName === 'A' || el.tagName === 'BUTTON' ||
el.tagName === 'INPUT' || el.tagName === 'SELECT' ||
el.getAttribute('role') === 'button' || el.getAttribute('onclick'));
const isEditable = el.isContentEditable || tag === 'textarea' ||
(tag === 'input' && !['button','submit','checkbox','radio','range','color','file','image','reset']
.includes((el.type || '').toLowerCase()));
const children = [];
for (const child of el.children) {
children.push(buildTree(child, false));
@@ -238,6 +260,7 @@ func (d *Driver) Hierarchy(_ context.Context) (string, error) {
focused: document.activeElement === el || null,
checked: el.checked || null,
selected: el.selected || null,
editable: isEditable || null,
};
}
return buildTree(document.body, true);
@@ -263,6 +286,21 @@ func (d *Driver) Screenshot(_ context.Context) (driver.Image, error) {
return driver.Image{PNG: buf, Width: w, Height: h}, nil
}
// Snapshot pairs hierarchy and screenshot back-to-back. The chromedp tab
// is single-threaded so the two CDP round-trips are already serialized:
// pairing them here matches the DeviceDriver contract without extra locking.
func (d *Driver) Snapshot(ctx context.Context) (string, driver.Image, error) {
hierarchy, err := d.Hierarchy(ctx)
if err != nil {
return "", driver.Image{}, err
}
image, err := d.Screenshot(ctx)
if err != nil {
return hierarchy, driver.Image{}, err
}
return hierarchy, image, nil
}
func (d *Driver) RecentLogs(_ context.Context, since time.Time, minLevel string) ([]driver.LogEntry, error) {
sinceMillis := since.UnixMilli()
d.logsMu.Lock()
+61
View File
@@ -2,10 +2,71 @@ package chrome
import (
"context"
"encoding/json"
"testing"
"time"
)
// TestHierarchy_EditableFlag confirms the injected hierarchy script marks text
// inputs, textareas, and contenteditable elements editable while leaving
// buttons and non-text inputs alone.
func TestHierarchy_EditableFlag(t *testing.T) {
const html = `<body>` +
`<input id="name">` +
`<textarea id="bio"></textarea>` +
`<button id="go">go</button>` +
`<div id="rich" contenteditable="true">x</div>` +
`<input id="chk" type="checkbox">` +
`</body>`
d := New()
defer d.Terminate(context.Background())
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := d.Launch(ctx, "data:text/html,"+html, false, nil); err != nil {
t.Fatalf("Launch: %v", err)
}
dump, err := d.Hierarchy(ctx)
if err != nil {
t.Fatalf("Hierarchy: %v", err)
}
type node struct {
Attributes map[string]string `json:"attributes"`
Children []node `json:"children"`
Editable *bool `json:"editable"`
}
var root node
if err := json.Unmarshal([]byte(dump), &root); err != nil {
t.Fatalf("unmarshal hierarchy: %v", err)
}
editableByID := map[string]*bool{}
var walk func(n node)
walk = func(n node) {
if id := n.Attributes["resource-id"]; id != "" {
editableByID[id] = n.Editable
}
for _, c := range n.Children {
walk(c)
}
}
walk(root)
isEditable := func(id string) bool {
return editableByID[id] != nil && *editableByID[id]
}
for _, id := range []string{"name", "bio", "rich"} {
if !isEditable(id) {
t.Errorf("%q: editable = %v, want true", id, editableByID[id])
}
}
for _, id := range []string{"go", "chk"} {
if isEditable(id) {
t.Errorf("%q: editable = true, want false/absent", id)
}
}
}
// TestRunCtx_CallerCancelPropagates confirms that cancelling the caller's
// context cancels the chromedp-bound context returned by runCtx. This is the
// channel by which step deadlines and Ctrl-C reach in-flight CDP calls.
+30
View File
@@ -19,9 +19,15 @@ type DeviceDriver interface {
InputText(ctx context.Context, text string) error
Swipe(ctx context.Context, fromX, fromY, toX, toY int, duration time.Duration) error
PressKey(ctx context.Context, key string) error
LongPress(ctx context.Context, x, y int) error
Hierarchy(ctx context.Context) (string, error)
Screenshot(ctx context.Context) (Image, error)
// Snapshot returns the hierarchy and screenshot captured back-to-back
// under a backend-side mutex, so the pair describes the same on-device
// frame. Prefer this over calling Hierarchy and Screenshot separately:
// independent reads can land on different frames during transitions.
Snapshot(ctx context.Context) (string, Image, error)
// RecentLogs returns log entries at or after `since`, filtered to
// `minLevel` or above. An empty minLevel defaults to "E".
RecentLogs(ctx context.Context, since time.Time, minLevel string) ([]LogEntry, error)
@@ -35,6 +41,30 @@ type DeviceDriver interface {
Metrics(ctx context.Context, bundleID string) (Metrics, error)
}
// ForegroundChecker is the optional capability for reporting which app is
// currently in the foreground. The runner uses it to keep exploration scoped
// to the app under test: when an action backs out of (or otherwise leaves) the
// app, the runner relaunches it before acting again. Drivers that cannot
// determine the foreground app simply do not implement this interface.
type ForegroundChecker interface {
// ForegroundApp returns the bundle id / package of the app currently in
// the foreground. An empty string means "unknown" and the runner skips
// enforcement for that step rather than relaunching blindly.
ForegroundApp(ctx context.Context) (string, error)
}
// FocusedWindowChecker is the optional capability for reporting which app owns
// the focused (on-screen) window. The startup gate prefers it over
// ForegroundChecker: the resumed-activity signal flips to a freshly launched
// app before its first frame draws, so observing on it alone can capture the
// previous app's screen. The focused window only names the app once its window
// is actually up.
type FocusedWindowChecker interface {
// FocusedWindowApp returns the package owning the focused window, or ""
// when no window is focused yet (e.g. mid-launch transition).
FocusedWindowApp(ctx context.Context) (string, error)
}
type LogEntry struct {
UnixMillis int64
Level string
+77
View File
@@ -18,8 +18,10 @@ const (
ActionInputText ActionKind = "input_text"
ActionSwipe ActionKind = "swipe"
ActionPressKey ActionKind = "press_key"
ActionLongPress ActionKind = "long_press"
ActionHierarchy ActionKind = "hierarchy"
ActionScreenshot ActionKind = "screenshot"
ActionSnapshot ActionKind = "snapshot"
ActionRecentLogs ActionKind = "recent_logs"
ActionWaitForIdle ActionKind = "wait_for_idle"
ActionHealth ActionKind = "health"
@@ -55,6 +57,21 @@ type Driver struct {
LogEntries []driver.LogEntry
MetricsData driver.Metrics
Failures map[ActionKind]error
// ForegroundResults is consumed one entry per ForegroundApp call (the
// last entry repeats). Empty yields "", which disables the runner's
// app-scope guard so tests that don't care are unaffected.
ForegroundResults []string
foregroundIndex int
// FocusedWindowResults is consumed one entry per FocusedWindowApp call
// (the last entry repeats). When empty, FocusedWindowApp mirrors the
// last ForegroundApp result, so the startup gate treats the window as
// already drawn and tests that don't care are unaffected.
FocusedWindowResults []string
focusedWindowIndex int
focusedWindowCalls int
lastForeground string
}
func New() *Driver {
@@ -96,6 +113,44 @@ func (d *Driver) Launch(_ context.Context, bundleID string, clearState bool, _ m
return nil
}
func (d *Driver) ForegroundApp(_ context.Context) (string, error) {
d.mutex.Lock()
defer d.mutex.Unlock()
if len(d.ForegroundResults) == 0 {
d.lastForeground = ""
return "", nil
}
index := d.foregroundIndex
if index >= len(d.ForegroundResults) {
index = len(d.ForegroundResults) - 1
}
d.foregroundIndex++
d.lastForeground = d.ForegroundResults[index]
return d.lastForeground, nil
}
func (d *Driver) FocusedWindowApp(_ context.Context) (string, error) {
d.mutex.Lock()
defer d.mutex.Unlock()
d.focusedWindowCalls++
if len(d.FocusedWindowResults) == 0 {
return d.lastForeground, nil
}
index := d.focusedWindowIndex
if index >= len(d.FocusedWindowResults) {
index = len(d.FocusedWindowResults) - 1
}
d.focusedWindowIndex++
return d.FocusedWindowResults[index], nil
}
// FocusedWindowCalls reports how many times FocusedWindowApp has been called.
func (d *Driver) FocusedWindowCalls() int {
d.mutex.Lock()
defer d.mutex.Unlock()
return d.focusedWindowCalls
}
func (d *Driver) Terminate(ctx context.Context) error {
if err := d.failure(ActionTerminate); err != nil {
return err
@@ -112,6 +167,14 @@ func (d *Driver) Tap(ctx context.Context, x, y int) error {
return nil
}
func (d *Driver) LongPress(ctx context.Context, x, y int) error {
if err := d.failure(ActionLongPress); err != nil {
return err
}
d.record(Action{Kind: ActionLongPress, X: x, Y: y})
return nil
}
func (d *Driver) TapSelector(ctx context.Context, selector string) error {
if err := d.failure(ActionTapSelector); err != nil {
return err
@@ -181,6 +244,20 @@ func (d *Driver) Screenshot(ctx context.Context) (driver.Image, error) {
return d.ImageData, nil
}
// Snapshot returns the hierarchy + screenshot pair atomically, mirroring
// the real driver's contract. It records a single ActionSnapshot so tests
// can assert the runner reached for the paired RPC instead of racing the
// two reads.
func (d *Driver) Snapshot(ctx context.Context) (string, driver.Image, error) {
if err := d.failure(ActionSnapshot); err != nil {
return "", driver.Image{}, err
}
d.record(Action{Kind: ActionSnapshot})
d.mutex.Lock()
defer d.mutex.Unlock()
return d.HierarchyJSON, d.ImageData, nil
}
func (d *Driver) WaitForIdle(ctx context.Context, duration time.Duration) error {
if err := d.failure(ActionWaitForIdle); err != nil {
return err
+48
View File
@@ -8,6 +8,7 @@ import (
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"github.com/priyanshujain/sanderling/internal/android"
"github.com/priyanshujain/sanderling/internal/driver"
driverpb "github.com/priyanshujain/sanderling/proto/driverpb"
)
@@ -15,6 +16,31 @@ import (
type Client struct {
connection *grpc.ClientConn
stub driverpb.DriverClient
platform string
}
// SetPlatform records the target platform so capability methods (e.g.
// ForegroundApp) can pick the right backend. The caller sets this right after
// Dial.
func (c *Client) SetPlatform(platform string) { c.platform = platform }
// ForegroundApp reports the foreground package. Only Android is supported (via
// adb); other platforms return "" so the runner skips app-scope enforcement.
func (c *Client) ForegroundApp(ctx context.Context) (string, error) {
if c.platform != "android" {
return "", nil
}
return android.ForegroundPackage(ctx)
}
// FocusedWindowApp reports the package owning the focused window. Only Android
// is supported (via adb); other platforms return "" so the startup gate falls
// back to the foreground-app signal.
func (c *Client) FocusedWindowApp(ctx context.Context) (string, error) {
if c.platform != "android" {
return "", nil
}
return android.FocusedWindowPackage(ctx)
}
// Dial connects to the sidecar gRPC server at the given address.
@@ -67,6 +93,11 @@ func (c *Client) Tap(ctx context.Context, x, y int) error {
return err
}
func (c *Client) LongPress(ctx context.Context, x, y int) error {
_, err := c.stub.LongPress(ctx, &driverpb.Point{X: int32(x), Y: int32(y)})
return err
}
func (c *Client) TapSelector(ctx context.Context, selector string) error {
_, err := c.stub.TapSelector(ctx, &driverpb.Selector{Value: selector})
return err
@@ -136,6 +167,23 @@ func (c *Client) Screenshot(ctx context.Context) (driver.Image, error) {
}, nil
}
// Snapshot fetches hierarchy and screenshot in a single sidecar round-trip.
// The sidecar serializes the two reads behind a mutex so the returned pair
// describes the same on-device frame, removing the cross-fade race the
// runner used to see when fetching them as independent goroutines.
func (c *Client) Snapshot(ctx context.Context) (string, driver.Image, error) {
response, err := c.stub.Snapshot(ctx, &driverpb.Empty{})
if err != nil {
return "", driver.Image{}, err
}
image := response.GetScreenshot()
return response.GetHierarchy().GetJson(), driver.Image{
PNG: image.GetPng(),
Width: int(image.GetWidth()),
Height: int(image.GetHeight()),
}, nil
}
func (c *Client) WaitForIdle(ctx context.Context, duration time.Duration) error {
_, err := c.stub.WaitForIdle(ctx, &driverpb.Duration{Millis: duration.Milliseconds()})
return err
+26
View File
@@ -105,6 +105,15 @@ func (s *fakeServer) Screenshot(_ context.Context, _ *driverpb.Empty) (*driverpb
return &driverpb.Image{Png: s.imagePNG, Width: s.imageWidth, Height: s.imageHeight}, nil
}
func (s *fakeServer) Snapshot(_ context.Context, _ *driverpb.Empty) (*driverpb.SnapshotResponse, error) {
s.mutex.Lock()
defer s.mutex.Unlock()
return &driverpb.SnapshotResponse{
Hierarchy: &driverpb.HierarchyJSON{Json: s.hierarchy},
Screenshot: &driverpb.Image{Png: s.imagePNG, Width: s.imageWidth, Height: s.imageHeight},
}, nil
}
type harness struct {
server *grpc.Server
fake *fakeServer
@@ -260,6 +269,23 @@ func TestClient_HierarchyAndScreenshot(t *testing.T) {
}
}
func TestClient_SnapshotPairsHierarchyAndScreenshot(t *testing.T) {
state := newHarness(t)
client, _ := Dial(state.address)
defer client.Close()
hierarchyJSON, image, err := client.Snapshot(context.Background())
if err != nil {
t.Fatal(err)
}
if hierarchyJSON != `{"x":1}` {
t.Errorf("hierarchy wrong: %q", hierarchyJSON)
}
if image.Width != 1080 || image.Height != 2340 || len(image.PNG) != 1 {
t.Errorf("image wrong: %+v", image)
}
}
func TestClient_WaitForIdleForwardsMillis(t *testing.T) {
state := newHarness(t)
client, _ := Dial(state.address)
+20
View File
@@ -66,6 +66,7 @@ type Element struct {
Checked bool `json:"checked,omitempty"`
Focused bool `json:"focused,omitempty"`
Selected bool `json:"selected,omitempty"`
Editable bool `json:"editable,omitempty"`
Bounds Bounds `json:"bounds"`
Attributes map[string]string `json:"attrs,omitempty"`
}
@@ -91,6 +92,7 @@ type treeNodeJSON struct {
Focused *bool `json:"focused"`
Checked *bool `json:"checked"`
Selected *bool `json:"selected"`
Editable *bool `json:"editable"`
}
// Selector describes a multi-attribute AND match.
@@ -204,6 +206,18 @@ func elementFromNode(node *treeNodeJSON) *Element {
}
element.Class = attrs["class"]
element.Package = attrs["package"]
if element.Package == "" {
// Android omits an explicit package attribute, but native views carry
// it as the resource-id prefix (`com.android.systemui:id/...`). Compose
// testTags are colon-less and leave the package empty, which keeps them
// in scope. This lets target selection tell the app apart from the soft
// keyboard and system UI.
if resourceID := attrs["resource-id"]; resourceID != "" {
if colon := strings.IndexByte(resourceID, ':'); colon > 0 {
element.Package = resourceID[:colon]
}
}
}
element.Screen = attrs["sanderling-screen"]
if node.Clickable != nil {
@@ -221,6 +235,11 @@ func elementFromNode(node *treeNodeJSON) *Element {
if node.Selected != nil {
element.Selected = *node.Selected
}
if node.Editable != nil {
element.Editable = *node.Editable
} else {
element.Editable = strings.Contains(element.Class, "EditText") || attrs["hintText"] != ""
}
if b, ok := attrs["bounds"]; ok && b != "" {
bounds, err := parseBounds(b)
@@ -246,6 +265,7 @@ func elementFromNode(node *treeNodeJSON) *Element {
if node.Selected != nil {
element.Attributes["selected"] = strconv.FormatBool(*node.Selected)
}
element.Attributes["editable"] = strconv.FormatBool(element.Editable)
return element
}
+103
View File
@@ -141,6 +141,61 @@ func TestBoolFieldsFromNode(t *testing.T) {
}
}
func TestEditableDerivation(t *testing.T) {
cases := []struct {
name string
node string
want bool
}{
{"driver flag", `{"attributes": {"bounds": "[0,0,10,10]"}, "editable": true}`, true},
{"driver flag false", `{"attributes": {"bounds": "[0,0,10,10]"}, "editable": false}`, false},
{"native EditText class", `{"attributes": {"class": "android.widget.EditText", "bounds": "[0,0,10,10]"}}`, true},
{"hintText attr", `{"attributes": {"hintText": "Enter amount", "bounds": "[0,0,10,10]"}}`, true},
{"plain button", `{"attributes": {"class": "android.widget.Button", "bounds": "[0,0,10,10]"}}`, false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
tree, err := Parse(tc.node)
if err != nil {
t.Fatalf("Parse: %v", err)
}
el := tree.Elements[0]
if el.Editable != tc.want {
t.Errorf("Editable = %v, want %v", el.Editable, tc.want)
}
if got := el.Attributes["editable"]; got != boolString(tc.want) {
t.Errorf("attrs[editable] = %q, want %q", got, boolString(tc.want))
}
})
}
}
func boolString(b bool) string {
if b {
return "true"
}
return "false"
}
// TestFindByEditableSelector confirms find({editable:true}) matches via the
// mirrored attrs entry, the same path clickable uses.
func TestFindByEditableSelector(t *testing.T) {
input := `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"class": "android.widget.EditText", "bounds": "[0,0,100,40]"}, "children": []},
{"attributes": {"class": "android.widget.Button", "bounds": "[0,40,100,80]"}, "children": []}
]
}`
tree, _ := Parse(input)
if el := tree.Find("editable:true"); el == nil {
t.Fatal("expected to find the EditText via editable:true")
}
if matches := tree.FindAll("editable:true"); len(matches) != 1 {
t.Fatalf("editable:true matched %d elements, want 1", len(matches))
}
}
func TestIdentifierFallback(t *testing.T) {
input := `{
"attributes": {"identifier": "my-button", "bounds": "[0,0,100,100]"},
@@ -627,3 +682,51 @@ func TestNodeFindDoesNotReturnSiblings(t *testing.T) {
t.Fatal("Node.Find should not return nodes from sibling subtrees")
}
}
// TestPackageDerivedFromResourceIDPrefix verifies that when the sidecar omits an
// explicit package attribute, native nodes pick it up from the resource-id
// prefix while colon-less Compose testTags stay empty (in scope for the app).
func TestPackageDerivedFromResourceIDPrefix(t *testing.T) {
const dump = `{
"attributes": {"class": "android.widget.FrameLayout", "bounds": "[0,0,320,640]"},
"children": [
{"attributes": {"resource-id": "AddAccountScreen", "bounds": "[0,0,320,400]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "com.google.android.inputmethod.latin:id/key_pos_0_0", "bounds": "[0,400,40,440]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "android:id/content", "bounds": "[0,0,320,640]"}, "children": []}
]
}`
tree, err := Parse(dump)
if err != nil {
t.Fatalf("Parse: %v", err)
}
want := map[string]string{
"AddAccountScreen": "",
"com.google.android.inputmethod.latin:id/key_pos_0_0": "com.google.android.inputmethod.latin",
"android:id/content": "android",
}
for _, element := range tree.Elements {
expected, ok := want[element.ResourceID]
if !ok {
continue
}
if element.Package != expected {
t.Errorf("resource-id %q: package = %q, want %q", element.ResourceID, element.Package, expected)
}
}
}
// TestExplicitPackageAttributeWins verifies an explicit package attribute is not
// overridden by the resource-id prefix fallback.
func TestExplicitPackageAttributeWins(t *testing.T) {
const dump = `{
"attributes": {"resource-id": "android:id/content", "package": "app.folio", "bounds": "[0,0,320,640]"},
"children": []
}`
tree, err := Parse(dump)
if err != nil {
t.Fatalf("Parse: %v", err)
}
if got := tree.Elements[0].Package; got != "app.folio" {
t.Errorf("package = %q, want app.folio (explicit attr should win)", got)
}
}
+18 -18
View File
@@ -61,7 +61,7 @@ func decodeStepSummary(line []byte) (StepSummary, int, error) {
Index int `json:"step"`
Timestamp time.Time `json:"timestamp"`
Screen string `json:"screen,omitempty"`
Action *struct {
NextAction *struct {
Kind string `json:"kind"`
X int `json:"x,omitempty"`
Y int `json:"y,omitempty"`
@@ -73,7 +73,7 @@ func decodeStepSummary(line []byte) (StepSummary, int, error) {
Text string `json:"text,omitempty"`
Selector string `json:"selector,omitempty"`
DurationMillis int `json:"duration_millis,omitempty"`
} `json:"action,omitempty"`
} `json:"next_action,omitempty"`
Exceptions []json.RawMessage `json:"exceptions,omitempty"`
Violations []string `json:"violations,omitempty"`
}
@@ -87,29 +87,29 @@ func decodeStepSummary(line []byte) (StepSummary, int, error) {
HasViolations: len(partial.Violations) > 0,
HasExceptions: len(partial.Exceptions) > 0,
}
if partial.Action != nil {
summary.ActionKind = partial.Action.Kind
switch partial.Action.Kind {
case "Tap":
if partial.Action.Selector != "" {
summary.ActionLabel = partial.Action.Selector
} else if partial.Action.Text != "" {
summary.ActionLabel = partial.Action.Text
} else if partial.Action.X != 0 || partial.Action.Y != 0 {
summary.ActionLabel = fmt.Sprintf("(%d,%d)", partial.Action.X, partial.Action.Y)
if partial.NextAction != nil {
summary.ActionKind = partial.NextAction.Kind
switch partial.NextAction.Kind {
case "Tap", "DoubleTap":
if partial.NextAction.Selector != "" {
summary.ActionLabel = partial.NextAction.Selector
} else if partial.NextAction.Text != "" {
summary.ActionLabel = partial.NextAction.Text
} else if partial.NextAction.X != 0 || partial.NextAction.Y != 0 {
summary.ActionLabel = fmt.Sprintf("(%d,%d)", partial.NextAction.X, partial.NextAction.Y)
}
case "InputText":
summary.ActionLabel = fmt.Sprintf("%q", partial.Action.Text)
summary.ActionLabel = fmt.Sprintf("%q", partial.NextAction.Text)
case "Swipe":
summary.ActionLabel = swipeDirectionLabel(
partial.Action.FromX, partial.Action.FromY,
partial.Action.ToX, partial.Action.ToY,
partial.NextAction.FromX, partial.NextAction.FromY,
partial.NextAction.ToX, partial.NextAction.ToY,
)
case "PressKey":
summary.ActionLabel = partial.Action.Key
summary.ActionLabel = partial.NextAction.Key
case "Wait":
if partial.Action.DurationMillis > 0 {
summary.ActionLabel = fmt.Sprintf("%dms", partial.Action.DurationMillis)
if partial.NextAction.DurationMillis > 0 {
summary.ActionLabel = fmt.Sprintf("%dms", partial.NextAction.DurationMillis)
}
}
}
+8 -8
View File
@@ -104,7 +104,7 @@ func TestCacheStep_LazyDecodeReturnsFullStep(t *testing.T) {
startedAt := time.Now().UTC()
steps := []trace.Step{
{Index: 1, Timestamp: startedAt, Screen: "A"},
{Index: 2, Timestamp: startedAt.Add(time.Second), Screen: "B", Action: &trace.Action{Kind: "tap"}},
{Index: 2, Timestamp: startedAt.Add(time.Second), Screen: "B", NextAction: &trace.Action{Kind: "tap"}},
{Index: 3, Timestamp: startedAt.Add(2 * time.Second), Screen: "C", Violations: []string{"prop1"}},
}
writeRun(t, root, "r1", trace.Meta{StartedAt: startedAt, EndedAt: timePointer(startedAt.Add(3 * time.Second))}, steps)
@@ -144,13 +144,13 @@ func TestDecodeStepSummary_ActionLabelPerKind(t *testing.T) {
wantKind string
wantLabel string
}{
{`{"step":1,"timestamp":"2026-04-20T10:00:00Z","action":{"kind":"Tap","selector":"id:save"}}`, "Tap", "id:save"},
{`{"step":2,"timestamp":"2026-04-20T10:00:01Z","action":{"kind":"Tap","x":140,"y":220}}`, "Tap", "(140,220)"},
{`{"step":3,"timestamp":"2026-04-20T10:00:02Z","action":{"kind":"InputText","text":"alice"}}`, "InputText", `"alice"`},
{`{"step":4,"timestamp":"2026-04-20T10:00:03Z","action":{"kind":"Swipe","from_x":10,"from_y":500,"to_x":10,"to_y":50}}`, "Swipe", "up"},
{`{"step":5,"timestamp":"2026-04-20T10:00:04Z","action":{"kind":"Swipe","from_x":100,"from_y":50,"to_x":600,"to_y":50}}`, "Swipe", "right"},
{`{"step":6,"timestamp":"2026-04-20T10:00:05Z","action":{"kind":"PressKey","key":"back"}}`, "PressKey", "back"},
{`{"step":7,"timestamp":"2026-04-20T10:00:06Z","action":{"kind":"Wait","duration_millis":500}}`, "Wait", "500ms"},
{`{"step":1,"timestamp":"2026-04-20T10:00:00Z","next_action":{"kind":"Tap","selector":"id:save"}}`, "Tap", "id:save"},
{`{"step":2,"timestamp":"2026-04-20T10:00:01Z","next_action":{"kind":"Tap","x":140,"y":220}}`, "Tap", "(140,220)"},
{`{"step":3,"timestamp":"2026-04-20T10:00:02Z","next_action":{"kind":"InputText","text":"alice"}}`, "InputText", `"alice"`},
{`{"step":4,"timestamp":"2026-04-20T10:00:03Z","next_action":{"kind":"Swipe","from_x":10,"from_y":500,"to_x":10,"to_y":50}}`, "Swipe", "up"},
{`{"step":5,"timestamp":"2026-04-20T10:00:04Z","next_action":{"kind":"Swipe","from_x":100,"from_y":50,"to_x":600,"to_y":50}}`, "Swipe", "right"},
{`{"step":6,"timestamp":"2026-04-20T10:00:05Z","next_action":{"kind":"PressKey","key":"back"}}`, "PressKey", "back"},
{`{"step":7,"timestamp":"2026-04-20T10:00:06Z","next_action":{"kind":"Wait","duration_millis":500}}`, "Wait", "500ms"},
}
for _, tc := range cases {
summary, _, err := decodeStepSummary([]byte(tc.line))
+1 -1
View File
@@ -38,7 +38,7 @@ func newFixtureServer(t *testing.T) (*Server, string) {
BundleID: "com.example",
}, []trace.Step{
{Index: 1, Timestamp: startedAt, Screen: "Home"},
{Index: 2, Timestamp: startedAt.Add(time.Second), Screen: "Home", Action: &trace.Action{Kind: "tap"}},
{Index: 2, Timestamp: startedAt.Add(time.Second), Screen: "Home", NextAction: &trace.Action{Kind: "tap"}},
{Index: 3, Timestamp: startedAt.Add(2 * time.Second), Screen: "Cart", Violations: []string{"propA"}},
})
writeRun(t, root, "run-b", trace.Meta{
+431 -101
View File
@@ -10,6 +10,8 @@ import (
"time"
"golang.org/x/sync/errgroup"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"github.com/priyanshujain/sanderling/internal/driver"
"github.com/priyanshujain/sanderling/internal/hierarchy"
@@ -18,6 +20,11 @@ import (
"github.com/priyanshujain/sanderling/internal/verifier"
)
// doubleTapGap is the inter-tap delay for ActionKindDoubleTap: short enough to
// land both events inside a sub-100 ms race window, long enough for adb
// `input tap` to serialize two MotionEvent streams.
const doubleTapGap = 50 * time.Millisecond
type Options struct {
Duration time.Duration
IdleTimeout time.Duration
@@ -53,13 +60,16 @@ func Run(ctx context.Context, options Options) (Summary, error) {
logger = slog.Default()
}
// Gate on the app actually being on top before acting, so the first
// action never fires against a leftover screen or a system dialog. Done
// before the deadline is set so the settle time does not eat the run.
waitForForeground(ctx, options, logger)
summary := Summary{StartTime: time.Now()}
deadline := summary.StartTime.Add(options.Duration)
stepIndex := 0
var lastAction *verifier.Action
var lastLogTime time.Time
var pendingPostScreenshotStep int
pendingPostScreenshot := false
for time.Now().Before(deadline) {
if err := ctx.Err(); err != nil {
break
@@ -67,10 +77,19 @@ func Run(ctx context.Context, options Options) (Summary, error) {
stepIndex++
stepStart := time.Now()
// Hierarchy, metrics, and logs are independent device reads — run
// Keep exploration scoped to the app under test. If a prior action
// backed out of (or otherwise left) the app, relaunch it before we
// observe or act, so properties never evaluate against a foreign app
// and actions never land outside the app.
if ensureForeground(ctx, options, logger, stepIndex) {
lastAction = nil
}
// Hierarchy, metrics, and logs are independent device reads. Run
// them concurrently so metrics+logs hide behind the hierarchy fetch.
var tree *hierarchy.Tree
var hierarchyErr error
var transitional bool
var metrics *trace.Metrics
var logs []verifier.LogEntry
@@ -79,11 +98,14 @@ func Run(ctx context.Context, options Options) (Summary, error) {
// goroutine, whose CDP round-trip can otherwise outrun the step
// budget on a hung tab.
g, gctx := errgroup.WithContext(ctx)
si := stepIndex
// fetchSyncedState issues a single Snapshot RPC so hierarchy and
// screenshot describe the same frame, then re-fetches the pair
// while the tree still looks transitional.
g.Go(func() error {
tree, hierarchyErr = fetchHierarchy(gctx, options.Driver)
tree, transitional, hierarchyErr = fetchSyncedState(gctx, options, logger, si)
return nil
})
si := stepIndex
g.Go(func() error {
metrics = captureMetrics(gctx, options, logger, si)
return nil
@@ -105,14 +127,6 @@ func Run(ctx context.Context, options Options) (Summary, error) {
return nil
})
}
if pendingPostScreenshot {
postStep := pendingPostScreenshotStep
g.Go(func() error {
captureScreenshot(gctx, options, logger, postStep, true)
return nil
})
pendingPostScreenshot = false
}
// All goroutines write to local variables and return nil, so the Wait
// error is always nil; ignored intentionally.
_ = g.Wait()
@@ -127,38 +141,62 @@ func Run(ctx context.Context, options Options) (Summary, error) {
if tree != nil {
treeSize = len(tree.Elements)
}
// A nil or empty tree means the sidecar's hierarchy fetch failed or
// returned nothing (e.g. transient device-side timeout). Pushing it
// would let spec extractors call findAll() and chain .map() on a null
// result; treat it like a transitional capture so the verifier is
// skipped, the step is still recorded, and the loop progresses.
if treeSize == 0 {
transitional = true
}
lastLogTime = stepStart
if err := options.Verifier.PushSnapshot(verifier.SnapshotInput{
Tree: tree,
LastAction: lastAction,
StepTime: stepStart,
RunStart: summary.StartTime,
Logs: logs,
}); err != nil {
return summary, fmt.Errorf("step %d push: %w", stepIndex, err)
}
skipped, overrideErr := options.Verifier.OverrideExtractorValues(v8Overrides)
if overrideErr != nil {
logger.Warn("v8 override apply failed", "step", stepIndex, "err", overrideErr)
}
if skipped > 0 {
logger.Warn("v8 override skipped out-of-range entries",
"step", stepIndex, "skipped", skipped, "have", len(v8Overrides))
}
screen := ""
if tree != nil && len(tree.Elements) > 0 {
screen = tree.Elements[0].Screen
}
logger.Info("step", "index", stepIndex, "screen", screen, "nodes", treeSize)
verdicts := options.Verifier.EvaluateProperties()
violations := violationNames(verdicts)
for _, name := range violations {
if predicateErr := options.Verifier.PredicateError(name); predicateErr != nil {
logger.Warn("predicate error", "step", stepIndex, "property", name, "err", predicateErr)
// Transitional trees describe a NavHost mid cross-fade. Pushing
// one would poison the verifier's previous/current extractor
// advance, so the next clean step would compare against this
// transient state and emit false-positive violations. We still
// record the step (hierarchy + screenshot) for inspect-side
// debugging, but skip the verifier entirely and pick the next
// action against the unchanged prior state to keep the loop
// progressing.
var violations []string
var extractorChanges map[string]trace.ExtractorChange
if !transitional {
if err := options.Verifier.PushSnapshot(verifier.SnapshotInput{
Tree: tree,
LastAction: lastAction,
StepTime: stepStart,
RunStart: summary.StartTime,
Logs: logs,
}); err != nil {
return summary, fmt.Errorf("step %d push: %w", stepIndex, err)
}
skipped, overrideErr := options.Verifier.OverrideExtractorValues(v8Overrides)
if overrideErr != nil {
logger.Warn("v8 override apply failed", "step", stepIndex, "err", overrideErr)
}
if skipped > 0 {
logger.Warn("v8 override skipped out-of-range entries",
"step", stepIndex, "skipped", skipped, "have", len(v8Overrides))
}
options.Verifier.EvaluateProperties()
violations = options.Verifier.NewlyViolatedProperties()
for _, name := range violations {
if predicateErr := options.Verifier.PredicateError(name); predicateErr != nil {
logger.Warn("predicate error", "step", stepIndex, "property", name, "err", predicateErr)
}
}
extractorChanges = encodeExtractorChanges(options.Verifier.ChangedExtractors())
} else {
logger.Warn("transitional tree after retry budget; skipping verifier",
"step", stepIndex, "screen", screen, "nodes", treeSize)
}
logger.Info("step", "index", stepIndex, "screen", screen, "nodes", treeSize)
var nextAction verifier.Action
var nextErr error
@@ -179,20 +217,43 @@ func Run(ctx context.Context, options Options) (Summary, error) {
logger.Warn("residual encode failed", "step", stepIndex, "err", residualErr)
}
applySkipped := false
if nextErr == nil {
if err := applyAction(ctx, options.Driver, nextAction, tree); err != nil {
if isWDADrop(err) {
return summary, fmt.Errorf("step %d: iOS XCTest runner lost connection - known WDA startup flake, re-run the test: %w", stepIndex, err)
}
if isTransientApplyError(ctx, err) {
logger.Warn("transient apply error; marking step transitional", "step", stepIndex, "err", err)
transitional = true
applySkipped = true
lastAction = nil
} else {
return summary, fmt.Errorf("step %d apply: %w", stepIndex, err)
}
} else {
actionCopy := nextAction
lastAction = &actionCopy
}
} else {
lastAction = nil
}
step := trace.Step{
Index: stepIndex,
Timestamp: stepStart,
Screen: screen,
Action: traceAction,
Violations: violations,
Hierarchy: tree,
Residuals: residuals,
Metrics: metrics,
Index: stepIndex,
Timestamp: stepStart,
Screen: screen,
NextAction: traceAction,
Violations: violations,
Hierarchy: tree,
Residuals: residuals,
Metrics: metrics,
ExtractorChanges: extractorChanges,
Transitional: transitional,
}
if err := options.TraceWriter.WriteStep(step); err != nil {
return summary, fmt.Errorf("step %d trace: %w", stepIndex, err)
}
captureScreenshot(ctx, options, logger, stepIndex, false)
summary.Steps = stepIndex
if len(violations) > 0 {
summary.Violations = append(summary.Violations, ViolationRecord{
@@ -200,34 +261,19 @@ func Run(ctx context.Context, options Options) (Summary, error) {
Properties: violations,
})
}
if nextErr == nil {
if err := applyAction(ctx, options.Driver, nextAction, tree); err != nil {
if isWDADrop(err) {
return summary, fmt.Errorf("step %d: iOS XCTest runner lost connection - known WDA startup flake, re-run the test: %w", stepIndex, err)
}
return summary, fmt.Errorf("step %d apply: %w", stepIndex, err)
// Wait actions are themselves a settling: skip the idle poll. Actions
// that mutate the UI fall through to WaitForIdle so the next step's
// concurrent fetches observe a stable post-action state. A transient
// apply error means nothing landed, so the idle poll has nothing to
// settle and may itself hang on the same device condition.
if nextErr == nil && !applySkipped && nextAction.Kind != verifier.ActionKindWait {
idleCtx, idleCancel := context.WithTimeout(ctx, options.IdleTimeout)
idleErr := options.Driver.WaitForIdle(idleCtx, options.IdleTimeout)
if idleErr != nil && idleCtx.Err() == nil {
logger.Warn("wait_for_idle failed", "step", stepIndex, "err", idleErr)
}
actionCopy := nextAction
lastAction = &actionCopy
} else {
lastAction = nil
idleCancel()
}
idleCtx, idleCancel := context.WithTimeout(ctx, options.IdleTimeout)
idleErr := options.Driver.WaitForIdle(idleCtx, options.IdleTimeout)
if nextErr == nil {
pendingPostScreenshot = true
pendingPostScreenshotStep = stepIndex
}
if idleErr != nil && idleCtx.Err() == nil {
logger.Warn("wait_for_idle failed", "step", stepIndex, "err", idleErr)
}
idleCancel()
}
if pendingPostScreenshot {
captureScreenshot(ctx, options, logger, pendingPostScreenshotStep, true)
}
summary.EndTime = time.Now()
@@ -253,14 +299,109 @@ func validate(options Options) error {
return nil
}
func violationNames(verdicts map[string]ltl.Verdict) []string {
var names []string
for name, verdict := range verdicts {
if verdict == ltl.VerdictViolated {
names = append(names, name)
}
// ensureForeground keeps the app under test in the foreground. When the driver
// can report the foreground app and it no longer matches the bundle under test,
// the app is relaunched. Returns true when a relaunch happened so the caller
// can drop the now-stale lastAction. Drivers without ForegroundChecker (web,
// iOS) are a no-op.
func ensureForeground(ctx context.Context, options Options, logger *slog.Logger, stepIndex int) bool {
checker, ok := options.Driver.(driver.ForegroundChecker)
if !ok || options.BundleID == "" {
return false
}
return names
foreground, err := checker.ForegroundApp(ctx)
if err != nil {
logger.Warn("foreground check failed", "step", stepIndex, "err", err)
return false
}
if foreground == "" || foreground == options.BundleID {
return false
}
logger.Warn("app left foreground; relaunching",
"step", stepIndex, "foreground", foreground, "want", options.BundleID)
return bringToForeground(ctx, options, logger, stepIndex)
}
// foregroundReadyAttempts bounds how many times waitForForeground tries to
// bring the app forward before the first step, so a stuck system dialog can
// never hang the run.
const foregroundReadyAttempts = 8
// waitForForeground blocks until the app under test is actually on screen, so
// the first observe never captures a leftover screen or a freshly-booted
// device's system dialog (e.g. Android's "set a screen lock" prompt). Drivers
// without ForegroundChecker (web) and an unknown foreground both skip the gate.
//
// It is not enough that the app is the resumed activity: ResumedActivity flips
// to a freshly launched app ~before its first frame draws, so gating on it
// alone lets the first observe read the outgoing app. When the driver can also
// report the focused window, the gate additionally waits for that window to
// name the app, which only happens once it is genuinely drawn.
func waitForForeground(ctx context.Context, options Options, logger *slog.Logger) {
checker, ok := options.Driver.(driver.ForegroundChecker)
if !ok || options.BundleID == "" {
return
}
focusChecker, hasFocus := options.Driver.(driver.FocusedWindowChecker)
for attempt := range foregroundReadyAttempts {
if err := ctx.Err(); err != nil {
return
}
foreground, err := checker.ForegroundApp(ctx)
if err != nil {
logger.Warn("foreground check failed before first step", "err", err)
return
}
if foreground == "" {
return // foreground unknowable (e.g. iOS); don't block the run
}
if foreground != options.BundleID {
logger.Warn("app not in foreground at start; bringing it forward",
"foreground", foreground, "want", options.BundleID, "attempt", attempt)
bringToForeground(ctx, options, logger, 0)
continue
}
if !hasFocus {
return // resumed is the app and no finer signal exists
}
focused, err := focusChecker.FocusedWindowApp(ctx)
if err != nil {
logger.Warn("focus check failed before first step", "err", err)
return
}
if focused == options.BundleID {
return // window is drawn; safe to observe
}
logger.Warn("app resumed but window not yet drawn; waiting",
"focused", focused, "want", options.BundleID, "attempt", attempt)
settleForForeground(ctx, options)
}
logger.Warn("app never reached foreground before first step; proceeding anyway",
"want", options.BundleID)
}
// bringToForeground returns the app under test to the foreground. It first
// presses BACK to dismiss any modal system dialog (a relaunch alone does not
// close one), then relaunches and waits for the UI to settle. Returns true
// when the relaunch itself succeeded.
func bringToForeground(ctx context.Context, options Options, logger *slog.Logger, stepIndex int) bool {
if err := options.Driver.PressKey(ctx, "back"); err != nil {
logger.Warn("dismiss key before relaunch failed", "step", stepIndex, "err", err)
}
if err := options.Driver.Launch(ctx, options.BundleID, false, nil); err != nil {
logger.Warn("relaunch failed", "step", stepIndex, "err", err)
return false
}
settleForForeground(ctx, options)
return true
}
// settleForForeground waits one idle window for the UI to settle, bounding the
// wait by the driver's idle timeout.
func settleForForeground(ctx context.Context, options Options) {
idleCtx, cancel := context.WithTimeout(ctx, options.IdleTimeout)
_ = options.Driver.WaitForIdle(idleCtx, options.IdleTimeout)
cancel()
}
func applyAction(ctx context.Context, drv driver.DeviceDriver, action verifier.Action, tree *hierarchy.Tree) error {
@@ -274,6 +415,43 @@ func applyAction(ctx context.Context, drv driver.DeviceDriver, action verifier.A
return drv.TapSelector(ctx, action.On)
}
return drv.Tap(ctx, x, y)
case verifier.ActionKindDoubleTap:
x, y, ok := resolveCoordinates(action, tree)
tap := func() error {
if !ok {
if action.On == "" {
return nil
}
return drv.TapSelector(ctx, action.On)
}
return drv.Tap(ctx, x, y)
}
if err := tap(); err != nil {
return err
}
timer := time.NewTimer(doubleTapGap)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
}
return tap()
case verifier.ActionKindLongPress:
x, y, ok := resolveCoordinates(action, tree)
if !ok {
// No long-press-by-selector RPC exists, so an unresolved target is
// nothing we can dispatch; skip rather than error.
return nil
}
return drv.LongPress(ctx, x, y)
case verifier.ActionKindScroll:
fromX, fromY, toX, toY := scrollEndpoints(action, tree)
duration := time.Duration(action.DurationMillis) * time.Millisecond
if duration <= 0 {
duration = 300 * time.Millisecond
}
return drv.Swipe(ctx, fromX, fromY, toX, toY, duration)
case verifier.ActionKindInputText:
if x, y, ok := resolveCoordinates(action, tree); ok {
if err := drv.Tap(ctx, x, y); err != nil {
@@ -360,18 +538,139 @@ func resolveCoordinates(action verifier.Action, tree *hierarchy.Tree) (int, int,
return 0, 0, false
}
func fetchHierarchy(ctx context.Context, drv driver.DeviceDriver) (*hierarchy.Tree, error) {
xmlText, err := drv.Hierarchy(ctx)
if err != nil {
return nil, err
// scrollEndpoints lowers a Scroll to a swipe's from/to points. Pre-computed
// endpoints (from the generator) win. Otherwise it derives them from the
// container bounds: the named node when On resolves, else the whole screen.
func scrollEndpoints(action verifier.Action, tree *hierarchy.Tree) (fromX, fromY, toX, toY int) {
if action.FromX != 0 || action.FromY != 0 || action.ToX != 0 || action.ToY != 0 {
return action.FromX, action.FromY, action.ToX, action.ToY
}
return hierarchy.Parse(xmlText)
bounds := scrollBounds(action, tree)
cx, cy := bounds.Center()
width := bounds.Width()
height := bounds.Height()
toX, toY = cx, cy
// Scroll direction names content motion; the gesture swipes the opposite
// way. Revealing lower content ("down") drags the finger up, so toY drops.
switch action.Direction {
case "down":
toY = cy - (4*height)/10
case "up":
toY = cy + (4*height)/10
case "left":
toX = cx + (4*width)/10
case "right":
toX = cx - (4*width)/10
}
if toX < 0 {
toX = 0
}
if toY < 0 {
toY = 0
}
return cx, cy, toX, toY
}
// scrollBounds returns the container bounds for an authored Scroll: the node
// named by On when it resolves, otherwise the root (whole-screen) bounds.
func scrollBounds(action verifier.Action, tree *hierarchy.Tree) hierarchy.Bounds {
if tree == nil {
return hierarchy.Bounds{}
}
if action.On != "" {
if element := tree.Find(action.On); element != nil {
return element.Bounds
}
}
if tree.Root != nil {
return tree.Root.Bounds
}
return hierarchy.Bounds{}
}
// transitionalRetryAttempts caps how many times we re-fetch hierarchy when a
// tree carries more than one route-level Screen tag (NavHost cross-fade in
// flight). Each retry pauses transitionalRetrySleep before the next fetch.
const (
transitionalRetryAttempts = 4
transitionalRetrySleep = 200 * time.Millisecond
)
// fetchSyncedState fetches hierarchy and screenshot together so the recorded
// pair shows the same UI moment. If the hierarchy looks like a NavHost
// cross-fade (multiple route-level *Screen tags), the function waits briefly
// and re-fetches the pair, up to transitionalRetryAttempts times. This
// handles transitions whose async work begins after the sidecar's settle
// poll has already exited.
//
// The driver's Snapshot RPC captures both reads under a backend-side mutex
// so they describe the same on-device frame; the retry exists for the
// orthogonal case where the frame itself is transitional.
//
// The transitional return reports whether the retry budget was exhausted
// on a still-transitional tree. Callers use it to skip the verifier for
// that step so the previous/current extractor advance does not absorb
// transient state.
func fetchSyncedState(ctx context.Context, options Options, logger *slog.Logger, stepIndex int) (tree *hierarchy.Tree, transitional bool, err error) {
var pngBytes []byte
retryLoop:
for attempt := range transitionalRetryAttempts {
hierarchyJSON, image, snapshotErr := options.Driver.Snapshot(ctx)
if snapshotErr != nil {
err = snapshotErr
tree = nil
} else {
tree, err = hierarchy.Parse(hierarchyJSON)
pngBytes = image.PNG
}
if err != nil || !isTransitionalHierarchy(tree) {
break
}
if attempt == transitionalRetryAttempts-1 {
transitional = true
break
}
timer := time.NewTimer(transitionalRetrySleep)
select {
case <-ctx.Done():
timer.Stop()
break retryLoop
case <-timer.C:
}
}
if len(pngBytes) > 0 {
if writeErr := options.TraceWriter.WriteScreenshot(stepIndex, pngBytes); writeErr != nil {
logger.Warn("screenshot write failed", "step", stepIndex, "err", writeErr)
}
}
return tree, transitional, err
}
// isTransitionalHierarchy returns true when the tree carries more than one
// resource-id ending in "Screen" - the marker of a Compose NavHost mid
// cross-fade where both source and destination route composables are alive.
// Mirrors the sidecar's stabilitySnapshot heuristic so runner-side rejection
// stays consistent with the settle poll.
func isTransitionalHierarchy(tree *hierarchy.Tree) bool {
if tree == nil {
return false
}
screens := 0
for _, element := range tree.Elements {
if strings.HasSuffix(element.ResourceID, "Screen") {
screens++
if screens > 1 {
return true
}
}
}
return false
}
func traceActionFor(action verifier.Action, tree *hierarchy.Tree) *trace.Action {
traceAction := &trace.Action{Kind: string(action.Kind), X: action.X, Y: action.Y}
switch action.Kind {
case verifier.ActionKindTap:
case verifier.ActionKindTap, verifier.ActionKindDoubleTap, verifier.ActionKindLongPress:
traceAction.Selector = action.On
stampSelectorTarget(traceAction, action, tree)
case verifier.ActionKindInputText:
@@ -386,6 +685,15 @@ func traceActionFor(action verifier.Action, tree *hierarchy.Tree) *trace.Action
traceAction.DurationMillis = action.DurationMillis
traceAction.X = 0
traceAction.Y = 0
case verifier.ActionKindScroll:
fromX, fromY, toX, toY := scrollEndpoints(action, tree)
traceAction.FromX = fromX
traceAction.FromY = fromY
traceAction.ToX = toX
traceAction.ToY = toY
traceAction.DurationMillis = action.DurationMillis
traceAction.X = 0
traceAction.Y = 0
case verifier.ActionKindPressKey:
traceAction.Key = action.Key
case verifier.ActionKindWait:
@@ -469,6 +777,8 @@ func nextActionFromV8(ctx context.Context, web driver.WebDriver) (verifier.Actio
switch decoded.Kind {
case "Tap":
return verifier.Action{Kind: verifier.ActionKindTap, X: decoded.X, Y: decoded.Y}, nil
case "DoubleTap":
return verifier.Action{Kind: verifier.ActionKindDoubleTap, X: decoded.X, Y: decoded.Y}, nil
case "InputText":
return verifier.Action{
Kind: verifier.ActionKindInputText,
@@ -493,24 +803,18 @@ func nextActionFromV8(ctx context.Context, web driver.WebDriver) (verifier.Actio
}
}
func captureScreenshot(ctx context.Context, options Options, logger *slog.Logger, stepIndex int, after bool) {
image, err := options.Driver.Screenshot(ctx)
if err != nil {
logger.Warn("screenshot capture failed", "step", stepIndex, "after", after, "err", err)
return
func encodeExtractorChanges(changes map[string]verifier.ExtractorChange) map[string]trace.ExtractorChange {
if len(changes) == 0 {
return nil
}
if len(image.PNG) == 0 {
return
}
var writeErr error
if after {
writeErr = options.TraceWriter.WriteScreenshotAfter(stepIndex, image.PNG)
} else {
writeErr = options.TraceWriter.WriteScreenshot(stepIndex, image.PNG)
}
if writeErr != nil {
logger.Warn("screenshot write failed", "step", stepIndex, "after", after, "err", writeErr)
out := make(map[string]trace.ExtractorChange, len(changes))
for name, change := range changes {
out[name] = trace.ExtractorChange{
Prev: json.RawMessage(change.Prev),
Curr: json.RawMessage(change.Curr),
}
}
return out
}
func encodeResiduals(residuals map[string]ltl.Formula) (map[string]json.RawMessage, error) {
@@ -537,3 +841,29 @@ func isWDADrop(err error) bool {
return strings.Contains(msg, "ConnectException") ||
(strings.Contains(msg, "code = Internal") && strings.Contains(msg, "SocketException"))
}
// isTransientApplyError reports whether an applyAction failure is a transient
// device-side hang (sidecar RPC deadline, momentary unavailability) rather than
// a fatal condition. Such steps are recorded as transitional and the loop
// continues. The run context being cancelled is never transient: it means the
// caller wants to stop.
func isTransientApplyError(runCtx context.Context, err error) bool {
if err == nil || runCtx.Err() != nil {
return false
}
if s, ok := status.FromError(err); ok {
switch s.Code() {
case codes.DeadlineExceeded, codes.Unavailable:
return true
case codes.Internal:
message := s.Message()
if strings.Contains(message, "DEADLINE_EXCEEDED") || strings.Contains(message, "UNAVAILABLE") {
return true
}
}
}
if errors.Is(err, context.DeadlineExceeded) {
return true
}
return false
}
+764 -17
View File
@@ -1,8 +1,10 @@
package runner
import (
"bufio"
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
@@ -13,8 +15,12 @@ import (
"testing"
"time"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"github.com/priyanshujain/sanderling/internal/driver"
mockdriver "github.com/priyanshujain/sanderling/internal/driver/mock"
"github.com/priyanshujain/sanderling/internal/hierarchy"
"github.com/priyanshujain/sanderling/internal/trace"
"github.com/priyanshujain/sanderling/internal/verifier"
)
@@ -118,6 +124,76 @@ func TestRunner_ViolationSurfacesInSummary(t *testing.T) {
}
}
func TestRunner_ViolationSurfacesOnlyOnOnsetStep(t *testing.T) {
// violationSpec uses always(() => false): onset fires on step 1 and the
// residual stays violated forever. The runner must record the violation
// exactly once (at the onset step) in both summary.Violations and trace
// lines, not on every subsequent step.
state := newHarnessWithSpec(t, violationSpec)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
summary, err := Run(ctx, Options{
Duration: 200 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
if summary.Steps < 2 {
t.Fatalf("need at least 2 steps to prove onset-only behavior, got %d", summary.Steps)
}
if len(summary.Violations) != 1 {
t.Fatalf("expected exactly one ViolationRecord (onset only), got %d: %v",
len(summary.Violations), summary.Violations)
}
if summary.Violations[0].StepIndex != 1 {
t.Errorf("onset step: got %d, want 1 (always(()=>false) violates immediately)",
summary.Violations[0].StepIndex)
}
if !slices.Equal(summary.Violations[0].Properties, []string{"balanceNonNegative"}) {
t.Errorf("onset properties: got %v, want [balanceNonNegative]",
summary.Violations[0].Properties)
}
file, err := os.Open(filepath.Join(state.writer.Directory(), "trace.jsonl"))
if err != nil {
t.Fatal(err)
}
defer file.Close()
type traceLine struct {
Step int `json:"step"`
Violations []string `json:"violations"`
}
linesWithViolations := 0
scanner := bufio.NewScanner(file)
scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024)
for scanner.Scan() {
var line traceLine
if err := json.Unmarshal(scanner.Bytes(), &line); err != nil {
t.Fatalf("trace line decode: %v", err)
}
if len(line.Violations) == 0 {
continue
}
linesWithViolations++
if line.Step != 1 {
t.Errorf("step %d unexpectedly emitted violations %v (should be onset-only at step 1)",
line.Step, line.Violations)
}
}
if err := scanner.Err(); err != nil {
t.Fatalf("scan trace: %v", err)
}
if linesWithViolations != 1 {
t.Errorf("expected exactly 1 trace line with violations, got %d", linesWithViolations)
}
}
func TestRunner_ThrowingPredicateIsLoggedNotPanic(t *testing.T) {
const throwingSpec = `
globalThis.properties = {
@@ -294,6 +370,151 @@ func TestApplyAction_V8InputTextAtOriginStillTaps(t *testing.T) {
}
}
func TestApplyAction_DoubleTapDispatchesTwoTapsAtCoordinates(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindDoubleTap, X: 100, Y: 200}
start := time.Now()
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("apply action: %v", err)
}
elapsed := time.Since(start)
if elapsed < 40*time.Millisecond {
t.Errorf("expected >= 40ms gap between taps, elapsed %v", elapsed)
}
taps := 0
for _, a := range driverMock.Actions() {
if a.Kind == mockdriver.ActionTap && a.X == 100 && a.Y == 200 {
taps++
}
}
if taps != 2 {
t.Errorf("expected 2 Tap calls at (100,200), got %d in %v", taps, driverMock.Actions())
}
}
func TestApplyAction_DoubleTapDispatchesTwoSelectorTaps(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindDoubleTap, On: "id:save"}
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("apply action: %v", err)
}
taps := 0
for _, a := range driverMock.Actions() {
if a.Kind == mockdriver.ActionTapSelector && a.Selector == "id:save" {
taps++
}
}
if taps != 2 {
t.Errorf("expected 2 TapSelector calls with id:save, got %d in %v", taps, driverMock.Actions())
}
}
func TestApplyAction_LongPressDispatchesAtResolvedCoordinates(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindLongPress, X: 120, Y: 240}
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("apply action: %v", err)
}
found := false
for _, a := range driverMock.Actions() {
if a.Kind == mockdriver.ActionLongPress && a.X == 120 && a.Y == 240 {
found = true
}
}
if !found {
t.Errorf("expected LongPress at (120,240), got %v", driverMock.Actions())
}
}
func TestApplyAction_ScrollWithPrecomputedEndpointsSwipes(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{
Kind: verifier.ActionKindScroll,
Direction: "down",
FromX: 100,
FromY: 500,
ToX: 100,
ToY: 300,
DurationMillis: 300,
}
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("apply action: %v", err)
}
found := false
for _, a := range driverMock.Actions() {
if a.Kind == mockdriver.ActionSwipe && a.FromX == 100 && a.FromY == 500 && a.ToX == 100 && a.ToY == 300 {
found = true
}
}
if !found {
t.Errorf("expected Swipe with precomputed endpoints, got %v", driverMock.Actions())
}
}
func TestApplyAction_ScrollDirectionUsesInversion(t *testing.T) {
driverMock := mockdriver.New()
treeJSON := `{"attributes":{"resource-id":"com.fixture:id/list","bounds":"[0,0,400,800]"},"children":[],"enabled":true}`
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatalf("parse tree: %v", err)
}
action := verifier.Action{Kind: verifier.ActionKindScroll, Direction: "down", On: "id:list"}
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
t.Fatalf("apply action: %v", err)
}
var swipe *mockdriver.Action
for i := range driverMock.Actions() {
if driverMock.Actions()[i].Kind == mockdriver.ActionSwipe {
a := driverMock.Actions()[i]
swipe = &a
}
}
if swipe == nil {
t.Fatalf("expected a Swipe, got %v", driverMock.Actions())
}
// "down" reveals lower content by dragging the finger up, so toY < fromY.
if swipe.ToY >= swipe.FromY {
t.Errorf("expected toY < fromY for scroll down, got from=%d to=%d", swipe.FromY, swipe.ToY)
}
}
func TestApplyAction_ScrollScreenFallback(t *testing.T) {
driverMock := mockdriver.New()
treeJSON := `{"attributes":{"bounds":"[0,0,400,800]"},"children":[],"enabled":true}`
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatalf("parse tree: %v", err)
}
// On unset: container falls back to whole-screen (root) bounds.
action := verifier.Action{Kind: verifier.ActionKindScroll, Direction: "up"}
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
t.Fatalf("apply action: %v", err)
}
var swipe *mockdriver.Action
for i := range driverMock.Actions() {
if driverMock.Actions()[i].Kind == mockdriver.ActionSwipe {
a := driverMock.Actions()[i]
swipe = &a
}
}
if swipe == nil {
t.Fatalf("expected a Swipe, got %v", driverMock.Actions())
}
if swipe.FromX != 200 || swipe.FromY != 400 {
t.Errorf("expected swipe to start at screen center (200,400), got (%d,%d)", swipe.FromX, swipe.FromY)
}
// "up" reveals upper content by dragging the finger down, so toY > fromY.
if swipe.ToY <= swipe.FromY {
t.Errorf("expected toY > fromY for scroll up, got from=%d to=%d", swipe.FromY, swipe.ToY)
}
}
func TestRunner_ParallelFetchCallsAllDriverMethods(t *testing.T) {
state := newHarness(t)
state.mock.MetricsData = driver.Metrics{CPUPercent: 5.0, HeapBytes: 1024, TotalMemoryBytes: 4096}
@@ -316,19 +537,19 @@ func TestRunner_ParallelFetchCallsAllDriverMethods(t *testing.T) {
}
actions := state.mock.Actions()
var hasHierarchy, hasMetrics, hasLogs bool
var hasSnapshot, hasMetrics, hasLogs bool
for _, a := range actions {
switch a.Kind {
case mockdriver.ActionHierarchy:
hasHierarchy = true
case mockdriver.ActionSnapshot:
hasSnapshot = true
case mockdriver.ActionMetrics:
hasMetrics = true
case mockdriver.ActionRecentLogs:
hasLogs = true
}
}
if !hasHierarchy {
t.Error("expected Hierarchy call in mock actions")
if !hasSnapshot {
t.Error("expected Snapshot call in mock actions")
}
if !hasMetrics {
t.Error("expected Metrics call in mock actions")
@@ -338,7 +559,56 @@ func TestRunner_ParallelFetchCallsAllDriverMethods(t *testing.T) {
}
}
func TestRunner_PipelinedPostScreenshotWritten(t *testing.T) {
// TestRunner_UsesAtomicSnapshot ensures the runner observes a step's UI
// through the paired Snapshot RPC instead of racing two independent
// hierarchy + screenshot reads. The pair must come from one on-device
// frame; a regression to separate calls is what this test catches.
func TestRunner_UsesAtomicSnapshot(t *testing.T) {
state := newHarness(t)
state.mock.ImageData = driver.Image{PNG: []byte("png"), Width: 1, Height: 1}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
summary, err := Run(ctx, Options{
Duration: 100 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
if summary.Steps == 0 {
t.Fatal("expected at least one step")
}
var snapshotCalls, hierarchyCalls, screenshotCalls int
for _, action := range state.mock.Actions() {
switch action.Kind {
case mockdriver.ActionSnapshot:
snapshotCalls++
case mockdriver.ActionHierarchy:
hierarchyCalls++
case mockdriver.ActionScreenshot:
screenshotCalls++
}
}
if snapshotCalls == 0 {
t.Errorf("expected at least one Snapshot call, got %d", snapshotCalls)
}
if hierarchyCalls != 0 {
t.Errorf("expected zero standalone Hierarchy calls (runner must use Snapshot), got %d", hierarchyCalls)
}
if screenshotCalls != 0 {
t.Errorf("expected zero standalone Screenshot calls (runner must use Snapshot), got %d", screenshotCalls)
}
}
// TestRunner_OneScreenshotPerStep verifies the runner writes a single
// screenshot per step, captured concurrently with hierarchy so the two
// observations describe the same UI moment.
func TestRunner_OneScreenshotPerStep(t *testing.T) {
state := newHarness(t)
state.mock.ImageData = driver.Image{PNG: []byte("fakepng"), Width: 100, Height: 200}
@@ -355,24 +625,362 @@ func TestRunner_PipelinedPostScreenshotWritten(t *testing.T) {
t.Fatalf("Run: %v", err)
}
if summary.Steps < 2 {
t.Fatalf("need at least 2 steps for pipelining test, got %d", summary.Steps)
t.Fatalf("need at least 2 steps for screenshot test, got %d", summary.Steps)
}
screenshotDir := filepath.Join(state.writer.Directory(), "screenshots")
preFile := filepath.Join(screenshotDir, "step-00001.png")
if _, err := os.Stat(preFile); os.IsNotExist(err) {
t.Errorf("expected pre-screenshot for step 1: %s", preFile)
for step := 1; step <= summary.Steps; step++ {
path := filepath.Join(screenshotDir, fmt.Sprintf("step-%05d.png", step))
if _, err := os.Stat(path); os.IsNotExist(err) {
t.Errorf("expected screenshot for step %d at %s", step, path)
}
}
postFile := filepath.Join(screenshotDir, "step-00001-after.png")
if _, err := os.Stat(postFile); os.IsNotExist(err) {
t.Errorf("expected pipelined post-screenshot for step 1: %s", postFile)
entries, err := os.ReadDir(screenshotDir)
if err != nil {
t.Fatal(err)
}
for _, entry := range entries {
if strings.Contains(entry.Name(), "-after") {
t.Errorf("unexpected -after screenshot remains: %s", entry.Name())
}
}
}
// TestIsTransitionalHierarchy_DetectsMultipleScreens covers the runner-side
// guard that re-fetches when the hierarchy still carries two route-level
// *Screen ids - the NavHost cross-fade signature.
func TestIsTransitionalHierarchy_DetectsMultipleScreens(t *testing.T) {
multi, err := hierarchy.Parse(`{"attributes":{"resource-id":"root"},"children":[
{"attributes":{"resource-id":"AddAccountScreen"},"children":[]},
{"attributes":{"resource-id":"HomeScreen"},"children":[]}
]}`)
if err != nil {
t.Fatal(err)
}
if !isTransitionalHierarchy(multi) {
t.Error("expected multi-screen tree to be flagged as transitional")
}
lastAfter := filepath.Join(screenshotDir, fmt.Sprintf("step-%05d-after.png", summary.Steps))
if _, err := os.Stat(lastAfter); os.IsNotExist(err) {
t.Errorf("expected flushed post-screenshot for last step %d: %s", summary.Steps, lastAfter)
single, err := hierarchy.Parse(`{"attributes":{"resource-id":"HomeScreen"},"children":[]}`)
if err != nil {
t.Fatal(err)
}
if isTransitionalHierarchy(single) {
t.Error("single-screen tree must not be flagged as transitional")
}
if isTransitionalHierarchy(nil) {
t.Error("nil tree must not be flagged as transitional")
}
}
// TestRunner_TransitionalSkipsVerifier feeds a driver whose hierarchy stays
// transitional (multiple route-level *Screen ids) on every Snapshot call.
// Every step must be marked transitional in the trace, no violations may be
// emitted (the verifier never ran), and the summary must stay clean even
// though the spec is a guaranteed always-false predicate.
func TestRunner_TransitionalSkipsVerifier(t *testing.T) {
state := newHarnessWithSpec(t, violationSpec)
state.mock.HierarchyJSON = `{"attributes":{"resource-id":"root"},"children":[
{"attributes":{"resource-id":"AddAccountScreen"},"children":[]},
{"attributes":{"resource-id":"HomeScreen"},"children":[]}
]}`
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
summary, err := Run(ctx, Options{
Duration: 200 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
if summary.Steps == 0 {
t.Fatal("expected at least one step")
}
if len(summary.Violations) != 0 {
t.Fatalf("verifier must be skipped on transitional steps; got %v", summary.Violations)
}
type traceLine struct {
Step int `json:"step"`
Transitional bool `json:"transitional"`
Violations []string `json:"violations"`
}
body, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
if err != nil {
t.Fatal(err)
}
lines := 0
for _, raw := range bytes.Split(bytes.TrimSpace(body), []byte("\n")) {
var line traceLine
if err := json.Unmarshal(raw, &line); err != nil {
t.Fatalf("decode trace line: %v", err)
}
lines++
if !line.Transitional {
t.Errorf("step %d: expected transitional=true on every step, got false", line.Step)
}
if len(line.Violations) != 0 {
t.Errorf("step %d: verifier must be skipped, got violations %v", line.Step, line.Violations)
}
}
if lines == 0 {
t.Fatal("expected trace lines, got none")
}
}
// TestRunner_CleanTreeStillVerified is the control: a single-screen hierarchy
// must not be marked transitional and the verifier must still run, surfacing
// the always-false predicate's violation on the onset step.
func TestRunner_CleanTreeStillVerified(t *testing.T) {
state := newHarnessWithSpec(t, violationSpec)
state.mock.HierarchyJSON = `{"attributes":{"resource-id":"HomeScreen"},"children":[]}`
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
summary, err := Run(ctx, Options{
Duration: 200 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
if !containsProperty(summary.Violations, "balanceNonNegative") {
t.Fatalf("expected verifier to surface balanceNonNegative on a clean tree, got %v", summary.Violations)
}
type traceLine struct {
Step int `json:"step"`
Transitional bool `json:"transitional"`
}
body, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
if err != nil {
t.Fatal(err)
}
for _, raw := range bytes.Split(bytes.TrimSpace(body), []byte("\n")) {
var line traceLine
if err := json.Unmarshal(raw, &line); err != nil {
t.Fatalf("decode trace line: %v", err)
}
if line.Transitional {
t.Errorf("step %d: clean tree must not be marked transitional", line.Step)
}
}
}
// snapshotFailFirst wraps a mock driver so the first Snapshot call returns an
// error (mimicking a sidecar timeout while fetching view hierarchy), then
// delegates every subsequent call back to the mock.
type snapshotFailFirst struct {
*mockdriver.Driver
calls int
}
func (d *snapshotFailFirst) Snapshot(ctx context.Context) (string, driver.Image, error) {
d.calls++
if d.calls == 1 {
return "", driver.Image{}, errors.New("Timeout while fetching view hierarchy")
}
return d.Driver.Snapshot(ctx)
}
// TestRunner_NilHierarchyMarksTransitional verifies that when the sidecar's
// hierarchy fetch fails (nil tree), the runner marks the step transitional and
// skips the verifier instead of pushing a nil tree that would crash the spec.
// Subsequent steps with a clean tree still drive the verifier normally.
func TestRunner_NilHierarchyMarksTransitional(t *testing.T) {
state := newHarnessWithSpec(t, violationSpec)
state.mock.HierarchyJSON = `{"attributes":{"resource-id":"HomeScreen"},"children":[]}`
wrapped := &snapshotFailFirst{Driver: state.mock}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
summary, err := Run(ctx, Options{
Duration: 200 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
Driver: wrapped,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
if summary.Steps < 2 {
t.Fatalf("need at least 2 steps to verify the first is skipped and the second runs, got %d", summary.Steps)
}
// violationSpec always() => false fires on the first verifier push. With
// step 1's verifier skipped, onset moves to step 2.
if len(summary.Violations) != 1 {
t.Fatalf("expected exactly one onset record, got %d: %v", len(summary.Violations), summary.Violations)
}
if summary.Violations[0].StepIndex != 2 {
t.Errorf("onset step: got %d, want 2 (step 1 verifier skipped due to nil tree)", summary.Violations[0].StepIndex)
}
type traceLine struct {
Step int `json:"step"`
Transitional bool `json:"transitional"`
Violations []string `json:"violations"`
}
body, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
if err != nil {
t.Fatal(err)
}
var first traceLine
if err := json.Unmarshal(bytes.SplitN(bytes.TrimSpace(body), []byte("\n"), 2)[0], &first); err != nil {
t.Fatalf("decode first trace line: %v", err)
}
if first.Step != 1 {
t.Fatalf("first trace line step: got %d, want 1", first.Step)
}
if !first.Transitional {
t.Error("first step must be marked transitional when the hierarchy fetch failed")
}
if len(first.Violations) != 0 {
t.Errorf("step 1 must skip the verifier; got violations %v", first.Violations)
}
}
// tapSelectorFailFirst wraps a mock driver so the first TapSelector call
// returns a gRPC DeadlineExceeded error (mimicking a sidecar-side RPC hang),
// then delegates every subsequent call back to the mock.
type tapSelectorFailFirst struct {
*mockdriver.Driver
calls int
}
func (d *tapSelectorFailFirst) TapSelector(ctx context.Context, selector string) error {
d.calls++
if d.calls == 1 {
return status.Error(codes.DeadlineExceeded, "boom")
}
return d.Driver.TapSelector(ctx, selector)
}
// TestRunner_TransientApplyErrorMarksTransitional verifies that a transient
// gRPC error from applyAction (e.g. sidecar RPC deadline) does not kill the
// run: the step is marked transitional, the verifier is skipped for it, and
// the loop continues with the next step running cleanly.
func TestRunner_TransientApplyErrorMarksTransitional(t *testing.T) {
state := newHarness(t)
wrapped := &tapSelectorFailFirst{Driver: state.mock}
var logBuf bytes.Buffer
logger := slog.New(slog.NewTextHandler(&logBuf, &slog.HandlerOptions{Level: slog.LevelWarn}))
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
summary, err := Run(ctx, Options{
Duration: 300 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
Driver: wrapped,
Verifier: state.verifier,
TraceWriter: state.writer,
Logger: logger,
})
if err != nil {
t.Fatalf("Run must not return on transient apply error, got %v", err)
}
if summary.Steps < 2 {
t.Fatalf("need at least 2 steps to prove the loop continued past the failed apply, got %d", summary.Steps)
}
if len(summary.Violations) != 0 {
t.Errorf("transient apply error must not surface as a violation, got %v", summary.Violations)
}
if !strings.Contains(logBuf.String(), "transient apply error") {
t.Errorf("expected transient-apply WARN log, got %q", logBuf.String())
}
type traceLine struct {
Step int `json:"step"`
Transitional bool `json:"transitional"`
Violations []string `json:"violations"`
}
body, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
if err != nil {
t.Fatal(err)
}
lines := bytes.Split(bytes.TrimSpace(body), []byte("\n"))
var first, second traceLine
if err := json.Unmarshal(lines[0], &first); err != nil {
t.Fatalf("decode first trace line: %v", err)
}
if err := json.Unmarshal(lines[1], &second); err != nil {
t.Fatalf("decode second trace line: %v", err)
}
if first.Step != 1 || !first.Transitional {
t.Errorf("step 1 must be transitional after transient apply error, got step=%d transitional=%v", first.Step, first.Transitional)
}
if len(first.Violations) != 0 {
t.Errorf("transient apply step must have no violations, got %v", first.Violations)
}
if second.Step != 2 || second.Transitional {
t.Errorf("step 2 must run cleanly after the transient step, got step=%d transitional=%v", second.Step, second.Transitional)
}
}
// TestIsTransientApplyError_Classification covers the helper's matching rules
// directly so future code changes don't quietly drop a transient case.
func TestIsTransientApplyError_Classification(t *testing.T) {
cleanCtx := context.Background()
cancelledCtx, cancel := context.WithCancel(context.Background())
cancel()
cases := []struct {
name string
ctx context.Context
err error
want bool
}{
{"nil error", cleanCtx, nil, false},
{"deadline exceeded", cleanCtx, status.Error(codes.DeadlineExceeded, "boom"), true},
{"unavailable", cleanCtx, status.Error(codes.Unavailable, "boom"), true},
{"internal wrapping deadline", cleanCtx, status.Error(codes.Internal, "io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ..."), true},
{"internal wrapping unavailable", cleanCtx, status.Error(codes.Internal, "io.grpc.StatusRuntimeException: UNAVAILABLE: ..."), true},
{"internal generic", cleanCtx, status.Error(codes.Internal, "boom"), false},
{"raw context deadline", cleanCtx, context.DeadlineExceeded, true},
{"run context cancelled overrides", cancelledCtx, status.Error(codes.DeadlineExceeded, "boom"), false},
}
for _, testCase := range cases {
t.Run(testCase.name, func(t *testing.T) {
if got := isTransientApplyError(testCase.ctx, testCase.err); got != testCase.want {
t.Errorf("got %v, want %v", got, testCase.want)
}
})
}
}
// TestRunner_WaitActionSkipsIdle ensures the runner does not call WaitForIdle
// after a Wait action - the action already provides settling time.
func TestRunner_WaitActionSkipsIdle(t *testing.T) {
const waitSpec = `
globalThis.actions = __sanderling__.actions(() => [__sanderling__.wait({ durationMillis: 5 })]);
`
state := newHarnessWithSpec(t, waitSpec)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_, err := Run(ctx, Options{
Duration: 150 * time.Millisecond,
IdleTimeout: 50 * time.Millisecond,
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
for _, action := range state.mock.Actions() {
if action.Kind == mockdriver.ActionWaitForIdle {
t.Fatalf("Wait action must skip WaitForIdle, got: %v", action)
}
}
}
@@ -426,3 +1034,142 @@ func containsProperty(records []ViolationRecord, property string) bool {
}
return false
}
func TestRunner_RelaunchesWhenAppLeavesForeground(t *testing.T) {
state := newHarness(t)
// Always report a foreign app, so every step's guard must relaunch.
state.mock.ForegroundResults = []string{"com.android.chrome"}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_, err := Run(ctx, Options{
Duration: 100 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
BundleID: "app.folio",
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
relaunches := 0
for _, a := range state.mock.Actions() {
if a.Kind == mockdriver.ActionLaunch && a.BundleID == "app.folio" && !a.ClearState {
relaunches++
}
}
if relaunches == 0 {
t.Fatal("expected runner to relaunch app.folio when foreground escaped, got none")
}
}
func TestRunner_NoRelaunchWhenAppInForeground(t *testing.T) {
state := newHarness(t)
state.mock.ForegroundResults = []string{"app.folio"}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_, err := Run(ctx, Options{
Duration: 100 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
BundleID: "app.folio",
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
for _, a := range state.mock.Actions() {
if a.Kind == mockdriver.ActionLaunch {
t.Fatalf("expected no relaunch while app in foreground, got %v", a)
}
}
}
// TestRunner_WaitsForForegroundBeforeFirstAction verifies the startup gate
// brings the app forward (back-press + relaunch) before any tap fires when the
// device boots showing a system dialog.
func TestRunner_WaitsForForegroundBeforeFirstAction(t *testing.T) {
state := newHarness(t)
// First the device shows a system setup screen, then the app is on top.
state.mock.ForegroundResults = []string{"com.google.android.setupwizard", "app.folio"}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_, err := Run(ctx, Options{
Duration: 100 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
BundleID: "app.folio",
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
actions := state.mock.Actions()
firstLaunch, firstTap := -1, -1
backPressed := false
for i, a := range actions {
switch {
case a.Kind == mockdriver.ActionLaunch && firstLaunch < 0:
firstLaunch = i
case a.Kind == mockdriver.ActionTap && firstTap < 0:
firstTap = i
case a.Kind == mockdriver.ActionPressKey && a.Key == "back":
backPressed = true
}
}
if firstLaunch < 0 {
t.Fatal("expected a relaunch to bring the app forward, got none")
}
if !backPressed {
t.Fatal("expected a back-press to dismiss the system dialog, got none")
}
if firstTap >= 0 && firstLaunch > firstTap {
t.Fatalf("expected the foreground gate (launch at %d) before the first tap (at %d)", firstLaunch, firstTap)
}
}
// TestRunner_WaitsForWindowDrawnBeforeFirstAction verifies the startup gate
// keeps waiting while the app is the resumed activity but its window has not
// drawn yet (a leftover screen still focused). It must poll the focused-window
// signal rather than relaunching, and only proceed once the window names the
// app.
func TestRunner_WaitsForWindowDrawnBeforeFirstAction(t *testing.T) {
state := newHarness(t)
// The app is resumed immediately, but its window lags: the outgoing
// settings screen stays focused for two checks before the app draws.
state.mock.ForegroundResults = []string{"app.folio"}
state.mock.FocusedWindowResults = []string{"com.android.settings", "com.android.settings", "app.folio"}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_, err := Run(ctx, Options{
Duration: 100 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
BundleID: "app.folio",
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
// The gate must have polled the focused window until it named the app,
// i.e. at least the three queued results were consumed.
if calls := state.mock.FocusedWindowCalls(); calls < 3 {
t.Fatalf("expected the gate to poll the focused window until drawn (>=3 calls), got %d", calls)
}
// The resumed app was never a foreign app, so the gate must not relaunch
// or back-press to "fix" a window that simply had not drawn yet.
for _, a := range state.mock.Actions() {
if a.Kind == mockdriver.ActionPressKey && a.Key == "back" {
t.Fatal("expected no back-press while waiting for the window to draw")
}
}
}
+1
View File
@@ -63,6 +63,7 @@ func buildDriver(ctx context.Context, options Options, stdout io.Writer) (driver
_ = sidecarCommand.Process.Kill()
return nil, nil, fmt.Errorf("dial sidecar: %w", err)
}
driverClient.SetPlatform(options.Platform)
// WaitForHealth confirms the gRPC sidecar is up. For iOS, the WDA warmup
// (absorbing the XCUITest startup race) runs inside IosDriverBackend.init
// in the sidecar - no additional sleep needed here.
+5 -1
View File
@@ -50,6 +50,7 @@ func Execute(ctx context.Context, options Options, stdout io.Writer) error {
if specAPIPath != "" {
aliases["@sanderling/spec"] = specAPIPath
base := filepath.Dir(specAPIPath)
aliases["@sanderling/spec/defaults"] = filepath.Join(base, "defaults/index.ts")
aliases["@sanderling/spec/defaults/properties"] = filepath.Join(base, "defaults/properties.ts")
}
defines := map[string]string{
@@ -104,7 +105,10 @@ func Execute(ctx context.Context, options Options, stdout io.Writer) error {
if seed == 0 {
seed = time.Now().UnixNano()
}
verifierInstance, err := verifier.New(verifier.WithRand(rand.New(rand.NewPCG(uint64(seed), 0))))
verifierInstance, err := verifier.New(
verifier.WithRand(rand.New(rand.NewPCG(uint64(seed), 0))),
verifier.WithAppPackage(options.BundleID),
)
if err != nil {
return fmt.Errorf("verifier: %w", err)
}
+26 -17
View File
@@ -13,16 +13,32 @@ import (
)
type Step struct {
Index int `json:"step"`
Timestamp time.Time `json:"timestamp"`
Screen string `json:"screen,omitempty"`
Snapshots map[string]json.RawMessage `json:"snapshots,omitempty"`
Action *Action `json:"action,omitempty"`
Exceptions []Exception `json:"exceptions,omitempty"`
Violations []string `json:"violations,omitempty"`
Hierarchy *hierarchy.Tree `json:"hierarchy,omitempty"`
Residuals map[string]json.RawMessage `json:"residuals,omitempty"`
Metrics *Metrics `json:"metrics,omitempty"`
Index int `json:"step"`
Timestamp time.Time `json:"timestamp"`
Screen string `json:"screen,omitempty"`
Snapshots map[string]json.RawMessage `json:"snapshots,omitempty"`
// NextAction is the action chosen for the next iteration based on observing this step.
NextAction *Action `json:"next_action,omitempty"`
Exceptions []Exception `json:"exceptions,omitempty"`
Violations []string `json:"violations,omitempty"`
Hierarchy *hierarchy.Tree `json:"hierarchy,omitempty"`
Residuals map[string]json.RawMessage `json:"residuals,omitempty"`
Metrics *Metrics `json:"metrics,omitempty"`
ExtractorChanges map[string]ExtractorChange `json:"extractor_changes,omitempty"`
// Transitional marks a step whose hierarchy still showed a NavHost
// cross-fade (multiple route-level *Screen ids) after the runner's
// retry budget. The verifier is skipped for these steps so transient
// state does not poison the previous/current extractor advance.
Transitional bool `json:"transitional,omitempty"`
}
// ExtractorChange records the prev/curr JSON values of an extractor whose
// observation differed between two consecutive steps. Surfaced under
// violation rows in the inspect UI as a "what changed at this step"
// breadcrumb.
type ExtractorChange struct {
Prev json.RawMessage `json:"prev"`
Curr json.RawMessage `json:"curr"`
}
type Metrics struct {
@@ -125,13 +141,6 @@ func (w *Writer) WriteScreenshot(stepIndex int, png []byte) error {
return w.writePNG(fmt.Sprintf("step-%05d.png", stepIndex), png)
}
// WriteScreenshotAfter writes the post-action screenshot for a step.
// Callers use this after applyAction + waitForIdle so the UI can show a
// before/after pair.
func (w *Writer) WriteScreenshotAfter(stepIndex int, png []byte) error {
return w.writePNG(fmt.Sprintf("step-%05d-after.png", stepIndex), png)
}
func (w *Writer) writePNG(name string, png []byte) error {
if len(png) == 0 {
return nil
+9 -9
View File
@@ -102,7 +102,7 @@ func TestWriteStep_HierarchyAndResidualsRoundTrip(t *testing.T) {
step := Step{
Index: 1,
Timestamp: time.Now().UTC(),
Action: &Action{
NextAction: &Action{
Kind: "tap",
Selector: "id:next",
ResolvedBounds: &BoundsRecord{X: 10, Y: 20, Width: 100, Height: 50},
@@ -120,14 +120,14 @@ func TestWriteStep_HierarchyAndResidualsRoundTrip(t *testing.T) {
if err := json.Unmarshal(body, &got); err != nil {
t.Fatalf("bad jsonl: %v\n%s", err, body)
}
if got.Action.Selector != "id:next" {
t.Errorf("selector = %q", got.Action.Selector)
if got.NextAction.Selector != "id:next" {
t.Errorf("selector = %q", got.NextAction.Selector)
}
if got.Action.ResolvedBounds == nil || got.Action.ResolvedBounds.Width != 100 {
t.Errorf("resolved_bounds round-trip wrong: %+v", got.Action.ResolvedBounds)
if got.NextAction.ResolvedBounds == nil || got.NextAction.ResolvedBounds.Width != 100 {
t.Errorf("resolved_bounds round-trip wrong: %+v", got.NextAction.ResolvedBounds)
}
if got.Action.TapPoint == nil || got.Action.TapPoint.X != 60 {
t.Errorf("tap_point round-trip wrong: %+v", got.Action.TapPoint)
if got.NextAction.TapPoint == nil || got.NextAction.TapPoint.X != 60 {
t.Errorf("tap_point round-trip wrong: %+v", got.NextAction.TapPoint)
}
if string(got.Residuals["prop1"]) != `{"op":"true"}` {
t.Errorf("residuals round-trip wrong: %s", got.Residuals["prop1"])
@@ -162,7 +162,7 @@ func TestWriteStep_AppendsOneJsonLine(t *testing.T) {
Snapshots: map[string]json.RawMessage{
"ledger.balance": json.RawMessage(`1500`),
},
Action: &Action{Kind: "tap", X: 100, Y: 200},
NextAction: &Action{Kind: "tap", X: 100, Y: 200},
Violations: []string{"ledgerBalanceMatchesTxns"},
}
if err := writer.WriteStep(step); err != nil {
@@ -177,7 +177,7 @@ func TestWriteStep_AppendsOneJsonLine(t *testing.T) {
if err := json.Unmarshal([]byte(lines[0]), &got); err != nil {
t.Fatalf("invalid JSONL line: %v\n%s", err, lines[0])
}
if got.Index != 1 || got.Screen != "customer_ledger" || got.Action.X != 100 || got.Violations[0] != "ledgerBalanceMatchesTxns" {
if got.Index != 1 || got.Screen != "customer_ledger" || got.NextAction.X != 100 || got.Violations[0] != "ledgerBalanceMatchesTxns" {
t.Errorf("step round-trip wrong: %+v", got)
}
}
+84 -9
View File
@@ -10,6 +10,12 @@ import (
type extractorState struct {
getter goja.Callable
handle *goja.Object
name string
// prev/curr cache the JSON-encoded extractor values from the prior and
// current PushSnapshot, used by ChangedExtractors to surface per-step
// diffs in the trace.
prev []byte
curr []byte
}
type formulaState struct {
@@ -60,12 +66,16 @@ const (
tagInternalKind = "__sanderlingKind"
tagSelector = "__sanderlingSelector"
internalKindActions = "actions"
internalKindWeighted = "weighted"
internalKindBuiltinTaps = "taps"
internalKindBuiltinSwipes = "swipes"
internalKindBuiltinWaitOnce = "waitOnce"
internalKindBuiltinPressKey = "pressKey"
internalKindActions = "actions"
internalKindWeighted = "weighted"
internalKindBuiltinTaps = "taps"
internalKindBuiltinDoubleTaps = "doubleTaps"
internalKindBuiltinTyping = "typing"
internalKindBuiltinSwipes = "swipes"
internalKindBuiltinWaitOnce = "waitOnce"
internalKindBuiltinPressKey = "pressKey"
internalKindBuiltinLongPresses = "longPresses"
internalKindBuiltinScrolls = "scrolls"
)
// installRuntimeBindings exposes globalThis.__sanderling__ to the loaded spec.
@@ -99,6 +109,15 @@ func (v *Verifier) installRuntimeBindings() error {
if err := sanderling.Set("tap", v.bindTap); err != nil {
return err
}
if err := sanderling.Set("doubleTap", v.bindDoubleTap); err != nil {
return err
}
if err := sanderling.Set("longPress", v.bindLongPress); err != nil {
return err
}
if err := sanderling.Set("scroll", v.bindScroll); err != nil {
return err
}
if err := sanderling.Set("inputText", v.bindInputText); err != nil {
return err
}
@@ -114,6 +133,12 @@ func (v *Verifier) installRuntimeBindings() error {
if err := sanderling.Set("taps", v.builtinGenerator(internalKindBuiltinTaps)); err != nil {
return err
}
if err := sanderling.Set("doubleTaps", v.builtinGenerator(internalKindBuiltinDoubleTaps)); err != nil {
return err
}
if err := sanderling.Set("typing", v.builtinGenerator(internalKindBuiltinTyping)); err != nil {
return err
}
if err := sanderling.Set("swipes", v.builtinGenerator(internalKindBuiltinSwipes)); err != nil {
return err
}
@@ -123,24 +148,40 @@ func (v *Verifier) installRuntimeBindings() error {
if err := sanderling.Set("pressKeys", v.builtinGenerator(internalKindBuiltinPressKey)); err != nil {
return err
}
if err := sanderling.Set("longPresses", v.builtinGenerator(internalKindBuiltinLongPresses)); err != nil {
return err
}
if err := sanderling.Set("scrolls", v.builtinGenerator(internalKindBuiltinScrolls)); err != nil {
return err
}
return v.runtime.GlobalObject().Set("__sanderling__", sanderling)
}
func (v *Verifier) bindExtract(call goja.FunctionCall) goja.Value {
if len(call.Arguments) != 1 {
panic(v.runtime.NewTypeError("extract requires exactly one argument"))
if len(call.Arguments) < 1 || len(call.Arguments) > 2 {
panic(v.runtime.NewTypeError("extract requires (getter) or (getter, name)"))
}
getter, ok := goja.AssertFunction(call.Arguments[0])
if !ok {
panic(v.runtime.NewTypeError("extract argument must be a function"))
}
name := ""
if len(call.Arguments) == 2 {
arg := call.Arguments[1]
if !goja.IsUndefined(arg) && !goja.IsNull(arg) {
name = arg.String()
}
}
if name == "" {
name = fmt.Sprintf("extractor_%d", len(v.extractors))
}
handle := v.runtime.NewObject()
_ = handle.Set("current", goja.Undefined())
_ = handle.Set("previous", goja.Undefined())
v.extractors = append(v.extractors, &extractorState{getter: getter, handle: handle})
v.extractors = append(v.extractors, &extractorState{getter: getter, handle: handle, name: name})
return handle
}
@@ -387,6 +428,40 @@ func (v *Verifier) bindTap(call goja.FunctionCall) goja.Value {
return handle
}
func (v *Verifier) bindDoubleTap(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("DoubleTap requires {on}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "DoubleTap")
_ = handle.Set("on", parameters.Get("on"))
return handle
}
func (v *Verifier) bindLongPress(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("LongPress requires {on}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "LongPress")
_ = handle.Set("on", parameters.Get("on"))
return handle
}
func (v *Verifier) bindScroll(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("Scroll requires {direction}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "Scroll")
_ = handle.Set("direction", parameters.Get("direction"))
_ = handle.Set("in", parameters.Get("in"))
return handle
}
func (v *Verifier) bindInputText(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
+85 -8
View File
@@ -3,6 +3,7 @@ package verifier
import (
"encoding/json"
"fmt"
"strings"
"time"
"github.com/dop251/goja"
@@ -69,7 +70,7 @@ func accessibilityObject(runtime *goja.Runtime, tree *hierarchy.Tree) *goja.Obje
if node == nil {
return goja.Undefined()
}
return nodeObject(runtime, node, selectorStringFromJS(call.Argument(0)))
return nodeObject(runtime, node, selectorStringFromJS(runtime, call.Argument(0)))
}
findAll := func(call goja.FunctionCall) goja.Value {
if tree == nil {
@@ -78,7 +79,7 @@ func accessibilityObject(runtime *goja.Runtime, tree *hierarchy.Tree) *goja.Obje
nodes := findAllNodesFromJS(runtime, tree, call.Argument(0))
array := runtime.NewArray()
for i, n := range nodes {
_ = array.Set(fmt.Sprintf("%d", i), nodeObject(runtime, n, selectorStringFromJS(call.Argument(0))))
_ = array.Set(fmt.Sprintf("%d", i), nodeObject(runtime, n, selectorStringFromJS(runtime, call.Argument(0))))
}
return array
}
@@ -100,6 +101,7 @@ func nodeObject(runtime *goja.Runtime, node *hierarchy.Node, selector string) go
_ = object.Set("checked", element.Checked)
_ = object.Set("focused", element.Focused)
_ = object.Set("selected", element.Selected)
_ = object.Set("editable", element.Editable)
_ = object.Set("x", centerX)
_ = object.Set("y", centerY)
_ = object.Set(tagSelector, selector)
@@ -120,14 +122,14 @@ func nodeObject(runtime *goja.Runtime, node *hierarchy.Node, selector string) go
if childNode == nil {
return goja.Undefined()
}
return nodeObject(runtime, childNode, selectorStringFromJS(arg))
return nodeObject(runtime, childNode, selectorStringFromJS(runtime, arg))
}
childFindAll := func(call goja.FunctionCall) goja.Value {
arg := call.Argument(0)
childNodes := findAllNodesInSubtreeFromJS(runtime, node, arg)
array := runtime.NewArray()
for i, n := range childNodes {
_ = array.Set(fmt.Sprintf("%d", i), nodeObject(runtime, n, selectorStringFromJS(arg)))
_ = array.Set(fmt.Sprintf("%d", i), nodeObject(runtime, n, selectorStringFromJS(runtime, arg)))
}
return array
}
@@ -265,15 +267,62 @@ func selectorPathFromJS(runtime *goja.Runtime, arg goja.Value) ([]hierarchy.Sele
// selectorStringFromJS returns a string representation of the selector argument
// for tagging returned element objects (used by selectorOf to reconstruct the
// selector when the element is passed back as an action target).
func selectorStringFromJS(arg goja.Value) string {
if goja.IsUndefined(arg) || goja.IsNull(arg) {
// selector when the element is passed back as an action target). Output
// follows the canonical hierarchy selector grammar: "k:v" pairs space-joined
// per object, chains joined by " > ".
func selectorStringFromJS(runtime *goja.Runtime, arg goja.Value) string {
if arg == nil || goja.IsUndefined(arg) || goja.IsNull(arg) {
return ""
}
if s, ok := arg.Export().(string); ok {
return s
}
return arg.String()
exported := arg.Export()
if slice, ok := exported.([]any); ok {
object := arg.ToObject(runtime)
if object == nil {
return ""
}
parts := make([]string, 0, len(slice))
for index := range slice {
entry := object.Get(fmt.Sprintf("%d", index))
if entry == nil || goja.IsUndefined(entry) || goja.IsNull(entry) {
continue
}
segment := selectorObjectToString(runtime, entry)
if segment == "" {
continue
}
parts = append(parts, segment)
}
return strings.Join(parts, " > ")
}
return selectorObjectToString(runtime, arg)
}
// selectorObjectToString formats a single JS object as a space-joined sequence
// of "k:v" pairs, mirroring the hierarchy package's predicate grammar.
func selectorObjectToString(runtime *goja.Runtime, arg goja.Value) string {
if arg == nil || goja.IsUndefined(arg) || goja.IsNull(arg) {
return ""
}
object := arg.ToObject(runtime)
if object == nil {
return ""
}
keys := object.Keys()
parts := make([]string, 0, len(keys))
for _, key := range keys {
if key == tagSelector {
continue
}
value := object.Get(key)
if value == nil || goja.IsUndefined(value) || goja.IsNull(value) {
continue
}
parts = append(parts, fmt.Sprintf("%s:%s", key, value.String()))
}
return strings.Join(parts, " ")
}
func lastActionObject(runtime *goja.Runtime, action *Action) goja.Value {
@@ -301,6 +350,16 @@ func lastActionObject(runtime *goja.Runtime, action *Action) goja.Value {
if action.DurationMillis > 0 {
_ = object.Set("durationMillis", action.DurationMillis)
}
case ActionKindScroll:
_ = object.Set("direction", action.Direction)
from := runtime.NewObject()
_ = from.Set("x", action.FromX)
_ = from.Set("y", action.FromY)
to := runtime.NewObject()
_ = to.Set("x", action.ToX)
_ = to.Set("y", action.ToY)
_ = object.Set("from", from)
_ = object.Set("to", to)
case ActionKindPressKey:
_ = object.Set("key", action.Key)
case ActionKindWait:
@@ -371,6 +430,10 @@ func jsValueToAction(runtime *goja.Runtime, value goja.Value) (Action, error) {
on := object.Get("on")
x, y := coordinatesOf(runtime, on)
return Action{Kind: ActionKindTap, On: selectorOf(runtime, on), X: x, Y: y}, nil
case "DoubleTap":
on := object.Get("on")
x, y := coordinatesOf(runtime, on)
return Action{Kind: ActionKindDoubleTap, On: selectorOf(runtime, on), X: x, Y: y}, nil
case "InputText":
into := object.Get("into")
text := object.Get("text")
@@ -395,6 +458,20 @@ func jsValueToAction(runtime *goja.Runtime, value goja.Value) (Action, error) {
ToY: toY,
DurationMillis: intField(object, "durationMillis"),
}, nil
case "LongPress":
on := object.Get("on")
x, y := coordinatesOf(runtime, on)
return Action{Kind: ActionKindLongPress, On: selectorOf(runtime, on), X: x, Y: y}, nil
case "Scroll":
in := object.Get("in")
x, y := coordinatesOf(runtime, in)
return Action{
Kind: ActionKindScroll,
Direction: stringOf(object.Get("direction")),
On: selectorOf(runtime, in),
X: x,
Y: y,
}, nil
case "PressKey":
return Action{Kind: ActionKindPressKey, Key: stringOf(object.Get("key"))}, nil
case "Wait":
+232
View File
@@ -0,0 +1,232 @@
package verifier
import (
"errors"
"testing"
"github.com/priyanshujain/sanderling/internal/hierarchy"
)
// A clickable app button and a clickable soft-keyboard key sharing the screen.
// The keyboard key sits where a random tap would otherwise insert a glyph.
const scopedTreeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,500]", "package": "com.folio"},
"children": [
{"attributes": {"testTag": "SubmitButton", "bounds": "[0,40,100,80]", "package": "com.folio"}, "clickable": true, "editable": false, "enabled": true, "children": []},
{"attributes": {"testTag": "Emoticon", "bounds": "[0,400,100,440]", "package": "com.google.android.inputmethod.latin"}, "clickable": true, "enabled": true, "children": []}
]
}`
func pushTree(t *testing.T, v *Verifier, treeJSON string) {
t.Helper()
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
if err := v.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
}
// TestTaps_ExcludeOffAppPackage proves the tap generator never targets the soft
// keyboard: with the app package set, only the in-app button is a candidate, so
// the result lands on its center regardless of seed.
func TestTaps_ExcludeOffAppPackage(t *testing.T) {
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.taps;`)
pushTree(t, verifier, scopedTreeJSON)
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.X != 50 || action.Y != 60 {
t.Errorf("coords = (%d,%d), want (50,60) at SubmitButton; keyboard key leaked into targets", action.X, action.Y)
}
}
// TestTyping_ExcludeOffAppPackage proves keyboard glyph buttons that report as
// editable never become typing targets once the app package is set.
func TestTyping_ExcludeOffAppPackage(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,500]", "package": "com.folio"},
"children": [
{"attributes": {"testTag": "NameField", "bounds": "[0,0,100,40]", "package": "com.folio"}, "editable": true, "enabled": true, "children": []},
{"attributes": {"testTag": "SearchBox", "bounds": "[0,400,100,440]", "package": "com.google.android.inputmethod.latin"}, "editable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.typing;`)
pushTree(t, verifier, treeJSON)
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.X != 50 || action.Y != 20 {
t.Errorf("coords = (%d,%d), want (50,20) at NameField; off-app field leaked into targets", action.X, action.Y)
}
}
// TestSwipes_ExcludeOffAppPackage proves swipes anchor on app nodes only, so
// exploration never scrolls the keyboard's emoji list instead of the app.
func TestSwipes_ExcludeOffAppPackage(t *testing.T) {
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.swipes;`)
pushTree(t, verifier, scopedTreeJSON)
// Both the root and SubmitButton (com.folio) are valid anchors; only the
// keyboard key at center (50,420) must be excluded. Draw many times so the
// invariant is not satisfied by a lucky seed.
for i := range 200 {
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindSwipe {
t.Fatalf("kind = %v, want Swipe", action.Kind)
}
if action.FromX == 50 && action.FromY == 420 {
t.Fatalf("draw %d anchored on the keyboard key (50,420); off-app node leaked into swipe targets", i)
}
}
}
// TestTaps_AllOffAppYieldsErrNoAction proves the generator declines when every
// clickable node belongs to another package, so a weighted layer falls through
// instead of fuzzing the keyboard.
func TestTaps_AllOffAppYieldsErrNoAction(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,500]", "package": "com.folio"},
"children": [
{"attributes": {"testTag": "Emoticon", "bounds": "[0,400,100,440]", "package": "com.google.android.inputmethod.latin"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.taps;`)
pushTree(t, verifier, treeJSON)
if _, err := verifier.NextAction(); !errors.Is(err, ErrNoAction) {
t.Fatalf("err = %v, want ErrNoAction", err)
}
}
// TestTaps_UnsetAppPackageKeepsAllNodes proves the filter is opt-in: with no app
// package configured, an off-app node is still a valid target (prior behavior).
func TestTaps_UnsetAppPackageKeepsAllNodes(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,500]", "package": "com.folio"},
"children": [
{"attributes": {"testTag": "Emoticon", "bounds": "[0,400,100,440]", "package": "com.google.android.inputmethod.latin"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `globalThis.actions = __sanderling__.taps;`)
pushTree(t, verifier, treeJSON)
action, err := verifier.NextAction()
if err != nil {
t.Fatalf("err = %v, want a tap on the only node when unscoped", err)
}
if action.X != 50 || action.Y != 420 {
t.Errorf("coords = (%d,%d), want (50,420); unscoped run should target any node", action.X, action.Y)
}
}
// TestLongPresses_TargetsClickableElement proves the longPresses generator
// mirrors taps: it yields a LongPress on the only clickable in-app node.
func TestLongPresses_TargetsClickableElement(t *testing.T) {
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.longPresses;`)
pushTree(t, verifier, scopedTreeJSON)
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindLongPress {
t.Fatalf("kind = %v, want LongPress", action.Kind)
}
if action.X != 50 || action.Y != 60 {
t.Errorf("coords = (%d,%d), want (50,60) at SubmitButton", action.X, action.Y)
}
}
// TestScrolls_TargetsScrollableContainer proves the scrolls generator anchors on
// a scrollable container and pre-computes swipe endpoints inside its bounds.
func TestScrolls_TargetsScrollableContainer(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,500]", "package": "com.folio"},
"children": [
{"attributes": {"testTag": "Feed", "bounds": "[0,0,100,400]", "scrollable": "true", "package": "com.folio"}, "enabled": true, "children": []},
{"attributes": {"testTag": "Header", "bounds": "[0,400,100,440]", "package": "com.folio"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.scrolls;`)
pushTree(t, verifier, treeJSON)
for range 50 {
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindScroll {
t.Fatalf("kind = %v, want Scroll", action.Kind)
}
// Feed center is (50,200); both endpoints must stay anchored there.
if action.FromX != 50 || action.FromY != 200 {
t.Fatalf("from = (%d,%d), want (50,200) at Feed center", action.FromX, action.FromY)
}
switch action.Direction {
case "up", "down":
if action.ToX != 50 {
t.Fatalf("vertical scroll moved x: to = (%d,%d)", action.ToX, action.ToY)
}
case "left", "right":
if action.ToY != 200 {
t.Fatalf("horizontal scroll moved y: to = (%d,%d)", action.ToX, action.ToY)
}
default:
t.Fatalf("unexpected direction %q", action.Direction)
}
if action.DurationMillis != 300 {
t.Fatalf("durationMillis = %d, want 300", action.DurationMillis)
}
}
}
// TestScrolls_NoScrollableYieldsErrNoAction proves the generator declines when
// no scrollable container is present.
func TestScrolls_NoScrollableYieldsErrNoAction(t *testing.T) {
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.scrolls;`)
pushTree(t, verifier, scopedTreeJSON)
if _, err := verifier.NextAction(); !errors.Is(err, ErrNoAction) {
t.Fatalf("err = %v, want ErrNoAction", err)
}
}
// TestTaps_EmptyPackageNodeStaysInScope proves nodes that omit the package
// attribute (e.g. iOS, decor views) are kept, so the filter never empties a
// legitimate app screen.
func TestTaps_EmptyPackageNodeStaysInScope(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,500]"},
"children": [
{"attributes": {"testTag": "SubmitButton", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.taps;`)
pushTree(t, verifier, treeJSON)
action, err := verifier.NextAction()
if err != nil {
t.Fatalf("err = %v, want the empty-package node kept in scope", err)
}
if action.X != 50 || action.Y != 60 {
t.Errorf("coords = (%d,%d), want (50,60)", action.X, action.Y)
}
}
+14
View File
@@ -5,10 +5,13 @@ type ActionKind string
const (
ActionKindTap ActionKind = "Tap"
ActionKindDoubleTap ActionKind = "DoubleTap"
ActionKindInputText ActionKind = "InputText"
ActionKindSwipe ActionKind = "Swipe"
ActionKindPressKey ActionKind = "PressKey"
ActionKindWait ActionKind = "Wait"
ActionKindLongPress ActionKind = "LongPress"
ActionKindScroll ActionKind = "Scroll"
)
// Action is a single UI interaction produced by the spec's action generator.
@@ -27,6 +30,9 @@ type Action struct {
DurationMillis int
// Key is the logical key name for ActionKindPressKey.
Key string
// Direction is the scroll direction for ActionKindScroll: one of "up",
// "down", "left", "right". Empty for every other kind.
Direction string
}
// LogEntry mirrors a logcat line captured between steps.
@@ -44,3 +50,11 @@ type Exception struct {
StackTrace string
UnixMillis int64
}
// ExtractorChange records a single extractor's value transition across one
// step. Used to surface "what changed at this step" breadcrumbs at violation
// markers in the inspect UI.
type ExtractorChange struct {
Prev []byte
Curr []byte
}
+384
View File
@@ -4,6 +4,7 @@ import (
"encoding/json"
"errors"
"math/rand/v2"
"slices"
"strings"
"testing"
@@ -118,6 +119,117 @@ func TestEvaluateProperties_HoldsThenViolates(t *testing.T) {
}
}
func TestNewlyViolatedProperties_OnsetOnly(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
// Balance trajectory: holds, holds, violates, stays violated, stays violated.
// Onset must appear only on step 3 even though the residual stays false
// through steps 4 and 5 (LTL `always` sticky semantics).
balances := []int{1500, 1500, -1, 500, 500}
for index, balance := range balances {
raw, _ := json.Marshal(balance)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": raw}}); err != nil {
t.Fatal(err)
}
_ = verifier.EvaluateProperties()
got := verifier.NewlyViolatedProperties()
step := index + 1
if step == 3 {
want := []string{"balanceNonNegative"}
if !slices.Equal(got, want) {
t.Errorf("step %d (onset): got %v, want %v", step, got, want)
}
} else if len(got) != 0 {
t.Errorf("step %d: expected empty onset set, got %v", step, got)
}
}
}
func TestNewlyViolatedProperties_FirstStepViolation(t *testing.T) {
const spec = `
globalThis.properties = {
alwaysFalse: __sanderling__.always(() => false),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, spec)
for step := 1; step <= 3; step++ {
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}}); err != nil {
t.Fatal(err)
}
_ = verifier.EvaluateProperties()
got := verifier.NewlyViolatedProperties()
if step == 1 {
want := []string{"alwaysFalse"}
if !slices.Equal(got, want) {
t.Errorf("step 1 (onset): got %v, want %v", got, want)
}
} else if len(got) != 0 {
t.Errorf("step %d: expected empty onset set after first-step violation, got %v", step, got)
}
}
}
func TestNewlyViolatedProperties_MultipleProperties(t *testing.T) {
const spec = `
const a = __sanderling__.extract(state => state.snapshots["a"] ?? 0);
const b = __sanderling__.extract(state => state.snapshots["b"] ?? 0);
globalThis.properties = {
propA: __sanderling__.always(() => a.current >= 0),
propB: __sanderling__.always(() => b.current >= 0),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, spec)
// propA violates at step 2, propB violates at step 4. Each must surface
// only on its own onset step.
aValues := []int{1, -1, -1, -1}
bValues := []int{1, 1, 1, -1}
expectOnset := map[int][]string{
2: {"propA"},
4: {"propB"},
}
for index := range aValues {
aRaw, _ := json.Marshal(aValues[index])
bRaw, _ := json.Marshal(bValues[index])
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"a": aRaw, "b": bRaw}}); err != nil {
t.Fatal(err)
}
_ = verifier.EvaluateProperties()
got := verifier.NewlyViolatedProperties()
step := index + 1
want := expectOnset[step]
if !slices.Equal(got, want) {
t.Errorf("step %d: got %v, want %v", step, got, want)
}
}
}
func TestNewlyViolatedProperties_DeterministicOrder(t *testing.T) {
const spec = `
globalThis.properties = {
zebra: __sanderling__.always(() => false),
apple: __sanderling__.always(() => false),
mango: __sanderling__.always(() => false),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, spec)
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}}); err != nil {
t.Fatal(err)
}
_ = verifier.EvaluateProperties()
got := verifier.NewlyViolatedProperties()
want := []string{"apple", "mango", "zebra"}
if !slices.Equal(got, want) {
t.Errorf("onset order: got %v, want %v (sorted lexicographically)", got, want)
}
}
func TestNextAction_FromActionsGenerator(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
@@ -281,6 +393,75 @@ func TestNextAction_NoSetupRegistered(t *testing.T) {
}
}
// TestDoubleTapsBuiltin_TargetsClickable verifies the doubleTaps builtin emits
// a DoubleTap action targeting a clickable, enabled element's center.
func TestDoubleTapsBuiltin_TargetsClickable(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"testTag": "SubmitButton", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `globalThis.actions = __sanderling__.doubleTaps;`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindDoubleTap {
t.Fatalf("kind = %v, want DoubleTap", action.Kind)
}
if action.X != 50 || action.Y != 60 {
t.Errorf("coords = (%d,%d), want (50,60) at SubmitButton center", action.X, action.Y)
}
// Action-gated properties read lastAction.on to tell which target the
// chooser hit. An empty On reduces those properties to vacuously-true and
// they never fire on the real tap event.
if action.On == "" {
t.Fatal("On must be populated so action-gated properties can identify the target")
}
if !strings.Contains(action.On, "SubmitButton") {
t.Errorf("On = %q, want a selector containing SubmitButton", action.On)
}
resolved := tree.Find(action.On)
if resolved == nil {
t.Fatalf("On = %q does not resolve in the same tree", action.On)
}
rx, ry := resolved.Bounds.Center()
if rx != action.X || ry != action.Y {
t.Errorf("On %q resolves to (%d,%d), want the picked element's center (%d,%d)",
action.On, rx, ry, action.X, action.Y)
}
}
func TestDoubleTap_RoundTrip(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.actions = __sanderling__.actions(() => [
__sanderling__.doubleTap({ on: "id:save" }),
]);
`)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindDoubleTap {
t.Errorf("kind: got %v, want DoubleTap", action.Kind)
}
if action.On != "id:save" {
t.Errorf("selector: got %q, want id:save", action.On)
}
}
func TestInputText_RoundTrip(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
@@ -302,6 +483,65 @@ func TestInputText_RoundTrip(t *testing.T) {
}
}
// TestTypingBuiltin_TargetsEditableField verifies the typing generator emits an
// InputText action aimed at an editable, enabled element's center and fills it
// with a corpus value, ignoring clickable-but-not-editable elements.
func TestTypingBuiltin_TargetsEditableField(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"testTag": "EmailField", "bounds": "[0,0,100,40]"}, "editable": true, "enabled": true, "children": []},
{"attributes": {"testTag": "SubmitButton", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `globalThis.actions = __sanderling__.typing;`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
}
if action.Kind != ActionKindInputText {
t.Fatalf("kind = %v, want InputText", action.Kind)
}
if action.X != 50 || action.Y != 20 {
t.Errorf("coords = (%d,%d), want (50,20) at EmailField center", action.X, action.Y)
}
if !slices.Contains(inputCorpus, action.Text) {
t.Errorf("text %q not drawn from inputCorpus", action.Text)
}
}
// TestTypingBuiltin_NoEditableYieldsErrNoAction verifies the typing generator
// declines (ErrNoAction) when no editable element is present, so a weighted
// layer falls through to another generator.
func TestTypingBuiltin_NoEditableYieldsErrNoAction(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"testTag": "SubmitButton", "bounds": "[0,0,100,40]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `globalThis.actions = __sanderling__.typing;`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
if _, err := verifier.NextAction(); !errors.Is(err, ErrNoAction) {
t.Fatalf("err = %v, want ErrNoAction", err)
}
}
func TestPushSnapshot_FeedsSnapshotsToExtractorState(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
@@ -609,6 +849,150 @@ globalThis.properties = {
globalThis.actions = __sanderling__.actions(() => []);
`
// TestSelectorStringFromJS_CanonicalGrammar guarantees the selector tag stamped
// on returned AX nodes round-trips back to a parseable selector string when the
// node is later used as an action target. Without this, an action emitted from
// `tap({ on: state.ax.find({ testTag: "LoginEmail" }) })` ends up with
// `action.selector = "[object Object]"` in the trace.
func TestSelectorStringFromJS_CanonicalGrammar(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"testTag": "LoginScreen", "bounds": "[0,0,100,40]"},
"children": [
{"attributes": {"testTag": "LoginEmail", "bounds": "[0,0,100,20]"}, "editable": true, "enabled": true, "children": []}
]}
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.objectSelector = __sanderling__.extract(state =>
state.ax.find({ testTag: "LoginScreen" })
);
globalThis.chainSelector = __sanderling__.extract(state =>
state.ax.find([{ testTag: "LoginScreen" }, { testTag: "LoginEmail" }])
);
globalThis.stringSelector = __sanderling__.extract(state =>
state.ax.find("testTag:LoginScreen")
);
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
read := func(name string) string {
handle := verifier.runtime.GlobalObject().Get(name).ToObject(verifier.runtime)
current := handle.Get("current")
if goja.IsUndefined(current) || goja.IsNull(current) {
return ""
}
object := current.ToObject(verifier.runtime)
return object.Get(tagSelector).String()
}
cases := []struct {
name string
want string
}{
{"objectSelector", "testTag:LoginScreen"},
{"chainSelector", "testTag:LoginScreen > testTag:LoginEmail"},
{"stringSelector", "testTag:LoginScreen"},
}
for _, testCase := range cases {
got := read(testCase.name)
if got != testCase.want {
t.Errorf("%s: got %q, want %q", testCase.name, got, testCase.want)
}
if strings.Contains(got, "[object") {
t.Errorf("%s: selector contains garbage %q", testCase.name, got)
}
}
}
// TestChangedExtractors_DiffsBetweenSnapshots verifies the per-step diff
// surfaces only extractors whose value actually changed, keyed by name (or
// extractor_N fallback when unnamed).
func TestChangedExtractors_DiffsBetweenSnapshots(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
__sanderling__.extract(state => state.snapshots["a"] ?? 0, "alpha");
__sanderling__.extract(state => state.snapshots["b"] ?? 0);
`)
push := func(a, b int) {
raw := func(n int) json.RawMessage {
body, _ := json.Marshal(n)
return body
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"a": raw(a), "b": raw(b)}}); err != nil {
t.Fatal(err)
}
}
push(1, 1)
first := verifier.ChangedExtractors()
if _, ok := first["alpha"]; !ok {
t.Errorf("step 1: expected alpha in diff (initial value), got %+v", first)
}
if _, ok := first["extractor_1"]; !ok {
t.Errorf("step 1: expected extractor_1 fallback name in diff, got %+v", first)
}
push(1, 2)
second := verifier.ChangedExtractors()
if _, ok := second["alpha"]; ok {
t.Errorf("step 2: alpha did not change, should not appear: %+v", second)
}
change, ok := second["extractor_1"]
if !ok {
t.Fatalf("step 2: expected extractor_1 in diff, got %+v", second)
}
if string(change.Prev) != "1" || string(change.Curr) != "2" {
t.Errorf("step 2: extractor_1 diff prev=%s curr=%s, want 1 -> 2", change.Prev, change.Curr)
}
push(1, 2)
third := verifier.ChangedExtractors()
if len(third) != 0 {
t.Errorf("step 3: nothing changed, diff should be empty, got %+v", third)
}
}
// TestExtract_DefaultsAndNamedNames verifies bindExtract assigns a fallback
// `extractor_N` name when no name is supplied and respects an explicit one.
func TestExtract_DefaultsAndNamedNames(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
__sanderling__.extract(state => 1);
__sanderling__.extract(state => 2, "ledgerRows");
__sanderling__.extract(state => 3);
`)
if len(verifier.extractors) != 3 {
t.Fatalf("extractors registered: got %d, want 3", len(verifier.extractors))
}
want := []string{"extractor_0", "ledgerRows", "extractor_2"}
for i, name := range want {
if got := verifier.extractors[i].name; got != name {
t.Errorf("extractor %d name: got %q, want %q", i, got, name)
}
}
}
// TestSelectorStringFromJS_NullEmpty verifies that nil/undefined args produce
// an empty string instead of "null"/"undefined" garbage.
func TestSelectorStringFromJS_NullEmpty(t *testing.T) {
verifier := newVerifier(t)
if got := selectorStringFromJS(verifier.runtime, goja.Undefined()); got != "" {
t.Errorf("undefined: got %q, want empty", got)
}
if got := selectorStringFromJS(verifier.runtime, goja.Null()); got != "" {
t.Errorf("null: got %q, want empty", got)
}
}
func TestOverrideExtractorValues_PropagatesNestedObjectFields(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, objectExtractorSpec)
+303 -6
View File
@@ -1,10 +1,14 @@
package verifier
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"maps"
"math/rand/v2"
"sort"
"strings"
"time"
"github.com/dop251/goja"
@@ -25,6 +29,9 @@ type Verifier struct {
evaluators map[string]*ltl.Evaluator
priorVerdicts map[string]ltl.Verdict
newlyViolated []string
lastTree *hierarchy.Tree
lastAction *Action
lastLogs []LogEntry
@@ -32,6 +39,8 @@ type Verifier struct {
stepTime time.Time
runStart time.Time
appPackage string
rng *rand.Rand
}
@@ -41,12 +50,21 @@ func WithRand(rng *rand.Rand) Option {
return func(v *Verifier) { v.rng = rng }
}
// WithAppPackage scopes random-action target selection to the app under test.
// Nodes belonging to another package (the soft keyboard, system UI, permission
// dialogs) are excluded so exploration never spends steps fuzzing the IME or
// inserting keyboard glyphs into fields. Empty package keeps current behavior.
func WithAppPackage(appPackage string) Option {
return func(v *Verifier) { v.appPackage = appPackage }
}
func New(options ...Option) (*Verifier, error) {
verifier := &Verifier{
runtime: goja.New(),
properties: map[string]int{},
evaluators: map[string]*ltl.Evaluator{},
rng: rand.New(rand.NewPCG(0, 0)),
runtime: goja.New(),
properties: map[string]int{},
evaluators: map[string]*ltl.Evaluator{},
priorVerdicts: map[string]ltl.Verdict{},
rng: rand.New(rand.NewPCG(0, 0)),
}
for _, option := range options {
option(verifier)
@@ -235,10 +253,55 @@ func (v *Verifier) PushSnapshot(input SnapshotInput) error {
return fmt.Errorf("extractor %d: %w", index, err)
}
_ = extractor.handle.Set("current", newValue)
extractor.prev = extractor.curr
extractor.curr = encodeExtractorValue(newValue)
}
return nil
}
// encodeExtractorValue produces a stable JSON encoding of an extractor's
// current value for diff comparison. goja values that don't survive Export
// (e.g. wrapped host functions) yield nil; callers treat nil as "unknown" and
// emit no diff entry.
func encodeExtractorValue(value goja.Value) []byte {
if value == nil || goja.IsUndefined(value) || goja.IsNull(value) {
return []byte("null")
}
exported := value.Export()
body, err := json.Marshal(exported)
if err != nil {
return nil
}
return body
}
// ChangedExtractors returns the named extractors whose value changed between
// the prior PushSnapshot and the current one. The map is keyed by extractor
// name; unnamed extractors (extractor_N fallback) are included so the inspect
// UI can still display them under a numeric label. The very first snapshot
// emits every non-null extractor as a change (Prev=null, Curr=current) since
// the runner can otherwise misread "no diff yet" as "nothing initialized".
func (v *Verifier) ChangedExtractors() map[string]ExtractorChange {
changes := map[string]ExtractorChange{}
for _, extractor := range v.extractors {
if extractor.curr == nil {
continue
}
prev := extractor.prev
if prev == nil {
prev = []byte("null")
}
if bytes.Equal(prev, extractor.curr) {
continue
}
changes[extractor.name] = ExtractorChange{
Prev: append([]byte(nil), prev...),
Curr: append([]byte(nil), extractor.curr...),
}
}
return changes
}
// OverrideExtractorValues replaces each extractor's `current` slot with a
// caller-supplied value, keyed by registration index. Used by the web tick
// path so extractor bodies that ran in V8 (against the real DOM) drive the
@@ -287,6 +350,9 @@ type SnapshotInput struct {
// after the most recent PushSnapshot. The step time passed in PushSnapshot is
// forwarded to each evaluator so deadline-bound operators see the snapshot's
// wall clock rather than time.Now().
//
// As a side effect, the set of properties that newly transitioned to violated
// on this call is recorded; see NewlyViolatedProperties.
func (v *Verifier) EvaluateProperties() map[string]ltl.Verdict {
verdicts := map[string]ltl.Verdict{}
stepTime := v.stepTime
@@ -297,9 +363,38 @@ func (v *Verifier) EvaluateProperties() map[string]ltl.Verdict {
verdicts[name] = evaluator.ObserveAt(stepTime)
}
v.refreshPredicateErrors()
var onset []string
for name, verdict := range verdicts {
if verdict == ltl.VerdictViolated && v.priorVerdicts[name] != ltl.VerdictViolated {
onset = append(onset, name)
}
}
sort.Strings(onset)
v.newlyViolated = onset
next := make(map[string]ltl.Verdict, len(verdicts))
maps.Copy(next, verdicts)
v.priorVerdicts = next
return verdicts
}
// NewlyViolatedProperties returns the names of properties whose verdict
// transitioned from non-Violated to Violated on the most recent
// EvaluateProperties call, sorted lexicographically. Returns nil if no
// transition occurred or EvaluateProperties has not been called.
//
// This is the onset set: each property name appears at most once across a
// run's traces, at the step where the violation first fired. Subsequent
// steps where the property remains violated (LTL `always` sticky semantics)
// will not list it. Use this for trace emission and summary reporting so the
// onset is the only step that surfaces the violation event; use
// EvaluateProperties for residual / current-verdict needs.
func (v *Verifier) NewlyViolatedProperties() []string {
return append([]string(nil), v.newlyViolated...)
}
// Residuals returns the residual formula for each registered property after
// the most recent EvaluateProperties call. Properties that errored during
// predicate evaluation surface as ErrorFormula so the inspect UI can render
@@ -451,12 +546,20 @@ func (v *Verifier) resolveGenerator(generator goja.Value) (Action, error) {
return v.resolveGenerator(picked)
case internalKindBuiltinTaps:
return v.generateRandomTap()
case internalKindBuiltinDoubleTaps:
return v.generateRandomDoubleTap()
case internalKindBuiltinTyping:
return v.generateRandomInput()
case internalKindBuiltinSwipes:
return v.generateRandomSwipe()
case internalKindBuiltinWaitOnce:
return Action{Kind: ActionKindWait, DurationMillis: 500}, nil
case internalKindBuiltinPressKey:
return v.generateRandomPressKey()
case internalKindBuiltinLongPresses:
return v.generateRandomLongPress()
case internalKindBuiltinScrolls:
return v.generateRandomScroll()
default:
return Action{}, fmt.Errorf("unknown generator kind %q", kindValue.String())
}
@@ -465,6 +568,29 @@ func (v *Verifier) resolveGenerator(generator goja.Value) (Action, error) {
// generateRandomTap picks a visible, tappable element from the last
// hierarchy snapshot and returns a Tap action targeting its center.
func (v *Verifier) generateRandomTap() (Action, error) {
return v.generateRandomTapKind(ActionKindTap)
}
// generateRandomDoubleTap is the DoubleTap counterpart of generateRandomTap.
// Real user gestures include double-tap (image zoom, like-to-favorite,
// play/pause); a fuzzer that never emits one cannot exercise either those
// features or the sub-100ms race windows that single-step Tap cadence misses.
func (v *Verifier) generateRandomDoubleTap() (Action, error) {
return v.generateRandomTapKind(ActionKindDoubleTap)
}
// inScope reports whether an element belongs to the app under test. Nodes from
// another package (the soft keyboard, system UI, permission dialogs) are out of
// scope. An unset app package or an element with no package falls through to in
// scope, preserving behavior on platforms that omit the attribute (e.g. iOS).
func (v *Verifier) inScope(element *hierarchy.Element) bool {
if v.appPackage == "" || element.Package == "" {
return true
}
return element.Package == v.appPackage
}
func (v *Verifier) generateRandomTapKind(kind ActionKind) (Action, error) {
if v.lastTree == nil {
return Action{}, ErrNoAction
}
@@ -473,6 +599,9 @@ func (v *Verifier) generateRandomTap() (Action, error) {
if !element.Clickable || !element.Enabled {
continue
}
if !v.inScope(element) {
continue
}
if element.Bounds.Right-element.Bounds.Left <= 0 || element.Bounds.Bottom-element.Bounds.Top <= 0 {
continue
}
@@ -483,7 +612,99 @@ func (v *Verifier) generateRandomTap() (Action, error) {
}
picked := candidates[v.rng.IntN(len(candidates))]
x, y := picked.Bounds.Center()
return Action{Kind: ActionKindTap, X: x, Y: y}, nil
return Action{Kind: kind, On: selectorForElement(v.lastTree, picked), X: x, Y: y}, nil
}
// selectorForElement builds a canonical "key:value" selector that resolves
// back to the given element via hierarchy.Tree.Find. Prefers resource-id (the
// testTag carrier on Android / accessibilityIdentifier on iOS), falling back
// to text and content-description so action-gated properties can still tell
// what was tapped even on legacy nodes without a testTag. Returns "" when no
// candidate selector uniquely resolves to the picked element so the runner
// keeps using the action's coordinates without re-routing to a sibling that
// shares the same id/text.
func selectorForElement(tree *hierarchy.Tree, element *hierarchy.Element) string {
if element == nil || tree == nil {
return ""
}
candidates := make([]string, 0, 4)
if element.ResourceID != "" {
candidates = append(candidates, "id:"+element.ResourceID)
}
// Some platforms surface the Compose testTag only in the attributes map
// (the sidecar doesn't always promote it to resource-id). Try the raw
// attribute keys before falling back to text-based selectors so an
// element with a unique testTag still gets identified.
for _, key := range []string{"testTag", "identifier", "accessibilityIdentifier"} {
if value := element.Attributes[key]; value != "" {
candidates = append(candidates, key+":"+value)
}
}
if element.Text != "" {
candidates = append(candidates, "text:"+element.Text)
}
if element.Description != "" {
candidates = append(candidates, "desc:"+element.Description)
}
for _, selector := range candidates {
resolved := tree.Find(selector)
if resolved == nil || resolved != element {
continue
}
return selector
}
return ""
}
// inputCorpus is the edge-case string pool the typing builtin draws from to
// stress field parsing: empty, whitespace, overflow length, unicode, numeric
// boundaries, and common injection payloads.
var inputCorpus = []string{
"",
"a",
strings.Repeat("a", 4096),
"🙂🔥💸",
" ",
"\t\n",
"-1",
"999999999999999999999",
"0.0000001",
"1e10",
"'; DROP TABLE--",
"<script>alert(1)</script>",
"../../etc/passwd",
"%s%n",
"NaN",
}
// generateRandomInput picks a visible, editable, enabled element from the last
// hierarchy snapshot and types a random edge-case value into it. The runner
// taps the target coordinates to focus before typing, so this works on both
// native and web with no driver-side dispatch change.
func (v *Verifier) generateRandomInput() (Action, error) {
if v.lastTree == nil {
return Action{}, ErrNoAction
}
candidates := make([]*hierarchy.Element, 0, len(v.lastTree.Elements))
for _, element := range v.lastTree.Elements {
if !element.Editable || !element.Enabled {
continue
}
if !v.inScope(element) {
continue
}
if element.Bounds.Right-element.Bounds.Left <= 0 || element.Bounds.Bottom-element.Bounds.Top <= 0 {
continue
}
candidates = append(candidates, element)
}
if len(candidates) == 0 {
return Action{}, ErrNoAction
}
picked := candidates[v.rng.IntN(len(candidates))]
x, y := picked.Bounds.Center()
value := inputCorpus[v.rng.IntN(len(inputCorpus))]
return Action{Kind: ActionKindInputText, X: x, Y: y, Text: value}, nil
}
// generateRandomSwipe emits a swipe over a random enabled element or the
@@ -493,7 +714,16 @@ func (v *Verifier) generateRandomSwipe() (Action, error) {
if v.lastTree == nil || len(v.lastTree.Elements) == 0 {
return Action{}, ErrNoAction
}
element := v.lastTree.Elements[v.rng.IntN(len(v.lastTree.Elements))]
candidates := make([]*hierarchy.Element, 0, len(v.lastTree.Elements))
for _, element := range v.lastTree.Elements {
if v.inScope(element) {
candidates = append(candidates, element)
}
}
if len(candidates) == 0 {
return Action{}, ErrNoAction
}
element := candidates[v.rng.IntN(len(candidates))]
cx, cy := element.Bounds.Center()
if cx <= 0 || cy <= 0 {
return Action{}, ErrNoAction
@@ -527,6 +757,73 @@ func (v *Verifier) generateRandomSwipe() (Action, error) {
}, nil
}
// generateRandomLongPress mirrors generateRandomTap: it picks a visible,
// clickable, enabled, in-scope element and targets its center. Real users
// long-press (context menus, reorder handles, multi-select), so a fuzzer that
// never emits one cannot reach those affordances.
func (v *Verifier) generateRandomLongPress() (Action, error) {
return v.generateRandomTapKind(ActionKindLongPress)
}
// generateRandomScroll picks a scrollable, in-scope container and emits a swipe
// across it in a random direction.
func (v *Verifier) generateRandomScroll() (Action, error) {
if v.lastTree == nil {
return Action{}, ErrNoAction
}
candidates := make([]*hierarchy.Element, 0, len(v.lastTree.Elements))
for _, element := range v.lastTree.Elements {
if element.Attributes["scrollable"] != "true" {
continue
}
if !v.inScope(element) {
continue
}
if element.Bounds.Width() <= 0 || element.Bounds.Height() <= 0 {
continue
}
candidates = append(candidates, element)
}
if len(candidates) == 0 {
return Action{}, ErrNoAction
}
picked := candidates[v.rng.IntN(len(candidates))]
cx, cy := picked.Bounds.Center()
width := picked.Bounds.Width()
height := picked.Bounds.Height()
directions := []string{"up", "down", "left", "right"}
dir := directions[v.rng.IntN(len(directions))]
toX, toY := cx, cy
// Scroll direction names the content motion; the gesture swipes the
// opposite way. Revealing content below ("down") means dragging the
// finger up, so toY decreases, and likewise for the other directions.
switch dir {
case "down":
toY = cy - (4*height)/10
case "up":
toY = cy + (4*height)/10
case "left":
toX = cx + (4*width)/10
case "right":
toX = cx - (4*width)/10
}
if toX < 0 {
toX = 0
}
if toY < 0 {
toY = 0
}
return Action{
Kind: ActionKindScroll,
Direction: dir,
FromX: cx,
FromY: cy,
ToX: toX,
ToY: toY,
DurationMillis: 300,
}, nil
}
func (v *Verifier) generateRandomPressKey() (Action, error) {
// Keep exploration gentle: only "back" for now. Home/menu would navigate
// away from the app under test.