mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
* docs: add the apache 2.0 license text
package.json has declared Apache-2.0 since the first release and .goreleaser.yaml
globs LICENSE* into the archives, so that glob has been matching nothing. npm
only picks up a license from the package directory, hence the copy under
pkg/spec.
* fix(sidecar): close the soft keyboard after typing on android
* test(sidecar): pin the guarded ime dismissal
* fix(sidecar): treat a failed ime probe as no keyboard open
* ci(folio): let the ios leg clear state for itself
* ci(folio): drop the stale frontboard note from the ios job
* docs(ci): record what the ios calibration assumes and where it was measured
* fix(ios): replace the session when a launch blows its bound
a launch the simulator refuses is never reported: xctest records it as a test
failure the runner cannot see, then holds the session's main thread for about
four minutes on a diagnostic chain. so the only signal is the expired bound,
and every later call queues behind the same wedge. restart the session once and
launch again, bounded so the launch path stays inside testrun's backstop.
* test(ios): cover the session replacement a wedged launch needs
* fix(ios): share one deadline across the restart and the second launch
the recovery a blown bound triggers now costs at most launchRecoveryTimeout
whatever it spends it on, so the launch path tops out at 150s and testrun's
three minute backstop stays a backstop.
* test(ios): the restart a blown launch triggers has to be bounded
* docs(ci): the ios leg does not convict on the runner, and a seed cannot fix it
seed 28 reproduced its walk on macos-15 and reached the bug at the step it
convicts at locally. it still could not be judged: the run did not return
home between step 19 and step 136, so the counting invariant saw a rise of
15 against a window of 37 submits.
* docs(sidecar): record why the stale ime flag stays out of reach
* test(folio): add commonTest source sets to core and shared
* fix(folio): reject amounts parseCents cannot represent
* fix(folio): cap a transaction at one million dollars
* test(spec): give the fake dom a real tree and a walking querySelectorAll
* style(sidecar): make ktlint clean, formatting only
ktlint -F over every kotlin file except DriverBackend.kt, then hand
fixes where the reflow read worse and for the long lines ktlint cannot
break. No behaviour changes.
DriverBackend.kt is left untouched to avoid a conflict with concurrent
work; its three over-long lines still fail fmt-kotlin.
* fix(spec): deepQueryAll returns matches in document order
* ci(folio): say what the android gate found, not why
The gate proves only that AddTransactionScreen is absent from the trace.
Claiming the run never got past login was an inference it cannot make: the
run that produced it had logged in and was stuck on the new account screen.
Name the routes the trace does record instead.
* test(runner): a relaunch must not convict the submit counting property
* test(browser): compare ax.find across both hosts on one page
* test(spec): name the shadow match in the grammar both hosts parse
* ci: move every action off the node20 runtime
checkout v4->v7, setup-go v5->v7, setup-node v4->v7, setup-java v4->v5,
upload-artifact v4->v7, cache v4->v6, upload-pages-artifact v3->v5,
deploy-pages v4->v5, setup-chrome v1->v2, setup-android v3->v4,
goreleaser-action v6->v7. setup-bun and android-emulator-runner are
already node24; buf-setup-action stays on its deliberate SHA pin.
setup-chrome v2 resolves stable from Chrome for Testing rather than the
official installer, so the ci.yml comment about the action's default no
longer held.
* feat(verifier): report a relaunch on state.lastAction
* test(verifier): pin the relaunch field on both hosts
* fix(runner): keep the action the app was relaunched after
* test: pin that a nested undefined does not survive the wire
* fix(folio): uninstall before installing in just ios
folio's signed-in session lives in the data container, which an install
over the top keeps, so a local run started right after just ios opened on
the previous run's Home screen and diverged at step 1. On CI's fresh
simulator the uninstall is a no-op, so the ios leg is unchanged.
* style(sidecar): bring the last three lines under the line limit
* fix(ios): the runner must not answer ok for a launch that failed
XCTest records a refused launch as a test issue that never throws, so the
companion returned ok for an app that never started. Check the state the
app actually reached and report the refusal instead.
* test(ios): a refusal the runner names costs no session restart
The session restart is for a launch that never answers. A launch that
reports the app's state has already said what a fresh session would.
* fix(folio): attribute a created account by its whole key, not a suffix
createdAccountHasNonZeroBalance matched the created card with endsWith, so
an older account whose name ends with the typed one ("Emergency Fund" for a
typed "Fund") was judged instead whenever the new card was clipped out of
the reading. Build both keys the card can carry, the plain name and web's
initials + name, and compare them whole.
* fix(hierarchy): object selectors resolve by the same rule as string ones
* test(verifier): both ax.find selector forms resolve the same element
* test(browser): the cross-host fixture uses the object selector form
* fix(replay-ui): wrap the tab strip so its last tabs stay clickable
* test(replay-ui): drive the fuzzer onto a violating step with a panel
* feat(folio): judge a submit against the account's own balance
The counting invariant can only close its window on Home, and the iOS run
in #78 went 117 steps between two Home readings: 37 submits against a rise
of 15 transactions is no evidence about the double tap sitting inside it.
The ledger and the add-transaction screen both show the account's own
balance, and an accepted submit pops back to the ledger, so a window
bounded by those readings holds one action.
The bound is an upper one: a balance that has not moved is a commit still
in flight, a rejected submit or a tap that never landed, and none of those
is a violation. Moving by more than the one submit in the window typed is.
* test(runner): an overlay dismissal must not convict the counting property
* docs(runner): point the guard comment at the renamed test
* docs(replay-ui): name the viewport the tab overflow was measured at
* feat(folio): close the submit window on the account's own screens
submitCommitsOneTransactionPerAction now states its rule over two windows:
the Home counts it already compared, and the account balance the ledger and
the add-transaction screen redraw on nearly every frame of the transaction
flow. Same rule, and the second window is usually one action wide.
* fix(sidecar): reach the adb server the environment names
buildDadb hardcoded localhost:5037, so a serial-addressed device always
resolved through this machine's adb server and ADB_SERVER_SOCKET was
ignored. Read the endpoint the way the adb CLI does instead.
Fixes #79
* test(sidecar): pin the adb server endpoint parsing
* fix(sidecar): close a keyboard standing in the snapshot
A tap on a text field raises the keyboard and nothing closed it, so the
tree the picker chooses from was missing every app node underneath it,
the submit control included. Close it before the read rather than after
the tap: the picker only ever sees snapshots, and the keyboard is still
on its way up when the tap returns.
Fixes #78
* test(sidecar): pin the tree-guarded keyboard dismissal
* chore(make): a target that runs folio's unit tests
* chore(folio): a just recipe for the unit tests
* ci: run folio's unit tests on every pr
* ci: switch to jdk 21 only for the folio step
* docs(sidecar): put the measured read cost in the dismissal bound
* fix(folio): decline the two demanding properties across a relaunch
The runner now keeps lastAction and marks it relaunched: true where it used
to report nothing at all, so the two properties that demand an effect judge
a step whose process may have died before the write landed.
submitChangesBalanceByTypedAmount and createdAccountHasNonZeroBalance both
decline there. The counting bound does not: a relaunch cannot manufacture a
transaction, and the submit is counted, so declining would throw away the
detection the runner fix restored.
* docs(folio): say why the merged card key cannot be made injective
Folio rejects a duplicate account name, so the twin the drop rule guards
against is two names the web key cannot tell apart, not two accounts
sharing a name. State what closing the rest would cost and what the tree
would have to carry to close it properly.
* test(folio): pin that two accounts can render the same card text
The proof behind the comment: "Travel1" holding 25 transactions and
"Travel12" holding 5 merge to the same string, so no identity key read off
a web card can tell them apart.
* fix(sidecar): bound the diagnostic adb reads
adbOutput and readLogcat read to EOF and then waited with no timeout, so
a wedged adb held the step for as long as it liked; one stall over a
remote adb server measured ~100s. The bound has to sit on the read, not
on waitFor: a wedged adb never reaches EOF, so a bounded waitFor after
the read is a line that never runs.
* test(sidecar): pin the bound on a wedged adb read
* fix(sidecar): an unreadable animation count is not idle
Defaulting the count to zero made a dumpsys that said nothing mean
nothing is animating, so a degraded link broke out of the settle early
and handed the runner a frame caught mid-animation. Unknown now waits,
inside the deadline waitForIdle already holds.
* test(sidecar): unknown animation state must not read as idle
* fix(folio): stop spending the submit budget on taps the app refused
The window is an upper bound on the transactions an interval could hold, and
a bound inflated by taps that commit nothing is a bound the app can never
exceed: #78 read a rise of 15 transactions against 37 submits. TxnSubmit is
clickable(enabled = amount.isNotBlank()) and parseCents refuses anything its
regex misses, so a tap whose landing frame shows a refused amount cannot have
committed. Over four recorded android runs that is 19, 11, 25 and 25 of 35,
26, 42 and 42 submit taps.
A relaunch is excepted: a fresh process draws an empty field whatever was
submitted.
* feat(folio): read the amount field into every submit window
Each of the three windows asks whether the tap could have committed, off the
field as the landing frame shows it.
* fix(sidecar): a foreground read that fails degrades the typing guard
An unreadable dumpsys passed a null owner to typeChunks, which switches
the mid-type focus guard off outright and lets the rest of the string
spray into whatever holds the foreground. Fall back to the launched
bundle instead: the guard stays armed, typing still happens, and the
degradation is said out loud rather than assumed away.
* test(sidecar): pin the degraded typing guard both ways
* test(runner): answer Snapshot and Hierarchy off one tree in the fakes
* feat(runner): skip a step whose tree changed between two reads
* test(runner): cover the reread's cost to the existing snapshot rules
* fix(ios): clear app state before the automation session attaches
New performs the clear-state reset, so the uninstall and reinstall no
longer land underneath a live XCTest session that is already bound to
the app. Launch refuses a clear-state request the driver was not built
for rather than reinstalling under its own session.
* fix(ios): the device path clears before its runner session too
* fix(testrun): thread clear-data into the ios drivers
* test(runner): a skipped step must not swallow the action before it
* fix(runner): hold the action back on a step nothing verified
* refactor(runner): drop the empty branch from the hold path
* docs(runner): describe both modes of the composing test driver
* fix(sidecar): erase a field by selecting it, not one delete per character
maestro's eraseText sends one delete per character through its
instrumentation, measured 29.6 ms/char on the API 34 emulator. The
4096-character string the corpus types cost ~121s to clear, a fifth of a
20 minute run spent on one step, and it recurred every time that field
was typed into again.
Select the content and delete the selection instead: two key events at
any length, measured 0.15s to 1.16s for 4096 characters across API 34,
35 and 36. The result is read back off the tree, and a field that is not
empty, or that the tree cannot report on, is finished off per character
in batches rather than assumed clear.
Fixes #80
* test(sidecar): pin the constant-cost erase and its residue check
* fix(sidecar): find the erased field by class, past the keyboard's own focus
The check that decides whether the select-all worked looked for an
"editable" attribute maestro's tree does not carry, so it answered
"cannot tell" every time and every erase paid the per-character
fallback. Worse, an open keyboard puts a second focused node in the
tree, one of the IME's own keys, carrying no text: taking the first
focused node would read a field still holding 4096 characters as empty,
which is the one answer that stops the erase early.
Match the text field by class instead. Measured against the real
backend, 4096 characters now clear in 385ms on API 34, 409ms on API 35
and 870ms on API 36, verified empty, where the fallback took ~4s.
* test(sidecar): use the tree the device really returns
* docs(ci): the android step number describes a local emulator, not ci
the leg disables animations and the number was measured with them on. the
first real dispatch carries 4 transitional steps over 200, so the cross-fade
wait does still fire in ci, just far less often.
* fix(android): say what the sdk lookup checked, not just to set ANDROID_HOME
* fix(doctor): resolve adb and emulator the way a run does
* docs(cli): the android doctor checks are not path-only
* fix(testrun): preflight resolves adb through the sdk, not just PATH
* docs(skills): add a spec review skill and the skills index
* fix(testrun): report a sidecar that dies at startup as the exit it was
* test(testrun): cover the sidecar shutdown path after an early exit
* docs(skills): add a property patterns catalogue skill
* fix(folio): bound the total-balance move instead of demanding it exactly
The write finishes before AddTransactionViewModel navigates, but nothing
establishes that Home's total has re-rendered before the frame is read, and
an equality convicts a healthy app for a total one frame behind. A delta of
zero is exactly the shape nine of the eleven measured android false
convictions had. 2x still exceeds x, so all four recorded convictions
survive, checked against the traces.
The trade is real: a balance that moves by LESS than the amount typed is no
longer judged anywhere in this spec.
* docs(folio): say what property 2 demands now that it is a bound
* docs(skills): add a spec authoring skill
covers hooks, extractors, selectors, properties, actions and the order to write them in, with a complete sample spec that typechecks against the real export surface.
* docs(skills): ground the property patterns catalogue in the merged specs
* docs(skills): name the selector keys that still substring match
* docs(skills): add a setup skill for adopting sanderling
* docs(skills): add a run triage skill
* docs(skills): point the setup skill at its siblings
* docs(manual): correct the flags the cli reference gets wrong
--launcher-activity does not exist in cmd/sanderling/main.go. --device,
--android-app-path and --arm do and were undocumented. runs.md still listed
--max-steps and --exit-on-violation as unshipped, and described --clear-data
as opt-in when the default is already true, contradicting itself ten lines on.
* test(folio): pin that a commit stays in the window until Home reads it
The interaction that keeps a stale Home card list from ever banking counts
the budget has already forgotten: a submit lands on the ledger, so the
reading that resets the window is a whole action later and the submit is
still in it. Characterization, not a regression: no code changed and it
cannot go red first.
* docs(folio): record why a banked card reading can be trusted as current
The freshness rule rests on the app popping one entry back to the ledger,
not on anything the frame carries, so the assumption and the measurements
behind it belong next to it.
* refactor(folio): name the balance property for the bound it asserts
it stopped being an equality and became |delta| <= typed, so the old name
demanded more than the property does. renamed with the ci gate's
GATED_PROPERTIES in the same commit so the gate never sees a name it does
not know.
* fix(android): a refused uninstall must not pass for clear-state
adb uninstall answers Failure [DELETE_FAILED_INTERNAL_ERROR] both when the package was never installed and when it refuses to remove one, so the failure text cannot say which happened and the old code installed over the top either way, keeping the data clear-state was asked to drop. Ask pm path instead, and fall back to pm clear when the app is still there.
* fix(ios): a failed simctl uninstall must fail the reinstall
simctl install over an installed app carries its data container across, so discarding the uninstall error reported a clear-state that never happened. Uninstalling an app that is not installed exits 0 on a booted simulator, so every failure here is a real one.
* fix(ios): a failed devicectl uninstall must fail the reinstall
same hole as the simulator path: devicectl install over an app keeps its data, and the discarded uninstall error hid it. Uninstalling a bundle id that is not installed exits 0 with 'App uninstalled.' on a paired iPhone, so a failure here is always real.
* docs(android): say why the uninstall text cannot be read
* test(android): name the uninstall failure for what it says, not why
* docs(ci): the ios leg convicts on the runner now, and why it did not before
* docs(ci): the cross-fade wait does not fire on ci, say so
* fix(runner): a bounded hold puts the swallow back one step later
the hold carries one action; letting the runner act again while the verifier
is still skipped overwrites it, so the carried action reaches no spec. hold
for as long as the verifier is skipped, and settle on a held step so the
reread pair is not tighter than the window the detector was measured over.
* test(runner): pin what the two reads are compared on
structuralShape excluding text and bounds is the decision separating this
feature from a run that verifies nothing, and only prose held it. adding
either field back now turns a case red.
* fix(ci): close shell injection into the npm publish job
A refname is attacker-controlled and git permits backtick, $, (, ; and |
in it. Three sites substituted it into a run: block, and NODE_AUTH_TOKEN
sat at job level, so a pushed tag ran arbitrary commands with the publish
credential in reach.
The tag now goes through env:, is validated against an anchored version
pattern before anything consumes it, and reaches the other jobs as a job
output. The token is scoped to the publish step. release-npm declares
contents: read instead of inheriting the repo default.
* fix(ios): recognise every shape a blown launch bound arrives in
The runner transport reports a blown budget two ways, its own comment says
so: the context's error once cancellation has landed, and the connection's
i/o timeout when the deadline armed from that context fires first. The
legacy transport reports it as a gRPC status. errors.Is against
context.DeadlineExceeded only matches the first, so the session restart
never fired for the other two and a wedged session stayed wedged.
* test(ios): drive the launch recovery with what the transports return
The wedged-session fake answered with ctx.Err() raw, which is the one
shape the guard already matched. The recovery now runs against the error
each transport really produces for the same expiry, taken from a runner
and a legacy companion that never answer.
* test(runner): pin both guard writes to what the spec reads
deleting lastAction.Relaunched or lastAction.Applied left the whole suite
green, so the only producer of the two fields every spec-side guard reads
had nothing holding it. both now assert the value out of the trace.
* fix(testrun): a run that judged nothing is not a green run
every step skipped means no property ever evaluated, so no violations is the
absence of a verdict rather than a clean one. the hold makes that reachable
now, so the run says it instead of exiting 0.
* fix(ios): stop the app before clearing its state
Launch terminated and then cleared; the clear moved to construction and
left nothing stopping the app first. The container wipe deletes files a
live app still holds open, and the CI ios leg passes no app path so the
wipe is the path it takes. simctl stops it, since the clear now runs
before any automation session exists. On a device the uninstall that is
its only clear takes the running app with it.
* test(ios): pin the stop that has to precede a clear
The ordering probe now records the stop, and a scripted xcrun holds what
reaches the tool: terminate before get_app_container, with the previous
run's files gone after. A simctl terminate that finds nothing to stop
still leaves the clear a success.
* docs(spec): an unbounded eventually is violated at run end
* docs(skills): an unreached eventually convicts at run end
* docs(skills): noUncaughtExceptions only fires on web
* fix(ios): a device clear-state that cannot happen must fail
--clear-data on a physical device with no --ios-app-path warned and then
ran anyway, so the run started on the previous run's data while the flag
said it started clean. There is no data-container wipe on a device, so
there is nothing to fall back to.
* test(ios): a device clear-state without an app path ends the run
* docs(skills): the stock properties each cover one platform
* docs(ci): the balance property demands a bound, not an equality
* fix(ios): the clear-state guard checks the bundle that was cleared
A bool only said that something was cleared, so Launch(ctx, otherBundle,
clearState=true) passed the guard and reported a reset that had reached a
different app. Record what was cleared and compare against the bundle
being launched.
* test(ios): a clear-state launch for an uncleared bundle is refused
* docs(manual): the flagship property is a bound, and say what that costs
* fix(ios): one address picker for every bring-up
bringUpRunner reads the picker from a field, and NewDevice only ever set
the device one, so a device driver that reached bringUpRunner would call
nil. The two fields held the same function; keeping one leaves no path
that can be wired without it.
* test(ios): a device driver can bring a runner up
* docs(skills): both shipped balance forms are bounds now
* docs(skills): name the balance predicate that still exists
* docs(skills): quote the doctor the binary actually prints
* docs(skills): screen= is the chrome driver's url, web only
* docs(skills): substring selector matching is native only
* docs(skills): web selectors are exact, native ones are substrings
* fix(ci): a run that wrote no trace is not evidence about folio
run_dir is empty when the run produced no output directory, and the
fallback made trace ./trace.jsonl. A stray trace in the working directory
was then read as this run's, so a run that wrote nothing reported 'found
the submit bug' and exited 0, defeating the missing-trace check below it.
* fix(ci): fail folio when a gated property is not in the spec
Nothing tied GATED_PROPERTIES to the spec it gates. Renaming a property
left the classifier matching nothing: ios and web blamed the spec for
finding a different bug, and android silently reclassified a real
conviction as 'judging health only' and stayed green.
replay-ui-summary.sh already makes this check for its own list. The spec
path becomes SPEC-overridable the same way, so the check is testable.
* test(ci): cover the folio classifier's verdicts
21 cases through a stubbed sanderling: every exit path, the drift check,
a missing trace, a zero-byte trace, an empty glob and a truncated line.
Asserts the flags that reached the binary, not just the exit code.
Invoked as bash -eo pipefail -c, which is what a run: block does. Running
folio-run.sh itself under -e would kill it at the first non-zero
sanderling test, which is the exit code it exists to read.
* docs(skills): defaultActions bundles five of the eight generators
* test(ios): name the picker test for what it covers
* docs(skills): three of the replay-ui properties are cross-panel
* docs(manual): state.exceptions is web only and reportError does not exist
* fix(folio): the bound carries no unconfirmed-submit guard
deleting confirmedApplied here broke 0 of 355 tests: under a bound a submit
that may not have landed moves the balance by 0, which the bound already
permits, so the guard could only ever drop the double commit it exists to
catch. the relaunch guard stays for a reason the bound does not cover, and
both tests now assert a verdict that changes when their guard does.
* docs(ci): three of the replay-ui properties are cross-panel
* docs(manual): the starter property only fires on web
* test(folio): judge the conjunct on the landings a real run produces
three of the 18 frames the recorded ios run drove it down, each with the
second commit the bound is there to catch. neutering the comparison reddens
it: a second commit on 357900 went unjudged.
* test(folio): the walk drives the composition the spec runs
countSubmitsInWindow never saw an amountText here, so every walk test counted
submits the app must have refused. with the field passed, a refused submit no
longer buys a later double tap an alibi: without it the window reads 3, not 1.
* docs(folio): say which double submit the conjunct can see, and which it cannot
the home landing is the counting invariant's, three of three in the recorded
ios run; this one gets the interleaving whose second pop is cancelled. it is
still the only judge on the 18 ledger landings that run produced.
* docs(folio): the narrow window is not where the detection comes from
the double taps land on home, so the counting form convicts them; what turned
0 convictions into 4 on the recorded ios run is submitCouldCommit, which drops
the windows at those three steps from 5/4/7 to 2/1/2.
* docs(skills): folio drives three platforms from one spec
* fix(folio): an amount over the app's cap spends no window budget
the corpus reaches TxnSubmit with 999999999999999999999, AMOUNT_REGEX takes it
and AddTransactionViewModel refuses it against MAX_TRANSACTION_AMOUNT_CENTS, so
counting it was budget a double submit could hide behind.
* docs(folio): say which form judged one step, not which node was read once
* docs: a bound still needs the relaunch guard, and eventually does convict
* fix(sidecar): the hierarchy rpc serves the tree the snapshot reads
the runner compares the two per step, but snapshot settles and closes a
keyboard while hierarchy was a bare contentDescriptor. measured on emulator
-5556 (api 34) with an ime open: 489 nodes against the snapshot's 134. both
now come off snapshotTree under the same lock; the reread still costs ~75ms
when no keyboard is up.
* docs(runner): say what makes the two reads comparable
the reread's comment claimed the round trip was the only interval between
them; what it left out is that the two rpcs have to read the same way, which
the repo's own android backend did not do.
* refactor(runner): name the settle predicate for what it means
* ci: add a headless-chrome composite action
The setup-chrome / apparmor sysctl / launch-check trio is copied across
three jobs. The old comment described setup-chrome v1 semantics: under v2
stable is the default and the alternative is Chrome for Testing latest,
not a dev Chromium, so it is restated for what the pin actually does.
* ci(examples): add the folio setup actions
folio-app holds the per-platform toolchain and app build, so a caller
guards one step instead of eight. folio-simulator boots the simulator,
installs folio and leaves the app stopped.
* ci(examples): add the replay-ui fixture action
Records a trace and serves it with sanderling replay. The step page URL
is a composite output rather than GITHUB_ENV, so it is scoped to the one
step that drives it.
* ci(examples): one dispatch workflow for every example
folio.yml and replay-ui.yml ran the same operation: build sanderling for
a platform, bring a target up, run a spec against it, classify the trace,
upload the run. They are now one matrix over four examples, each naming
its own runner.
The job is named for what it fuzzes. 'dogfood' named why we run it, not
what runs, the same error as a diagnostic that reports a motivation
instead of an observation.
The matrix is computed by a plan job because jobs.<id>.if cannot read the
matrix context, so a static matrix has no way to leave a leg out. Seeds,
budgets, timeouts, runners and artifact names are unchanged.
* ci: reuse the headless-chrome action in the browser job
Same three steps the examples workflow needs, and the comment explaining
the AppArmor sysctl now lives in one place.
* ci: move the folio jdk step to setup-java v5
The only setup-java left on v4; every other one moved.
* ci: pin third-party actions to commit shas
buf-setup-action was already pinned with a comment saying why; the other
five rode mutable major tags, so a tag move is an unreviewed change to
what runs. Each major currently resolves to the release named in the
comment, so this freezes today's behaviour rather than changing it.
actions/* stay on major tags: they are first-party to the runner.
* ci(replay-ui): name the run directory for what it fuzzes
runs/dogfood and the '### replay-ui dogfood' heading carried the same
naming error as the job name: dogfooding is why the run exists, not what
it fuzzes.
* docs(driver): state the log level scale on LogEntry
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* docs(sidecar): name the device the node counts came off
* fix(chrome): keep a log entry the level scale cannot rank
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* fix(chrome): record console levels on the logcat scale
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* docs(ci): say why upload-pages-artifact needs no include-hidden-files
v3 to v5 crossed v4's change to exclude dot-files. build/site has none,
so nothing was dropped, and the underscore directory is not hidden.
* test(browser): drive a console error through to the spec
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* docs(ioscompanion): name the vacuity behind the empty log slice
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: run the folio classifier's test in make test-ci-scripts
* fix(chrome): keep the message of an object console argument
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(browser): cover console.error with an error object
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* feat(spec): let the runner install state.logs in the page
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* feat(verifier): encode state.logs for the web host
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* feat(chrome): install the step's logs in the page
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(ci): pin three real folio traces from run 31902501859
ios convicted on submitCommitsOneTransactionPerAction, web on both gated
properties, android ran its full 200 steps healthy. Every step is kept;
of each step only step, violations, witnesses and residuals survive.
hierarchy is replaced by the quoted "...Screen" resource ids it held, in
order. It cannot just be dropped: it is 95% of the bytes and also the
only place the android route gate's grep can match, so dropping it flips
that leg from healthy to 'never reached'. 8.4MB to 59KB.
* test(ci): drive the classifier over the real traces
Four cases on real data: each leg's real verdict, plus the android trace
cut before it reached the transaction screen, which is what proves the
route gate reads a real hierarchy dump.
Also corrects the hand-written fixtures. They set is_error to false on a
plain violation; internal/trace/writer.go tags that field omitempty, so a
real trace omits it entirely. Harmless to the classifier, but a fixture
that does not look like reality is the thing that hides drift.
* test(runner): teach the web fakes to take the step's logs
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* fix(runner): install the step's logs before the page extracts
On web every extractor reading is replaced by the one the page computed,
and the page answered logs: [], so noLogcatErrors counted an empty array
however full of errors the console was.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(runner): cover the logs reaching the page and failing to
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(spec): cover the host pushing state.logs into the page
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(browser): drive console.error through to a fired property
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* fix(runner): report a log fetch the driver could not make
The comment claimed the failure was warned about; nothing warned, so a
device whose log fetch failed every step held noLogcatErrors on evidence
nobody collected.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(runner): cover the silently dropped log fetch
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* fix(ci): read the route the spec reports, not the hierarchy dump
The android health gate grepped the trace for "AddTransactionScreen",
which occurs in exactly one place: the hierarchy dump, as a resource-id.
That is a debug artifact standing in for a fact the spec already reports,
and it was wrong in both directions. Against the real 8.4MB trace with
hierarchy stripped, the old gate failed a healthy 200-step run; against a
trace carrying the marker on a transition frame without the route ever
being reported, it passed and called it healthy.
It reads extractor_changes.route now, whose values come from SCREENS in
the spec, so the gate and the app agree on what being on a screen means.
routeOf answers null on a frame showing two screens, which is exactly the
frame the marker was matching.
The drift check grows to cover both new names: extract("route") and the
SCREENS key. The fixtures are re-derived keeping the route entry and no
hierarchy at all; the full artifacts and the fixtures give byte-identical
verdicts, which is what proves the coupling is gone.
Script and fixtures move together: either alone leaves the suite red.
* ci: check that the workflow references resolve
actionlint reads a local action's inputs but never checks its path
exists: uses: ./.github/actions/typo lints clean and fails only when the
job runs. Covers composite action paths, make targets including the ones
the examples matrix builds from $SANDERLING, and the scripts a run: step
invokes plus their executable bit.
Fails when it parses fewer references out of a file than that file
mentions, because a checker that matches nothing reports a safety it
never looked for.
* ci: lint the workflows on every pr
The workflows that fuzz the examples are dispatch-only, and GitHub will
not dispatch a workflow that is not on the default branch, so their first
real run is after merge. actionlint and the reference checker are the
only things that can fail before that.
actionlint is pinned by commit, and its tool version is pinned too so a
new release cannot change what CI enforces.
* ci: collapse the four workflows into one
Nine jobs written out one by one, each with its own steps and its own
calibrated seed and budget as literals. Triggers are pull requests, master
and v* tags, and a dispatch with no inputs.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: inline the two composite actions with one caller each
Both existed to give the matrix a per-target hook. folio-app and
headless-chrome stay: three and three callers.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: check that no run: block interpolates an expression
A ${{ }} lands in the script text before bash reads the line, and
actionlint only flags the contexts it already knows are attacker
controlled. Nothing enforced the rule the workflow follows. Also drops the
matrix table lookup, which has no table to read now.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci(folio): name the run, not the fuzzer, in the clean-run message
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* docs: point at the workflow that holds the release secrets now
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: name every job Category (variant), and gate the lot on one check
Follows the convention in antithesishq/bombadil: the display name is what
groups a run in the Actions UI, so Check (tests), Check (browser),
Check (workflows), Folio (android), Folio (ios), Folio (web), Replay UI,
Release and Docs. Every job carries a name, so none of them falls back to
its kebab-case id.
All checks passed needs all nine and runs with if: always(), so branch
protection has one check to point at and a skipped job cannot read as a
pass. Release and docs now gate on startsWith(github.ref, 'refs/tags/v')
alongside master, which is the form the trigger filter already uses.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: split the release job back in two
Collapsing them left the npm publish steps in a job holding contents:
write, because GoReleaser needs it, so npm ci ran its dependency lifecycle
scripts with a write-capable GITHUB_TOKEN in reach of the same job as a
live NPM_TOKEN. Release (npm) is back on contents: read and Release (cli)
keeps contents: write, which is what they each had before.
Each validates the tag from its own copy of the pattern rather than
waiting on a job that exists only to pass a string. Release (cli) is tags
only: there is no CLI to cut on a merge.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: run folio on pull requests
folio was skipped on pull requests, so ios, android and web only ever ran
after a merge. The three legs are 3 to 19 minutes and run in parallel, and
a superseded pull request run already cancels itself.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: draw each group as its own box in the run graph
The run graph boxes jobs together when they share the same dependencies and
the same dependents. All ten jobs fed only all-checks-passed, so all ten drew
as one pile. A gate per group gives each group a dependent that is exactly
that group.
Release and docs now need the checks, which they should have all along: npm
publish and the pages deploy ran on a merge without waiting for the test job.
Folio stays unblocked so a 20 minute leg does not wait on a 3 minute one.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
2101 lines
72 KiB
Go
2101 lines
72 KiB
Go
package runner
|
|
|
|
import (
|
|
"bufio"
|
|
"bytes"
|
|
"context"
|
|
"encoding/json"
|
|
"errors"
|
|
"fmt"
|
|
"io"
|
|
"log/slog"
|
|
"os"
|
|
"path/filepath"
|
|
"slices"
|
|
"strings"
|
|
"testing"
|
|
"time"
|
|
|
|
"google.golang.org/grpc/codes"
|
|
"google.golang.org/grpc/status"
|
|
|
|
"github.com/priyanshujain/sanderling/internal/bundler"
|
|
"github.com/priyanshujain/sanderling/internal/driver"
|
|
mockdriver "github.com/priyanshujain/sanderling/internal/driver/mock"
|
|
"github.com/priyanshujain/sanderling/internal/hierarchy"
|
|
"github.com/priyanshujain/sanderling/internal/trace"
|
|
"github.com/priyanshujain/sanderling/internal/verifier"
|
|
)
|
|
|
|
const fixtureSpec = `
|
|
import { actions, always, extract, Tap } from "@sanderling/spec";
|
|
const balance = extract(state => state.snapshots.balance ?? 0);
|
|
globalThis.properties = {
|
|
balanceNonNegative: always(() => balance.current >= 0),
|
|
};
|
|
globalThis.actions = actions(() => [Tap({ on: "id:next" })]);
|
|
`
|
|
|
|
const violationSpec = `
|
|
import { actions, always } from "@sanderling/spec";
|
|
globalThis.properties = {
|
|
balanceNonNegative: always(() => false),
|
|
};
|
|
globalThis.actions = actions(() => []);
|
|
`
|
|
|
|
type harness struct {
|
|
mock *mockdriver.Driver
|
|
verifier *verifier.Verifier
|
|
writer *trace.Writer
|
|
}
|
|
|
|
func newHarness(t *testing.T) *harness {
|
|
return newHarnessWithSpec(t, fixtureSpec)
|
|
}
|
|
|
|
func fastFocusSettle(t *testing.T) {
|
|
prev := focusTapSettle
|
|
focusTapSettle = time.Millisecond
|
|
t.Cleanup(func() { focusTapSettle = prev })
|
|
}
|
|
|
|
// bundleSpec compiles an authored TS spec with the goja runtime entry so the
|
|
// loaded bundle installs __sanderlingNextAction__ (the shared picker).
|
|
func bundleSpec(t *testing.T, specSource string) string {
|
|
t.Helper()
|
|
dir := t.TempDir()
|
|
specPath := filepath.Join(dir, "spec.ts")
|
|
if err := os.WriteFile(specPath, []byte(specSource), 0o600); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
apiPath, err := filepath.Abs("../../pkg/spec/src/index.ts")
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
runtimePath, err := filepath.Abs("../../pkg/spec/src/goja-runtime.ts")
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
bundle, err := bundler.Bundle(bundler.Options{
|
|
EntryFile: specPath,
|
|
RuntimeFile: runtimePath,
|
|
Aliases: map[string]string{"@sanderling/spec": apiPath},
|
|
})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
return string(bundle.JavaScript)
|
|
}
|
|
|
|
func newHarnessWithSpec(t *testing.T, spec string) *harness {
|
|
t.Helper()
|
|
directory := t.TempDir()
|
|
writer, err := trace.NewWriter(directory)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
verifierInstance, err := verifier.New()
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if err := verifierInstance.Load(bundleSpec(t, spec)); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
state := &harness{
|
|
mock: mockdriver.New(),
|
|
verifier: verifierInstance,
|
|
writer: writer,
|
|
}
|
|
t.Cleanup(func() { _ = writer.Close() })
|
|
return state
|
|
}
|
|
|
|
func TestRunner_HappyPathStepsAndTraces(t *testing.T) {
|
|
state := newHarness(t)
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 50 * time.Millisecond,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if summary.Steps == 0 {
|
|
t.Errorf("expected at least one step, got 0")
|
|
}
|
|
if len(summary.Violations) != 0 {
|
|
t.Errorf("no violations expected, got %v", summary.Violations)
|
|
}
|
|
// Every builtin verb is supported on every platform, so a clean run must
|
|
// report no unsupported verbs (the runner still wires the field through).
|
|
if len(summary.UnsupportedVerbs) != 0 {
|
|
t.Errorf("expected no unsupported verbs, got %v", summary.UnsupportedVerbs)
|
|
}
|
|
|
|
actions := state.mock.Actions()
|
|
if !containsAction(actions, mockdriver.ActionTapSelector, "id:next") {
|
|
t.Errorf("expected TapSelector with id:next, got %v", actions)
|
|
}
|
|
}
|
|
|
|
func TestRunner_MaxStepsStopsAfterExactlyNSteps(t *testing.T) {
|
|
state := newHarness(t)
|
|
|
|
const maxSteps = 3
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
// A long duration ensures MaxSteps, not the deadline, ends the run.
|
|
summary, err := Run(ctx, Options{
|
|
Duration: time.Hour,
|
|
IdleTimeout: 10 * time.Millisecond,
|
|
MaxSteps: maxSteps,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if summary.Steps != maxSteps {
|
|
t.Errorf("expected exactly %d steps, got %d", maxSteps, summary.Steps)
|
|
}
|
|
}
|
|
|
|
// TestRenderSummary_SurfacesUnsupportedVerbs exercises the real path that takes
|
|
// verbs the picker requested but the platform cannot dispatch and puts them in
|
|
// front of the operator. TestRunner_HappyPath only ever asserts the field stays
|
|
// empty (every builtin is supported, so its non-empty arm can never fire), so
|
|
// without this the "unsupported on %s: ..." branch could be deleted and every
|
|
// unsupported-verb regression would pass silently.
|
|
func TestRenderSummary_SurfacesUnsupportedVerbs(t *testing.T) {
|
|
summary := Summary{Steps: 3, UnsupportedVerbs: []string{"longPresses", "scrolls"}}
|
|
|
|
var out bytes.Buffer
|
|
RenderSummary(&out, summary, "ios")
|
|
|
|
if !strings.Contains(out.String(), "unsupported on ios: longPresses, scrolls") {
|
|
t.Errorf("expected unsupported verbs line for ios, got:\n%s", out.String())
|
|
}
|
|
}
|
|
|
|
// TestRenderSummary_OmitsUnsupportedLineWhenNone guards the inverse: a clean run
|
|
// must not print a stray "unsupported on" line, so a future refactor cannot
|
|
// start emitting an empty list and alarm the operator on every run.
|
|
func TestRenderSummary_OmitsUnsupportedLineWhenNone(t *testing.T) {
|
|
var out bytes.Buffer
|
|
RenderSummary(&out, Summary{Steps: 2}, "android")
|
|
|
|
if strings.Contains(out.String(), "unsupported on") {
|
|
t.Errorf("did not expect an unsupported-verbs line, got:\n%s", out.String())
|
|
}
|
|
}
|
|
|
|
// A step nothing judged is not a step that passed. The run prints its count so
|
|
// a green summary cannot hide a run that skipped most of its steps, which is
|
|
// what a screen that keeps moving under the reads would produce.
|
|
func TestRenderSummary_CountsTheStepsNothingJudged(t *testing.T) {
|
|
var out bytes.Buffer
|
|
RenderSummary(&out, Summary{Steps: 10, SkippedVerification: 4}, "android")
|
|
if !strings.Contains(out.String(), "4 step(s) judged by nothing") {
|
|
t.Errorf("expected the unjudged-step count, got:\n%s", out.String())
|
|
}
|
|
|
|
out.Reset()
|
|
RenderSummary(&out, Summary{Steps: 10}, "android")
|
|
if strings.Contains(out.String(), "judged by nothing") {
|
|
t.Errorf("a run that judged every step must not print the line, got:\n%s", out.String())
|
|
}
|
|
}
|
|
|
|
func TestRunner_ViolationSurfacesInSummary(t *testing.T) {
|
|
state := newHarnessWithSpec(t, violationSpec)
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 50 * time.Millisecond,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if len(summary.Violations) == 0 {
|
|
t.Errorf("expected at least one violation, got %v", summary.Violations)
|
|
}
|
|
if !containsProperty(summary.Violations, "balanceNonNegative") {
|
|
t.Errorf("expected balanceNonNegative in violations: %v", summary.Violations)
|
|
}
|
|
}
|
|
|
|
func TestRunner_ViolationSurfacesOnlyOnOnsetStep(t *testing.T) {
|
|
// violationSpec uses always(() => false): onset fires on step 1 and the
|
|
// residual stays violated forever. The runner must record the violation
|
|
// exactly once (at the onset step) in both summary.Violations and trace
|
|
// lines, not on every subsequent step.
|
|
state := newHarnessWithSpec(t, violationSpec)
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 200 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if summary.Steps < 2 {
|
|
t.Fatalf("need at least 2 steps to prove onset-only behavior, got %d", summary.Steps)
|
|
}
|
|
if len(summary.Violations) != 1 {
|
|
t.Fatalf("expected exactly one ViolationRecord (onset only), got %d: %v",
|
|
len(summary.Violations), summary.Violations)
|
|
}
|
|
if summary.Violations[0].StepIndex != 1 {
|
|
t.Errorf("onset step: got %d, want 1 (always(()=>false) violates immediately)",
|
|
summary.Violations[0].StepIndex)
|
|
}
|
|
if !slices.Equal(summary.Violations[0].Properties, []string{"balanceNonNegative"}) {
|
|
t.Errorf("onset properties: got %v, want [balanceNonNegative]",
|
|
summary.Violations[0].Properties)
|
|
}
|
|
|
|
file, err := os.Open(filepath.Join(state.writer.Directory(), "trace.jsonl"))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
defer file.Close()
|
|
|
|
type traceLine struct {
|
|
Step int `json:"step"`
|
|
Violations []string `json:"violations"`
|
|
}
|
|
linesWithViolations := 0
|
|
scanner := bufio.NewScanner(file)
|
|
scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024)
|
|
for scanner.Scan() {
|
|
var line traceLine
|
|
if err := json.Unmarshal(scanner.Bytes(), &line); err != nil {
|
|
t.Fatalf("trace line decode: %v", err)
|
|
}
|
|
if len(line.Violations) == 0 {
|
|
continue
|
|
}
|
|
linesWithViolations++
|
|
if line.Step != 1 {
|
|
t.Errorf("step %d unexpectedly emitted violations %v (should be onset-only at step 1)",
|
|
line.Step, line.Violations)
|
|
}
|
|
}
|
|
if err := scanner.Err(); err != nil {
|
|
t.Fatalf("scan trace: %v", err)
|
|
}
|
|
if linesWithViolations != 1 {
|
|
t.Errorf("expected exactly 1 trace line with violations, got %d", linesWithViolations)
|
|
}
|
|
}
|
|
|
|
func TestRunner_NextViolationAttributedToCausingStep(t *testing.T) {
|
|
// always(next(p)): the obligation spawned at step 2 fails against step 3's
|
|
// state. The summary record and the trace witness must attribute the
|
|
// violation to step 2 (the causing step); the trace line that carries it is
|
|
// still step 3, where the failure was detected.
|
|
const nextViolationSpec = `
|
|
import { actions, always, next, extract } from "@sanderling/spec";
|
|
let observed = 0;
|
|
const tick = extract(() => ++observed);
|
|
globalThis.properties = {
|
|
nextHolds: always(next(() => tick.current < 3)),
|
|
};
|
|
globalThis.actions = actions(() => []);
|
|
`
|
|
state := newHarnessWithSpec(t, nextViolationSpec)
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: time.Hour,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
MaxSteps: 5,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if len(summary.Violations) != 1 {
|
|
t.Fatalf("expected exactly one ViolationRecord, got %d: %v",
|
|
len(summary.Violations), summary.Violations)
|
|
}
|
|
if summary.Violations[0].StepIndex != 2 {
|
|
t.Errorf("summary step: got %d, want 2 (the step that spawned the next obligation)",
|
|
summary.Violations[0].StepIndex)
|
|
}
|
|
if !slices.Equal(summary.Violations[0].Properties, []string{"nextHolds"}) {
|
|
t.Errorf("properties: got %v, want [nextHolds]", summary.Violations[0].Properties)
|
|
}
|
|
|
|
file, err := os.Open(filepath.Join(state.writer.Directory(), "trace.jsonl"))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
defer file.Close()
|
|
|
|
type traceLine struct {
|
|
Step int `json:"step"`
|
|
Violations []string `json:"violations"`
|
|
Witnesses map[string]trace.Witness `json:"witnesses"`
|
|
}
|
|
found := false
|
|
scanner := bufio.NewScanner(file)
|
|
scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024)
|
|
for scanner.Scan() {
|
|
var line traceLine
|
|
if err := json.Unmarshal(scanner.Bytes(), &line); err != nil {
|
|
t.Fatalf("trace line decode: %v", err)
|
|
}
|
|
if len(line.Violations) == 0 {
|
|
continue
|
|
}
|
|
found = true
|
|
if line.Step != 3 {
|
|
t.Errorf("violation detected on step %d, want 3", line.Step)
|
|
}
|
|
witness, ok := line.Witnesses["nextHolds"]
|
|
if !ok {
|
|
t.Fatalf("step %d carries no witness for nextHolds", line.Step)
|
|
}
|
|
if witness.Step != 2 {
|
|
t.Errorf("witness step: got %d, want 2 (causing step)", witness.Step)
|
|
}
|
|
// The two indices the witness spans are recorded separately: the
|
|
// extractor snapshot it carries is step 3's state, not step 2's.
|
|
if witness.DetectedStep != 3 {
|
|
t.Errorf("witness detected step: got %d, want 3", witness.DetectedStep)
|
|
}
|
|
}
|
|
if err := scanner.Err(); err != nil {
|
|
t.Fatalf("scan trace: %v", err)
|
|
}
|
|
if !found {
|
|
t.Error("no trace line carried the violation")
|
|
}
|
|
}
|
|
|
|
func TestRunner_AlwaysNextLeavesNoEndOfRunViolation(t *testing.T) {
|
|
// always(next(p)) ends every run with a pending deferred check. That
|
|
// residue is vacuous (no successor state to check), so neither the
|
|
// summary nor the trace may report an end-of-run violation.
|
|
const spec = `
|
|
import { actions, always, next } from "@sanderling/spec";
|
|
globalThis.properties = {
|
|
nextHolds: always(next(() => true)),
|
|
};
|
|
globalThis.actions = actions(() => []);
|
|
`
|
|
state := newHarnessWithSpec(t, spec)
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: time.Hour,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
MaxSteps: 3,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if len(summary.Violations) != 0 {
|
|
t.Errorf("expected no violations, got %v", summary.Violations)
|
|
}
|
|
}
|
|
|
|
func TestRunner_FinalizeRecordUsesDistinctStepIndex(t *testing.T) {
|
|
// An eventually that never fires is reported at run end through a
|
|
// synthetic trace record. That record must carry its own step index so no
|
|
// two trace lines share one (duplicate indices made the replay UI select
|
|
// two rows at once).
|
|
const spec = `
|
|
import { actions, eventually } from "@sanderling/spec";
|
|
globalThis.properties = {
|
|
neverFires: eventually(() => false),
|
|
};
|
|
globalThis.actions = actions(() => []);
|
|
`
|
|
state := newHarnessWithSpec(t, spec)
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: time.Hour,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
MaxSteps: 3,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if !containsProperty(summary.Violations, "neverFires") {
|
|
t.Fatalf("expected neverFires in violations: %v", summary.Violations)
|
|
}
|
|
|
|
file, err := os.Open(filepath.Join(state.writer.Directory(), "trace.jsonl"))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
defer file.Close()
|
|
|
|
type traceLine struct {
|
|
Step int `json:"step"`
|
|
Violations []string `json:"violations"`
|
|
}
|
|
seen := map[int]bool{}
|
|
finalizeStep := 0
|
|
scanner := bufio.NewScanner(file)
|
|
scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024)
|
|
for scanner.Scan() {
|
|
var line traceLine
|
|
if err := json.Unmarshal(scanner.Bytes(), &line); err != nil {
|
|
t.Fatalf("trace line decode: %v", err)
|
|
}
|
|
if seen[line.Step] {
|
|
t.Errorf("duplicate step index %d in trace", line.Step)
|
|
}
|
|
seen[line.Step] = true
|
|
if slices.Contains(line.Violations, "neverFires") {
|
|
finalizeStep = line.Step
|
|
}
|
|
}
|
|
if err := scanner.Err(); err != nil {
|
|
t.Fatalf("scan trace: %v", err)
|
|
}
|
|
if finalizeStep != summary.Steps+1 {
|
|
t.Errorf("finalize record step = %d, want %d (steps+1)", finalizeStep, summary.Steps+1)
|
|
}
|
|
}
|
|
|
|
func TestRunner_ThrowingPredicateIsLoggedNotPanic(t *testing.T) {
|
|
const throwingSpec = `
|
|
import { actions, always, Tap } from "@sanderling/spec";
|
|
globalThis.properties = {
|
|
broken: always(() => { throw new Error("bad predicate"); }),
|
|
};
|
|
globalThis.actions = actions(() => [Tap({ on: "id:next" })]);
|
|
`
|
|
state := newHarnessWithSpec(t, throwingSpec)
|
|
|
|
var buffer bytes.Buffer
|
|
logger := slog.New(slog.NewTextHandler(&buffer, &slog.HandlerOptions{Level: slog.LevelWarn}))
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 50 * time.Millisecond,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
Logger: logger,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if !containsProperty(summary.Violations, "broken") {
|
|
t.Errorf("expected broken in violations: %v", summary.Violations)
|
|
}
|
|
if !strings.Contains(buffer.String(), "bad predicate") {
|
|
t.Errorf("expected predicate error in log, got %q", buffer.String())
|
|
}
|
|
}
|
|
|
|
func TestRunner_RejectsMissingFields(t *testing.T) {
|
|
_, err := Run(context.Background(), Options{Duration: time.Second})
|
|
if err == nil || !strings.Contains(err.Error(), "Driver") {
|
|
t.Errorf("expected Driver-required error, got %v", err)
|
|
}
|
|
}
|
|
|
|
func TestRunner_RejectsZeroDuration(t *testing.T) {
|
|
_, err := Run(context.Background(), Options{
|
|
Driver: mockdriver.New(),
|
|
Verifier: mustNewVerifier(t),
|
|
TraceWriter: mustNewTraceWriter(t),
|
|
})
|
|
if err == nil || !strings.Contains(err.Error(), "Duration") {
|
|
t.Errorf("expected Duration-required error, got %v", err)
|
|
}
|
|
}
|
|
|
|
func TestRunner_StampsHierarchyResolvedBoundsAndResiduals(t *testing.T) {
|
|
state := newHarness(t)
|
|
state.mock.HierarchyJSON = `{"attributes":{"resource-id":"com.fixture:id/next","bounds":"[40,80,240,160]"},"children":[],"clickable":true,"enabled":true}`
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
if _, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 50 * time.Millisecond,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
}); err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
body, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
text := string(body)
|
|
if !strings.Contains(text, `"selector":"id:next"`) {
|
|
t.Errorf("expected selector in trace: %s", text)
|
|
}
|
|
if !strings.Contains(text, `"resolved_bounds":{"x":40,"y":80,"width":200,"height":80}`) {
|
|
t.Errorf("expected resolved_bounds in trace: %s", text)
|
|
}
|
|
if !strings.Contains(text, `"tap_point":{"x":140,"y":120}`) {
|
|
t.Errorf("expected tap_point in trace: %s", text)
|
|
}
|
|
if !strings.Contains(text, `"hierarchy":{"elements":`) {
|
|
t.Errorf("expected hierarchy in trace: %s", text)
|
|
}
|
|
if !strings.Contains(text, `"residuals":{`) {
|
|
t.Errorf("expected residuals in trace: %s", text)
|
|
}
|
|
}
|
|
|
|
// TestTraceActionFor_StaleCoordinatesDoNotOverrideTreeCenter pins the stamp
|
|
// to applyAction's resolution rule: when On resolves in the tree, the trace
|
|
// tap point must be the tree center even if the action carries stale X/Y
|
|
// from an earlier tick.
|
|
func TestTraceActionFor_StaleCoordinatesDoNotOverrideTreeCenter(t *testing.T) {
|
|
tree, err := hierarchy.Parse(`{"attributes":{"resource-id":"root","bounds":"[0,0,1080,2340]"},"children":[
|
|
{"attributes":{"resource-id":"next","bounds":"[100,200,300,400]"},"children":[]}
|
|
]}`)
|
|
if err != nil {
|
|
t.Fatalf("Parse: %v", err)
|
|
}
|
|
action := verifier.Action{Kind: verifier.ActionKindTap, On: "id:next", X: 50, Y: 60}
|
|
|
|
traceAction := traceActionFor(action, tree)
|
|
if traceAction.TapPoint == nil {
|
|
t.Fatal("expected a tap point")
|
|
}
|
|
if traceAction.TapPoint.X != 200 || traceAction.TapPoint.Y != 300 {
|
|
t.Errorf("tap point = (%d,%d), want tree center (200,300)",
|
|
traceAction.TapPoint.X, traceAction.TapPoint.Y)
|
|
}
|
|
if traceAction.ResolvedBounds == nil {
|
|
t.Fatal("expected resolved bounds")
|
|
}
|
|
if traceAction.ResolvedBounds.X != 100 || traceAction.ResolvedBounds.Y != 200 {
|
|
t.Errorf("resolved bounds origin = (%d,%d), want (100,200)",
|
|
traceAction.ResolvedBounds.X, traceAction.ResolvedBounds.Y)
|
|
}
|
|
}
|
|
|
|
// TestTraceActionFor_RecordsKindSpecificFields locks each action kind's trace
|
|
// encoding. PressKey must carry its Key and Wait its DurationMillis; if either
|
|
// branch of traceActionFor drops the field (or a field rename desyncs from the
|
|
// trace.Action struct) the replay UI silently renders a key-less PressKey or a
|
|
// zero-duration Wait. Swipe's endpoint encoding is covered separately via
|
|
// applyAction (TestApplyAction_ScrollWithPrecomputedEndpointsSwipes).
|
|
func TestTraceActionFor_RecordsKindSpecificFields(t *testing.T) {
|
|
cases := []struct {
|
|
name string
|
|
action verifier.Action
|
|
check func(*testing.T, *trace.Action)
|
|
}{
|
|
{
|
|
"PressKey records key",
|
|
verifier.Action{Kind: verifier.ActionKindPressKey, Key: "back"},
|
|
func(t *testing.T, a *trace.Action) {
|
|
if a.Key != "back" {
|
|
t.Errorf("Key = %q, want %q", a.Key, "back")
|
|
}
|
|
},
|
|
},
|
|
{
|
|
"Wait records duration",
|
|
verifier.Action{Kind: verifier.ActionKindWait, DurationMillis: 250},
|
|
func(t *testing.T, a *trace.Action) {
|
|
if a.DurationMillis != 250 {
|
|
t.Errorf("DurationMillis = %d, want 250", a.DurationMillis)
|
|
}
|
|
},
|
|
},
|
|
}
|
|
for _, testCase := range cases {
|
|
t.Run(testCase.name, func(t *testing.T) {
|
|
traceAction := traceActionFor(testCase.action, nil)
|
|
if traceAction.Kind != string(testCase.action.Kind) {
|
|
t.Errorf("Kind = %q, want %q", traceAction.Kind, testCase.action.Kind)
|
|
}
|
|
testCase.check(t, traceAction)
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestRunner_LogsWaitForIdleDriverErrors(t *testing.T) {
|
|
state := newHarness(t)
|
|
state.mock.Failures[mockdriver.ActionWaitForIdle] = errors.New("sidecar lost gRPC stream")
|
|
|
|
var logBuf bytes.Buffer
|
|
logger := slog.New(slog.NewTextHandler(&logBuf, &slog.HandlerOptions{Level: slog.LevelWarn}))
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
if _, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 50 * time.Millisecond,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
Logger: logger,
|
|
}); err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
output := logBuf.String()
|
|
if !strings.Contains(output, "wait_for_idle failed") {
|
|
t.Errorf("expected wait_for_idle warning, got: %q", output)
|
|
}
|
|
if !strings.Contains(output, "sidecar lost gRPC stream") {
|
|
t.Errorf("expected driver error message in warning, got: %q", output)
|
|
}
|
|
}
|
|
|
|
func TestApplyAction_InputTextErasesExistingTextBeforeTyping(t *testing.T) {
|
|
fastFocusSettle(t)
|
|
tree, err := hierarchy.Parse(`{"attributes":{"resource-id":"root","bounds":"[0,0,1080,2340]"},"children":[
|
|
{"attributes":{"resource-id":"username","text":"stale-value","bounds":"[10,10,500,100]"},"children":[]}
|
|
]}`)
|
|
if err != nil {
|
|
t.Fatalf("Parse: %v", err)
|
|
}
|
|
driverMock := mockdriver.New()
|
|
action := verifier.Action{Kind: verifier.ActionKindInputText, On: "id:username", Text: "alice"}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
|
|
t.Fatalf("applyAction: %v", err)
|
|
}
|
|
// The post-tap settle is now a brief internal sleep, not a WaitForIdle RPC,
|
|
// so the recorded driver actions are tap, erase, input.
|
|
actions := driverMock.Actions()
|
|
if len(actions) != 3 {
|
|
t.Fatalf("want tap, erase, input; got %v", actions)
|
|
}
|
|
if actions[0].Kind != mockdriver.ActionTap {
|
|
t.Errorf("first action = %v, want tap", actions[0].Kind)
|
|
}
|
|
if actions[1].Kind != mockdriver.ActionEraseText || actions[1].CharacterCount != len("stale-value") {
|
|
t.Errorf("second action = %+v, want erase_text of %d characters", actions[1], len("stale-value"))
|
|
}
|
|
if actions[2].Kind != mockdriver.ActionInputText || actions[2].Text != "alice" {
|
|
t.Errorf("third action = %+v, want input_text alice", actions[2])
|
|
}
|
|
}
|
|
|
|
// TestApplyAction_InputTextWithoutTargetSkipsFocusTap pins that with no
|
|
// resolvable target there is no focus tap (and so no settle), and InputText
|
|
// still runs at the cursor.
|
|
func TestApplyAction_InputTextWithoutTargetSkipsFocusTap(t *testing.T) {
|
|
driverMock := mockdriver.New()
|
|
action := verifier.Action{Kind: verifier.ActionKindInputText, X: -1, Y: -1, Text: "alice"}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
|
|
t.Fatalf("applyAction: %v", err)
|
|
}
|
|
actions := driverMock.Actions()
|
|
if len(actions) != 1 || actions[0].Kind != mockdriver.ActionInputText {
|
|
t.Errorf("no target: want input_text only (no focus tap), got %v", actions)
|
|
}
|
|
}
|
|
|
|
// TestApplyAction_InputTextSkipsEraseForReplacingDriver pins that a driver
|
|
// asserting the TextReplacer capability never pays the pre-erase round-trip:
|
|
// its InputText already replaces the field's content.
|
|
func TestApplyAction_InputTextSkipsEraseForReplacingDriver(t *testing.T) {
|
|
fastFocusSettle(t)
|
|
tree, err := hierarchy.Parse(`{"attributes":{"resource-id":"root","bounds":"[0,0,1080,2340]"},"children":[
|
|
{"attributes":{"resource-id":"username","text":"stale-value","bounds":"[10,10,500,100]"},"children":[]}
|
|
]}`)
|
|
if err != nil {
|
|
t.Fatalf("Parse: %v", err)
|
|
}
|
|
driverMock := mockdriver.New()
|
|
driverMock.ReplacesText = true
|
|
action := verifier.Action{Kind: verifier.ActionKindInputText, On: "id:username", Text: "alice"}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
|
|
t.Fatalf("applyAction: %v", err)
|
|
}
|
|
if containsAction(driverMock.Actions(), mockdriver.ActionEraseText, "") {
|
|
t.Errorf("replacing driver must not be asked to erase: %v", driverMock.Actions())
|
|
}
|
|
if !containsAction(driverMock.Actions(), mockdriver.ActionInputText, "") {
|
|
t.Errorf("expected InputText, got %v", driverMock.Actions())
|
|
}
|
|
}
|
|
|
|
func TestApplyAction_InputTextSkipsEraseWhenTargetEmpty(t *testing.T) {
|
|
fastFocusSettle(t)
|
|
tree, err := hierarchy.Parse(`{"attributes":{"resource-id":"root","bounds":"[0,0,1080,2340]"},"children":[
|
|
{"attributes":{"resource-id":"username","bounds":"[10,10,500,100]"},"children":[]}
|
|
]}`)
|
|
if err != nil {
|
|
t.Fatalf("Parse: %v", err)
|
|
}
|
|
driverMock := mockdriver.New()
|
|
action := verifier.Action{Kind: verifier.ActionKindInputText, On: "id:username", Text: "alice"}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
|
|
t.Fatalf("applyAction: %v", err)
|
|
}
|
|
if containsAction(driverMock.Actions(), mockdriver.ActionEraseText, "") {
|
|
t.Errorf("empty field must not be erased: %v", driverMock.Actions())
|
|
}
|
|
}
|
|
|
|
func TestApplyAction_InputTextSurfacesFocusTapError(t *testing.T) {
|
|
t.Run("selector focus tap fails", func(t *testing.T) {
|
|
driverMock := mockdriver.New()
|
|
driverMock.Failures[mockdriver.ActionTapSelector] = errors.New("adb unreachable")
|
|
action := verifier.Action{Kind: verifier.ActionKindInputText, On: "id:username", Text: "alice"}
|
|
|
|
err := applyAction(context.Background(), driverMock, action, nil)
|
|
if err == nil {
|
|
t.Fatalf("expected focus tap failure to surface, got nil")
|
|
}
|
|
if containsAction(driverMock.Actions(), mockdriver.ActionInputText, "") {
|
|
t.Errorf("InputText must not run after focus tap failed: %v", driverMock.Actions())
|
|
}
|
|
})
|
|
t.Run("coordinate focus tap fails", func(t *testing.T) {
|
|
driverMock := mockdriver.New()
|
|
driverMock.Failures[mockdriver.ActionTap] = errors.New("tap driver error")
|
|
action := verifier.Action{Kind: verifier.ActionKindInputText, X: 10, Y: 20, Text: "alice"}
|
|
|
|
err := applyAction(context.Background(), driverMock, action, nil)
|
|
if err == nil {
|
|
t.Fatalf("expected focus tap failure to surface, got nil")
|
|
}
|
|
if containsAction(driverMock.Actions(), mockdriver.ActionInputText, "") {
|
|
t.Errorf("InputText must not run after focus tap failed: %v", driverMock.Actions())
|
|
}
|
|
})
|
|
}
|
|
|
|
func TestApplyAction_V8InputTextTapsAtCoordinates(t *testing.T) {
|
|
fastFocusSettle(t)
|
|
driverMock := mockdriver.New()
|
|
action := verifier.Action{Kind: verifier.ActionKindInputText, X: 50, Y: 100, Text: "alice"}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
|
|
t.Fatalf("apply action: %v", err)
|
|
}
|
|
actions := driverMock.Actions()
|
|
if !containsAction(actions, mockdriver.ActionTap, "") {
|
|
t.Errorf("expected focus Tap before InputText, got %v", actions)
|
|
}
|
|
if !containsAction(actions, mockdriver.ActionInputText, "") {
|
|
t.Errorf("expected InputText after focus Tap, got %v", actions)
|
|
}
|
|
}
|
|
|
|
func TestApplyAction_V8InputTextAtOriginStillTaps(t *testing.T) {
|
|
fastFocusSettle(t)
|
|
driverMock := mockdriver.New()
|
|
// V8 emits real (0,0) coordinates for an element at viewport top-left
|
|
// (post-#15 the runtime nullifies unresolved actions, so a non-null
|
|
// InputText with (0,0) is a deliberate edge tap, not a sentinel).
|
|
action := verifier.Action{Kind: verifier.ActionKindInputText, X: 0, Y: 0, Text: "alice"}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
|
|
t.Fatalf("apply action: %v", err)
|
|
}
|
|
if !containsAction(driverMock.Actions(), mockdriver.ActionTap, "") {
|
|
t.Errorf("expected focus Tap at (0,0), got %v", driverMock.Actions())
|
|
}
|
|
}
|
|
|
|
func TestApplyAction_DoubleTapDispatchesDoubleTapAtCoordinates(t *testing.T) {
|
|
driverMock := mockdriver.New()
|
|
action := verifier.Action{Kind: verifier.ActionKindDoubleTap, X: 100, Y: 200}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
|
|
t.Fatalf("apply action: %v", err)
|
|
}
|
|
taps := 0
|
|
for _, a := range driverMock.Actions() {
|
|
if a.Kind == mockdriver.ActionDoubleTap && a.X == 100 && a.Y == 200 {
|
|
taps++
|
|
}
|
|
}
|
|
if taps != 1 {
|
|
t.Errorf("expected 1 DoubleTap call at (100,200), got %d in %v", taps, driverMock.Actions())
|
|
}
|
|
}
|
|
|
|
func TestApplyAction_DoubleTapDispatchesDoubleTapSelector(t *testing.T) {
|
|
driverMock := mockdriver.New()
|
|
action := verifier.Action{Kind: verifier.ActionKindDoubleTap, On: "id:save"}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
|
|
t.Fatalf("apply action: %v", err)
|
|
}
|
|
taps := 0
|
|
for _, a := range driverMock.Actions() {
|
|
if a.Kind == mockdriver.ActionDoubleTapSelector && a.Selector == "id:save" {
|
|
taps++
|
|
}
|
|
}
|
|
if taps != 1 {
|
|
t.Errorf("expected 1 DoubleTapSelector call with id:save, got %d in %v", taps, driverMock.Actions())
|
|
}
|
|
}
|
|
|
|
func TestApplyAction_LongPressDispatchesAtResolvedCoordinates(t *testing.T) {
|
|
driverMock := mockdriver.New()
|
|
action := verifier.Action{Kind: verifier.ActionKindLongPress, X: 120, Y: 240}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
|
|
t.Fatalf("apply action: %v", err)
|
|
}
|
|
found := false
|
|
for _, a := range driverMock.Actions() {
|
|
if a.Kind == mockdriver.ActionLongPress && a.X == 120 && a.Y == 240 {
|
|
found = true
|
|
}
|
|
}
|
|
if !found {
|
|
t.Errorf("expected LongPress at (120,240), got %v", driverMock.Actions())
|
|
}
|
|
}
|
|
|
|
func TestApplyAction_ScrollWithPrecomputedEndpointsSwipes(t *testing.T) {
|
|
driverMock := mockdriver.New()
|
|
action := verifier.Action{
|
|
Kind: verifier.ActionKindScroll,
|
|
Direction: "down",
|
|
FromX: 100,
|
|
FromY: 500,
|
|
ToX: 100,
|
|
ToY: 300,
|
|
DurationMillis: 300,
|
|
}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
|
|
t.Fatalf("apply action: %v", err)
|
|
}
|
|
found := false
|
|
for _, a := range driverMock.Actions() {
|
|
if a.Kind == mockdriver.ActionSwipe && a.FromX == 100 && a.FromY == 500 && a.ToX == 100 && a.ToY == 300 {
|
|
found = true
|
|
}
|
|
}
|
|
if !found {
|
|
t.Errorf("expected Swipe with precomputed endpoints, got %v", driverMock.Actions())
|
|
}
|
|
}
|
|
|
|
func TestApplyAction_ScrollDirectionUsesInversion(t *testing.T) {
|
|
driverMock := mockdriver.New()
|
|
treeJSON := `{"attributes":{"resource-id":"com.fixture:id/list","bounds":"[0,0,400,800]"},"children":[],"enabled":true}`
|
|
tree, err := hierarchy.Parse(treeJSON)
|
|
if err != nil {
|
|
t.Fatalf("parse tree: %v", err)
|
|
}
|
|
action := verifier.Action{Kind: verifier.ActionKindScroll, Direction: "down", On: "id:list"}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
|
|
t.Fatalf("apply action: %v", err)
|
|
}
|
|
var swipe *mockdriver.Action
|
|
for i := range driverMock.Actions() {
|
|
if driverMock.Actions()[i].Kind == mockdriver.ActionSwipe {
|
|
a := driverMock.Actions()[i]
|
|
swipe = &a
|
|
}
|
|
}
|
|
if swipe == nil {
|
|
t.Fatalf("expected a Swipe, got %v", driverMock.Actions())
|
|
}
|
|
// "down" reveals lower content by dragging the finger up, so toY < fromY.
|
|
if swipe.ToY >= swipe.FromY {
|
|
t.Errorf("expected toY < fromY for scroll down, got from=%d to=%d", swipe.FromY, swipe.ToY)
|
|
}
|
|
}
|
|
|
|
func TestApplyAction_ScrollNearTopKeepsDirectionAfterClamp(t *testing.T) {
|
|
driverMock := mockdriver.New()
|
|
// Full-screen root sets the 1080x2400 screen (marginY=200); the scrollable
|
|
// list sits inside the top margin (y 20..180), where the clamp must fire.
|
|
treeJSON := `{"attributes":{"bounds":"[0,0,1080,2400]"},"children":[
|
|
{"attributes":{"resource-id":"com.fixture:id/toplist","scrollable":"true","bounds":"[0,20,1080,180]"},"children":[],"enabled":true}
|
|
]}`
|
|
tree, err := hierarchy.Parse(treeJSON)
|
|
if err != nil {
|
|
t.Fatalf("parse tree: %v", err)
|
|
}
|
|
action := verifier.Action{Kind: verifier.ActionKindScroll, Direction: "up", On: "id:toplist"}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
|
|
t.Fatalf("apply action: %v", err)
|
|
}
|
|
var swipe *mockdriver.Action
|
|
for i := range driverMock.Actions() {
|
|
if driverMock.Actions()[i].Kind == mockdriver.ActionSwipe {
|
|
a := driverMock.Actions()[i]
|
|
swipe = &a
|
|
}
|
|
}
|
|
if swipe == nil {
|
|
t.Fatalf("expected a Swipe, got %v", driverMock.Actions())
|
|
}
|
|
if swipe.FromY != 200 {
|
|
t.Errorf("origin not pushed below the shade strip, got fromY=%d want 200", swipe.FromY)
|
|
}
|
|
if swipe.ToY <= swipe.FromY {
|
|
t.Errorf("scroll up reversed by the clamp: from=%d to=%d (want toY > fromY)", swipe.FromY, swipe.ToY)
|
|
}
|
|
}
|
|
|
|
func TestApplyAction_ScrollScreenFallback(t *testing.T) {
|
|
driverMock := mockdriver.New()
|
|
treeJSON := `{"attributes":{"bounds":"[0,0,400,800]"},"children":[],"enabled":true}`
|
|
tree, err := hierarchy.Parse(treeJSON)
|
|
if err != nil {
|
|
t.Fatalf("parse tree: %v", err)
|
|
}
|
|
// On unset: container falls back to whole-screen (root) bounds.
|
|
action := verifier.Action{Kind: verifier.ActionKindScroll, Direction: "up"}
|
|
|
|
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
|
|
t.Fatalf("apply action: %v", err)
|
|
}
|
|
var swipe *mockdriver.Action
|
|
for i := range driverMock.Actions() {
|
|
if driverMock.Actions()[i].Kind == mockdriver.ActionSwipe {
|
|
a := driverMock.Actions()[i]
|
|
swipe = &a
|
|
}
|
|
}
|
|
if swipe == nil {
|
|
t.Fatalf("expected a Swipe, got %v", driverMock.Actions())
|
|
}
|
|
if swipe.FromX != 200 || swipe.FromY != 400 {
|
|
t.Errorf("expected swipe to start at screen center (200,400), got (%d,%d)", swipe.FromX, swipe.FromY)
|
|
}
|
|
// "up" reveals upper content by dragging the finger down, so toY > fromY.
|
|
if swipe.ToY <= swipe.FromY {
|
|
t.Errorf("expected toY > fromY for scroll up, got from=%d to=%d", swipe.FromY, swipe.ToY)
|
|
}
|
|
}
|
|
|
|
func TestRunner_ParallelFetchCallsAllDriverMethods(t *testing.T) {
|
|
state := newHarness(t)
|
|
state.mock.MetricsData = driver.Metrics{CPUPercent: 5.0, HeapBytes: 1024, TotalMemoryBytes: 4096}
|
|
state.mock.LogEntries = []driver.LogEntry{
|
|
{UnixMillis: 1000, Level: "E", Tag: "test", Message: "boom"},
|
|
}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
_, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 50 * time.Millisecond,
|
|
BundleID: "com.fixture",
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
|
|
actions := state.mock.Actions()
|
|
var hasSnapshot, hasMetrics, hasLogs bool
|
|
for _, a := range actions {
|
|
switch a.Kind {
|
|
case mockdriver.ActionSnapshot:
|
|
hasSnapshot = true
|
|
case mockdriver.ActionMetrics:
|
|
hasMetrics = true
|
|
case mockdriver.ActionRecentLogs:
|
|
hasLogs = true
|
|
}
|
|
}
|
|
if !hasSnapshot {
|
|
t.Error("expected Snapshot call in mock actions")
|
|
}
|
|
if !hasMetrics {
|
|
t.Error("expected Metrics call in mock actions")
|
|
}
|
|
if !hasLogs {
|
|
t.Error("expected RecentLogs call in mock actions")
|
|
}
|
|
}
|
|
|
|
// TestRunner_UsesAtomicSnapshot ensures the runner observes a step's UI
|
|
// through the paired Snapshot RPC instead of racing two independent
|
|
// hierarchy + screenshot reads. The pair must come from one on-device
|
|
// frame; a regression to separate calls is what this test catches.
|
|
func TestRunner_UsesAtomicSnapshot(t *testing.T) {
|
|
state := newHarness(t)
|
|
state.mock.ImageData = driver.Image{PNG: []byte("png"), Width: 1, Height: 1}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if summary.Steps == 0 {
|
|
t.Fatal("expected at least one step")
|
|
}
|
|
|
|
var snapshotCalls, hierarchyCalls, screenshotCalls int
|
|
for _, action := range state.mock.Actions() {
|
|
switch action.Kind {
|
|
case mockdriver.ActionSnapshot:
|
|
snapshotCalls++
|
|
case mockdriver.ActionHierarchy:
|
|
hierarchyCalls++
|
|
case mockdriver.ActionScreenshot:
|
|
screenshotCalls++
|
|
}
|
|
}
|
|
if snapshotCalls == 0 {
|
|
t.Errorf("expected at least one Snapshot call, got %d", snapshotCalls)
|
|
}
|
|
// The recorded pair still comes from Snapshot. The standalone hierarchy
|
|
// reads are the composition detector (changedOnReread), one per step at
|
|
// most, and they are never the source of what the step records.
|
|
if hierarchyCalls > summary.Steps {
|
|
t.Errorf("expected at most one standalone Hierarchy call per step (runner must observe through Snapshot), got %d over %d steps",
|
|
hierarchyCalls, summary.Steps)
|
|
}
|
|
if snapshotCalls < summary.Steps {
|
|
t.Errorf("expected a Snapshot per step, got %d over %d steps", snapshotCalls, summary.Steps)
|
|
}
|
|
if screenshotCalls != 0 {
|
|
t.Errorf("expected zero standalone Screenshot calls (runner must use Snapshot), got %d", screenshotCalls)
|
|
}
|
|
}
|
|
|
|
// TestRunner_OneScreenshotPerStep verifies the runner writes a single
|
|
// screenshot per step, captured concurrently with hierarchy so the two
|
|
// observations describe the same UI moment.
|
|
func TestRunner_OneScreenshotPerStep(t *testing.T) {
|
|
state := newHarness(t)
|
|
state.mock.ImageData = driver.Image{PNG: []byte("fakepng"), Width: 100, Height: 200}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 200 * time.Millisecond,
|
|
IdleTimeout: 50 * time.Millisecond,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if summary.Steps < 2 {
|
|
t.Fatalf("need at least 2 steps for screenshot test, got %d", summary.Steps)
|
|
}
|
|
|
|
screenshotDir := filepath.Join(state.writer.Directory(), "screenshots")
|
|
for step := 1; step <= summary.Steps; step++ {
|
|
path := filepath.Join(screenshotDir, fmt.Sprintf("step-%05d.png", step))
|
|
if _, err := os.Stat(path); os.IsNotExist(err) {
|
|
t.Errorf("expected screenshot for step %d at %s", step, path)
|
|
}
|
|
}
|
|
|
|
entries, err := os.ReadDir(screenshotDir)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
for _, entry := range entries {
|
|
if strings.Contains(entry.Name(), "-after") {
|
|
t.Errorf("unexpected -after screenshot remains: %s", entry.Name())
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestRunner_StableTransitionalTreeIsVerified feeds a driver whose hierarchy
|
|
// constantly carries two route-level *Screen ids but never changes between
|
|
// retry attempts. Such a tree is a settled state that merely matches the
|
|
// transitional heuristic, so the runner must verify it (the always-false
|
|
// predicate's violation surfaces) instead of skipping the verifier forever.
|
|
func TestRunner_StableTransitionalTreeIsVerified(t *testing.T) {
|
|
state := newHarnessWithSpec(t, violationSpec)
|
|
state.mock.HierarchyJSON = `{"attributes":{"resource-id":"root"},"children":[
|
|
{"attributes":{"resource-id":"AddAccountScreen"},"children":[]},
|
|
{"attributes":{"resource-id":"HomeScreen"},"children":[]}
|
|
]}`
|
|
state.mock.ImageData = driver.Image{PNG: []byte("fakepng"), Width: 100, Height: 200}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 200 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if summary.Steps == 0 {
|
|
t.Fatal("expected at least one step")
|
|
}
|
|
if !containsProperty(summary.Violations, "balanceNonNegative") {
|
|
t.Fatalf("expected verifier to run on a stable two-screen tree, got %v", summary.Violations)
|
|
}
|
|
|
|
type traceLine struct {
|
|
Step int `json:"step"`
|
|
Transitional bool `json:"transitional"`
|
|
}
|
|
body, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
for _, raw := range bytes.Split(bytes.TrimSpace(body), []byte("\n")) {
|
|
var line traceLine
|
|
if err := json.Unmarshal(raw, &line); err != nil {
|
|
t.Fatalf("decode trace line: %v", err)
|
|
}
|
|
if line.Transitional {
|
|
t.Errorf("step %d: stable tree must not be marked transitional", line.Step)
|
|
}
|
|
}
|
|
|
|
// The early break must still persist the step's screenshot.
|
|
screenshotPath := filepath.Join(state.writer.Directory(), "screenshots", "step-00001.png")
|
|
if _, err := os.Stat(screenshotPath); err != nil {
|
|
t.Errorf("expected screenshot for step 1 at %s: %v", screenshotPath, err)
|
|
}
|
|
}
|
|
|
|
// snapshotCrossFade wraps a mock driver so every Snapshot call returns a
|
|
// transitional two-screen tree whose JSON differs from the previous call,
|
|
// mimicking a genuine cross-fade in flight.
|
|
type snapshotCrossFade struct {
|
|
*mockdriver.Driver
|
|
calls int
|
|
}
|
|
|
|
func (d *snapshotCrossFade) Snapshot(ctx context.Context) (string, driver.Image, error) {
|
|
d.calls++
|
|
_, image, err := d.Driver.Snapshot(ctx)
|
|
hierarchyJSON := fmt.Sprintf(`{"attributes":{"resource-id":"root"},"children":[
|
|
{"attributes":{"resource-id":"AddAccountScreen","text":"frame-%d"},"children":[]},
|
|
{"attributes":{"resource-id":"HomeScreen"},"children":[]}
|
|
]}`, d.calls)
|
|
return hierarchyJSON, image, err
|
|
}
|
|
|
|
// TestRunner_GenuineCrossFadeStillRetried pins the existing behavior for real
|
|
// transitions: a tree that keeps changing between retry attempts exhausts the
|
|
// budget, stays transitional, and the verifier is skipped for the step.
|
|
func TestRunner_GenuineCrossFadeStillRetried(t *testing.T) {
|
|
state := newHarnessWithSpec(t, violationSpec)
|
|
wrapped := &snapshotCrossFade{Driver: state.mock}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 200 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
Driver: wrapped,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if summary.Steps == 0 {
|
|
t.Fatal("expected at least one step")
|
|
}
|
|
if len(summary.Violations) != 0 {
|
|
t.Fatalf("verifier must be skipped on cross-fade steps; got %v", summary.Violations)
|
|
}
|
|
|
|
type traceLine struct {
|
|
Step int `json:"step"`
|
|
Transitional bool `json:"transitional"`
|
|
Violations []string `json:"violations"`
|
|
}
|
|
body, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
lines := 0
|
|
for _, raw := range bytes.Split(bytes.TrimSpace(body), []byte("\n")) {
|
|
var line traceLine
|
|
if err := json.Unmarshal(raw, &line); err != nil {
|
|
t.Fatalf("decode trace line: %v", err)
|
|
}
|
|
lines++
|
|
if !line.Transitional {
|
|
t.Errorf("step %d: expected transitional=true on every step, got false", line.Step)
|
|
}
|
|
if len(line.Violations) != 0 {
|
|
t.Errorf("step %d: verifier must be skipped, got violations %v", line.Step, line.Violations)
|
|
}
|
|
}
|
|
if lines == 0 {
|
|
t.Fatal("expected trace lines, got none")
|
|
}
|
|
}
|
|
|
|
// TestRunner_CleanTreeStillVerified is the control: a single-screen hierarchy
|
|
// must not be marked transitional and the verifier must still run, surfacing
|
|
// the always-false predicate's violation on the onset step.
|
|
func TestRunner_CleanTreeStillVerified(t *testing.T) {
|
|
state := newHarnessWithSpec(t, violationSpec)
|
|
state.mock.HierarchyJSON = `{"attributes":{"resource-id":"HomeScreen"},"children":[]}`
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 200 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if !containsProperty(summary.Violations, "balanceNonNegative") {
|
|
t.Fatalf("expected verifier to surface balanceNonNegative on a clean tree, got %v", summary.Violations)
|
|
}
|
|
|
|
type traceLine struct {
|
|
Step int `json:"step"`
|
|
Transitional bool `json:"transitional"`
|
|
}
|
|
body, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
for _, raw := range bytes.Split(bytes.TrimSpace(body), []byte("\n")) {
|
|
var line traceLine
|
|
if err := json.Unmarshal(raw, &line); err != nil {
|
|
t.Fatalf("decode trace line: %v", err)
|
|
}
|
|
if line.Transitional {
|
|
t.Errorf("step %d: clean tree must not be marked transitional", line.Step)
|
|
}
|
|
}
|
|
}
|
|
|
|
// snapshotFailFirst wraps a mock driver so the first Snapshot call returns an
|
|
// error (mimicking a sidecar timeout while fetching view hierarchy), then
|
|
// delegates every subsequent call back to the mock.
|
|
type snapshotFailFirst struct {
|
|
*mockdriver.Driver
|
|
calls int
|
|
}
|
|
|
|
func (d *snapshotFailFirst) Snapshot(ctx context.Context) (string, driver.Image, error) {
|
|
d.calls++
|
|
if d.calls == 1 {
|
|
return "", driver.Image{}, errors.New("Timeout while fetching view hierarchy")
|
|
}
|
|
return d.Driver.Snapshot(ctx)
|
|
}
|
|
|
|
// TestRunner_NilHierarchyMarksTransitional verifies that when the sidecar's
|
|
// hierarchy fetch fails (nil tree), the runner marks the step transitional and
|
|
// skips the verifier instead of pushing a nil tree that would crash the spec.
|
|
// Subsequent steps with a clean tree still drive the verifier normally.
|
|
func TestRunner_NilHierarchyMarksTransitional(t *testing.T) {
|
|
state := newHarnessWithSpec(t, violationSpec)
|
|
state.mock.HierarchyJSON = `{"attributes":{"resource-id":"HomeScreen"},"children":[]}`
|
|
wrapped := &snapshotFailFirst{Driver: state.mock}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 200 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
Driver: wrapped,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if summary.Steps < 2 {
|
|
t.Fatalf("need at least 2 steps to verify the first is skipped and the second runs, got %d", summary.Steps)
|
|
}
|
|
// violationSpec always() => false fires on the first verifier push. With
|
|
// step 1's verifier skipped, onset moves to step 2.
|
|
if len(summary.Violations) != 1 {
|
|
t.Fatalf("expected exactly one onset record, got %d: %v", len(summary.Violations), summary.Violations)
|
|
}
|
|
if summary.Violations[0].StepIndex != 2 {
|
|
t.Errorf("onset step: got %d, want 2 (step 1 verifier skipped due to nil tree)", summary.Violations[0].StepIndex)
|
|
}
|
|
|
|
type traceLine struct {
|
|
Step int `json:"step"`
|
|
Transitional bool `json:"transitional"`
|
|
Violations []string `json:"violations"`
|
|
}
|
|
body, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
var first traceLine
|
|
if err := json.Unmarshal(bytes.SplitN(bytes.TrimSpace(body), []byte("\n"), 2)[0], &first); err != nil {
|
|
t.Fatalf("decode first trace line: %v", err)
|
|
}
|
|
if first.Step != 1 {
|
|
t.Fatalf("first trace line step: got %d, want 1", first.Step)
|
|
}
|
|
if !first.Transitional {
|
|
t.Error("first step must be marked transitional when the hierarchy fetch failed")
|
|
}
|
|
if len(first.Violations) != 0 {
|
|
t.Errorf("step 1 must skip the verifier; got violations %v", first.Violations)
|
|
}
|
|
}
|
|
|
|
// tapSelectorFailFirst wraps a mock driver so the first TapSelector call
|
|
// returns a gRPC DeadlineExceeded error (mimicking a sidecar-side RPC hang),
|
|
// then delegates every subsequent call back to the mock.
|
|
type tapSelectorFailFirst struct {
|
|
*mockdriver.Driver
|
|
calls int
|
|
}
|
|
|
|
func (d *tapSelectorFailFirst) TapSelector(ctx context.Context, selector string) error {
|
|
d.calls++
|
|
if d.calls == 1 {
|
|
return status.Error(codes.DeadlineExceeded, "boom")
|
|
}
|
|
return d.Driver.TapSelector(ctx, selector)
|
|
}
|
|
|
|
// TestRunner_TransientApplyErrorMarksTransitional verifies that a transient
|
|
// gRPC error from applyAction (e.g. sidecar RPC deadline) does not kill the
|
|
// run: the step is marked transitional, the verifier is skipped for it, and
|
|
// the loop continues with the next step running cleanly.
|
|
func TestRunner_TransientApplyErrorMarksTransitional(t *testing.T) {
|
|
state := newHarness(t)
|
|
wrapped := &tapSelectorFailFirst{Driver: state.mock}
|
|
|
|
var logBuf bytes.Buffer
|
|
logger := slog.New(slog.NewTextHandler(&logBuf, &slog.HandlerOptions{Level: slog.LevelWarn}))
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 300 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
Driver: wrapped,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
Logger: logger,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run must not return on transient apply error, got %v", err)
|
|
}
|
|
if summary.Steps < 2 {
|
|
t.Fatalf("need at least 2 steps to prove the loop continued past the failed apply, got %d", summary.Steps)
|
|
}
|
|
if len(summary.Violations) != 0 {
|
|
t.Errorf("transient apply error must not surface as a violation, got %v", summary.Violations)
|
|
}
|
|
if !strings.Contains(logBuf.String(), "apply error; marking step transitional") {
|
|
t.Errorf("expected apply-error WARN log, got %q", logBuf.String())
|
|
}
|
|
|
|
type traceLine struct {
|
|
Step int `json:"step"`
|
|
Transitional bool `json:"transitional"`
|
|
Violations []string `json:"violations"`
|
|
}
|
|
body, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
lines := bytes.Split(bytes.TrimSpace(body), []byte("\n"))
|
|
var first, second traceLine
|
|
if err := json.Unmarshal(lines[0], &first); err != nil {
|
|
t.Fatalf("decode first trace line: %v", err)
|
|
}
|
|
if err := json.Unmarshal(lines[1], &second); err != nil {
|
|
t.Fatalf("decode second trace line: %v", err)
|
|
}
|
|
if first.Step != 1 || !first.Transitional {
|
|
t.Errorf("step 1 must be transitional after transient apply error, got step=%d transitional=%v", first.Step, first.Transitional)
|
|
}
|
|
if len(first.Violations) != 0 {
|
|
t.Errorf("transient apply step must have no violations, got %v", first.Violations)
|
|
}
|
|
if second.Step != 2 || second.Transitional {
|
|
t.Errorf("step 2 must run cleanly after the transient step, got step=%d transitional=%v", second.Step, second.Transitional)
|
|
}
|
|
}
|
|
|
|
// internalApplyErrorFailFirst wraps a mock driver so the first InputText call
|
|
// fails with the bare Internal error the iOS runner's input handler emits
|
|
// when it chokes (HTTP 500 with an empty body), then recovers.
|
|
type internalApplyErrorFailFirst struct {
|
|
*mockdriver.Driver
|
|
calls int
|
|
}
|
|
|
|
func (d *internalApplyErrorFailFirst) TapSelector(ctx context.Context, selector string) error {
|
|
d.calls++
|
|
if d.calls == 1 {
|
|
return status.Error(codes.Internal, "UnknownFailure(errorResponse=Request for inputText failed, code: 500, body: )")
|
|
}
|
|
return d.Driver.TapSelector(ctx, selector)
|
|
}
|
|
|
|
// TestRunner_InternalApplyErrorMarksTransitional pins the policy that a
|
|
// one-off device-side failure (e.g. the iOS input handler's bare 500) is
|
|
// absorbed as a transitional step instead of killing the run. Persistent
|
|
// failure is covered by the consecutive-failure cap.
|
|
func TestRunner_InternalApplyErrorMarksTransitional(t *testing.T) {
|
|
state := newHarness(t)
|
|
wrapped := &internalApplyErrorFailFirst{Driver: state.mock}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 300 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
Driver: wrapped,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run must not return on a one-off internal apply error, got %v", err)
|
|
}
|
|
if summary.Steps < 2 {
|
|
t.Fatalf("need at least 2 steps to prove the loop continued, got %d", summary.Steps)
|
|
}
|
|
}
|
|
|
|
// TestIsWDADrop_Classification pins the fatal-vs-recoverable boundary: only
|
|
// the sidecar's explicit reconnect-failure signal is a drop. A structured
|
|
// UNAVAILABLE that happens to embed raw exception text (e.g. ConnectException
|
|
// from the original failure) means the sidecar already recovered and the run
|
|
// must continue.
|
|
// TestIsWDADrop_Classification pins isWDADrop to the exact phrase the sidecar
|
|
// throws at its reconnect site (sidecar DriverBackend.kt):
|
|
//
|
|
// throw IllegalStateException("WDA reconnect failed: $restartErr", cause)
|
|
//
|
|
// If that message is reworded on the Kotlin side without updating the Go
|
|
// matcher, a fatal, unrecoverable WDA drop is misclassified as transient and
|
|
// the run burns its budget retrying a dead channel instead of aborting.
|
|
func TestIsWDADrop_Classification(t *testing.T) {
|
|
// sidecarReconnectFailedMessage mirrors the literal the sidecar emits; the
|
|
// matcher's contract is keyed on this exact prefix.
|
|
const sidecarReconnectFailedMessage = "WDA reconnect failed"
|
|
cases := []struct {
|
|
name string
|
|
err error
|
|
want bool
|
|
}{
|
|
{
|
|
"unavailable with embedded ConnectException is recovered",
|
|
status.Error(codes.Unavailable, "connection dropped mid-action; the action may have applied: java.net.ConnectException: Failed to connect to /127.0.0.1:22161"),
|
|
false,
|
|
},
|
|
{
|
|
"sidecar reconnect-failed message is a drop",
|
|
status.Error(codes.Internal, "java.lang.IllegalStateException: "+sidecarReconnectFailedMessage+": IOSDriverTimeoutException"),
|
|
true,
|
|
},
|
|
{"generic internal is not a drop", status.Error(codes.Internal, "boom"), false},
|
|
}
|
|
for _, testCase := range cases {
|
|
t.Run(testCase.name, func(t *testing.T) {
|
|
if got := isWDADrop(testCase.err); got != testCase.want {
|
|
t.Errorf("got %v, want %v", got, testCase.want)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// tapSelectorAlwaysUnavailable wraps a mock driver so every TapSelector call
|
|
// fails with a transient Unavailable error, mimicking a device whose channel
|
|
// never recovers between steps.
|
|
type tapSelectorAlwaysUnavailable struct {
|
|
*mockdriver.Driver
|
|
}
|
|
|
|
func (d *tapSelectorAlwaysUnavailable) TapSelector(ctx context.Context, selector string) error {
|
|
return status.Error(codes.Unavailable, "connection dropped mid-action; the action may have applied")
|
|
}
|
|
|
|
// TestRunner_ConsecutiveTransientApplyFailuresAbort verifies the run fails
|
|
// fast once transient apply errors form an unbroken streak instead of burning
|
|
// the whole budget on a wedged device.
|
|
func TestRunner_ConsecutiveTransientApplyFailuresAbort(t *testing.T) {
|
|
state := newHarness(t)
|
|
wrapped := &tapSelectorAlwaysUnavailable{Driver: state.mock}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 5 * time.Second,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
Driver: wrapped,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err == nil {
|
|
t.Fatal("Run must abort after consecutive transient apply failures")
|
|
}
|
|
if !strings.Contains(err.Error(), "consecutive failures") {
|
|
t.Errorf("expected consecutive-failure abort, got %v", err)
|
|
}
|
|
// The aborting step returns before it is recorded, so the summary holds
|
|
// the steps before the cap-hitting one.
|
|
if summary.Steps != maxConsecutiveApplyFailures-1 {
|
|
t.Errorf("run must stop at the failure cap, got %d recorded steps", summary.Steps)
|
|
}
|
|
}
|
|
|
|
// TestRunner_WaitActionSkipsIdle ensures the runner does not call WaitForIdle
|
|
// after a Wait action - the action already provides settling time.
|
|
func TestRunner_WaitActionSkipsIdle(t *testing.T) {
|
|
const waitSpec = `
|
|
import { actions, Wait } from "@sanderling/spec";
|
|
globalThis.actions = actions(() => [Wait({ durationMillis: 5 })]);
|
|
`
|
|
state := newHarnessWithSpec(t, waitSpec)
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
_, err := Run(ctx, Options{
|
|
Duration: 150 * time.Millisecond,
|
|
IdleTimeout: 50 * time.Millisecond,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
for _, action := range state.mock.Actions() {
|
|
if action.Kind == mockdriver.ActionWaitForIdle {
|
|
t.Fatalf("Wait action must skip WaitForIdle, got: %v", action)
|
|
}
|
|
}
|
|
}
|
|
|
|
func mustNewVerifier(t *testing.T) *verifier.Verifier {
|
|
t.Helper()
|
|
verifierInstance, err := verifier.New()
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
return verifierInstance
|
|
}
|
|
|
|
func mustNewTraceWriter(t *testing.T) *trace.Writer {
|
|
t.Helper()
|
|
writer, err := trace.NewWriter(t.TempDir())
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
t.Cleanup(func() { _ = writer.Close() })
|
|
return writer
|
|
}
|
|
|
|
func containsAction(actions []mockdriver.Action, kind mockdriver.ActionKind, payload string) bool {
|
|
for _, action := range actions {
|
|
if action.Kind != kind {
|
|
continue
|
|
}
|
|
switch kind {
|
|
case mockdriver.ActionLaunch:
|
|
if action.BundleID == payload {
|
|
return true
|
|
}
|
|
case mockdriver.ActionTapSelector:
|
|
if action.Selector == payload {
|
|
return true
|
|
}
|
|
case mockdriver.ActionTerminate:
|
|
return true
|
|
default:
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
func containsProperty(records []ViolationRecord, property string) bool {
|
|
for _, record := range records {
|
|
if slices.Contains(record.Properties, property) {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
func TestRunner_RelaunchesWhenAppLeavesForeground(t *testing.T) {
|
|
state := newHarness(t)
|
|
// Always report a foreign app, so every step's guard must relaunch.
|
|
state.mock.ForegroundResults = []string{"com.android.chrome"}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
_, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
BundleID: "app.folio",
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
relaunches := 0
|
|
for _, a := range state.mock.Actions() {
|
|
if a.Kind == mockdriver.ActionLaunch && a.BundleID == "app.folio" && !a.ClearState {
|
|
relaunches++
|
|
}
|
|
}
|
|
if relaunches == 0 {
|
|
t.Fatal("expected runner to relaunch app.folio when foreground escaped, got none")
|
|
}
|
|
}
|
|
|
|
func TestRunner_NoRelaunchWhenAppInForeground(t *testing.T) {
|
|
state := newHarness(t)
|
|
state.mock.ForegroundResults = []string{"app.folio"}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
_, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
BundleID: "app.folio",
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
for _, a := range state.mock.Actions() {
|
|
if a.Kind == mockdriver.ActionLaunch {
|
|
t.Fatalf("expected no relaunch while app in foreground, got %v", a)
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestRunner_WaitsForForegroundBeforeFirstAction verifies the startup gate
|
|
// brings the app forward (back-press + relaunch) before any tap fires when the
|
|
// device boots showing a system dialog.
|
|
func TestRunner_WaitsForForegroundBeforeFirstAction(t *testing.T) {
|
|
state := newHarness(t)
|
|
// First the device shows a system setup screen, then the app is on top.
|
|
state.mock.ForegroundResults = []string{"com.google.android.setupwizard", "app.folio"}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
_, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
BundleID: "app.folio",
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
|
|
actions := state.mock.Actions()
|
|
firstLaunch, firstTap := -1, -1
|
|
backPressed := false
|
|
for i, a := range actions {
|
|
switch {
|
|
case a.Kind == mockdriver.ActionLaunch && firstLaunch < 0:
|
|
firstLaunch = i
|
|
case a.Kind == mockdriver.ActionTap && firstTap < 0:
|
|
firstTap = i
|
|
case a.Kind == mockdriver.ActionPressKey && a.Key == "back":
|
|
backPressed = true
|
|
}
|
|
}
|
|
if firstLaunch < 0 {
|
|
t.Fatal("expected a relaunch to bring the app forward, got none")
|
|
}
|
|
if !backPressed {
|
|
t.Fatal("expected a back-press to dismiss the system dialog, got none")
|
|
}
|
|
if firstTap >= 0 && firstLaunch > firstTap {
|
|
t.Fatalf("expected the foreground gate (launch at %d) before the first tap (at %d)", firstLaunch, firstTap)
|
|
}
|
|
}
|
|
|
|
// TestRunner_WaitsForWindowDrawnBeforeFirstAction verifies the startup gate
|
|
// keeps waiting while the app is the resumed activity but its window has not
|
|
// drawn yet (a leftover screen still focused). It must poll the focused-window
|
|
// signal rather than relaunching, and only proceed once the window names the
|
|
// app.
|
|
func TestRunner_WaitsForWindowDrawnBeforeFirstAction(t *testing.T) {
|
|
state := newHarness(t)
|
|
// The app is resumed immediately, but its window lags: the outgoing
|
|
// settings screen stays focused for two checks before the app draws.
|
|
state.mock.ForegroundResults = []string{"app.folio"}
|
|
state.mock.FocusedWindowResults = []string{"com.android.settings", "com.android.settings", "app.folio"}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
_, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
BundleID: "app.folio",
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
|
|
// The gate must have polled the focused window until it named the app,
|
|
// i.e. at least the three queued results were consumed.
|
|
if calls := state.mock.FocusedWindowCalls(); calls < 3 {
|
|
t.Fatalf("expected the gate to poll the focused window until drawn (>=3 calls), got %d", calls)
|
|
}
|
|
// The resumed app was never a foreign app, so the gate must not relaunch
|
|
// or back-press to "fix" a window that simply had not drawn yet.
|
|
for _, a := range state.mock.Actions() {
|
|
if a.Kind == mockdriver.ActionPressKey && a.Key == "back" {
|
|
t.Fatal("expected no back-press while waiting for the window to draw")
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestAwaitForeground_RelaunchesThenWaitsForWindow locks the per-step scope
|
|
// guard's recovery: after the app leaves to the launcher, it must relaunch AND
|
|
// keep polling the focused window until it names the app, so the step never
|
|
// observes or acts while the launcher is on screen (where InputText would land
|
|
// in the launcher's type-to-search filter). A single fire-and-forget relaunch,
|
|
// which returns before the window draws on a slow physical device, is the bug
|
|
// this guards against.
|
|
func TestAwaitForeground_RelaunchesThenWaitsForWindow(t *testing.T) {
|
|
m := mockdriver.New()
|
|
// Foreground: launcher on the first poll (still gone), then the app. Focus:
|
|
// the launcher window lingers one extra poll before the app's window draws.
|
|
m.ForegroundResults = []string{"com.android.launcher", "app.folio"}
|
|
m.FocusedWindowResults = []string{"com.android.launcher", "app.folio"}
|
|
|
|
logger := slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{Level: slog.LevelWarn}))
|
|
options := Options{BundleID: "app.folio", Driver: m, IdleTimeout: 10 * time.Millisecond}
|
|
|
|
awaitForeground(context.Background(), options, logger, 7)
|
|
|
|
relaunches, backs := 0, 0
|
|
for _, a := range m.Actions() {
|
|
switch {
|
|
case a.Kind == mockdriver.ActionLaunch && a.BundleID == "app.folio" && !a.ClearState:
|
|
relaunches++
|
|
case a.Kind == mockdriver.ActionPressKey && a.Key == "back":
|
|
backs++
|
|
}
|
|
}
|
|
if relaunches != 1 {
|
|
t.Fatalf("expected exactly one relaunch while the app was gone, got %d", relaunches)
|
|
}
|
|
if backs != 1 {
|
|
t.Fatalf("expected one back-press to dismiss a possible dialog before relaunch, got %d", backs)
|
|
}
|
|
// The window lagged one poll behind the resumed activity, so the focused
|
|
// window must have been queried at least twice before the gate returned.
|
|
if calls := m.FocusedWindowCalls(); calls < 2 {
|
|
t.Fatalf("expected the guard to poll the focused window until drawn (>=2), got %d", calls)
|
|
}
|
|
}
|
|
|
|
func TestClampGestureToSafeArea_KeepsOriginBelowShadeStrip(t *testing.T) {
|
|
screen := hierarchy.Bounds{Left: 0, Top: 0, Right: 1080, Bottom: 2400} // marginY = 200
|
|
|
|
// Origin in the shade strip: the whole segment shifts down by 138, so the
|
|
// downward gesture stays downward (447 -> 585) instead of reversing.
|
|
fromX, fromY, toX, toY := clampGestureToSafeArea(802, 62, 802, 447, screen)
|
|
if fromX != 802 || fromY != 200 || toX != 802 || toY != 585 {
|
|
t.Errorf("segment not translated below the shade strip: from=(%d,%d) to=(%d,%d), want from=(802,200) to=(802,585)", fromX, fromY, toX, toY)
|
|
}
|
|
|
|
fromX, fromY, _, _ = clampGestureToSafeArea(5, 1200, 540, 1200, screen)
|
|
if fromX != 5 || fromY != 1200 {
|
|
t.Errorf("side origin must pass through, got (%d,%d), want (5,1200)", fromX, fromY)
|
|
}
|
|
|
|
fromX, fromY, _, _ = clampGestureToSafeArea(540, 2399, 540, 1200, screen)
|
|
if fromX != 540 || fromY != 2399 {
|
|
t.Errorf("bottom origin must pass through, got (%d,%d), want (540,2399)", fromX, fromY)
|
|
}
|
|
|
|
_, _, toX, toY = clampGestureToSafeArea(540, 1200, -50, 9999, screen)
|
|
if toX != 0 || toY != 2400 {
|
|
t.Errorf("off-screen destination not clamped to screen edges: got (%d,%d), want (0,2400)", toX, toY)
|
|
}
|
|
|
|
fromX, _, _, _ = clampGestureToSafeArea(-30, 1200, 540, 1200, screen)
|
|
if fromX != 0 {
|
|
t.Errorf("off-screen origin x must clamp to 0, got %d", fromX)
|
|
}
|
|
|
|
fromX, fromY, toX, toY = clampGestureToSafeArea(802, 62, 802, 447, hierarchy.Bounds{})
|
|
if fromX != 802 || fromY != 62 || toX != 802 || toY != 447 {
|
|
t.Error("coordinates must pass through unchanged when screen size is unknown")
|
|
}
|
|
}
|
|
|
|
// TestScreenBounds_UsesMaxExtentNotRoot guards the screen-size source: the
|
|
// Android hierarchy root reports zero bounds, so the screen rectangle must come
|
|
// from the maximum element extent or the gesture clamp silently no-ops.
|
|
func TestScreenBounds_UsesMaxExtentNotRoot(t *testing.T) {
|
|
tree := &hierarchy.Tree{
|
|
Root: &hierarchy.Node{Element: hierarchy.Element{Bounds: hierarchy.Bounds{}}},
|
|
Elements: []*hierarchy.Element{
|
|
{Bounds: hierarchy.Bounds{}},
|
|
{Bounds: hierarchy.Bounds{Left: 0, Top: 0, Right: 1080, Bottom: 2160}},
|
|
{Bounds: hierarchy.Bounds{Left: 0, Top: 2268, Right: 1080, Bottom: 2400}},
|
|
},
|
|
}
|
|
got := screenBounds(tree)
|
|
if got.Right != 1080 || got.Bottom != 2400 {
|
|
t.Fatalf("screenBounds = %+v, want right=1080 bottom=2400", got)
|
|
}
|
|
}
|
|
|
|
// TestEnsureForeground_DismissesSystemOverlay locks the shade fix: when the app
|
|
// is still the resumed activity but a system overlay (notification shade) holds
|
|
// the focused window, the guard must dismiss it with back rather than relaunch
|
|
// or act on the obscured app.
|
|
func TestEnsureForeground_DismissesSystemOverlay(t *testing.T) {
|
|
m := mockdriver.New()
|
|
// Resumed activity stays the app; the focused window is the shade.
|
|
m.ForegroundResults = []string{"app.folio"}
|
|
m.FocusedWindowResults = []string{"com.android.systemui"}
|
|
|
|
logger := slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{Level: slog.LevelWarn}))
|
|
options := Options{BundleID: "app.folio", Driver: m, IdleTimeout: 10 * time.Millisecond}
|
|
|
|
got := ensureForeground(context.Background(), options, logger, 5)
|
|
if got != foregroundOverlayDismissed {
|
|
t.Fatalf("the guard reported %v, want foregroundOverlayDismissed; "+
|
|
"an obscured app is not a relaunched one", got)
|
|
}
|
|
backs, relaunches := 0, 0
|
|
for _, a := range m.Actions() {
|
|
switch {
|
|
case a.Kind == mockdriver.ActionPressKey && a.Key == "back":
|
|
backs++
|
|
case a.Kind == mockdriver.ActionLaunch:
|
|
relaunches++
|
|
}
|
|
}
|
|
if backs != 1 {
|
|
t.Fatalf("expected one back-press to collapse the shade, got %d", backs)
|
|
}
|
|
if relaunches != 0 {
|
|
t.Fatalf("a resumed-but-obscured app must not be relaunched, got %d relaunches", relaunches)
|
|
}
|
|
}
|
|
|
|
func TestAppIsForeground(t *testing.T) {
|
|
readErr := errors.New("adb read failed")
|
|
cases := []struct {
|
|
name string
|
|
bundleID string
|
|
foreground []string
|
|
foregErr error
|
|
focused []string
|
|
focusErr error
|
|
want bool
|
|
}{
|
|
{name: "no bundle id", bundleID: "", foreground: []string{"app.folio"}, want: true},
|
|
{name: "foreground unknown", bundleID: "app.folio", foreground: nil, want: true},
|
|
{name: "foreground read error", bundleID: "app.folio", foregErr: readErr, want: true},
|
|
{name: "foreign foreground", bundleID: "app.folio", foreground: []string{"com.android.chrome"}, want: false},
|
|
{name: "app resumed and focused", bundleID: "app.folio", foreground: []string{"app.folio"}, focused: []string{"app.folio"}, want: true},
|
|
{name: "app resumed but overlay focused", bundleID: "app.folio", foreground: []string{"app.folio"}, focused: []string{"com.android.systemui"}, want: false},
|
|
{name: "app resumed, focus unknown", bundleID: "app.folio", foreground: []string{"app.folio"}, focused: []string{""}, want: true},
|
|
{name: "app resumed, focus read error", bundleID: "app.folio", foreground: []string{"app.folio"}, focusErr: readErr, want: true},
|
|
}
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
m := mockdriver.New()
|
|
m.ForegroundResults = tc.foreground
|
|
m.ForegroundErr = tc.foregErr
|
|
m.FocusedWindowResults = tc.focused
|
|
m.FocusedWindowErr = tc.focusErr
|
|
options := Options{BundleID: tc.bundleID, Driver: m}
|
|
if got := appIsForeground(context.Background(), options); got != tc.want {
|
|
t.Errorf("appIsForeground = %v, want %v", got, tc.want)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// A system overlay holds focus while the app stays resumed every step, so the
|
|
// fixture's id:next tap must never reach the driver.
|
|
func TestRunner_SkipsActionWhenOverlayStealsFocusAtApplyTime(t *testing.T) {
|
|
state := newHarness(t)
|
|
state.mock.ForegroundResults = []string{"app.folio"}
|
|
state.mock.FocusedWindowResults = []string{"app.folio", "com.android.systemui"}
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: 100 * time.Millisecond,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
BundleID: "app.folio",
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if summary.Steps == 0 {
|
|
t.Fatal("expected the loop to run steps")
|
|
}
|
|
if containsAction(state.mock.Actions(), mockdriver.ActionTapSelector, "id:next") {
|
|
t.Error("apply-time guard failed: a tap fired while a system overlay held focus")
|
|
}
|
|
}
|
|
|
|
// TestRunner_StopOnViolationEndsAtTheFirstViolation pins the gate CI runs on:
|
|
// a step budget of 8 against a spec that only violates on the third step must
|
|
// end on step 3 and write nothing after it, so the trace's last state is the
|
|
// one that produced the violation.
|
|
func TestRunner_StopOnViolationEndsAtTheFirstViolation(t *testing.T) {
|
|
const thirdStepViolationSpec = `
|
|
import { actions, always, extract } from "@sanderling/spec";
|
|
let observed = 0;
|
|
const tick = extract(() => ++observed);
|
|
globalThis.properties = {
|
|
staysUnderThree: always(() => tick.current < 3),
|
|
};
|
|
globalThis.actions = actions(() => []);
|
|
`
|
|
state := newHarnessWithSpec(t, thirdStepViolationSpec)
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: time.Hour,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
MaxSteps: 8,
|
|
StopOnViolation: true,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if !containsProperty(summary.Violations, "staysUnderThree") {
|
|
t.Fatalf("expected staysUnderThree to fire, got %v", summary.Violations)
|
|
}
|
|
if summary.Steps != 3 {
|
|
t.Errorf("steps: got %d, want 3 (the run must stop at the violating step, not run the 8-step budget)",
|
|
summary.Steps)
|
|
}
|
|
for _, step := range traceStepIndices(t, state.writer.Directory()) {
|
|
if step > summary.Steps {
|
|
t.Errorf("trace kept stepping after the violation: found step %d past step %d",
|
|
step, summary.Steps)
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestRunner_WithoutStopOnViolationRunsTheWholeBudget is the other half: the
|
|
// default must stay a full-budget fuzz run, so turning the flag on is the only
|
|
// thing that shortens a run.
|
|
func TestRunner_WithoutStopOnViolationRunsTheWholeBudget(t *testing.T) {
|
|
state := newHarnessWithSpec(t, violationSpec)
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
defer cancel()
|
|
summary, err := Run(ctx, Options{
|
|
Duration: time.Hour,
|
|
IdleTimeout: 20 * time.Millisecond,
|
|
MaxSteps: 4,
|
|
Driver: state.mock,
|
|
Verifier: state.verifier,
|
|
TraceWriter: state.writer,
|
|
})
|
|
if err != nil {
|
|
t.Fatalf("Run: %v", err)
|
|
}
|
|
if summary.Steps != 4 {
|
|
t.Errorf("steps: got %d, want 4; a violation must not shorten a default run", summary.Steps)
|
|
}
|
|
}
|
|
|
|
// traceStepIndices reads every step index the trace recorded, so a test can
|
|
// assert on what the run actually wrote rather than on the summary alone.
|
|
func traceStepIndices(t *testing.T, directory string) []int {
|
|
t.Helper()
|
|
file, err := os.Open(filepath.Join(directory, "trace.jsonl"))
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
defer file.Close()
|
|
var steps []int
|
|
scanner := bufio.NewScanner(file)
|
|
scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024)
|
|
for scanner.Scan() {
|
|
var line struct {
|
|
Step int `json:"step"`
|
|
}
|
|
if err := json.Unmarshal(scanner.Bytes(), &line); err != nil {
|
|
t.Fatalf("trace line decode: %v", err)
|
|
}
|
|
steps = append(steps, line.Step)
|
|
}
|
|
if err := scanner.Err(); err != nil {
|
|
t.Fatalf("scan trace: %v", err)
|
|
}
|
|
return steps
|
|
}
|