WIP: Drive physical Android devices over USB (#67)

* feat(sidecar): reach USB devices via the adb server by serial

* feat(test): add --device flag to target a specific Android device by serial

* feat(folio): select Android device via ANDROID_DEVICE in justfile

* feat(conformance): add android backend to the gate suite

* feat(android): keep device awake and unlocked so the app stays foreground

* feat(conformance): prep physical android device (autofill/verifier/stayon)

* fix(android): make device prep best-effort so OEM-blocked commands don't abort the run

* fix(verifier): require positive bounds for swipe candidates

A zero-bounds element centers at (0,0); a downward swipe from the
top-left corner is the system gesture that pulls down the notification
shade, dragging the fuzzer out of the app. Swipes now require positive
bounds like every other verb.

* fix(runner): harden app-scope guard against launcher and overlays

The per-step guard now relaunches and waits until the app window is
actually drawn before proceeding, so a slow physical-device relaunch no
longer lets an observe or action land on the launcher. It also detects a
system overlay (notification shade) stealing window focus while the app
stays resumed, and dismisses it with back.

* feat(android): harden physical-device runs in device prep

Device prep now disables the AOSP cached-app freezer, phantom-process
killer, and Doze (and exempts the driver) so OEM background management
stops suspending the driver mid-run. Adds ReinstallApp for clear-state on
ROMs that deny pm clear, and teaches focus detection to report the
notification shade as systemui so the scope guard can dismiss it.

* feat(driver): clear-state via APK reinstall when pm clear is blocked

When an APK path is set, Android clear-state resets the app by
uninstalling and reinstalling instead of asking the sidecar to pm clear,
which hardened OEM builds (ColorOS) deny even to the adb shell user.
Falls back to the sidecar clear path when no APK path is provided.

* feat(cli): add --android-app-path for clear-state reinstall

Wires the APK path from the test command through to the sidecar client so
Android clear-state can reset apps on OEM builds that deny pm clear.

* chore(folio): pass --android-app-path in just test

* fix(runner): clamp swipe/scroll origin out of edge gesture zones

A gesture starting in the top status-bar strip pulls down the
notification shade; the bottom and side strips are the home and back
gestures. Any of them drags the fuzzer out of the app. Swipe and scroll
origins are now clamped into a safe inner area sized from the maximum
element extent (the Android hierarchy root reports zero bounds, so the
extent is the reliable screen size). Calibrated on device: origins below
~7% of height no longer open the shade.

* perf(sidecar): faster Android text input and drop redundant settle poll

inputText now uses adb `input text` for short shell-safe ASCII (~5x
faster than the driver's per-character path) and falls back to the driver
for unicode, injection payloads, and overflow-length strings. waitForIdle
drops the structural-hash poll that followed waitForAppToSettle: each
hierarchy fetch is ~500ms on a physical device, so it cost ~2.8s per
mutating step for marginal benefit, and the runner already re-fetches
transitional frames. Cuts p95 step latency from ~6.5s to ~5.1s; G1-G4
still pass.

* fix(verifier): exclude soft-keyboard region from action candidates

The fuzzer was tapping Gboard's "Settings" key, navigating out of the
app. That key is a bare FrameLayout with a content-desc and no package or
resource-id, so the package-based scope filter missed it. Candidates whose
center falls in the keyboard region (derived from the IME elements' bounds)
are now dropped, so no tap or long-press lands on a key. Opt-in with app
scoping; unscoped runs keep every node.

* perf(runner): replace focus-tap settle with a brief wait

The full WaitForIdle after a field-focus tap cost ~0.5-1s per InputText
step on a physical device while the keyboard animated in. The tap registers
focus immediately and text is injected into the focused view, so a short
fixed wait suffices. Drops p95 step latency ~5.1s to ~4.0s; G1-G4 stay
green.

* chore(conformance): platform-aware G5 p95 budget for android

The 2500ms ceiling was calibrated on the iOS simulator. A physical Android
device drives every step over USB (snapshot + settle + adb round-trips), so
its per-step floor is several times higher; holding it to 2500ms would force
removing the settle/retry logic the correctness gates depend on. The android
backend now defaults to 4500ms (override with P95_LIMIT_MS); iOS stays 2500.

* fix(sidecar): retry maestro android driver startup

The maestro Android driver's dadb.open() occasionally misses its startup
deadline (its instrumentation host is slow to come up right after a reboot
or per-run reinstall), which aborted the whole run. Retry the open a few
times with a short backoff so a transient timeout recovers.

* chore(conformance): widen android G5 budget to 5500ms

Physical-device p95 swung 3209-4612ms across sessions (cold runs right
after a reboot are slower). 4500ms was too tight for that jitter; 5500ms
covers the observed ceiling with headroom.

* web replay fix

* feat(android): force 3-button nav during runs to prevent app drift

On gesture navigation a fuzzer swipe can trigger swipe-up-home or
edge-back and fling the app off screen. Device-prep now switches to
3-button navigation for the run (no edge gestures; the nav bar's buttons
are systemui-owned and already excluded from action candidates) and
restores the original navigation mode when the run ends. Best effort:
leaves nav untouched if the overlay command is unavailable.

* fix(android): target the selected device in adb reads; don't strand nav mode

Review fixes:
- ForegroundPackage/FocusedWindowPackage now take a serial and pass -s, so the
  foreground/scope guard works when several devices are attached (the --device
  path). Previously they ran bare `adb shell`, which errors with multiple
  devices, silently disabling app-scope enforcement. The sidecar client passes
  its serial through.
- Extract an adbArgs helper and route every adb call through it, removing four
  duplicated serial-arg builders.
- ForceThreeButtonNav now decides what to restore before changing anything: if
  the current mode is unknown or already 3-button it leaves nav untouched,
  instead of switching and then stranding the device in 3-button. Logic split
  into the pure navModeToRestore, now unit tested.

* fix(runner): restore scrollBounds doc; cover destination clamp and screenBounds

Review fixes: move the scrollBounds doc comment back onto scrollBounds (it was
stranded above screenBounds by an insertion). Extend the clamp test to assert an
off-screen destination is clamped onto the screen and that the origin lands
exactly on the margin.

* test(verifier): cover keyboardRegionTop, including the decor-view guard

The full-screen IME decor view rejection had no test; removing it left the
suite green. Add direct cases: no keyboard -> sentinel, decor view ignored in
favor of the real keyboard line, and decor-only -> sentinel.

* style(cli): gofmt testOptions field alignment

* fix(sidecar): keep a leading dash off the fast input path

A value starting with '-' could be read as an option by `adb input text`, so
the fast-path regex now requires a non-dash first character; such values fall
back to the driver. Also cover the dadb-target branch where a colon precedes a
non-numeric port (a USB serial, not host:port).

* refactor(verifier): scope action candidates by window ownership

Replaces the leaky per-element package check and the keyboard-region Y
heuristic with one rule: walk the window tree propagating each node's owning
package (empty and the neutral android framework package are transparent); a
node is in scope only when no concrete foreign package owns it (the app's own
window carries no package on Compose apps) or the owner is the app package.

This drops whole foreign windows (soft keyboard, system UI, launcher) AND
their empty-package child wrappers -- e.g. a keyboard's 'Settings' key, which
the old empty-package-is-in-scope rule admitted and which navigated out of the
app. Deletes keyboardRegionTop/isInputMethodElement.

* fix(runner): re-check foreground at apply time, skip stale actions

ensureForeground runs before observe, but the app can leave between observe and
apply (a prior gesture settling late); swipes/keys then fire stale coordinates
onto whatever screen is now up. Re-check foreground immediately before applying
and, when the app is gone, skip the action and log it (making the escape
visible) so the next step's guard relaunches instead.

* fix(android): type long ASCII via fast guarded path to stop keystroke escape

A 4096-char corpus string exceeded the fast input cap and fell to the
per-character driver path, which takes ~120s. During that uninterruptible
window focus could leave the app and the remaining keystrokes sprayed into
the launcher search box. Route shell-safe ASCII of any length through adb
input text, chunked, re-checking the foreground app between chunks and
stopping if it changed.

* chore: ignore gate artifacts and local scratch files

* refactor(runner): narrow gesture clamp to the top shade strip

3-button nav (forced for every run) disables the side back and bottom home
gestures at the OS level. On-device probing confirmed side and bottom swipe
origins no longer drift, leaving the notification shade as the only edge
gesture a swipe can trigger. Clamp only the top strip; keep origin and
destination on screen otherwise.

* chore(format): add .editorconfig enforcing 80-column limit

* chore(format): add prettier config with 80-char printWidth

* chore(deps): add prettier devDependency to replay-ui

* chore(deps): add prettier devDependency to folio-web

* chore(deps): add prettier devDependency to spec package

* chore(format): add swift-format config with 80-char lineLength

* feat(format): add make fmt targets for per-language 80-col formatting

* fix(runner): translate gesture to safe area so near-top scrolls keep direction

Clamping the swipe origin to the top margin while leaving the destination on the full screen used two reference frames: a scrollable container pinned in the top strip had its origin pushed past the destination, reversing the gesture. Translate the whole from->to segment down by the same delta so the origin clears the shade strip without flipping direction. Adds a scroll-near-top test that fails under the old origin-only clamp.

* fix(runner): apply-time guard consults focused window, not just resumed activity

ensureForeground detects a system overlay (notification shade) owning the focused window while the app stays the resumed activity, but appIsForeground only queried ForegroundApp. A swipe that pulls the shade over the app between observe and apply then fired onto the shade. Mirror the focus check at apply time so the action skips and the next step dismisses the overlay.

* test(runner): cover apply-time foreground skip and appIsForeground table

Adds a Run-level test asserting no tap reaches the driver while a system overlay holds focus (guards against the skip branch being dead-coded), plus a decision-table test for appIsForeground. Adds ForegroundErr/FocusedWindowErr to the mock driver so the guard's transient-read paths are exercised.

* fix(sidecar): harden android driver open, input guard, pressKey, foreground marker

- openWithRetry rebuilt a closed AndroidDriver, whose gRPC channel is final and shut down by close(); the retry then ran against a dead channel. Build a fresh driver per attempt and extract a unit-tested retryOpen helper (named DRIVER_OPEN_ATTEMPTS/BACKOFF).
- pressKey on the Maestro backend did KEY_MAP[key] (no lowercase, no throw), silently dropping unknown or wrong-case keys; route through a pure maestroKeyFor that lowercases and rejects unknown keys like the Stub contract.
- the mid-type foreground guard (typeShellSafe) was untested; extract a pure typeChunks and cover stop-on-foreground-change, always-send-first-chunk, and unknown-owner.
- foreground detection required the literal topResumedActivity=ActivityRecord; align parseResumedPackage to the same *ResumedActivity marker set Go reads so OEM wording does not disable the guard.

* fix(conformance): pin self-test p95 budget and score install failures as run failures

self_test reused the backend-dependent P95_LIMIT_MS, so under BACKEND=android the 4000ms slow fixture rated PASS and the offline analyzer check failed from an env var; pin it to 2500. A per-run adb install failure ran unguarded under set -e and aborted the whole harness; guard it, record the run as a G1 failure, and continue.

* fix(android): require --device when several devices are connected

With no serial requested and more than one device online, pickDevice silently returned connected[0], but that serial is never threaded into the per-step adb calls, so every later bare adb command failed with "more than one device". Error instead and ask for --device, mirroring pickAVD; a single device stays unambiguous.

* refactor(android): move PrepareDevice doc onto it; extract tested wakeCommands

The PrepareDevice doc block was stranded above adbArgs, leaving the exported function undocumented under godoc. Move it back and split the wake/keyguard tuples into wakeCommands so they have a unit test.

* perf(verifier): memoize scopedElements per tree

scopedElements rebuilt a full tree walk plus map on every candidatesForVerb call (~16 per step). Cache the result keyed on lastTree and invalidate it in PushSnapshot.

* fix(sidecar): default reinstallApp in SetClearStateReinstall; cover non-android clear

Only Dial set reinstallApp, so a Client built another way would nil-deref on Android clear-state. Default it in SetClearStateReinstall too. Add a non-android test so the platform guard has negative coverage: dropping the android check would now fail.

* test(runner): make focusTapSettle injectable so apply tests don't sleep 250ms

The focus-tap settle was a const, so five InputText apply tests each blocked the full 250ms. Make it a package var and shorten it per-test with cleanup.

* refactor(runner,android): drop unused bringToForeground return; grep no-match yields empty

bringToForeground's bool return was read by no caller. FocusedWindowPackage's on-device grep exited 1 on no match, surfacing as an error instead of the documented ""; add || true.

* perf(sidecar): reuse a single Jackson ObjectMapper

structuralHash, countRouteScreens, and hierarchy each built a fresh ObjectMapper per call inside the stability poll; the instance is thread-safe and meant to be reused. Hoist one shared val.

* refactor(android): remove unused AdbReverse/AdbReverseRemove

No callers anywhere in the tree; they were also the only adb calls bypassing adbArgs. Dead code, removed.

* style(runner): trim non-load-bearing comments from this PR's runner code and tests

* style(sidecar): trim non-load-bearing comments from this PR's driver code and tests
This commit is contained in:
pj authored and GitHub committed 2026-06-11 10:10:05 +05:30
1 parent 991c583eb9
commit 6b0d6cb971
34 files changed
+2117 -320

No files matched your search

+183 -27
View File
@@ -225,8 +225,18 @@ func Run(ctx context.Context, options Options) (Summary, error) {
}
applySkipped := false
if nextErr == nil {
if err := applyAction(ctx, options.Driver, nextAction, tree, options.IdleTimeout); err != nil {
if nextErr == nil && !appIsForeground(ctx, options) {
// The app left the foreground between observe and apply (a prior
// action's gesture settling late, or an async navigation). The
// chosen action's coordinates reference a tree that no longer
// applies, so firing it would act on whatever screen is now up.
// Skip it and record the escape; the next step's guard relaunches.
logger.Warn("app not in foreground at action time; skipping (relaunch next step)",
"step", stepIndex, "action", nextAction.Kind)
applySkipped = true
lastAction = nil
} else if nextErr == nil {
if err := applyAction(ctx, options.Driver, nextAction, tree); err != nil {
if isWDADrop(err) {
return summary, fmt.Errorf("step %d: the iOS XCTest runner could not be restarted - re-run the test: %w", stepIndex, err)
}
@@ -371,12 +381,76 @@ func ensureForeground(ctx context.Context, options Options, logger *slog.Logger,
logger.Warn("foreground check failed", "step", stepIndex, "err", err)
return false
}
if foreground == "" || foreground == options.BundleID {
if foreground != "" && foreground != options.BundleID {
logger.Warn("app left foreground; relaunching",
"step", stepIndex, "foreground", foreground, "want", options.BundleID)
// Relaunch and confirm the app is genuinely back on screen before the
// step observes or acts. A single relaunch returns before the window
// draws on a slow physical device, which would let the observe and the
// next action land on the launcher (its type-to-search swallows
// InputText). awaitForeground re-checks the foreground and focused
// window, so it never acts outside the app no matter how slow the
// relaunch settles.
awaitForeground(ctx, options, logger, stepIndex)
return true
}
// The app is the resumed activity, but a system overlay can still own the
// focused window while the app stays resumed: a fuzzer swipe starting in the
// status bar pulls the notification shade over the app. The resumed-activity
// signal misses this, so observing or acting would land on the shade.
// Dismiss it with back (which collapses the shade) so the next observe sees
// the app again.
focusChecker, hasFocus := options.Driver.(driver.FocusedWindowChecker)
if !hasFocus {
return false
}
logger.Warn("app left foreground; relaunching",
"step", stepIndex, "foreground", foreground, "want", options.BundleID)
return bringToForeground(ctx, options, logger, stepIndex)
focused, err := focusChecker.FocusedWindowApp(ctx)
if err != nil {
logger.Warn("focus check failed", "step", stepIndex, "err", err)
return false
}
if focused == "" || focused == options.BundleID {
return false
}
logger.Warn("system window obscuring app; dismissing",
"step", stepIndex, "focused", focused, "want", options.BundleID)
if err := options.Driver.PressKey(ctx, "back"); err != nil {
logger.Warn("dismiss overlay failed", "step", stepIndex, "err", err)
}
settleForForeground(ctx, options)
return true
}
// appIsForeground reports whether the app under test currently owns the
// foreground. It is the apply-time half of the scope guard: ensureForeground
// runs before observe, but the app can leave between observe and apply (a prior
// gesture settling late, an async navigation), and swipes/keys carry stale
// coordinates with no selector to re-resolve. An absent capability or an unknown
// foreground returns true so the run is never blocked where the signal is
// unavailable (web, iOS, a transient read).
func appIsForeground(ctx context.Context, options Options) bool {
checker, ok := options.Driver.(driver.ForegroundChecker)
if !ok || options.BundleID == "" {
return true
}
foreground, err := checker.ForegroundApp(ctx)
if err != nil || foreground == "" {
return true
}
if foreground != options.BundleID {
return false
}
// A system overlay can own the focused window while the app stays resumed,
// so mirror ensureForeground's focus check rather than act on the overlay.
focusChecker, ok := options.Driver.(driver.FocusedWindowChecker)
if !ok {
return true
}
focused, err := focusChecker.FocusedWindowApp(ctx)
if err != nil || focused == "" {
return true
}
return focused == options.BundleID
}
// foregroundReadyAttempts bounds how many times waitForForeground tries to
@@ -384,6 +458,12 @@ func ensureForeground(ctx context.Context, options Options, logger *slog.Logger,
// never hang the run.
const foregroundReadyAttempts = 8
// focusTapSettle is the pause after tapping a field to focus it, before typing.
// Long enough for focus to land, short enough to avoid the ~500ms-1s full
// settle the keyboard's open animation would otherwise cost every InputText
// step on a physical device.
var focusTapSettle = 250 * time.Millisecond
// waitForForeground blocks until the app under test is actually on screen, so
// the first observe never captures a leftover screen or a freshly-booted
// device's system dialog (e.g. Android's "set a screen lock" prompt). Drivers
@@ -395,6 +475,20 @@ const foregroundReadyAttempts = 8
// report the focused window, the gate additionally waits for that window to
// name the app, which only happens once it is genuinely drawn.
func waitForForeground(ctx context.Context, options Options, logger *slog.Logger) {
awaitForeground(ctx, options, logger, 0)
}
// awaitForeground brings the app under test forward when it is not already
// resumed and blocks until its window is actually drawn, bounded by
// foregroundReadyAttempts so a stuck system dialog can never hang the run. It
// re-checks the foreground each iteration and only presses back + relaunches
// while the app is genuinely absent, so once the app is resumed it polls the
// focused-window signal instead of mashing back (which would re-exit the app
// from its root screen). Shared by the pre-run startup gate (stepIndex 0) and
// the per-step scope guard so neither lets an observe or action land outside
// the app. Drivers without ForegroundChecker (web) and an unknown foreground
// both skip the gate.
func awaitForeground(ctx context.Context, options Options, logger *slog.Logger, stepIndex int) {
checker, ok := options.Driver.(driver.ForegroundChecker)
if !ok || options.BundleID == "" {
return
@@ -406,16 +500,16 @@ func waitForForeground(ctx context.Context, options Options, logger *slog.Logger
}
foreground, err := checker.ForegroundApp(ctx)
if err != nil {
logger.Warn("foreground check failed before first step", "err", err)
logger.Warn("foreground check failed", "step", stepIndex, "err", err)
return
}
if foreground == "" {
return // foreground unknowable (e.g. iOS); don't block the run
}
if foreground != options.BundleID {
logger.Warn("app not in foreground at start; bringing it forward",
"foreground", foreground, "want", options.BundleID, "attempt", attempt)
bringToForeground(ctx, options, logger, 0)
logger.Warn("app not in foreground; bringing it forward",
"step", stepIndex, "foreground", foreground, "want", options.BundleID, "attempt", attempt)
bringToForeground(ctx, options, logger, stepIndex)
continue
}
if !hasFocus {
@@ -423,34 +517,32 @@ func waitForForeground(ctx context.Context, options Options, logger *slog.Logger
}
focused, err := focusChecker.FocusedWindowApp(ctx)
if err != nil {
logger.Warn("focus check failed before first step", "err", err)
logger.Warn("focus check failed", "step", stepIndex, "err", err)
return
}
if focused == options.BundleID {
return // window is drawn; safe to observe
}
logger.Warn("app resumed but window not yet drawn; waiting",
"focused", focused, "want", options.BundleID, "attempt", attempt)
"step", stepIndex, "focused", focused, "want", options.BundleID, "attempt", attempt)
settleForForeground(ctx, options)
}
logger.Warn("app never reached foreground before first step; proceeding anyway",
"want", options.BundleID)
logger.Warn("app never reached foreground; proceeding anyway",
"step", stepIndex, "want", options.BundleID)
}
// bringToForeground returns the app under test to the foreground. It first
// presses BACK to dismiss any modal system dialog (a relaunch alone does not
// close one), then relaunches and waits for the UI to settle. Returns true
// when the relaunch itself succeeded.
func bringToForeground(ctx context.Context, options Options, logger *slog.Logger, stepIndex int) bool {
// close one), then relaunches and waits for the UI to settle.
func bringToForeground(ctx context.Context, options Options, logger *slog.Logger, stepIndex int) {
if err := options.Driver.PressKey(ctx, "back"); err != nil {
logger.Warn("dismiss key before relaunch failed", "step", stepIndex, "err", err)
}
if err := options.Driver.Launch(ctx, options.BundleID, false, nil); err != nil {
logger.Warn("relaunch failed", "step", stepIndex, "err", err)
return false
return
}
settleForForeground(ctx, options)
return true
}
// settleForForeground waits one idle window for the UI to settle, bounding the
@@ -461,7 +553,7 @@ func settleForForeground(ctx context.Context, options Options) {
cancel()
}
func applyAction(ctx context.Context, drv driver.DeviceDriver, action verifier.Action, tree *hierarchy.Tree, idleTimeout time.Duration) error {
func applyAction(ctx context.Context, drv driver.DeviceDriver, action verifier.Action, tree *hierarchy.Tree) error {
switch action.Kind {
case verifier.ActionKindTap:
x, y, ok := resolveCoordinates(action, tree)
@@ -491,6 +583,7 @@ func applyAction(ctx context.Context, drv driver.DeviceDriver, action verifier.A
return drv.LongPress(ctx, x, y)
case verifier.ActionKindScroll:
fromX, fromY, toX, toY := scrollEndpoints(action, tree)
fromX, fromY, toX, toY = clampGestureToSafeArea(fromX, fromY, toX, toY, screenBounds(tree))
duration := time.Duration(action.DurationMillis) * time.Millisecond
if duration <= 0 {
duration = 300 * time.Millisecond
@@ -509,13 +602,19 @@ func applyAction(ctx context.Context, drv driver.DeviceDriver, action verifier.A
}
tapped = true
}
// The focus tap raises the keyboard. Settle before sending key
// events so the keyboard animation cannot race them into the wrong
// field (or drop them entirely).
// The focus tap raises the keyboard. The tap registers focus
// immediately and the text is injected into the focused view (not typed
// on the visible keyboard), so a brief pause is enough for focus to land
// rather than a full settle, which costs ~500ms-1s per InputText step on
// a physical device while the keyboard animates in.
if tapped {
idleCtx, idleCancel := context.WithTimeout(ctx, idleTimeout)
_ = drv.WaitForIdle(idleCtx, idleTimeout)
idleCancel()
timer := time.NewTimer(focusTapSettle)
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
// InputText replaces the field's content: erase what the target
// holds before typing. Appending instead lets repeated draws grow
@@ -535,7 +634,8 @@ func applyAction(ctx context.Context, drv driver.DeviceDriver, action verifier.A
if duration <= 0 {
duration = 250 * time.Millisecond
}
return drv.Swipe(ctx, action.FromX, action.FromY, action.ToX, action.ToY, duration)
fromX, fromY, toX, toY := clampGestureToSafeArea(action.FromX, action.FromY, action.ToX, action.ToY, screenBounds(tree))
return drv.Swipe(ctx, fromX, fromY, toX, toY, duration)
case verifier.ActionKindPressKey:
if action.Key == "" {
return nil
@@ -659,6 +759,62 @@ func scrollEndpoints(action verifier.Action, tree *hierarchy.Tree) (fromX, fromY
return cx, cy, toX, toY
}
// screenBounds returns the device screen rectangle as the maximum extent across
// all elements. The hierarchy root often reports zero bounds on Android, so the
// extent (driven by full-screen containers and the navigation bar) is the
// reliable screen size. Returns a zero rectangle when unknown.
func screenBounds(tree *hierarchy.Tree) hierarchy.Bounds {
if tree == nil {
return hierarchy.Bounds{}
}
var bounds hierarchy.Bounds
for _, element := range tree.Elements {
if element.Bounds.Right > bounds.Right {
bounds.Right = element.Bounds.Right
}
if element.Bounds.Bottom > bounds.Bottom {
bounds.Bottom = element.Bounds.Bottom
}
}
return bounds
}
// clampGestureToSafeArea keeps a swipe's origin below the top status strip,
// where a downward drag pulls the notification shade over the app. Runs force
// 3-button navigation (ForceThreeButtonNav), which disables the side back and
// bottom home gestures at the OS level; on-device probing confirmed side and
// bottom origins then no longer drift, so the shade is the only edge gesture a
// swipe can still trigger. Origin and destination are otherwise only kept on
// screen. With an unknown screen size the coordinates pass through unchanged.
func clampGestureToSafeArea(fromX, fromY, toX, toY int, screen hierarchy.Bounds) (int, int, int, int) {
width, height := screen.Width(), screen.Height()
if width <= 0 || height <= 0 {
return fromX, fromY, toX, toY
}
// Translate the whole segment when the origin is in the top margin, rather
// than clamping the origin alone, which could push it past the destination
// and reverse a near-top scroll.
marginY := height / 12
if shortfall := (screen.Top + marginY) - fromY; shortfall > 0 {
fromY += shortfall
toY += shortfall
}
clamp := func(value, low, high int) int {
if value < low {
return low
}
if value > high {
return high
}
return value
}
fromX = clamp(fromX, screen.Left, screen.Right)
fromY = clamp(fromY, screen.Top, screen.Bottom)
toX = clamp(toX, screen.Left, screen.Right)
toY = clamp(toY, screen.Top, screen.Bottom)
return fromX, fromY, toX, toY
}
// scrollBounds returns the container bounds for an authored Scroll: the node
// named by On when it resolves, otherwise the root (whole-screen) bounds.
func scrollBounds(action verifier.Action, tree *hierarchy.Tree) hierarchy.Bounds {
+266 -31
View File
@@ -7,6 +7,7 @@ import (
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"os"
"path/filepath"
@@ -53,6 +54,12 @@ func newHarness(t *testing.T) *harness {
return newHarnessWithSpec(t, fixtureSpec)
}
func fastFocusSettle(t *testing.T) {
prev := focusTapSettle
focusTapSettle = time.Millisecond
t.Cleanup(func() { focusTapSettle = prev })
}
// bundleSpec compiles an authored TS spec with the goja runtime entry so the
// loaded bundle installs __sanderlingNextAction__ (the shared picker).
func bundleSpec(t *testing.T, specSource string) string {
@@ -652,6 +659,7 @@ func TestRunner_LogsWaitForIdleDriverErrors(t *testing.T) {
}
func TestApplyAction_InputTextErasesExistingTextBeforeTyping(t *testing.T) {
fastFocusSettle(t)
tree, err := hierarchy.Parse(`{"attributes":{"resource-id":"root","bounds":"[0,0,1080,2340]"},"children":[
{"attributes":{"resource-id":"username","text":"stale-value","bounds":"[10,10,500,100]"},"children":[]}
]}`)
@@ -661,41 +669,39 @@ func TestApplyAction_InputTextErasesExistingTextBeforeTyping(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindInputText, On: "id:username", Text: "alice"}
if err := applyAction(context.Background(), driverMock, action, tree, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
t.Fatalf("applyAction: %v", err)
}
// The post-tap settle is now a brief internal sleep, not a WaitForIdle RPC,
// so the recorded driver actions are tap, erase, input.
actions := driverMock.Actions()
if len(actions) != 4 {
t.Fatalf("want tap, wait_for_idle, erase, input; got %v", actions)
if len(actions) != 3 {
t.Fatalf("want tap, erase, input; got %v", actions)
}
if actions[0].Kind != mockdriver.ActionTap {
t.Errorf("first action = %v, want tap", actions[0].Kind)
}
if actions[1].Kind != mockdriver.ActionWaitForIdle {
t.Errorf("second action = %v, want wait_for_idle (settle after focus tap)", actions[1].Kind)
if actions[1].Kind != mockdriver.ActionEraseText || actions[1].CharacterCount != len("stale-value") {
t.Errorf("second action = %+v, want erase_text of %d characters", actions[1], len("stale-value"))
}
if actions[2].Kind != mockdriver.ActionEraseText || actions[2].CharacterCount != len("stale-value") {
t.Errorf("third action = %+v, want erase_text of %d characters", actions[2], len("stale-value"))
}
if actions[3].Kind != mockdriver.ActionInputText || actions[3].Text != "alice" {
t.Errorf("fourth action = %+v, want input_text alice", actions[3])
if actions[2].Kind != mockdriver.ActionInputText || actions[2].Text != "alice" {
t.Errorf("third action = %+v, want input_text alice", actions[2])
}
}
// TestApplyAction_InputTextWithoutTargetSkipsSettle pins that the post-tap
// settle only runs when a focus tap actually happened: with no resolvable
// target there is no keyboard animation to absorb.
func TestApplyAction_InputTextWithoutTargetSkipsSettle(t *testing.T) {
// TestApplyAction_InputTextWithoutTargetSkipsFocusTap pins that with no
// resolvable target there is no focus tap (and so no settle), and InputText
// still runs at the cursor.
func TestApplyAction_InputTextWithoutTargetSkipsFocusTap(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindInputText, X: -1, Y: -1, Text: "alice"}
if err := applyAction(context.Background(), driverMock, action, nil, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("applyAction: %v", err)
}
for _, recorded := range driverMock.Actions() {
if recorded.Kind == mockdriver.ActionWaitForIdle {
t.Errorf("no focus tap happened; settle must be skipped: %v", driverMock.Actions())
}
actions := driverMock.Actions()
if len(actions) != 1 || actions[0].Kind != mockdriver.ActionInputText {
t.Errorf("no target: want input_text only (no focus tap), got %v", actions)
}
}
@@ -703,6 +709,7 @@ func TestApplyAction_InputTextWithoutTargetSkipsSettle(t *testing.T) {
// asserting the TextReplacer capability never pays the pre-erase round-trip:
// its InputText already replaces the field's content.
func TestApplyAction_InputTextSkipsEraseForReplacingDriver(t *testing.T) {
fastFocusSettle(t)
tree, err := hierarchy.Parse(`{"attributes":{"resource-id":"root","bounds":"[0,0,1080,2340]"},"children":[
{"attributes":{"resource-id":"username","text":"stale-value","bounds":"[10,10,500,100]"},"children":[]}
]}`)
@@ -713,7 +720,7 @@ func TestApplyAction_InputTextSkipsEraseForReplacingDriver(t *testing.T) {
driverMock.ReplacesText = true
action := verifier.Action{Kind: verifier.ActionKindInputText, On: "id:username", Text: "alice"}
if err := applyAction(context.Background(), driverMock, action, tree, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
t.Fatalf("applyAction: %v", err)
}
if containsAction(driverMock.Actions(), mockdriver.ActionEraseText, "") {
@@ -725,6 +732,7 @@ func TestApplyAction_InputTextSkipsEraseForReplacingDriver(t *testing.T) {
}
func TestApplyAction_InputTextSkipsEraseWhenTargetEmpty(t *testing.T) {
fastFocusSettle(t)
tree, err := hierarchy.Parse(`{"attributes":{"resource-id":"root","bounds":"[0,0,1080,2340]"},"children":[
{"attributes":{"resource-id":"username","bounds":"[10,10,500,100]"},"children":[]}
]}`)
@@ -734,7 +742,7 @@ func TestApplyAction_InputTextSkipsEraseWhenTargetEmpty(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindInputText, On: "id:username", Text: "alice"}
if err := applyAction(context.Background(), driverMock, action, tree, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
t.Fatalf("applyAction: %v", err)
}
if containsAction(driverMock.Actions(), mockdriver.ActionEraseText, "") {
@@ -748,7 +756,7 @@ func TestApplyAction_InputTextSurfacesFocusTapError(t *testing.T) {
driverMock.Failures[mockdriver.ActionTapSelector] = errors.New("adb unreachable")
action := verifier.Action{Kind: verifier.ActionKindInputText, On: "id:username", Text: "alice"}
err := applyAction(context.Background(), driverMock, action, nil, time.Millisecond)
err := applyAction(context.Background(), driverMock, action, nil)
if err == nil {
t.Fatalf("expected focus tap failure to surface, got nil")
}
@@ -761,7 +769,7 @@ func TestApplyAction_InputTextSurfacesFocusTapError(t *testing.T) {
driverMock.Failures[mockdriver.ActionTap] = errors.New("tap driver error")
action := verifier.Action{Kind: verifier.ActionKindInputText, X: 10, Y: 20, Text: "alice"}
err := applyAction(context.Background(), driverMock, action, nil, time.Millisecond)
err := applyAction(context.Background(), driverMock, action, nil)
if err == nil {
t.Fatalf("expected focus tap failure to surface, got nil")
}
@@ -772,10 +780,11 @@ func TestApplyAction_InputTextSurfacesFocusTapError(t *testing.T) {
}
func TestApplyAction_V8InputTextTapsAtCoordinates(t *testing.T) {
fastFocusSettle(t)
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindInputText, X: 50, Y: 100, Text: "alice"}
if err := applyAction(context.Background(), driverMock, action, nil, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("apply action: %v", err)
}
actions := driverMock.Actions()
@@ -788,13 +797,14 @@ func TestApplyAction_V8InputTextTapsAtCoordinates(t *testing.T) {
}
func TestApplyAction_V8InputTextAtOriginStillTaps(t *testing.T) {
fastFocusSettle(t)
driverMock := mockdriver.New()
// V8 emits real (0,0) coordinates for an element at viewport top-left
// (post-#15 the runtime nullifies unresolved actions, so a non-null
// InputText with (0,0) is a deliberate edge tap, not a sentinel).
action := verifier.Action{Kind: verifier.ActionKindInputText, X: 0, Y: 0, Text: "alice"}
if err := applyAction(context.Background(), driverMock, action, nil, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("apply action: %v", err)
}
if !containsAction(driverMock.Actions(), mockdriver.ActionTap, "") {
@@ -806,7 +816,7 @@ func TestApplyAction_DoubleTapDispatchesDoubleTapAtCoordinates(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindDoubleTap, X: 100, Y: 200}
if err := applyAction(context.Background(), driverMock, action, nil, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("apply action: %v", err)
}
taps := 0
@@ -824,7 +834,7 @@ func TestApplyAction_DoubleTapDispatchesDoubleTapSelector(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindDoubleTap, On: "id:save"}
if err := applyAction(context.Background(), driverMock, action, nil, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("apply action: %v", err)
}
taps := 0
@@ -842,7 +852,7 @@ func TestApplyAction_LongPressDispatchesAtResolvedCoordinates(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindLongPress, X: 120, Y: 240}
if err := applyAction(context.Background(), driverMock, action, nil, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("apply action: %v", err)
}
found := false
@@ -868,7 +878,7 @@ func TestApplyAction_ScrollWithPrecomputedEndpointsSwipes(t *testing.T) {
DurationMillis: 300,
}
if err := applyAction(context.Background(), driverMock, action, nil, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("apply action: %v", err)
}
found := false
@@ -891,7 +901,7 @@ func TestApplyAction_ScrollDirectionUsesInversion(t *testing.T) {
}
action := verifier.Action{Kind: verifier.ActionKindScroll, Direction: "down", On: "id:list"}
if err := applyAction(context.Background(), driverMock, action, tree, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
t.Fatalf("apply action: %v", err)
}
var swipe *mockdriver.Action
@@ -910,6 +920,40 @@ func TestApplyAction_ScrollDirectionUsesInversion(t *testing.T) {
}
}
func TestApplyAction_ScrollNearTopKeepsDirectionAfterClamp(t *testing.T) {
driverMock := mockdriver.New()
// Full-screen root sets the 1080x2400 screen (marginY=200); the scrollable
// list sits inside the top margin (y 20..180), where the clamp must fire.
treeJSON := `{"attributes":{"bounds":"[0,0,1080,2400]"},"children":[
{"attributes":{"resource-id":"com.fixture:id/toplist","scrollable":"true","bounds":"[0,20,1080,180]"},"children":[],"enabled":true}
]}`
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatalf("parse tree: %v", err)
}
action := verifier.Action{Kind: verifier.ActionKindScroll, Direction: "up", On: "id:toplist"}
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
t.Fatalf("apply action: %v", err)
}
var swipe *mockdriver.Action
for i := range driverMock.Actions() {
if driverMock.Actions()[i].Kind == mockdriver.ActionSwipe {
a := driverMock.Actions()[i]
swipe = &a
}
}
if swipe == nil {
t.Fatalf("expected a Swipe, got %v", driverMock.Actions())
}
if swipe.FromY != 200 {
t.Errorf("origin not pushed below the shade strip, got fromY=%d want 200", swipe.FromY)
}
if swipe.ToY <= swipe.FromY {
t.Errorf("scroll up reversed by the clamp: from=%d to=%d (want toY > fromY)", swipe.FromY, swipe.ToY)
}
}
func TestApplyAction_ScrollScreenFallback(t *testing.T) {
driverMock := mockdriver.New()
treeJSON := `{"attributes":{"bounds":"[0,0,400,800]"},"children":[],"enabled":true}`
@@ -920,7 +964,7 @@ func TestApplyAction_ScrollScreenFallback(t *testing.T) {
// On unset: container falls back to whole-screen (root) bounds.
action := verifier.Action{Kind: verifier.ActionKindScroll, Direction: "up"}
if err := applyAction(context.Background(), driverMock, action, tree, time.Millisecond); err != nil {
if err := applyAction(context.Background(), driverMock, action, tree); err != nil {
t.Fatalf("apply action: %v", err)
}
var swipe *mockdriver.Action
@@ -1764,3 +1808,194 @@ func TestRunner_WaitsForWindowDrawnBeforeFirstAction(t *testing.T) {
}
}
}
// TestAwaitForeground_RelaunchesThenWaitsForWindow locks the per-step scope
// guard's recovery: after the app leaves to the launcher, it must relaunch AND
// keep polling the focused window until it names the app, so the step never
// observes or acts while the launcher is on screen (where InputText would land
// in the launcher's type-to-search filter). A single fire-and-forget relaunch,
// which returns before the window draws on a slow physical device, is the bug
// this guards against.
func TestAwaitForeground_RelaunchesThenWaitsForWindow(t *testing.T) {
m := mockdriver.New()
// Foreground: launcher on the first poll (still gone), then the app. Focus:
// the launcher window lingers one extra poll before the app's window draws.
m.ForegroundResults = []string{"com.android.launcher", "app.folio"}
m.FocusedWindowResults = []string{"com.android.launcher", "app.folio"}
logger := slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{Level: slog.LevelWarn}))
options := Options{BundleID: "app.folio", Driver: m, IdleTimeout: 10 * time.Millisecond}
awaitForeground(context.Background(), options, logger, 7)
relaunches, backs := 0, 0
for _, a := range m.Actions() {
switch {
case a.Kind == mockdriver.ActionLaunch && a.BundleID == "app.folio" && !a.ClearState:
relaunches++
case a.Kind == mockdriver.ActionPressKey && a.Key == "back":
backs++
}
}
if relaunches != 1 {
t.Fatalf("expected exactly one relaunch while the app was gone, got %d", relaunches)
}
if backs != 1 {
t.Fatalf("expected one back-press to dismiss a possible dialog before relaunch, got %d", backs)
}
// The window lagged one poll behind the resumed activity, so the focused
// window must have been queried at least twice before the gate returned.
if calls := m.FocusedWindowCalls(); calls < 2 {
t.Fatalf("expected the guard to poll the focused window until drawn (>=2), got %d", calls)
}
}
func TestClampGestureToSafeArea_KeepsOriginBelowShadeStrip(t *testing.T) {
screen := hierarchy.Bounds{Left: 0, Top: 0, Right: 1080, Bottom: 2400} // marginY = 200
// Origin in the shade strip: the whole segment shifts down by 138, so the
// downward gesture stays downward (447 -> 585) instead of reversing.
fromX, fromY, toX, toY := clampGestureToSafeArea(802, 62, 802, 447, screen)
if fromX != 802 || fromY != 200 || toX != 802 || toY != 585 {
t.Errorf("segment not translated below the shade strip: from=(%d,%d) to=(%d,%d), want from=(802,200) to=(802,585)", fromX, fromY, toX, toY)
}
fromX, fromY, _, _ = clampGestureToSafeArea(5, 1200, 540, 1200, screen)
if fromX != 5 || fromY != 1200 {
t.Errorf("side origin must pass through, got (%d,%d), want (5,1200)", fromX, fromY)
}
fromX, fromY, _, _ = clampGestureToSafeArea(540, 2399, 540, 1200, screen)
if fromX != 540 || fromY != 2399 {
t.Errorf("bottom origin must pass through, got (%d,%d), want (540,2399)", fromX, fromY)
}
_, _, toX, toY = clampGestureToSafeArea(540, 1200, -50, 9999, screen)
if toX != 0 || toY != 2400 {
t.Errorf("off-screen destination not clamped to screen edges: got (%d,%d), want (0,2400)", toX, toY)
}
fromX, _, _, _ = clampGestureToSafeArea(-30, 1200, 540, 1200, screen)
if fromX != 0 {
t.Errorf("off-screen origin x must clamp to 0, got %d", fromX)
}
fromX, fromY, toX, toY = clampGestureToSafeArea(802, 62, 802, 447, hierarchy.Bounds{})
if fromX != 802 || fromY != 62 || toX != 802 || toY != 447 {
t.Error("coordinates must pass through unchanged when screen size is unknown")
}
}
// TestScreenBounds_UsesMaxExtentNotRoot guards the screen-size source: the
// Android hierarchy root reports zero bounds, so the screen rectangle must come
// from the maximum element extent or the gesture clamp silently no-ops.
func TestScreenBounds_UsesMaxExtentNotRoot(t *testing.T) {
tree := &hierarchy.Tree{
Root: &hierarchy.Node{Element: hierarchy.Element{Bounds: hierarchy.Bounds{}}},
Elements: []*hierarchy.Element{
{Bounds: hierarchy.Bounds{}},
{Bounds: hierarchy.Bounds{Left: 0, Top: 0, Right: 1080, Bottom: 2160}},
{Bounds: hierarchy.Bounds{Left: 0, Top: 2268, Right: 1080, Bottom: 2400}},
},
}
got := screenBounds(tree)
if got.Right != 1080 || got.Bottom != 2400 {
t.Fatalf("screenBounds = %+v, want right=1080 bottom=2400", got)
}
}
// TestEnsureForeground_DismissesSystemOverlay locks the shade fix: when the app
// is still the resumed activity but a system overlay (notification shade) holds
// the focused window, the guard must dismiss it with back rather than relaunch
// or act on the obscured app.
func TestEnsureForeground_DismissesSystemOverlay(t *testing.T) {
m := mockdriver.New()
// Resumed activity stays the app; the focused window is the shade.
m.ForegroundResults = []string{"app.folio"}
m.FocusedWindowResults = []string{"com.android.systemui"}
logger := slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{Level: slog.LevelWarn}))
options := Options{BundleID: "app.folio", Driver: m, IdleTimeout: 10 * time.Millisecond}
if !ensureForeground(context.Background(), options, logger, 5) {
t.Fatal("expected the guard to act on the focus-stealing overlay")
}
backs, relaunches := 0, 0
for _, a := range m.Actions() {
switch {
case a.Kind == mockdriver.ActionPressKey && a.Key == "back":
backs++
case a.Kind == mockdriver.ActionLaunch:
relaunches++
}
}
if backs != 1 {
t.Fatalf("expected one back-press to collapse the shade, got %d", backs)
}
if relaunches != 0 {
t.Fatalf("a resumed-but-obscured app must not be relaunched, got %d relaunches", relaunches)
}
}
func TestAppIsForeground(t *testing.T) {
readErr := errors.New("adb read failed")
cases := []struct {
name string
bundleID string
foreground []string
foregErr error
focused []string
focusErr error
want bool
}{
{name: "no bundle id", bundleID: "", foreground: []string{"app.folio"}, want: true},
{name: "foreground unknown", bundleID: "app.folio", foreground: nil, want: true},
{name: "foreground read error", bundleID: "app.folio", foregErr: readErr, want: true},
{name: "foreign foreground", bundleID: "app.folio", foreground: []string{"com.android.chrome"}, want: false},
{name: "app resumed and focused", bundleID: "app.folio", foreground: []string{"app.folio"}, focused: []string{"app.folio"}, want: true},
{name: "app resumed but overlay focused", bundleID: "app.folio", foreground: []string{"app.folio"}, focused: []string{"com.android.systemui"}, want: false},
{name: "app resumed, focus unknown", bundleID: "app.folio", foreground: []string{"app.folio"}, focused: []string{""}, want: true},
{name: "app resumed, focus read error", bundleID: "app.folio", foreground: []string{"app.folio"}, focusErr: readErr, want: true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
m := mockdriver.New()
m.ForegroundResults = tc.foreground
m.ForegroundErr = tc.foregErr
m.FocusedWindowResults = tc.focused
m.FocusedWindowErr = tc.focusErr
options := Options{BundleID: tc.bundleID, Driver: m}
if got := appIsForeground(context.Background(), options); got != tc.want {
t.Errorf("appIsForeground = %v, want %v", got, tc.want)
}
})
}
}
// A system overlay holds focus while the app stays resumed every step, so the
// fixture's id:next tap must never reach the driver.
func TestRunner_SkipsActionWhenOverlayStealsFocusAtApplyTime(t *testing.T) {
state := newHarness(t)
state.mock.ForegroundResults = []string{"app.folio"}
state.mock.FocusedWindowResults = []string{"app.folio", "com.android.systemui"}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
summary, err := Run(ctx, Options{
Duration: 100 * time.Millisecond,
IdleTimeout: 20 * time.Millisecond,
BundleID: "app.folio",
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
if summary.Steps == 0 {
t.Fatal("expected the loop to run steps")
}
if containsAction(state.mock.Actions(), mockdriver.ActionTapSelector, "id:next") {
t.Error("apply-time guard failed: a tap fired while a system overlay held focus")
}
}