mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
9fb121e9d09c1245a4ff1f3d00ccb25846f8ca85
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 pj
|
76dce1a75e |
experiment instrumentation: step budgets, arm labels, campaign runner (#72)
* feat(cli): add --max-steps for step-bounded runs runner.Options.MaxSteps already worked but was unreachable from the command line. A step budget is what makes two generators comparable: one making a model call per step and one drawing from a PRNG are not comparable per second. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * feat(trace): record arm membership and host in meta.json meta.json recorded the seed but not which picker ran, how it was configured, what budget it was given, or which machine produced it. A directory of runs cannot be attributed to an experiment cell without those, which makes any factorial computed from such a directory unanalysable after the fact. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * feat(cli): add --arm and populate run meta from it Model and instructions are recorded only when the LLM picker is the one that will actually run, so a spec declaring generator = llm() that is run under the seeded picker does not label its trace with a model it never called. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * feat(campaign): sweep seeds for one experiment cell campaign.json lists the seeds a sweep intended to run and is written before the first run, so a host that dropped runs shows up as missing seeds rather than as a smaller sample. Seed 0 is rejected: sanderling test reads it as "derive a seed from the clock", which is why conformance/gates.sh controls nothing today. Each run contributes one runs.jsonl line carrying steps to first violation by origin step, the step that armed the failed obligation, so the survival analysis never reopens a trace. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(runner): no silent generator fallback, and llm on web --generator llm against a spec declaring no generator = llm(...) logged a warning and ran the seeded picker. For a comparison campaign that is silent arm corruption: the run completes, the directory looks correct, and the wrong policy drove it. It is now fatal. pickSources also returned the V8 source for both action and extractor on web before it looked at the generator, so the llm policy was unreachable there. The two axes are now independent: the driver picks the extractor source, the flag picks the action source, and llmSource composes with either because the runner populates the candidate list and screenshot on every platform. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(chrome): make the hierarchy dump agree with the web runtime Three facts differed between the dump the goja host reads and the DOM the V8 host reads, so the two enumerated different candidates on one page. scrollable was never emitted, and worker.go reads exactly that attribute while targets.ts requires it for scrolls, so the goja host could not offer a single web scroll. clickable tested el.onclick, which React assigns to its root container for event delegation, making the whole viewport a tap target here and in no other enumeration. Both now resolve through the selector sets in pkg/spec/src/web-runtime.ts. The dump also rooted at body while collectTargets walks querySelectorAll("*"), so the goja host never saw html, where page-level scrolling lives. It now roots at documentElement and skips the head subtree, which is all zero-bounds and would otherwise carry script and title text into the trace. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(conformance): give the gate reproducible seeds SEED defaulted to 0 and sanderling test reads --seed 0 as "derive a seed from the clock", so the tunable controlled nothing and a gate failure could not be re-run. SEEDS now takes one explicit non-zero seed per run, recorded in the results table so a failing row names its stream. The five runs stay on five different streams: a gate that scored one path five times would catch less than one that scores five. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(chrome): emit editable as a plain boolean editable was emitted as `isEditable || null`, and an absent field sends internal/hierarchy into the native fallback, which reads any class name containing "EditText" as an Android text widget. On web that is just a CSS class, so a page styling a div with it was editable to the goja host and not to the web runtime, and the model policy could be offered typing into a div. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(spec): leave the head subtree out of the web target walk collectTargets walked querySelectorAll("*") while the hierarchy dump skips head, so the two hosts enumerated different element sets on every page with a <head>. No candidate changes: builtinCandidates pushes only for targets acceptsTarget admits, and head elements have no positive bounds, so the list the draw ranges over is untouched. What changes is that targetIndex now means the same thing on both hosts. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * test(chrome): compare the facts both hosts derive from one DOM The existing parity harness hand-authors the facts on both sides, so it proves that given identical facts both hosts select identical candidates, and says nothing about the two code paths that derive those facts from a real page. Four divergences lived in that blind spot and it passed throughout. This drives one real page and compares clickable, enabled, editable, scrollable and positiveBounds element by element, plus the element sets themselves, which is what catches a host that omits html or includes head. Reverting any of the four fixes makes it fail naming the element and the fact. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * chore(make): run the browser packages one at a time Both launch Chrome and launching two at once has failed with "Launch: context canceled". Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * style: remove every em-dash and en-dash Eighteen occurrences across fourteen files. Each sentence was repunctuated to suit what the dash was doing rather than swapped for a hyphen, which produces comma splices. The minus sign in folio-web's ledger is a minus sign and stays. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(chrome): honor the caller context in Launch Launch and clearState ran against d.tabCtx, so a target that accepts the connection and never answers wedged the process past its own --duration and through SIGTERM, needing SIGKILL. Unattended that is a campaign worker lost for the rest of the sweep with no diagnostic. The browser is still allocated against d.tabCtx first, because chromedp starts Chrome under whichever context calls Run first and allocating under a caller deadline would kill the browser when Launch returns. Everything after allocation goes through runCtx. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(sidecarassets): publish the extracted jar through a rename Extract wrote a 96 MB jar with a plain WriteFile into a temp path every sanderling process on the host shares. On a cold host several concurrent workers all miss the checksum and all write the same path, and O_TRUNC lets one spawn a JVM against another's half-written archive. A fresh experiment host is exactly a cold host. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * feat(campaign): kill a run that outlives --run-timeout A wedged run holds its worker for the rest of the sweep, and on an unattended host nothing else will send it a signal. Defaults to three times --duration and must exceed it. A killed run is recorded as timed_out rather than as a generic failure, so the analysis can tell a lost cell from a real crash. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * style(test): gofmt browser_test.go Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX |
||
|
pj
|
406b7516b3 |
iOS simulator driver: Go-native companion-backed backend (#62)
* perf(ios): use prebuilt XCTest runner to cut startup * chore(ioscompanion): add companion asset prepare script * feat(ioscompanion): embed and extract simulator companion bundle * test(ioscompanion): cover companion stub and embedded extraction * docs: add third party notices for vendored companion * chore: ignore vendored companion bundle artifact * build(proto): pin simulator companion proto v1.1.8 * build(proto): add dedicated buf module and gen template for pinned proto * build(proto): exclude pinned companion proto from root buf workspace * feat(ioscompanion): commit generated companion gRPC stubs * feat(ioscompanion): map flat companion describe dump to TreeNode JSON * test(ioscompanion): add hierarchy-map golden and unit tests * feat(ioscompanion): port screen-settle stability polling to Go * test(ioscompanion): cover settle transitional, hash, streak, and cap rules * feat(ioscompanion): add USB HID keymap module * test(ioscompanion): cover keymap branches and paste-chord constants * build: embed companion assets via withcompanion tag * feat(ioscompanion): add transport companion interface * feat(ioscompanion): add HID event wrapper and builders * feat(ioscompanion): wire gRPC companion client and Dial * test(ioscompanion): cover HID builders and unit conversions * test(ioscompanion): cover Dial, process-state mapping, and install archive * test(ioscompanion): add gated simulator integration smoke test * feat(ioscompanion): text input and gesture HID composition with pasteboard fallback * test(ioscompanion): cover input composers, paste dialog loop, and pure helpers * feat(ioscompanion): add Describe to companion transport * feat(ioscompanion): implement DeviceDriver with companion supervision * test(ioscompanion): unit tests with fake companion transport * test(ioscompanion): gated companion smoke test * feat(ios): add ResolveTarget for simulator vs physical-device routing * feat(testrun): route iOS simulators through the native companion driver * refactor(testrun): defer the java preflight check to the physical-device path * feat(cli): add --ios-app-path flag * feat(doctor): split iOS checks into simulator and physical-device paths * test(folio): add gate-analyzer fixtures for G1-G5 * feat(folio): add iOS conformance gate script * chore(folio): wire gates recipe, app path, and ignore gate output * style: gofmt struct alignment drift * fix(doctor): probe simctl via xcrun instead of PATH lookup * fix(ioscompanion): spawn companion under driver-lifetime context * test(ioscompanion): prove companion child outlives startup context * fix(ioscompanion): chunk install payload under companion message cap * test(ioscompanion): cover install payload chunking * fix(ioscompanion): reinstall via simctl and sanitize companion env * fix(ioscompanion): wait out unresolved accessibility values after launch * perf(ioscompanion): paste long text for atomic landing * test(ioscompanion): cover paste threshold, retry flow, and sentinel detection * fix(ioscompanion): treat unresolved bridge values as transitional, never as content * fix(ioscompanion): accept masked secure-field values as paste landing * test(ioscompanion): cover sentinel mapping and masked-field landing * fix(ioscompanion): atomic erase and single-send paste to prevent doubling * test(ioscompanion): cover atomic erase, single chord, unverifiable field * fix(ioscompanion): verify paste on a time budget that outlasts the bridge blackout * test(ioscompanion): cover bridge-blackout paste verification * fix(ioscompanion): drop unresolved-value settle gate that never let empty-field screens settle * refactor(ioscompanion): name the empty-editable-field sentinel for what it is * perf(ioscompanion): tighten settle streak for the fast companion transport * feat(ioscompanion): pre-grant pasteboard access so unicode input skips the OS prompt * refactor(ioscompanion): drop paste warm-up now that the grant suppresses the prompt * test(ioscompanion): cover pasteboard grant on launch, drop warm-up tests * fix(ioscompanion): retry describe past transient collapsed accessibility dumps * test(ioscompanion): cover collapsed-dump detection * perf(ioscompanion): split raw and retrying describe so settle does not double-wait collapses * perf(ioscompanion): tighten settle now that collapses are handled separately * fix(ioscompanion): replace field content on input so blackout-skipped erase cannot accumulate text * test(ioscompanion): cover replace-on-input and TextReplacer capability * refactor(ioscompanion): neutralize HID events behind the transport seam * feat(companion): add simulator runner project skeleton * feat(companion): serve accessibility snapshots over the wire protocol * feat(companion): synthesize timestamped touch gestures * feat(companion): type text with replace semantics * feat(companion): serve the wire protocol from a parked runner * feat(ioscompanion): add TextEditor capability and unavailable sentinel to the transport seam * feat(ioscompanion): route text input through a text-editing companion when available * fix(companion): bind listener by port and source screen size from snapshot * feat(ioscompanion): add runner companion JSON transport * test(ioscompanion): cover runner transport protocol mapping * fix(companion): synthesize gestures synchronously to avoid the async completion crash * fix(companion): type on the main thread and recover from focus assertions * fix(companion): keep serving after an automation failure * refactor(companion): tidy snapshot serialization * fix(companion): honor sequential tap gaps and survive synthesis exceptions * feat(ioscompanion): expose native typing with an explicit replace flag * chore(companion): add runner asset prepare script * feat(ioscompanion): embed and extract the runner test bundle * test(ioscompanion): cover runner asset extraction * build(ioscompanion): commit runner asset archive * feat(ioscompanion): pair the legacy companion with the in-simulator runner * test(ioscompanion): cover hybrid routing, paste-grant skip, and port binding * fix(ioscompanion): reconnect after interrupted runner calls instead of restarting * fix(ioscompanion): route hybrid lifecycle through the runner and harden restarts * feat(companion): launch and terminate apps through the automation session * build(ioscompanion): refresh runner asset with session lifecycle * fix(ioscompanion): classify connection deadline expiry as caller budget * fix(companion): capture snapshots on the main thread inside the catch bridge * build(ioscompanion): refresh runner asset with main-thread snapshots * perf(ioscompanion): count read spans toward settle and capture snapshots concurrently * feat(ioscompanion): make the hybrid simulator companion the default * test(folio): cover runner-session orphans in the gate harness * test(ioscompanion): pin the child-lifetime test to the legacy path * fix(ioscompanion): keep mappable text on one HID stream and verify unicode clears * fix(ioscompanion): pause the clear chord so selection applies before the delete * fix(companion): prune the keyboard subtree from snapshots * build(ioscompanion): refresh runner asset without keyboard elements * fix(ioscompanion): capture the screenshot transport before a recovery can reassign it * fix(companion): pin the runner listener to loopback * fix(companion): size the replace delete prefix to cover any focused field * build(ioscompanion): refresh runner asset with loopback bind and replace fix * fix(cli): cancel the run context on SIGINT so spawned children are reaped * fix(testrun): point the device java preflight hint at the ios-device doctor * fix(folio): word-bound the G2 ERROR scan and drop the dead objc allowlist glob * test(ioscompanion): cover stopProcess, restart, and failed bring-up supervision * chore: add test-companion target for the withcompanion-tagged suite * chore(ioscompanion): stop tracking the runner archive build artifact * build: produce the runner archive from source like the companion bundle * refactor(conformance): move the gate harness out of examples/folio * chore(folio): drop the gate harness wiring from the example app |
||
|
pj
|
94d9511312 |
test: full test-suite refactor sweep (#61)
* chore(test): start test-suite refactor sweep * test(ltl): pin exact multi-obligation residual AST * test(ltl): table-test finalize Kleene connective combinations * test(ltl): pin reduce over pending inner for bound, Or, Not * test(ltl): marshal bounded Always steps/duration/deadline * test(verifier): cover LTL combinator verdict transitions and within unit panic * test(verifier): table-test DecodeAction kinds and lastAction field exposure * test(verifier): assert WithPlatform(ios) reaches the picker host and key pool * test(verifier): widen weighted-selection assertion to a 5x skew margin * test(verifier): un-skip ax-find round trip with a committed tree fixture * test(runner): pin isWDADrop to sidecar reconnect-failed message origin * test(runner): assert PressKey/Wait trace encoding records kind-specific fields * test(runner): cover RenderSummary unsupported-verbs surfacing branch * test(trace): set Hierarchy in round-trip and lock lossy Tree contract Also add a -race concurrent WriteStep test that asserts N well-formed JSONL lines, catching torn lines if the writer mutex is dropped. * test(trace): round-trip witnesses/changes/metrics/exceptions, pin step-0 witness * test(trace): document ViolationsAreGreppable grep contract and lock-free WriteScreenshot * test(hierarchy): cover invalid-JSON and malformed-bounds parser paths * test(trace): guard writer mutex via WriteStep/Close race on w.file * test(replay): drop unfailable assets and devproxy assertions * test(replay): cache reuses on equal mtime, reparses after append * test(replay): violation marker falls back to detection step when attributed missing * test(replay): corrupt meta/trace dirs return 500 with error body * test(replay): SSE client receives runs.changed after a broadcast * test(replay): Run coalesces creates, ignores write/chmod, closes subs on cancel * fix(sidecar): synchronize health fixture writes and exercise healthError * test(sidecar): cover swipe/longpress/doubletap/erase/presskey/metrics/logs translations * test(sidecar): cover DoubleTapSelector composition and mid-gesture cancel * test(sidecar): assert gRPC error status surfaces from action RPC * fix(chrome): route action methods through runCtx so caller cancellation aborts CDP * fix(chrome): route hierarchy/screenshot/waitidle/metrics through runCtx * refactor(ios): extract pure simctl JSON parsers * refactor(ios): add command-runner seams for EnsureSimulator * test(ios): table-test simctl parsers and EnsureSimulator seams * test(sidecarassets): cover placeholder build path * test(sidecarassets): assert reuse via sentinel bytes not mtime * test(bundler): cover properties-only spec registration * refactor(testrun): extract prepareBundleInputs from Execute * test(testrun): cover prepareBundleInputs aliases and missing-runtime error * test(testrun): table-test resolveRuntimeSibling search edges * test(testrun): exact-output tests for progressHandler line format * fix(cmd): point bundle-check aliases at pkg/spec/src * test(cmd): smoke-test bundle-check resolves spec aliases * test(cmd): table-test hier-check parse and FindAll on fixture * test(cmd): unit-test buildBrowseURL deep-link vs root * test(cmd): drop flaky TestRun_Doctor that launched real Chromium * test(cmd): pin pipeline error to bundle resolution on web platform * test(replay-ui): add bun test script * ci(replay-ui): run bun test via make web-test target * ci(replay-ui): point bun cache key at replay-ui/bun.lock * test(replay-ui): exercise real URL encoding and non-ok throw in getJson * refactor(replay-ui): extract snapshot flatten/getAtPath into lib module * test(replay-ui): pin snapshot flatten/getAtPath path round-trip * refactor(replay-ui): extract action selector/format into lib module * test(replay-ui): pin action selector parse and row formatting * refactor(replay-ui): share one statusFor between panels * refactor(replay-ui): extract run-history derivation into lib module * test(replay-ui): pin shared statusFor precedence and ordering * test(replay-ui): pin run-history derivation alignment * refactor(replay-ui): export clampIndex for testing * refactor(replay-ui): extract keyboard-nav dispatch into pure module * refactor(replay-ui): extract metrics formatters into lib module * test(replay-ui): pin clampIndex step boundaries * test(replay-ui): pin keyboard-nav ownership and key routing * test(replay-ui): pin metrics formatters and path gap handling * refactor(sidecar): expose device-output parsers as internal for testing * test(sidecar): table-test device-output parsers against malformed input * test(sidecar): cover logcat parsing year inference and line skipping * test(sidecar): pin pressKey keycode mapping and unknown-key rejection * test(sidecar): metrics bundleId falls back to launched app and honors override * test(sidecar): loosen deadline upper bound to tolerate slow CI scheduling * test(web-runtime): export selector builders for unit tests * test(web-runtime): guard sanitize cycle, function, and depth limits * test(web-runtime): table-test selector builder quoting and escaping * test(sidecar): collapse scalar-forwarding RPC tests into a table * test(replay-ui): dedup step/summary fixtures into shared module * test(ios): collapse pickSimulator point-tests into a table |
||
|
pj
|
88db9653e5 |
refactoring default action layer (#51)
* feat(hierarchy): add editable signal with native derivation
* feat(chrome): emit editable flag in hierarchy dump
* feat(verifier): expose editable on ax element objects
* feat(spec): add editable to selector and element types
* feat(verifier): register typing builtin generator
* feat(verifier): typing builtin types edge-case corpus into editable fields
* feat(spec): export typing builtin generator
* feat(spec): add defaultActions bundle
* feat(spec): export @sanderling/spec/defaults subpath
* feat(folio): layer defaultActions breadth over targeted flows
* test(verifier): typing builtin targets editable fields, declines otherwise
* test(hierarchy): editable derivation and selector matching
* test(spec): defaultActions, typing, and defaults barrel resolve
* fix(testrun): alias @sanderling/spec/defaults for the bundler
* test(chrome): editable flag for inputs, textarea, contenteditable
* feat(spec): typing builtin for the web (V8) action path
* chore(folio): auto-boot a bootable AVD in just test/install when none connected
* feat(driver): add ForegroundChecker optional capability
* feat(android): detect foreground package via adb dumpsys
* feat(sidecar): implement ForegroundApp via adb for android
* feat(runner): relaunch app when foreground escapes during exploration
* fix(spec): drop hardware back from defaultActions to stay in-app
* feat(spec): add DoubleTap action type and constructor
* feat(spec): wire DoubleTap through web-runtime serializer
* feat(verifier): bind doubleTap and decode DoubleTap actions
* feat(runner): dispatch DoubleTap as two taps inside one step
* test(doubleTap): cover constructor, verifier round-trip, and runner dispatch
* feat(folio): add noDuplicateTxnPerStep invariant and doubleSubmitTxn action
* fix(folio): track ledger row count across non-ledger steps; pin reproducer seed
* feat(spec): add doubleTaps random-target builtin to defaultActions
* feat(verifier): add doubleTaps random-target generator
* refactor(folio): drop doubleSubmitTxn; fuzzer surfaces double-submit via defaultActions
* fix(folio): make ledgerRowsSeen monotonic to suppress transient-render false positives
* feat(verifier): track newly-violated property set per step
Sticky `always(P)` violations re-surfaced on every step after onset,
flooding traces and summaries with duplicate records. EvaluateProperties
now diffs against the prior verdict map and records the onset set; a new
NewlyViolatedProperties accessor exposes it so callers can emit each
violation exactly once at its onset step. The verdict-map return is
preserved for residual / current-verdict consumers.
* refactor(runner): emit onset-only violations to trace and summary
Switch the per-step violation list from the sticky verdict map to the
verifier's onset set. Each property now appears exactly once across a
run: at the step it first violates, not on every subsequent step where
the residual stays false. Removes the dead violationNames helper.
* style(verifier): use maps.Copy for verdict snapshot
* fix(folio): make login spec content-driven (idempotent across re-entries)
* fix(verifier): canonicalize selector strings
Object/chain JS selectors used to fall through to goja's default
stringification, producing "[object Object]" tags that surfaced as
garbage in trace.action.selector. Emit canonical "k:v" / " > "-joined
strings instead so the tag round-trips back through the hierarchy
selector grammar.
* refactor(folio): replace txn invariants with balanceMatchesAddedTxn
Collapse noDuplicateTxnPerStep and newTxnChangesBalance into a single
per-row property: every newly-appearing ledger row's signed amount must
match the ledger balance delta. A double-submit lands two rows whose
individual amounts cannot both equal the aggregate delta, so each row
fires the property, catching both the row-count and balance-math
classes of bug under one semantic invariant.
* refactor(trace): drop WriteScreenshotAfter
Only one screenshot per step is captured now (concurrently with
hierarchy after settle), so the -after.png variant is unused.
* refactor(runner): one concurrent screenshot per step
Move screenshot capture into the post-action errgroup so it observes
the same UI moment as the hierarchy fetch. Drop the pre-action and
deferred -after captures. Skip WaitForIdle when the action is Wait
since the wait itself provides settling time.
* refactor(inspect-ui): use next step's screenshot for state after
Each step now has one screenshot (the moment of observation). The
"state after" view of step N is the same moment as step (N+1)'s
observation, so reuse that file rather than expecting a separate
-after.png.
* feat(sidecar): structural-hash settle poll
Add pollUntilStable and structuralHash helpers; wire them into the
Stub, Maestro, and iOS backends' waitForIdle. The structural hash
ignores bounds-only flicker (measure passes) but trips on any change
in resource-id/class/content-desc/text, so a Compose cross-fade where
both source and destination composables are momentarily alive no
longer slips through Maestro's waitForAppToSettle and contaminates
the next hierarchy fetch.
* test(sidecar): cover pollUntilStable and structuralHash
Verify the poll returns on two equal snapshots, after transient
churn, and at the cap when never stable; assert the hash ignores
bounds-only flicker and detects content changes.
* feat(spec): accept optional name on extract()
Add an (name, getter) overload so each extractor handle carries a
debuggable label that future trace fields (per-step diffs) can key
off. The web-runtime falls back to extractor_\${index} when none is
supplied so existing call sites keep working unchanged.
* test(spec): cover extract name overload
Verify the runtime receives an undefined name in the legacy shape,
the supplied name in the (name, getter) shape, and that
extract("name") with no getter throws.
* feat(verifier): name extractors for diff surfacing
bindExtract accepts an optional name argument; falls back to
extractor_N when omitted. The name is stored on extractorState
alongside prev/curr value caches that the next change will use to
emit per-step diffs.
* chore(folio): name every extract() call
Give each extractor in the Folio spec a debuggable label so the
inspect UI can render extractor-value diffs at violation steps
keyed by intent (ledgerRows, route, ledgerBalance, ...) rather
than by registration index.
* feat(verifier): track extractor value transitions
Cache each extractor's prior and current JSON-encoded value during
PushSnapshot; expose ChangedExtractors to surface per-step diffs the
runner can emit into the trace. The first observation flushes every
non-null extractor as a change so the inspect UI shows initial state
breadcrumbs alongside later transitions.
* test(verifier): cover ChangedExtractors diffs
Verify initial snapshot reports both named and fallback-named
extractors, a subsequent change surfaces prev/curr, and a no-op
snapshot leaves the diff empty.
* feat(trace): emit extractor_changes per step
Add ExtractorChanges to trace.Step and a runner helper that converts
the verifier's diff map into the trace shape. The inspect UI keys
its violation breadcrumbs off this field.
* feat(inspect-ui): render extractor-change breadcrumbs at violations
Show prev -> curr for each extractor whose value changed on the
selected step, anchored under the violation row in ActionList.
Long values collapse into <details> so the inline diff stays
readable while the full payload is one click away.
* fix(sidecar): cap stability poll independently of settle budget
The previous shape halved durationMillis between waitForAppToSettle
and the structural poll, then hammered hierarchy() at 80ms intervals
- on Maestro this stacked enough RPCs that hierarchy fetches began
timing out under load and the run stalled. Pass the full budget to
waitForAppToSettle and cap the follow-up structural poll at 600ms
with a 120ms interval, so the device sees at most a handful of
extra hierarchy reads per step.
* feat(cli): default --clear-data on so runs start fresh
* feat(sidecar): streak-based settle with route-transition detection
Two changes layered into the stability poll:
1. stabilitySnapshot returns null while the tree carries more than one
route-level Screen tag (resource-id / testTag / identifier ending
in "Screen"), so the poll cannot declare a NavHost cross-fade
stable. Apps following the Compose route convention get this
detection for free; apps that don't fall through to the generic
signal below.
2. pollUntilStable now requires an uninterrupted stable streak of at
least MIN_STABLE_STREAK_MILLIS rather than just N consecutive
matches. A late transition that fires after a brief calm window
breaks the streak instead of slipping past. Interval widened to
250ms so UiAutomation isn't hammered under fuzz load.
* test(sidecar): cover streak reset and route-transition rejection
Verify the poll honors MIN_STABLE_STREAK_MILLIS, that a transient
mid-stream change resets the streak, that null returns block streak
progress through a NavHost cross-fade, and that stabilitySnapshot
counts only route-level attribute keys when summing Screen tags.
* feat(runner): re-fetch on transitional hierarchy capture
Some actions trigger async work (DB write, ViewModel coroutine) whose
navigation transition begins after the sidecar settle poll has already
exited. Without intervention, the next iteration's hierarchy fetch
lands mid cross-fade and the verifier observes a partial extractor
state which then surfaces as a false-positive violation at the step
where the transition completes.
fetchSyncedState pairs hierarchy + screenshot in one goroutine and
retries the pair (up to 4 times, 200ms apart) while the captured tree
contains more than one route-level *Screen tag. Steps that observe
no transition get no added cost; steps that catch a transition pay
up to ~600ms extra wall time but record a tree that matches the
post-transition state the property language expects to compare.
* feat(runner): gate first action on app reaching foreground
* test(runner): cover startup foreground gate and back-press
* feat(verifier): scope random-action targets to app package
Random tap/doubleTap/type/swipe candidates now exclude nodes whose package differs from the app under test, so exploration never fuzzes the soft keyboard, system UI, or permission dialogs. An unset app package or an element with no package stays in scope, preserving behavior on iOS.
* feat(testrun): pass app package into verifier scope filter
* test(verifier): cover package-scoped target selection
* feat(hierarchy): derive package from resource-id prefix
The Android sidecar omits an explicit package attribute, so the verifier's package scope filter was a no-op and the keyboard still leaked into targets. Native nodes carry their package as the resource-id prefix; derive it there when the attribute is absent. Compose testTags are colon-less and stay empty, keeping them in scope.
* test(hierarchy): cover package derivation from resource-id
* chore: stop tracking inspect-ui/dist build artifacts
* feat(android): detect focused-window package via dumpsys window
* feat(driver): add FocusedWindowChecker capability
* fix(runner): gate first observe on the app window being drawn, not just resumed
* test(mock): add FocusedWindowApp with foreground mirroring
* test(runner): cover startup gate waiting for app window to draw
* feat(proto): add Snapshot RPC for atomic hierarchy+screenshot
Pairs hierarchy and screenshot in a single response so the runner can
capture both under a backend mutex, avoiding the cross-fade race where
the two reads describe different frames.
* feat(sidecar): add snapshot default on DriverBackend
Default impl calls hierarchy() then screenshot(). The service layer wraps
the call in a mutex so concurrent runners observe a serialized pair.
* feat(sidecar): wire Snapshot handler with serialization lock
Synchronizes backend.snapshot() so concurrent runners observe a
serialized hierarchy+screenshot pair, eliminating the cross-fade race
where two parallel reads describe different frames.
* test(sidecar): cover Snapshot wire path and serialization lock
SnapshotHandlerTest asserts both fields are populated, concurrent calls
are serialized, and the default impl runs hierarchy then screenshot.
* feat(driver): expose Snapshot on DeviceDriver and sidecar client
Snapshot wraps the new atomic-snapshot gRPC: the runner gets hierarchy
and screenshot from one round-trip whose two reads are serialized on
the sidecar side.
* feat(driver): add Snapshot to chrome and mock drivers
The chrome tab is single-threaded so its Snapshot pairs the two reads
without extra locking. The mock records ActionSnapshot so tests can
assert the runner reaches for the paired RPC.
* refactor(runner): observe each step via the atomic Snapshot RPC
fetchSyncedState now issues one Snapshot per attempt so hierarchy and
screenshot describe the same on-device frame. The transitional retry
stays: that case handles a fully-captured but mid cross-fade frame,
which atomic capture cannot fix.
* test(runner): assert step uses Snapshot, not raw hierarchy/screenshot
TestRunner_UsesAtomicSnapshot catches regressions to the two-goroutine
race, and the existing parallel-fetch test now keys off ActionSnapshot.
* test(driver): cover Snapshot in proto descriptor and sidecar client
Adds Snapshot to the descriptor allowlist and a sidecar-client test that
asserts both fields come back over the wire.
* feat(trace): add Transitional flag to Step
* fix(runner): skip verifier for transitional trees after retry budget
When fetchSyncedState exits its retry loop with a tree that still shows a NavHost cross-fade, the runner now marks the step transitional, writes the step + screenshot to the trace, and skips Verifier.PushSnapshot / EvaluateProperties / ChangedExtractors so the previous-to-current extractor advance is not poisoned by transient state. The next clean step's previous still references the prior clean state. NextAction continues to run so the loop never deadlocks on a never-stabilizing screen.
* test(runner): cover transitional step skips verifier and clean control
* refactor(trace): rename Step.Action to Step.NextAction
The trace step's action field is the action chosen FOR THE NEXT iteration
based on observing this step's hierarchy, not the action that produced
this step. Rename Step.Action to Step.NextAction and the JSON tag to
next_action to make causality explicit at the data level.
* refactor(runner): assign trace action to Step.NextAction field
Follows the rename of trace.Step.Action to Step.NextAction. The runner
already computed the next iteration's action here; only the field name
changes.
* refactor(inspect): decode trace step's next_action JSON field
Mirrors the trace schema rename of action to next_action. The summary
shape exposed to the SPA (action_kind/action_label) keeps its current
JSON tags since these are derived labels, not the raw next-action.
* test(inspect): update fixtures to use next_action trace field
Aligns inspect tests with the trace schema rename. Step constructors
now set NextAction and the JSONL fixtures use the next_action tag.
* refactor(inspect-ui): rename Step.action to Step.next_action
Aligns the SPA type and consumers with the trace schema rename. The
StepSummary.action_kind/action_label labels stay unchanged since they
are derived labels, not the raw next-action.
* fix(folio): extract balanceMatchesAddedSum predicate as testable helper
Move the ledger-balance-vs-added-rows predicate into a pure helper module
so the property's logic is unit-testable in isolation. Marks the sanderling
example as an ES module so cross-package ESM imports resolve under node.
* fix(folio): use sum-of-added-rows in balanceMatchesAddedTxn
The old predicate (every row's signed amount equals delta) silently passed
the double-submit bug because two same-amount rows each match the delta in
isolation. Switching to the sum check (addedSum === delta) catches both the
double-submit case and any future multi-row append whose total drifts from
the balance change.
* test(spec): cover balanceMatchesAddedSum single, sum-match, over, under cases
Pins the sum-based predicate: a single new row matching delta and two new
rows summing to delta both hold; two-row over-sum (double-submit) and
under-sum cases both violate.
* fix(build): rebuild sidecar JAR when Kotlin sources change
Without source-file deps on $(SIDECAR_JAR), make never re-ran shadowJar
after a Kotlin edit, so a stale embedded JAR shipped on every install
and the new sidecar code was silently absent at runtime.
* fix(chrome): launch with no-sandbox so headless Chrome starts in CI
* fix(sidecar): type text at cursor instead of clearing the field
InputText now appends at the focus caret, matching the native driver
and the standard mobile-input contract, instead of deleting existing
content first. Adds an injectable command runner so the behavior is
testable without a device.
* test(sidecar): assert InputText types at cursor without clearing
Captures the adb command stream and verifies a single input-text call
with no preceding delete keyevents, plus the adb escaping cases.
* feat(proto): add LongPress RPC
* chore(proto): regenerate Go stubs for LongPress
* feat(driver): add LongPress to DeviceDriver interface
* feat(sidecar): add LongPress client method
* feat(mock): record LongPress action
* feat(chrome): implement LongPress as press-and-hold
* feat(sidecar): implement longPress across backends
* feat(sidecar): dispatch LongPress RPC to backend
* test(sidecar): cover LongPress dispatch
* test(sidecar): implement longPress in snapshot test backend
* feat(verifier): add LongPress and Scroll action kinds
* feat(folio-spec): predicate that gates balance check on TxnSubmit tap
Replaces the row-sum predicate (which always held by construction since
balance is derived from rows in Folio) with one that compares the typed
amount to the actual balance delta after a tap on TxnSubmit. Catches the
planted double-submit bug.
* feat(folio-spec): wire submitMovesBalanceByTypedAmount property
Adds lastAction and totalBalance extractors and uses them in the new
property. Drops ledgerRows/ledgerBalance extractors since nothing else
referenced them.
* feat(verifier): wire longPresses and scrolls generators
* test(verifier): cover longPresses and scrolls generators
* test(folio-spec): unit tests for submitChangesBalanceByTypedAmount
Covers single vs double submit, the DoubleTap variant, vacuous cases
(null action, wrong kind, wrong target, zero typed), and selector-as-
object coercion.
* feat(spec): add LongPress and Scroll authoring surface
* feat(spec): no-op LongPress and Scroll in web runtime
* feat(spec): re-export longPresses and scrolls as opt-in generators
* test(spec): cover LongPress and Scroll runtime members
* test(proto): expect LongPress in service descriptor
* feat(runner): dispatch LongPress and Scroll actions
* test(runner): cover LongPress and Scroll dispatch
* docs(action-space): move LongPress, Scroll, DoubleTap to current actions
* fix(runner): mark nil/empty hierarchy as transitional
A failed or empty sidecar hierarchy fetch was pushed straight to the
verifier, letting spec extractors crash with "Cannot read property 'map'
of undefined" when findAll returned null. Treat that case like a
transitional capture: skip the verifier push, still record the step, and
keep the loop progressing.
* fix(verifier): populate Action.On when tap chooser picks an element
Coordinate-targeted Taps/DoubleTaps left On empty, so action-gated
properties reading lastAction.on couldn't tell which target was hit and
were vacuously skipped. Resolve the picked element to a stable
key:value selector (resource-id, testTag, text, desc) and validate it
resolves back to the same element so we don't accidentally redirect the
tap to a sibling that shares the identifier.
* fix(folio): add parseTypedAmount helper matching app's parseCents
Raw user input like "50" must become 5000 cents, not 50. The existing
parseDollarCents helper strips non-digits and so reads "50" as 50 cents,
which is correct for formatted balance text but off by 100x for raw
input from the amount field.
* fix(folio): parse raw amount input as cents in submit predicate
txnAmountField holds raw user keystrokes, not formatted balance text.
Route it through parseTypedAmount so "50" reads as $50, matching how
the app commits the transaction.
* fix(folio): carry forward total balance across off-screen transitions
AddTransactionScreen shows neither AccountCard nor LedgerBalance, so the
extractor used to report 0 at the step before submit. That made every
non-zero current balance look like the full delta and tripped the typed
amount property on every honest submit. Remember the last-seen sum and
return it whenever the current snapshot has no balance signal.
* test(folio): cover submit predicate with raw typed-amount inputs
Pipes realistic raw keystrokes through parseTypedAmount + the predicate
so single submits clear and double submits fire as expected.
* feat(folio): add computeHomeTotalBalance helper
Pure helper that tracks Home multi-account total only and carries the last
Home sum across off-Home steps. Ledger's single-account balance is excluded
because mixing it would corrupt cross-screen scale comparisons.
* fix(folio): totalBalance carrier tracks only Home, not Ledger
Home cardSum is a multi-account total; Ledger's LedgerBalance is a single
account on a different scale. Blending them in the carrier produced bogus
cross-screen deltas (prev from Ledger, curr from Home), triggering false
positives in submitMovesBalanceByTypedAmount. Restrict the carrier to
Home AccountCard totals via the computeHomeTotalBalance helper.
* test(spec): cover computeHomeTotalBalance carrier behaviour
Tests Home sums, carrier passthrough on off-Home steps, the Ledger
scale-mismatch case, and a Home > off-Home > Home sequence.
* feat(runner): treat transient apply errors as transitional steps
Sidecar input RPCs occasionally hang with DEADLINE_EXCEEDED or
UNAVAILABLE on long fuzzing runs. The per-step loop previously
propagated any applyAction error and killed the run after a single
flake. Detect transient gRPC failures via status.FromError, mark the
step transitional, skip the post-action idle poll, and continue to the
next step. Fatal errors (outer ctx cancellation, non-transient codes,
verifier crashes) still propagate.
* test(runner): cover transient apply error resilience
TestRunner_TransientApplyErrorMarksTransitional drives the runner
through a wrapper that fails the first TapSelector with a gRPC
DeadlineExceeded then succeeds. Asserts the run does not exit, the
failed step is marked transitional with no violations, and the next
step runs cleanly. TestIsTransientApplyError_Classification covers the
helper's matching rules directly so future code changes don't quietly
drop a transient case.
* fix(folio): gate submit-balance property on Home route landing
totalBalance is only freshly computed when AccountCards are visible on
Home; off-Home landings return the carrier and would false-fire the
property, latching always(next(F)) to false and masking the real
double-submit bug. Skip vacuously when route is not "home".
* test(spec): cover route gate in submit-balance predicate
Adds route arg to existing cases (all use "home") and adds five new
cases: ledger landing with stale carrier, add-transaction with
double-insert delta, null route, plus home-landing positive and
double-insert negative cases anchoring the gate's allow path.
|
||
|
pj
|
8ccf95c1cf |
refactor: rename project uatu -> sanderling (#24)
* refactor: rename Go module path uatu -> sanderling
Module path github.com/priyanshujain/uatu -> github.com/priyanshujain/sanderling,
including all imports and the proto go_package option. Generated .pb.go files
rewritten in-place; safe to regenerate with protoc later.
* chore(proto): regenerate driverpb after module path rename
The previous sed-based module rename corrupted the embedded descriptor
byte lengths. buf generate rewrites them cleanly.
* refactor: rename CLI binary uatu -> sanderling
Updates Makefile target + UATU_BIN var, .goreleaser project/build IDs,
.gitignore comment, and all user-facing strings in the CLI help text,
error messages, and tests. Binary is now bin/sanderling.
* refactor(sdk): rename Kotlin package dev.uatu.sdk -> dev.sanderling.sdk
Moves sdk/android/src/{main,test}/kotlin/dev/uatu -> dev/sanderling and
rewrites package declarations, imports, and the Gradle namespace. Class
names (Uatu, UatuRuntime) are renamed in a follow-up commit.
* refactor(sidecar): rename Kotlin package dev.uatu.sidecar -> dev.sanderling.sidecar
Moves sidecar/src/{main,test}/kotlin/dev/uatu -> dev/sanderling and
rewrites package declarations, imports, and the application mainClass.
* refactor: rename Uatu API surface -> Sanderling
- Kotlin: Uatu -> Sanderling, UatuRuntime -> SanderlingRuntime (+ files).
- JS host binding: globalThis.__uatu__ -> __sanderling__ (Go verifier,
spec-api, tests).
- TS interface: UatuRuntime -> SanderlingRuntime; internal tags
__uatuFormula / __uatuActionGenerator -> __sanderling* variants.
- Go trace: UatuVersion field + uatu_version JSON tag renamed.
- Socket naming: uatu-agent / uatu-agent-reader -> sanderling-agent*.
- Sample app, docs, inline-JS test strings updated to match.
* refactor(examples): rename examples/folio/uatu -> examples/folio/sanderling
Renames the example spec directory; updates justfile paths + gitignore
entries accordingly. Package.json name/description and @uatu/spec
dependency are renamed in the npm + docs commits.
* chore(build): rename gradle property + rootProject.name uatu -> sanderling
- Renames the uatu.version gradle property and all its -P references in
Makefile, build.gradle.kts files, and .github/workflows/release.yml.
- settings.gradle.kts rootProject.name = "sanderling".
- Renames .env.local.example header + release-cli workflow job name.
* refactor(proto): rename proto package uatu.driver.v1 -> sanderling.driver.v1
Updates the proto package and java_package, regenerates driver.pb.go +
driver_grpc.pb.go, rewrites Kotlin imports and the gRPC ServiceName
assertion in driver_test.go.
* refactor: rename npm package @uatu/spec -> @sanderling/spec
Renames package name in pkg/spec-api/package.json + lockfile, all
consumer imports (examples/folio spec, testdata, verifier tests), the
esbuild alias in cmd/sanderling/test_run.go, and related doc references.
* docs: rename uatu -> sanderling in README, docs, and URLs
- README + docs/{manual,development}/*: narrative + GitHub + Pages URLs.
- POM + npm package.json repo/homepage/bugs URLs.
- .gitignore + embed_stub + Makefile-comment references updated to
'make sanderling'.
- Minor narrative comments in cmd/sanderling/test_run.go and
internal/inspect/server.go.
* refactor: rename remaining internal uatu strings -> sanderling
- SANDERLING_TEST_PHONE/OTP env vars (cmd + bundler tests).
- sanderling-sidecar runtime tmp dir + extracted JAR filename.
- Inspect web UI: @sanderling/inspect-web package, title, theme
localStorage key, RunList empty-state copy, uatu_version TS field.
- Sample app storage key sanderling.ledger.v1.
- Test data: sanderling_test AVD name + com.example.sanderling_test.
- Release docs tarball name template.
|
||
|
pj
|
13bb2feb82 |
feat: uatu inspect UI (web trace explorer) (#23)
* feat(trace): extend Step/Action/Meta schema for inspect UI
Add Step.Hierarchy, Step.Residuals, Action.Selector/ResolvedBounds/TapPoint,
Meta.EndedAt and JSON tags on hierarchy.Element/Bounds/Tree so trace.jsonl
can drive the upcoming uatu inspect web UI.
* test(trace): cover EndedAt + new step fields round-trip
* feat(ltl): MarshalJSON for Formula AST + Evaluator.Residual()
Each Formula concrete type now serializes to a closed-set residual node
(true/false/not/and/or/implies/always/now/next/eventually/predicate/error)
that mirrors the TS spec API surface. Evaluator.Residual() folds pending
obligations into a single Formula so the runner can stamp one residual
per property per step into trace.jsonl.
* feat(runner): stamp residuals, hierarchy, selector targets, ended_at
Each Step now carries the captured hierarchy, per-property residual ASTs,
and (for Tap/InputText) the selector + resolved bounds + tap point. The
test_run command writes meta.ended_at on graceful shutdown so the inspect
UI can distinguish completed runs from in-progress ones.
* feat(inspect): scaffold embed dist for SPA assets
Stage 2 stub for the inspect server. Real web bundle gets wired in
Stage 4 (Makefile copies web/dist into internal/inspect/dist).
* chore(web): ignore web/ build output in root .gitignore
* chore(web): add bun + vite + vitest scaffold config
* feat(web): monochrome design tokens, typography, app shell CSS
* chore(web): placeholder for self-hosted JetBrains Mono fonts
* feat(web): index.html entry with style links and root mount
* feat(web): typescript types mirroring run/step trace schema
* feat(web): typed fetchers for runs/steps/screenshots
* feat(web): App shell with router and run/step routes
* feat(inspect): runs scan, lazy step parse, mtime-aware cache
* feat(web): RunList route with table, loading, and error states
* feat(web): RunDetail route shell with three placeholder panels
* feat(inspect): fsnotify-backed runs watcher with debounce
* fix(web): use jest-dom/vitest entry so matchers register
* test(web): cover listRuns happy path and error response
* test(web): render RunList with mocked fetch and assert row
* chore(web): commit bun lockfile
* feat(inspect): http handlers for runs/steps/screenshots/SSE
* test(inspect): cover handlers, screenshot whitelist, SSE, dev proxy
* feat(cmd): add 'uatu inspect' subcommand
* fix(web): align TS types with snake_case wire format
Go inspect server serializes RunSummary, StepSummary, Step, Meta with
snake_case JSON tags (matching the on-disk trace.jsonl/meta.json). Update
the TS types and consumers to match so API responses parse without
runtime undefined fields. Action keeps resolvedBounds/tapPoint as camelCase
because those keys were defined that way in the trace schema.
* feat(web): add ActionList panel for run-detail step navigation
* feat(web): add SnapshotTable panel with diff highlighting
Renders snapshots dictionary as a flat sorted dotted-path tree.
Changed leaves get data-changed plus a hover title with the previous value.
* test(web): cover SnapshotTable rendering and diff behavior
Eight cases: empty state, sort order, dotted-path expansion,
changed/unchanged/missing-previous flagging, and inline-vs-expanded arrays.
* feat(web): add Screenshot panel with bounds and tap overlays
Center column of run-detail page. Renders the device screenshot
scaled to fit, with an SVG overlay drawing resolvedBounds as a
violation-colored rect, tapPoint as a contrast ring, and swipes
as an arrow. Falls back to a placeholder when src is missing or
the image fails to load.
* fix(web): guard scrollIntoView call for jsdom compatibility
* test(web): cover Screenshot panel rendering and overlays
* test(web): cover ActionList rendering, selection, keyboard, and markers
* feat(web): add ExceptionsPanel component
* test(web): add ExceptionsPanel tests
* feat(web): add Timeline panel with property swimlanes
Renders SVG swimlanes per property with violated/pending/holds cells,
action-marker dots, click-to-seek, and a selected-step highlight bar.
* test(web): cover Timeline empty state, cells, status, click, highlight
* feat(web): add ResidualNode recursive AST renderer
* test(web): cover ResidualNode operators, predicate, and error chip
* feat(web): add ViolationsPanel with status badges and jump button
* test(web): cover ViolationsPanel rows, status grouping, and jump button
* test(web): register testing-library cleanup globally
All six panel test files added local afterEach(cleanup); centralize it in
the shared setup so future tests inherit DOM isolation by default.
* feat(web): hooks for url/keyboard/theme/sse
* feat(web): wire all panels into run-detail with phone-dominant grid
ActionList left, Screenshot center, Snapshots/Properties/Exceptions
stacked right, Timeline bottom. URL-synced step index (useStep), keyboard
shortcuts (j/k/arrows/g/G/.), light+dark theme toggle stored in
localStorage, SSE auto-refresh on the run index.
* test(web): add three reference run fixtures (clean, violation, exception)
* build: web targets in Makefile + bun in CI; docs(inspect)
- Makefile: web-build/web-dev/inspect-dev/test-web targets; uatu and
install now depend on web-build so the binary embeds the latest SPA.
- ci.yml: setup-bun + cache; existing make test now runs web typecheck +
vitest as part of the full suite.
- docs/manual/inspect.md: panel reference, keyboard shortcuts, URLs.
- docs/manual/cli.md: document uatu inspect.
- README: link to inspect docs.
* feat(runner): capture a screenshot per step
The driver already exposes Screenshot(ctx), but the runner never called
it. Each step now writes <run>/screenshots/step-NNNNN.png right after
the trace line, using the same failure-is-a-warning posture as other
best-effort observability hooks. Makes the inspect UI's center panel
actually useful.
* feat(inspect): include action_label in StepSummary
Tap/InputText/Swipe/PressKey/Wait each get a short human-readable
label (selector, quoted text, swipe direction, key name, duration) so
the action list panel can render readable rows instead of just 'Tap'
with no target.
* test(inspect): accept either #app or #root in SPA shell fallback
* feat(web): render action_label and screen in ActionList rows
Step rows now show 'Tap id:save', 'InputText "alice"', 'Swipe up',
'PressKey back', etc. Steps with no action fall back to
'observe @ <screen>' so the list reads as a flow instead of a wall
of '--' placeholders.
* feat(sidecar): implement screencap for android driver backend
Was stubbed to return an empty byte array, which made the runner's
per-step screenshot capture a no-op. Shell out to 'adb exec-out
screencap -p' and stream the PNG bytes back. Width/height stay zero
because the PNG header carries them; the Go side can parse if needed.
* feat(proto): add Metrics RPC for per-step CPU and memory capture
* feat(driver): Metrics(bundleID) returns cpu_percent + heap/total bytes
* feat(sidecar): implement Metrics RPC via adb top + /proc/<pid>/status
* feat(runner): capture metrics + before/after screenshots per step
Each step now writes step-NNNNN.png (before applyAction) and
step-NNNNN-after.png (after the action + wait-for-idle). The runner
samples Driver.Metrics(bundleID) before writing the trace line and
stamps Step.Metrics with cpu_percent, heap_bytes, total_memory_bytes
so the inspect UI can chart CPU and heap over the run.
* fix(runner,sidecar): measure CPU across step via /proc stat delta
'top -d 0.3 -n 2' measures CPU in a 300ms window that coincides with
the SDK-paused app, always reporting 0%. Switch to reading
/proc/<pid>/stat utime+stime and computing the delta between successive
calls; the natural step cadence gives a 2-5s measurement window that
captures the action response and render cycle. Also moved the sample
to before snapshotStep so the delta starts before the SDK pause.
* feat(web): add Metrics type for per-step cpu and memory
* refactor(web): replace --accent-change with --accent-positive token
* refactor(web): recolor chip-progress as neutral outlined chip
* refactor(web): use neutral border for changed snapshot rows
* feat(web): add MetricsChart panel with HEAP and CPU lanes
SVG-based time-series chart rendering heap bytes and CPU percent per
step across two stacked lanes, with a shared step axis below. Lines are
monochrome; a vertical highlight marks the selected step; per-step hit
rects make any click seek to that step.
* feat(web): revamp ActionList with tag targets, elapsed time, and expandable rows
Render selector-based Tap actions as <tag/> markup, show zero-padded MM:SS.mmm
elapsed time per row, and expand the active row with Position/Content sub-rows
when a full Step is available. Adds formatActionRow/formatElapsed helpers and
covers both with unit tests.
* fix(runner): stop copying Tap selector into action.text
The 'Content' inspect row should show the user-supplied text for
InputText actions and stay empty for Taps. Previously the runner copied
action.On into traceAction.Text for both, so the inspect UI showed the
selector as the tap's 'Content'.
* fix(web): use text-muted for swipe arrow after accent-change removal
* fix(web): snapshot values truncate with ellipsis + title tooltip
Long JSON values were breaking one character per line due to
overflow-wrap:anywhere in a narrow column. Switch to single-line ellipsis
with the full value exposed via the title attribute on hover.
* feat(web): state-before/after columns + metrics chart at bottom
RunDetail now renders a four-column grid:
actions | state-before | state-after | side (exceptions + timeline)
with MetricsChart spanning the bottom row. Each state column shows its
own screenshot (step-NNNNN.png vs step-NNNNN-after.png), snapshot table,
and violations panel. ActionList now receives runStartMillis and the
selected Step so the active row can expand Position/Content sub-rows.
* fix(web): skip zero-value ticks + add exception markers to metrics
HEAP '0B' and CPU '100%' labels overlapped at the lane boundary. Drop
the bottom-of-range tick on both lanes (baseline is implied) and widen
LANE_GAP so the remaining labels have breathing room. Accept an
exceptionStepIndices prop and draw a dashed red vertical line at each
to surface exception spikes directly on the CPU/heap chart.
* fix(web): let action body column shrink below its content
Required minmax(0, 1fr) so the row grid honours the column's min-size of
0 instead of the implicit 'auto', preventing the action-list from
overflowing its parent when the target string is long.
* feat(web): bigger state screenshots + single properties row
Collapse snapshots into a summary chip ('SNAPSHOTS · N violations') so
the screenshot fills its state card. Deduplicate ViolationsPanel —
show it once in a new full-width 'properties' row between the state
cards and the timeline. Drop the right sidebar; exceptions now surface
as dashed markers on the metrics chart with the ExceptionsPanel only
rendering when there are actual exceptions to report.
* feat(web): add minimal Tabs component
Monochrome tab strip with underline-on-active. Used by state-before
and state-after cards to swap between Screenshot, Snapshots, Properties.
Pane scrolls internally so the outer grid stays fixed-height.
* feat(web): fold timeline into MetricsChart as STEPS lane
Adds a thin per-step status row above HEAP showing violated (red),
pending (dim gray) or holds (green-tinted). Extends highlight +
exception markers to span the status lane. Frees a whole row in the
detail grid so the page can fit in 100vh.
* refactor(web): tabbed state cards, drop standalone Timeline panel
State-before/after now use Tabs (Screenshot / Snapshots / Properties,
default Screenshot). Removes the dedicated timeline row; status lane
lives on the metrics chart. Banner is gone from the shell.
* feat(web): lock app shell to 100vh with no page scroll
html/body/#root fill the viewport, body gets overflow:hidden, and the
detail grid uses minmax(0, 1fr) rows so inner panels own their scroll.
Tightens toolbar + panel padding for a denser feel.
* feat(web): arrow-key nav + badges on Tabs (WAI-ARIA tablist)
Roving tabindex, ArrowLeft/Right/Up/Down/Home/End navigation, explicit
aria-selected/aria-controls/id wiring, and support for an optional
badge inside each tab (used for violation counts).
* feat(web): ViolationsPanel supports violationsOnly filter
* feat(web): ActionList arrow-key nav + listbox semantics + smaller font
Promote the list to role=listbox with role=option rows; roving tabindex
lets ArrowUp/Down (and Home/End) seek between steps with focus. Font
size dropped to 11px and padding tightened so long selector-tag labels
fit in the 340px actions column.
* fix(web): useKeyboardNav yields arrow keys to tablist/listbox targets
Previously pressing ArrowRight on a focused tab switched tabs AND
advanced the step. Skip arrow handling when the event target is inside
an element with an arrow-owning ARIA role.
* feat(web): fourth 'Violations' tab + wider actions + shorter metrics
Adds a Violations tab to each state card showing only violated properties
(with count badge on the tab label when > 0). Actions column widened
from 280px to 340px, bottom metrics strip trimmed from 220px to 140px
with tighter lane heights, so the whole page still fits in 100vh with
no scrollbar.
* feat(web): compact RunDetail layout using 1px borders instead of panel padding
* refactor(inspect): simplify MetricsChart to HEAP+CPU with time axis
Drop the STEPS status lane and per-sample circle markers, switch the
x-axis from step indices to mm:ss clock time, trim y-axis ticks to
min/max with compact units, rotate lane labels into the left gutter,
and replace the thin playhead line with a wider dotted red band.
Traces stay grayscale; red appears only on the playhead pattern.
* fix(web): RunList rows no longer stretch to fill viewport height
Tables inherited flex: 1 1 auto from .app-main > * and distributed extra
vertical space across rows. Override with flex: 0 0 auto + align-self.
* misc changes
* fix(web): hoist useState above early return in MetricsChart
Calling useState after an unconditional early return violates React's
Rules of Hooks: the empty-samples branch renders 0 hooks while the
populated branch calls 1. On the initial null->loaded transition of
history the hook count changes and React throws.
* fix(web): subscribe to named SSE event instead of 'message'
Server emits 'event: runs.changed' frames; the WHATWG EventSource spec
dispatches those as events of type 'runs.changed', not 'message'. The
listener registered on 'message' was never fired, so RunList never
auto-refreshed on run create/finish/delete.
* fix(inspect): unsubscribe SSE clients on disconnect
Watcher.Subscribe appended to a slice with no matching removal path,
so every closed EventSource connection leaked its channel. Over a
long-running server the slice grew unbounded and every fs event paid
O(N) iterating dead channels. Add Unsubscribe + defer it in
handleEvents.
Unsubscribe does not close the channel: broadcast snapshots the
slice without holding the mutex, so a concurrent close would race
with its non-blocking send.
* fix(trace): rename resolvedBounds/tapPoint to snake_case
Every other json tag in the trace schema (from_x, duration_millis,
bundle_sha256, etc.) uses snake_case. The two new Action fields
introduced with the inspect UI broke that pattern. Rename them
before the format ships to external consumers.
* chore(web): drop vitest and remove UI tests from CI
No UI tests wanted in web. Removes vitest, jsdom, testing-library
devDeps and the vitest.setup.ts + vite.config.ts test block.
Makefile test-web becomes web-typecheck (typecheck only).
Fixes CI failure where `vitest run` exits 1 with no test files.
* chore(make): dedupe sidecar embed and drop recursive make
Make $(SIDECAR_JAR) the real recipe and $(SIDECAR_EMBED) a file
target, so uatu/install/inspect-dev share one copy step and
sidecar/release-cli just depend on the jar instead of re-invoking make.
|
||
|
pj
|
a2e96af1af |
WIP: rename sample app to Folio (#20)
* refactor: rename examples/sample-app to examples/folio Directory-level rename and path references in Go tests, bundle-check, top-level README, and getting-started docs. Package declarations, Gradle config, iOS bundle IDs, and class names follow in later commits. * refactor(folio): rename Kotlin package dev.uatu.sample to app.folio Moves source dirs and sqldelight schema from dev/uatu/sample to app/folio, updates package declarations and imports, and switches Android namespace/applicationId, iOS binaryOption bundleId, and sqldelight database packageName to the new identifier. * refactor(folio): rename SampleApplication to FolioApplication Android manifest now points at .FolioApplication with label 'Folio' instead of 'Uatu Sample'. * refactor(folio): set iOS bundle id and display name to Folio bundleIdPrefix + PRODUCT_BUNDLE_IDENTIFIER -> app.folio. CFBundleName + CFBundleDisplayName -> 'Folio'. * refactor(folio): update demo email to [email protected] * refactor(folio): point justfile at app.folio bundle id Updates xcrun simctl launch target, uatu test --bundle-id, and the build/uninstall comments to reference folio instead of sample. * test: update fixture package ids to app.folio Sidecar activity-resolver test and verifier spec-integration XML fixtures referenced the old dev.uatu.sample Android package. Updates them to match the folio app's real package id so the tests stay representative of what the CLI sees on-device. * test(verifier): rename SampleApp identifiers to Folio Renames TestSampleAppSpec* functions, bundleSampleAppSpec helper, and sampleAppHierarchyXML const (now loginHierarchyXML for consistency with the other per-screen fixtures). Updates trailing sample-app mentions in comments and assertion messages. * refactor(folio): rename Gradle/npm/wasm project identifiers to folio settings.gradle.kts rootProject.name, package.json + package-lock.json name, and the WasmJS index.html <title> all still read 'uatu-sample' / 'Uatu Sample'. Realigns them with the Folio brand. * docs(folio): rewrite README title + getting-started bundle id examples/folio/README.md is now titled 'Folio' with the Kotlin source paths corrected to app/folio. Getting-started example uses --bundle-id app.folio. Harness launch message is now generic ('app under test') since uatu-sample-harness is not specific to folio. * chore(folio): drop trailing 'sample' reference in gradle.properties * refactor(folio): rename LoginPage composable to LoginScreen Align with KMP/Android industry convention (NowInAndroid, Cash App, JetBrains samples use Screen, not Page). * refactor(folio): rename HomePage composable to HomeScreen * refactor(folio): rename AddAccountPage composable to AddAccountScreen * refactor(folio): rename LedgerPage composable to LedgerScreen * refactor(folio): rename AddTransactionPage composable to AddTransactionScreen * refactor(folio): split Models.kt into app.folio.data package Account, Transaction (with TxnType), and Session move into their own files under app.folio.data, matching NowInAndroid-style per-type organization. * refactor(folio): move data layer into app.folio.data package Repository, LedgerStore (expect + interface), SqlLedgerStore, WebLedgerStore, DriverFactory (expect + actuals), AndroidLedgerContext, and Snapshot move into app.folio.data. Update all consumer imports. * refactor(folio): move Navigation into app.folio.navigation package Split the former Navigation.kt into Route.kt (sealed interface) and Navigator.kt (singleton). Update consumer imports across screens, App.kt, and FolioApplication. * refactor(folio): move Platform and Format into app.folio.platform Both files carry expect declarations (Platform object, formatDate); grouping them into a dedicated platform package makes the KMP seam obvious and mirrors the structure used by JetBrains samples. * refactor(folio): move login into feature/auth package Create app.folio.feature.auth with LoginScreen + LoginUiState. Inline the former Auth.kt (DEMO_EMAIL, DEMO_PASSWORD, checkCredentials) into LoginScreen since it is the sole caller. * refactor(folio): move HomeScreen into feature/home package * refactor(folio): move account creation into feature/account package AddAccountScreen gets its own AddAccountUiState colocated with the screen, replacing the shared UiState.addAccountError. * refactor(folio): move ledger screens into feature/ledger package LedgerScreen and AddTransactionScreen move into app.folio.feature.ledger with AddTransactionUiState (txnError, txnFormType) colocated. The former catch-all UiState.kt is removed now that each screen owns its state alongside its UI. * refactor(folio): split Theme.kt; move theme and icons to subpackages Theme split into Theme.kt (tokens, layout dims, LedgerTheme) and Type.kt (typography) under app.folio.ui.theme. Icons moves to app.folio.ui.icon. Update every consumer's imports to match. * refactor(folio): split ui components into per-file under ui/component Former Widgets.kt and Components.kt become 10 focused files: AppButton, Card, EmptyState, ErrorText, FieldLabel, Header, IconButton (w/ BackButton), Screen, Segmented, TextInput. Matches NowInAndroid style of one composable per file in a designsystem/component package. * chore(folio): consolidate uatu testing files under uatu/ folder Move spec.ts, package.json, package-lock.json into examples/folio/uatu so all uatu-specific testing artifacts live in one place. runs/ and node_modules/ follow the same convention (both remain gitignored). Update justfile, README, and the two Go consumers (bundle-check tool + verifier/trace tests) that referenced the old path. * refactor(trace): drop folio path in writer test Round-trip only needs a non-empty string; neutralize to keep the library free of folio references. * refactor(sidecar): neutralize ResolveActivity test fixtures Swap app.folio for com.example.app in the fixture strings so the sidecar tests don't reference the example app by name. * refactor(bundle-check): take spec path as argument Previously the tool hardcoded examples/folio/uatu/spec.ts. Accept a positional spec path instead so the tool works for any example and leaves no folio reference in the library surface. * test(verifier): add neutral integration spec and hierarchy fixtures Adds testdata/integration_spec.ts with two routes ("list", "form"), an InputText on text_field, a Tap on primary/secondary_action, a safety property (itemCountNonNegative), and a liveness property (submitEventually). Adds hierarchies_test.go with matching XML fixtures. Constants intentionally go in a _test.go at package root rather than testdata/hierarchies.go because go skips .go files under testdata/. * refactor(verifier): replace folio integration tests with neutral ones Renames bundleFolioSpec -> bundleIntegrationSpec and the three Test* entry points to TestIntegrationSpec*. Uses the synthetic spec and hierarchies added in the previous commit so the library's test suite no longer references examples/folio at all. Folio-specific coverage remains covered by examples/folio/justfile's 'just test' which exercises the real spec on device/emulator. * chore: remove cmd/uatu-sample-harness Not referenced by Makefile, docs, CI, or any script. Duplicates the adb reverse helpers already in cmd/uatu/test_run.go, and its name implies ownership by the sample app which violates the library/ example decoupling. If a bare-protocol debugging tool is later needed it belongs inside cmd/uatu/. * docs(folio): drop Layout section and KMP layout paragraph; fix AVD override syntax The directory-tree Layout section rots faster than the code and duplicates what ls shows for free. The expect/actual paragraph in Stack was the same kind of filler. The README also claimed 'just AVD=Pixel_7 test' but the justfile reads AVD as an env var via env_var_or_default, so the correct invocation is 'AVD=Pixel_7 just test'. |
||
|
pj
|
0c775c199d |
feat(examples): add sample-app spec
Introduce examples/sample-app/spec.ts — a minimal property-based spec that taps the sample app's "Click me" button and asserts click_count is monotonic. Bundle-check and the trace writer test now reference the new path. |
||
|
pj
|
72c4e4c4d3 |
feat(trace): JSONL writer for steps + meta + screenshots
Each WriteStep appends one JSON object per line so the trace can be filtered with jq directly. Screenshots land under screenshots/ with zero-padded indexes. Writer is concurrency-safe; Close is idempotent and a write after Close errors loudly rather than silently dropping. |