mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
b5ec7b2ce65290b21699f32bb2eb7ad33899524d
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 pj
|
b02e86b2e3 |
ci: dispatch workflows for folio and the replay ui (#73)
* feat(runner): stop the step loop at the first violation on request
* feat(testrun): report violations as a typed error under exit-on-violation
* feat(cli): add --exit-on-violation and exit 2 when it fires
* docs(cli): document --exit-on-violation, --max-steps, and exit codes
* fix(web): enumerate and query across shadow roots in both producers
* test(chrome): compare both producers on a shadow-dom parity page
* test(browser): drive a canvas-under-shadow-root fixture end to end
* fix(web): select the focused field inside a shadow root before typing
* fix(web): report the pathname as the screen when there is no hash route
* fix(web): settle on dom quiescence instead of returning at body ready
* feat(replay-ui): add data-testid hooks the dogfood spec drives
* feat(replay-ui): add the dogfood spec sanderling runs against the replay ui
* fix(replay-ui): scope the screenshot property to the named state panel
* chore(make): add per-platform sanderling build targets
* ci: add dispatch workflows for folio and the replay ui
* docs: describe the dispatch workflows and how to read a failure
* ci(folio): give the ios leg its jdk, android sdk, just, and a clean app start
* refactor(web): use max for the settle budget
* ci: pin calibrated seeds, skip the flaky ios reinstall, bound every job
* ci: authenticate and pin the buf setup step
the anonymous release download hit the shared runner ip rate limit and
failed the job with 'socket hang up' after three retries.
* docs: record that canvas apps need a dom proxy to be text-fuzzable
* fix(ios): bound lifecycle rpcs and claim the target device
a launch the simulator rejects sent the xctest session into a recovery
chain that answered minutes late or never, and the rpc had no deadline,
so the run hung with no trace and no error. also take a per-udid flock:
a second run's reinstall lands under the first's live automation session
and wedges it.
* docs(ci): correct the ios hang wording and note the device lock
* refactor(verifier): derive the lastAction shape from one field list
both hosts must show a spec the same lastAction. one ordered list now
feeds the goja object and the json the web host installs, so they
cannot drift.
* fix(web): install lastAction in the page before extractors read it
state.lastAction was hardcoded null on web, so every property reading it
was silently vacuous: a correct property passed without ever firing.
* fix(web): carry element identity on actions and fix findAll on paths
an action's target was coordinates only, so a property matching on which
element was acted upon could never fire. ax.findAll([a,b]) also returned
nothing on web.
* fix(chrome): wait out a route transition before sampling facts
the tree stays byte-identical and quiet across a cross-fade, so both the
quiet timer and the unchanged-tree escape called it settled mid-flight
and extractors read two screens at once.
* fix: bound the pre-run app launch
launch happens before the runner starts, so --duration never covered it
and a wedged driver hung with no trace and no error.
* fix(folio): read balances from merged cards and treat unreadable as unknown
compose for web merges the whole accountcard subtree, so the balance
child never exists there and every card parsed as 0. the property then
compared 0 to 0 and fired on any submit, which is a false positive
generator. unknown is now null and null is vacuously true.
* test(folio): cover merged-card parsing and unknown balances
* ci(folio): make web an expect-the-bug leg
the web runtime can observe the double submit now, so the health gate
understates it. seed 1 finds it at step 109, 3 runs out of 3.
* docs(ci): explain why a submit tap landing on home is the bug
* fix(ios): read a StaticText's label as its text
AXValue was the only source for text, but a StaticText carries its
string in AXLabel, so nothing on screen had .text on ios: a spec reading
it saw everything on android and nothing here.
* docs(ci): correct the calibrated step ranges
* fix(folio): stop convicting on arithmetic float64 cannot hold
past 2^53 cents the gap between representable values is 128, so a real
1600-cent move reads back as something else and the equality is false
for a healthy submit as readily as a double one. also match parseCents:
a sign or an oversized amount is rejected, not read as an amount.
* test(folio): pin the safe-integer guard and its boundary
* docs: stop teaching the zero-default that caused a false alarm
* docs: write down the silent-vacuity failure modes
* feat(folio): tag the home total and the card transaction count
the total was the only untagged node on the screen, so the spec had to
sum cards and a clipped card broke the sum.
* fix(folio): read the app's own total and refuse contaminated windows
summing cards went null when one was clipped, and the null poisoned the
carrier for the rest of the run. the balance window also spanned every
transaction since the last home visit, so the property convicted on
deltas it could not attribute: the old web witness was 3.16x the typed
amount, not 2x.
* test(folio): pin the window rules and the count invariant
* fix(folio): never read a frame that shows two screens
android dumps a cross-fade with both screens in the tree. the route said
add-transaction while an unscoped find said home, so the oracle took a
half-rendered total as fresh and convicted on a tap that committed
nothing. one function now decides the route and returns null when the
frame is ambiguous.
* test(folio): cover transition frames, card readings and creation
* fix(folio): only disambiguate counts that came from merged text
the equal-length digit rule exists because web merges the card and an
account named -1 makes '12' ambiguous. a dedicated count node has
nothing to disambiguate, so applying it there threw away real evidence.
* ci(folio): pin the recalibrated seeds and drop android to a health gate
web 3 and ios 7 convict 3 runs out of 3 with an exactly 2x witness.
android convicts 2 in 5 because the same seed does not walk the same
trajectory there, so it proves the app runs instead.
* docs(ci): describe the two properties and why android cannot convict
* fix(android): wait out a route cross-fade before snapshotting
the dump could hold two screens at once, and the runner refuses to act
on such a tree, so a quarter of android steps applied no action and the
count varied per run: the same seed never walked the same trajectory.
the ios companion and the chrome driver already do this.
* ci(folio): let the android leg run far enough to see its conviction
* docs: only the repo owner merges
* ci(folio): a thrown predicate is not a conviction
exit 2 means the run recorded a violation, and a predicate that throws
is recorded as one too. so was newAccountBalanceIsZero, an unrelated
property in the same spec. the gate read the exit code and went green
with detection dead.
* ci: install idb-companion from its tap and stop interpolating inputs
idb-companion is not in homebrew-core, so the ios leg died before it
built anything. replay-ui expanded dispatch inputs into the shell.
* docs: correct the snippets and numbers that drifted from the code
* test(sidecar): pin that a slow read counts toward the stability streak
* fix(web): read the page's extractors only on steps that count
the page advances the spec's carriers when it evaluates, but the runner
applied the result only on non-transitional steps. a discarded step
moved the window forward anyway, so the next accepted pair bracketed two
transactions while counting one submit, and convicted a healthy app.
extractor errors now fail the run instead of leaving goja's values in
current against v8's in previous.
* fix(chrome): anchor the transition deadline when the dom goes quiet
it was anchored at script start, so a page that churned past the window
reached the check already expired and returned mid cross-fade. the
driver now publishes the idle timeout it needs, since the caller's 1s
could never spend the 800ms window.
* fix(web): fail on a partial extractor override
same mixed-producer hazard as the install error: some extractors hold
the page's value and the rest hold goja's, and a property comparing
across that split fires on a healthy app.
* docs: six of seven, the seventh is the stock property
* fix(folio): drop a name two cards answer to
homeTxnCountsOf keyed on the account name and let the last card win, so
two accounts the fuzzer named the same collapsed into one entry. a
reading that saw one Travel card and a later one that saw both then
subtracted two different accounts' counts, and
submitCommitsOneTransactionPerAction convicted a healthy app of
double-submitting. it is a gated property in folio-run.sh, so that reads
as "found the submit bug" over a card scrolling into view.
same rule createdAccountHasNonZeroBalance already applies: a name
nothing can attribute is no evidence. counted over every card, since an
unreadable twin spoils the identity too.
* perf(folio): read each frame once
every extractor asked routeOf, and routeOf does five ax.find calls. on
web each find walks the document and every shadow root beneath it, so
the spec cost 110 tree walks a step; homeCards was parsed four times
over. now 5 and once.
keyed on the identity of the state object because both hosts build a new
one per step and hand that one object to every getter, so it cannot
outlive its frame. holding the reference is what keeps that true rather
than likely.
* fix(web): keep an undefined reading's index through JSON
json has no undefined, so an extractor whose getter returned one had its
whole index dropped by JSON.stringify. that index then kept goja's
dump-derived value while its neighbours held the page's, and a property
comparing previous to current across the split fires on a healthy app.
folio has nine on(route, tag) extractors, so this was most extractors on
most steps.
each reading is wrapped in a {value} envelope: the drop now happens
inside the entry, and an absent value means the getter returned
undefined, which is what the goja host records for the same getter. a
json null would instead claim it returned null and x.current ===
undefined would answer differently on the two hosts.
* feat(verifier): report the registered extractor count
the web path needs it to check the page sent one reading per extractor.
* fix(runner): fail when the page reports fewer readings than extractors
the comment here already claimed a partial override was fatal. it was
not: the skipped check only catches indices outside the extractor list,
so a page reporting values for some extractors and not others left the
rest holding goja's reading of the dump with nothing said.
* test(browser): drive an undefined reading through the whole web path
four layers carry it: the page's envelope, the driver's unwrap, the
runner's count check and the verifier's decode. each has a unit test and
only a run proves they compose. goes red both ways, decoding an absent
value as null and dropping the envelope.
* fix(web): offer the aria roles a user activates
only role=button was in the tappable set, so link, checkbox, radio,
switch, tab, option, the menuitems and treeitem were invisible to the
enumeration however plain the control looked. the replay ui builds its
step rows as <li role="option">, and the spec dogfooding it had to
hand-write an action to reach them because no default verb could see a
single row.
both producers build the set from the same role list, since the parity
test compares them element by element.
* test(browser): tap a role-based control end to end
every control on the page is an <li role="option">, the shape the
replay ui gives its step rows, and the spec carries no action of its
own: the property firing is the evidence the default enumeration offered
a tap on one.
* fix(web): read aria-disabled as disabled
the enabled fact came off the disabled property, which only real form
controls have. it reads undefined on the role-based controls the
tappable set now covers, so every one of them looked enabled however
plainly it was marked otherwise, and the fuzzer would spend actions on
inert ones.
both producers answer the same two ways, and the parity fixture carries
a disabled row so the comparison covers it: reverting one side alone
names the element and the fact.
* docs(replay-ui): the enumeration reaches step rows now
the comment said role="option" is not in the tappable selector set,
which stopped being true a few commits ago. selectAStep stays, for the
reason the tab weight below it stays: one row among the page's clickable
elements is a thin chance, and both step-facing properties go vacuous on
a run that never selects one.
* test(runner): bound the last-action test by steps, not wall clock
100ms of wall clock against an assertion that two steps ran fatals under
load with "the web path never installed it", which reads as a
regression. every sibling test in the package uses a long duration and
MaxSteps.
* ci: run the kotlin tests in make test
RouteTransitionTest and the stability poll cover the android settle and
nothing in ci ran them. :sidecar:test needs no android sdk, checked by
running it with ANDROID_HOME pointed at nothing.
* fix(sidecar): measure the stability streak as observed quiet
parameterising pollUntilStable also moved the clock to the start of the
read that opened a run of identical snapshots, so a read's own duration
counted as quiet. the pre-existing caller polls a real uiautomator dump:
at 400ms a read, 750ms of required quiet became 250ms of observed quiet
and the poll settled in two reads instead of four.
the parameters stay, the semantics go back.
* test(sidecar): pin the transition cap by driving it
it asserted 1500 >= 700 + 300, two constants, which can only fail if
someone edits a constant. it now drives awaitSettledTree against a fade
that lands after 700ms and asserts it hands back the settled tree before
the cap. cut the cap to 1000 and it goes red.
* ci: pin buf-setup-action to a commit
it takes a token now, so a floating tag is a token handed to whatever
that tag moves to. note v1 there is a branch, not a tag, so the ref
lookup that resolves it is matching-refs/heads/v1.
* ci: declare least-privilege permissions
none of the three declared any, so each got the repository default.
release.yml and docs.yml already do this. all three only check out,
build, test and upload artifacts.
* ci: fail fast when a server never comes up
the readiness loops fell through silently after 30 tries, so a server
that never started surfaced as an opaque driver failure minutes later.
each now says what did not answer and on which port.
* ci(folio): a missing trace is not a verdict
with no trace the android gate ran its grep against ./trace.jsonl and
reported "never reached AddTransactionScreen, so it never got past
login", which is not what happened. the web and ios branches had the
same misdiagnosis on exit 0.
same class, one line up: the classifier's own failure was swallowed, so
with the evidence reader dead the gate printed a healthy run and exited
0.
* ci(replay-ui): skip a run directory with no trace
the summarise step is if: always(), and under github's bash -eo pipefail
an unmatched glob stays literal, the redirect fails, pipefail carries it
into the assignment and -e kills the step. so a failed fuzz run went red
twice, once for the real reason.
|
||
|
pj
|
7343085614 |
llm action-selection backend (#68)
* feat(spec): add llm() action-backend marker * feat(spec): make llm marker inert on the JS picker * feat(spec): expose __sanderlingSampleInput__ corpus draw * feat(openrouter): minimal chat-completions client * test(openrouter): cover request shape, parse, and errors * feat(verifier): thread screenshot + capture corpus sampler * feat(verifier): LLM accessors — candidates, config, sampler * test(verifier): cover AllCandidates, LLMConfig, SampleInput * feat(trace): record action Source and LLMReasoning * feat(runner): thread step screenshot into PushSnapshot * feat(runner): llmSource selects actions via OpenRouter * feat(runner): wire llmSource selection and trace stamping * test(runner): cover llmSource selection, mapping, downscale * docs(folio): add llm action-backend example spec * docs(folio): document the LLM action backend run * feat(llmclient): support OPENAI_API_KEY, openrouter wins * refactor(runner): rename openrouter package to llmclient * docs: both api keys, example model gpt-5.4-nano * docs: add pr style rules to claude.md * fix(runner): explain action kinds in llm prompt to stop swipe loops * feat(trace): record llm ranked list and chosen rank * feat(runner): stamp llm ranked list and chosen rank on trace * fix(runner): tap by selector to survive layout shift after observe * revert(runner): drop selector-first tap; broke path/testTag selectors * feat(spec): llm() accepts optional instructions * feat(verifier): read llm instructions off config * feat(runner): append spec instructions to llm system prompt * docs(folio): describe app in llm spec instructions * feat(bundler): map generator export to globalThis.generator * feat(verifier): read llm config off globalThis.generator * feat(runner): gate llm source on --generator flag * feat(cmd): add --generator llm|seeded flag * test: cover --generator flag parsing and pickSources gating * feat(verifier): enumerate llm candidates by walking actionsRoot collect-walk the weighted action tree: recurse weighted branches accumulating selection probability, call authored leaves once for concrete actions, enumerate builtins per element. label controls by visible text (borrowing descendant text), fold gestures into directional scrolls over scrollable containers, drop disabled, dedup descriptions. * test(verifier): cover candidate enumeration walk * feat(verifier): add SetupAction to walk setup without the seeded root * test(verifier): cover SetupAction setup-only precedence * refactor(llmclient): make JSONSchema.Schema raw json for pinned field order * feat(trace): record llm choice number and chosen_action echo * feat(runner): llm picks one number from weighted candidates drop the seeded-root call for a setup-only precedence path, render a numbered weighted candidate list, pin a reasoning-first choice schema, strict-skip when chosen_action does not echo the numbered entry, and let the model supply typed values (corpus fallback when empty). * test(runner): cover choice schema, strict-skip, and setup precedence * refactor(verifier): drop the superseded AllCandidates enumeration * feat(folio): drive spec.ts under --generator llm; drop spec-llm.ts * fix(verifier): label editable fields by hint, not the typed value an editable field's own text is its transient content; prefer the hint so the field is named by purpose and the label stays stable. * test(runner): cover weight-suffixed echo and stripWeightSuffix * fix(runner): accept chosen_action echo that carries the weight suffix real runs showed the model copies the whole numbered line including the trailing (w34) weight annotation, so strict-skip rejected ~91% of picks and the llm was paralyzed. strip the weight suffix before comparing. also nudge the prompt to stress-test repeated submissions (idempotency). * fix(verifier): skip llm enumeration on cross-fade frames a navhost mid-transition carries >1 route *Screen in a collapsed coordinate space; acting on it taps garbage (soft keyboard). real runs showed the llm acting on 44% of steps being such frames. skip them so the llm re-observes a settled frame next step. * feat(folio): show current balance on the add-transaction screen renders the account's balance (testTag TxnCurrentBalance) below the account name, above the credit/debit toggle, so before/after screenshots carry comparison data. * fix(replay): derive device space from screen extent, not first node the first positive-bounds element is often a short status-bar node (320x24 on android); using it gave a 320/24 aspect ratio that squashed the screenshot overlay into a grey horizontal band. use the max extent across elements (like the runner's screenBounds) instead. * fix(folio): show balance as a compact one-line label per review: one line, account-name-sized, e.g. "Balance: $0.00" instead of a large balance card. * fix(folio): move balance into the header, one compact line under the account name * fix(replay): attribute deferred violations to the causing step, not detection * fix(replay): show a step's own violations in both panels, no next-step bleed * refactor(hierarchy): one Tree.Transitional, drop the duplicated cross-fade check * chore: ignore .playwright-mcp scratch output * docs: document the llm generator and --generator flag * docs(spec): correct the llm() comment; config reads off globalThis.generator * docs: add pr description rules |
||
|
pj
|
991c583eb9 |
docs update with case study (#63)
* docs(manual): add introduction page * docs(manual): rewrite getting started as guided first run * docs(manual): rewrite writing specs as a folio tutorial * docs(manual): document missing spec API in reference * docs(manual): plain-language rewrite of runs page * docs: real introductions on index pages and README * fix(docs): sibling links from directory-style pages need ../ * fix(docs): correct sampling and restart-cost claims to match implementation * docs: nav lists Introduction and Case study; roadmap points to milestone * docs(manual): make getting started target the reader's own app, not Folio * docs(manual): add Folio case study page * docs: point manual navigation at the case study * docs(readme): lead with the case study, fix roadmap link * docs: roadmap links to milestone, sync clear-data default and cross-links |
||
|
pj
|
90224dfd06 |
Physical-device iOS support (#64) (#66)
* feat(companion): add appState, eraseText, pressKey runner handlers
The Go runner transport already calls these methods; the in-device runner
implemented them only latently. They become load-bearing on the device
path, where the hybrid's legacy-companion fallback is absent. Backward
compatible: the simulator hybrid never calls them.
* feat(ios): resolve physical devices from devicectl
ResolveDevice parses xcrun devicectl list devices into Device{Name,
HardwareUDID, CoreDeviceID}: the hardware UDID feeds xcodebuild/iproxy
and the CoreDevice id feeds devicectl install. Matches by name or either
id; errors list candidates on none/ambiguous. Fixes the stale sidecar
comment on ResolveTarget.
* feat(ioscompanion): runner-only device driver mode
NewDevice reuses Driver with d.companion set to the runner dialed over an
iproxy usbmux tunnel, hybrid=false, runnerClient=nil. The existing accessor
seams then route launch/snapshot/text/gesture to the runner with no new
DeviceDriver methods. Device seams swap clear-state to a devicectl
reinstall, container reset to a warn-once no-op, and paste grant to a no-op.
realSpawnDeviceRunner builds and signs the runner at run time via the App
Store Connect API key (no Xcode UI), caching on a source hash.
* test(ioscompanion): cover device wiring, routing, and shell-out argv
Seam-driven NewDevice wiring + gesture/text routing (asserting no keyboard
HID), devicectl/build/test/iproxy argv builders, xctestrun test-target dict
name parsing, signing-credential env checks, and source-hash cache keying.
* feat(testrun): route physical-device iOS runs to the device driver
Execute resolves a non-simulator iOS target through ios.ResolveDevice into
its hardware UDID and CoreDevice id; buildDriver constructs NewDevice via a
seam instead of rejecting the device. Generalizes the --ios-device and
--ios-app-path help to cover the device path; signing stays env-read, never
a flag.
* feat(doctor): device prereqs replace java/sidecar for ios-device
iosDeviceChecks now verifies devicectl, iproxy on PATH, a connected+paired
device (via ios.ConnectedDevices), and App Store Connect signing creds (via
ioscompanion.VerifyDeviceSigning). The retired JVM sidecar checks stay only
under android.
* feat(conformance): device backend uses iphoneos app and tunnel orphan checks
The device backend now builds via just ios-device, points --ios-app-path at
the Debug-iphoneos bundle, and reinstalls each run for clear-state. The G5
orphan scan replaces the retired sidecar.jar check with lingering iproxy and
device test-without-building sessions (destination platform=iOS,id=).
* feat(folio): device build linking the iosArm64 framework
project.yml selects the Kotlin framework slice by SDK (iosArm64 for
iphoneos, iosSimulatorArm64 for simulator) and links via -framework Shared
on the SDK-conditional search path. New ios-device/test-ios-device recipes
mirror ios/test-ios, signing the Debug-iphoneos build with the .env API key.
* docs(cli): document ios-device doctor checks and the device flags
The --ios-device flag now also selects a connected device; --ios-app-path
covers the device install; the doctor gains an ios-device platform whose
checks are devicectl, iproxy, a paired device, and signing credentials.
Corrects the --clear-data default to true.
* fix(ioscompanion): resolve signing key path to absolute
xcodebuild's -authenticationKeyPath requires an absolute path, but .env
files commonly carry a repo-relative one. Resolve it against the working
directory before the stat so a relative ASC_API_KEY_PATH still signs.
* fix(ioscompanion): re-enable signing for the device runner build
companion/project.yml disables code signing for the simulator build, so
the device build inherited it and produced an unsigned runner that the
device rejected at install (0xe8008018). build-for-testing now forces
CODE_SIGNING_ALLOWED/REQUIRED=YES so automatic provisioning signs it.
* fix(ioscompanion): key the device build cache on signing identity
The cache marker hashed only sources, so switching signing team or key
reused a runner signed with the stale identity, which the device rejects at
install (0xe8008018). Fold team + key id into the cache key so a signing
change forces a rebuild.
* docs(getting-started): document physical iOS device setup
Lists the iproxy requirement and the App Store Connect signing env vars
(SANDERLING_IOS_TEAM, ASC_API_*) a device run needs, plus the
test-ios-device recipe and the doctor check.
* feat(ios): native usbmux client and in-process tunnel forwarder
Talk to macOS usbmuxd directly instead of shelling out to iproxy, so the
device path depends on nothing beyond macOS + Xcode.
* refactor(ios): drive device tunnel via io.Closer seam
Replace the tunnelChild *exec.Cmd and spawnTunnel seam with a tunnel
io.Closer and startTunnel seam backed by the in-process usbmux forwarder.
* refactor(ios): remove iproxy spawn from device runner
* test(ios): cover tunnel close via io.Closer not child process
* feat(doctor): check usbmuxd socket instead of iproxy on PATH
* chore(conformance): drop iproxy orphan check; tunnel is in-process
* docs(ios): device tunnel uses native usbmux, nothing to install
* chore: gitignore the signing keys directory
* feat(folio): add Android launcher icon (black bg, white dot)
* feat(folio): add iOS app icon (black bg, white dot)
* feat(folio): add web favicon (black bg, white dot)
* docs(ioscompanion): fix stale const comments
* refactor(ioscompanion): inline single-use devicectl argv builders
* refactor(ioscompanion): inline xcodegenArgs, drop tautological argv tests
* refactor(ioscompanion): inline firstNonEmpty
* refactor(doctor): dedup usbmuxd socket path via ioscompanion seam
* test(doctor): trim redundant signing-check test
* refactor(ioscompanion): deliver COMPANION_PORT via TEST_RUNNER_ env
* fix(testrun): seam preflight so iOS routing tests pass on CI without xcrun
|
||
|
pj
|
b44077afde |
replay ui fix (#56)
* refactor: rename inspect to replay across the codebase Renames inspect-ui/ to replay-ui/, internal/inspect/ to internal/replay/, the CLI subcommand from `sanderling inspect` to `sanderling replay`, and updates all references in docs, Makefile, README, and Go comments. * feat(replay-ui): show spec filename with full path on hover RunList and RunDetail now render the basename of spec_path (e.g. login.spec.ts) with the full path available as a title tooltip. |
||
|
pj
|
f572c8ba66 |
WIP: docs: refresh after iOS + web support (#50)
* docs: README covers iOS + web, surface both example apps * docs(cli): document --ios-device and per-platform doctor * docs: tighten README, fold examples into Docs list * docs(runs): correct --clear-data lifecycle wording Default behavior no longer wipes app data between runs; --clear-data is now opt-in. * docs(getting-started): add iOS path, separate folio and folio-web Document just test-ios under examples/folio, and distinguish the KMP sample from the React + Vite folio-web sample. * docs(inspect): document the eight panels Lists Screenshot, ActionList, Timeline, ViolationsPanel, HierarchyPanel, SnapshotTable, MetricsChart, ExceptionsPanel. Cross-links HierarchyPanel to the spec language reference. * docs(writing-specs): document setup export, flag noLogcatErrors as android-only Mirrors pkg/spec/README.md so the manual covers the runner's setup-first fall-through. Marks noLogcatErrors as Android-only so iOS/web spec authors know it silently no-ops. * docs(folio): document web target and iOS sanderling test recipe After the KMP refactor folio also runs on wasmJs and the justfile exposes just web, just web-build, and just test-ios. Surface all three. * docs(folio-web): add README Covers prerequisites, demo credentials, just test recipe, and how the React + Vite host exposes state to the sanderling spec via stable ids and data-* attributes. * docs: scrub driver-implementation name from user docs Drop the implementation tool name from README, cli.md doctor table, and spec-language.md. These docs should describe behaviour, not the specific underlying tool the native sidecar wraps. * docs(development): scrub driver-implementation name from dev docs architecture, design-principles, decisions now describe the native sidecar by role (gRPC surface over OS UI-test pipeline) rather than by the specific tool it wraps. |
||
|
pj
|
dd54c24c4e |
feat: --clear-data flag + typed attribute selectors (#48)
* feat(test): add --clear-data flag to clear app data on launch * test+docs: cover --clear-data flag in CLI parser test and reference * feat(spec): type AttrSelector with known attribute names Replace AttrSelector = Record<string, string> with KnownAttrSelectors plus a string|boolean index signature, so authors get autocomplete and type-checking on testTag / focused / clickable / etc. while raw driver attributes still type-check via the fallback. Boolean state attributes accept native booleans; goja stringifies them at the marshal boundary. AccessibilityElement.attrs becomes RawAttrs (typed string-valued shape of the same canonical names) so element.attrs.testTag autocompletes. * test(verifier): native boolean selector value matches focused=true * docs+folio: use native boolean for focused selector and document typed attrs |
||
|
pj
|
88db0cbea8 |
docs: web platform + clean URLs + dark/light mode (#37)
* chore(docs): replace d2 diagram pipeline with mermaid Remove docs/_diagrams/ and d2 build step from Makefile. The HTML template already initialises mermaid.js; diagrams are now inline code fences rendered client-side. * docs(architecture): add mermaid diagram + web/CDP platform docs Replace SVG img tag with inline mermaid flowchart showing both native (Maestro sidecar + in-app SDK) and web (Chrome CDP) paths. Update Processes, Transports table, and per-step cycle sections. * docs(design-principles): update principles 1-4 for web platform Principles 1, 2, 3, and 4 referenced Maestro and native-only concepts. Add web/CDP context and update driver-is-an-interface to name both sidecar and chrome implementations. * docs(manual): add web prerequisites and folio-web example Update --platform flag to list android, ios, web. Add web prerequisites section (Chrome, no SDK needed) and folio-web quick-start to getting-started. * chore(gitignore): untrack inspect dist/index.html build artifact index.html is regenerated by vite on every build with a new content hash, making it permanently dirty. Only .gitkeep is needed for //go:embed to compile on a fresh checkout. Also remove duplicate dist/* line and stale d2 diagram ignore entries. * feat(docs): click-to-zoom for mermaid diagrams * docs(architecture): change diagram layout from LR to TB * docs(getting-started): link npm and Maven Central package headers * update docs root * docs(spec): rewrite npm package README Update usage example to current API, drop stale version-compatibility and license sections. * build(docs): output pages as pagename/index.html for clean URLs Split DOCS_OUT into INDEX_OUT (index.md files stay as index.html) and PAGE_OUT (all other pages become pagename/index.html). The __ROOT__ depth computation already handles the extra directory level correctly. * chore(docs): update sidebar links to directory-style URLs * docs: update cross-links from .html to directory-style paths * ci(docs): remove d2 install step * feat(docs): dark/light mode toggle Add theme toggle button (top-right, fixed). Persists preference in localStorage; falls back to prefers-color-scheme. Flash-free via inline script in <head> that sets data-theme before first paint. * fix(docs): fix inspect image path broken by directory URL restructure * feat(docs): click-to-fullscreen for all article images * fix(inspect): allow AssetsFS override in ServerOptions; drop unused request param from serveIndex * fix(inspect): use in-memory FS in tests so TestAssets_FallbackToIndexHTML passes without web build |
||
|
pj
|
8ccf95c1cf |
refactor: rename project uatu -> sanderling (#24)
* refactor: rename Go module path uatu -> sanderling
Module path github.com/priyanshujain/uatu -> github.com/priyanshujain/sanderling,
including all imports and the proto go_package option. Generated .pb.go files
rewritten in-place; safe to regenerate with protoc later.
* chore(proto): regenerate driverpb after module path rename
The previous sed-based module rename corrupted the embedded descriptor
byte lengths. buf generate rewrites them cleanly.
* refactor: rename CLI binary uatu -> sanderling
Updates Makefile target + UATU_BIN var, .goreleaser project/build IDs,
.gitignore comment, and all user-facing strings in the CLI help text,
error messages, and tests. Binary is now bin/sanderling.
* refactor(sdk): rename Kotlin package dev.uatu.sdk -> dev.sanderling.sdk
Moves sdk/android/src/{main,test}/kotlin/dev/uatu -> dev/sanderling and
rewrites package declarations, imports, and the Gradle namespace. Class
names (Uatu, UatuRuntime) are renamed in a follow-up commit.
* refactor(sidecar): rename Kotlin package dev.uatu.sidecar -> dev.sanderling.sidecar
Moves sidecar/src/{main,test}/kotlin/dev/uatu -> dev/sanderling and
rewrites package declarations, imports, and the application mainClass.
* refactor: rename Uatu API surface -> Sanderling
- Kotlin: Uatu -> Sanderling, UatuRuntime -> SanderlingRuntime (+ files).
- JS host binding: globalThis.__uatu__ -> __sanderling__ (Go verifier,
spec-api, tests).
- TS interface: UatuRuntime -> SanderlingRuntime; internal tags
__uatuFormula / __uatuActionGenerator -> __sanderling* variants.
- Go trace: UatuVersion field + uatu_version JSON tag renamed.
- Socket naming: uatu-agent / uatu-agent-reader -> sanderling-agent*.
- Sample app, docs, inline-JS test strings updated to match.
* refactor(examples): rename examples/folio/uatu -> examples/folio/sanderling
Renames the example spec directory; updates justfile paths + gitignore
entries accordingly. Package.json name/description and @uatu/spec
dependency are renamed in the npm + docs commits.
* chore(build): rename gradle property + rootProject.name uatu -> sanderling
- Renames the uatu.version gradle property and all its -P references in
Makefile, build.gradle.kts files, and .github/workflows/release.yml.
- settings.gradle.kts rootProject.name = "sanderling".
- Renames .env.local.example header + release-cli workflow job name.
* refactor(proto): rename proto package uatu.driver.v1 -> sanderling.driver.v1
Updates the proto package and java_package, regenerates driver.pb.go +
driver_grpc.pb.go, rewrites Kotlin imports and the gRPC ServiceName
assertion in driver_test.go.
* refactor: rename npm package @uatu/spec -> @sanderling/spec
Renames package name in pkg/spec-api/package.json + lockfile, all
consumer imports (examples/folio spec, testdata, verifier tests), the
esbuild alias in cmd/sanderling/test_run.go, and related doc references.
* docs: rename uatu -> sanderling in README, docs, and URLs
- README + docs/{manual,development}/*: narrative + GitHub + Pages URLs.
- POM + npm package.json repo/homepage/bugs URLs.
- .gitignore + embed_stub + Makefile-comment references updated to
'make sanderling'.
- Minor narrative comments in cmd/sanderling/test_run.go and
internal/inspect/server.go.
* refactor: rename remaining internal uatu strings -> sanderling
- SANDERLING_TEST_PHONE/OTP env vars (cmd + bundler tests).
- sanderling-sidecar runtime tmp dir + extracted JAR filename.
- Inspect web UI: @sanderling/inspect-web package, title, theme
localStorage key, RunList empty-state copy, uatu_version TS field.
- Sample app storage key sanderling.ledger.v1.
- Test data: sanderling_test AVD name + com.example.sanderling_test.
- Release docs tarball name template.
|
||
|
pj
|
13bb2feb82 |
feat: uatu inspect UI (web trace explorer) (#23)
* feat(trace): extend Step/Action/Meta schema for inspect UI
Add Step.Hierarchy, Step.Residuals, Action.Selector/ResolvedBounds/TapPoint,
Meta.EndedAt and JSON tags on hierarchy.Element/Bounds/Tree so trace.jsonl
can drive the upcoming uatu inspect web UI.
* test(trace): cover EndedAt + new step fields round-trip
* feat(ltl): MarshalJSON for Formula AST + Evaluator.Residual()
Each Formula concrete type now serializes to a closed-set residual node
(true/false/not/and/or/implies/always/now/next/eventually/predicate/error)
that mirrors the TS spec API surface. Evaluator.Residual() folds pending
obligations into a single Formula so the runner can stamp one residual
per property per step into trace.jsonl.
* feat(runner): stamp residuals, hierarchy, selector targets, ended_at
Each Step now carries the captured hierarchy, per-property residual ASTs,
and (for Tap/InputText) the selector + resolved bounds + tap point. The
test_run command writes meta.ended_at on graceful shutdown so the inspect
UI can distinguish completed runs from in-progress ones.
* feat(inspect): scaffold embed dist for SPA assets
Stage 2 stub for the inspect server. Real web bundle gets wired in
Stage 4 (Makefile copies web/dist into internal/inspect/dist).
* chore(web): ignore web/ build output in root .gitignore
* chore(web): add bun + vite + vitest scaffold config
* feat(web): monochrome design tokens, typography, app shell CSS
* chore(web): placeholder for self-hosted JetBrains Mono fonts
* feat(web): index.html entry with style links and root mount
* feat(web): typescript types mirroring run/step trace schema
* feat(web): typed fetchers for runs/steps/screenshots
* feat(web): App shell with router and run/step routes
* feat(inspect): runs scan, lazy step parse, mtime-aware cache
* feat(web): RunList route with table, loading, and error states
* feat(web): RunDetail route shell with three placeholder panels
* feat(inspect): fsnotify-backed runs watcher with debounce
* fix(web): use jest-dom/vitest entry so matchers register
* test(web): cover listRuns happy path and error response
* test(web): render RunList with mocked fetch and assert row
* chore(web): commit bun lockfile
* feat(inspect): http handlers for runs/steps/screenshots/SSE
* test(inspect): cover handlers, screenshot whitelist, SSE, dev proxy
* feat(cmd): add 'uatu inspect' subcommand
* fix(web): align TS types with snake_case wire format
Go inspect server serializes RunSummary, StepSummary, Step, Meta with
snake_case JSON tags (matching the on-disk trace.jsonl/meta.json). Update
the TS types and consumers to match so API responses parse without
runtime undefined fields. Action keeps resolvedBounds/tapPoint as camelCase
because those keys were defined that way in the trace schema.
* feat(web): add ActionList panel for run-detail step navigation
* feat(web): add SnapshotTable panel with diff highlighting
Renders snapshots dictionary as a flat sorted dotted-path tree.
Changed leaves get data-changed plus a hover title with the previous value.
* test(web): cover SnapshotTable rendering and diff behavior
Eight cases: empty state, sort order, dotted-path expansion,
changed/unchanged/missing-previous flagging, and inline-vs-expanded arrays.
* feat(web): add Screenshot panel with bounds and tap overlays
Center column of run-detail page. Renders the device screenshot
scaled to fit, with an SVG overlay drawing resolvedBounds as a
violation-colored rect, tapPoint as a contrast ring, and swipes
as an arrow. Falls back to a placeholder when src is missing or
the image fails to load.
* fix(web): guard scrollIntoView call for jsdom compatibility
* test(web): cover Screenshot panel rendering and overlays
* test(web): cover ActionList rendering, selection, keyboard, and markers
* feat(web): add ExceptionsPanel component
* test(web): add ExceptionsPanel tests
* feat(web): add Timeline panel with property swimlanes
Renders SVG swimlanes per property with violated/pending/holds cells,
action-marker dots, click-to-seek, and a selected-step highlight bar.
* test(web): cover Timeline empty state, cells, status, click, highlight
* feat(web): add ResidualNode recursive AST renderer
* test(web): cover ResidualNode operators, predicate, and error chip
* feat(web): add ViolationsPanel with status badges and jump button
* test(web): cover ViolationsPanel rows, status grouping, and jump button
* test(web): register testing-library cleanup globally
All six panel test files added local afterEach(cleanup); centralize it in
the shared setup so future tests inherit DOM isolation by default.
* feat(web): hooks for url/keyboard/theme/sse
* feat(web): wire all panels into run-detail with phone-dominant grid
ActionList left, Screenshot center, Snapshots/Properties/Exceptions
stacked right, Timeline bottom. URL-synced step index (useStep), keyboard
shortcuts (j/k/arrows/g/G/.), light+dark theme toggle stored in
localStorage, SSE auto-refresh on the run index.
* test(web): add three reference run fixtures (clean, violation, exception)
* build: web targets in Makefile + bun in CI; docs(inspect)
- Makefile: web-build/web-dev/inspect-dev/test-web targets; uatu and
install now depend on web-build so the binary embeds the latest SPA.
- ci.yml: setup-bun + cache; existing make test now runs web typecheck +
vitest as part of the full suite.
- docs/manual/inspect.md: panel reference, keyboard shortcuts, URLs.
- docs/manual/cli.md: document uatu inspect.
- README: link to inspect docs.
* feat(runner): capture a screenshot per step
The driver already exposes Screenshot(ctx), but the runner never called
it. Each step now writes <run>/screenshots/step-NNNNN.png right after
the trace line, using the same failure-is-a-warning posture as other
best-effort observability hooks. Makes the inspect UI's center panel
actually useful.
* feat(inspect): include action_label in StepSummary
Tap/InputText/Swipe/PressKey/Wait each get a short human-readable
label (selector, quoted text, swipe direction, key name, duration) so
the action list panel can render readable rows instead of just 'Tap'
with no target.
* test(inspect): accept either #app or #root in SPA shell fallback
* feat(web): render action_label and screen in ActionList rows
Step rows now show 'Tap id:save', 'InputText "alice"', 'Swipe up',
'PressKey back', etc. Steps with no action fall back to
'observe @ <screen>' so the list reads as a flow instead of a wall
of '--' placeholders.
* feat(sidecar): implement screencap for android driver backend
Was stubbed to return an empty byte array, which made the runner's
per-step screenshot capture a no-op. Shell out to 'adb exec-out
screencap -p' and stream the PNG bytes back. Width/height stay zero
because the PNG header carries them; the Go side can parse if needed.
* feat(proto): add Metrics RPC for per-step CPU and memory capture
* feat(driver): Metrics(bundleID) returns cpu_percent + heap/total bytes
* feat(sidecar): implement Metrics RPC via adb top + /proc/<pid>/status
* feat(runner): capture metrics + before/after screenshots per step
Each step now writes step-NNNNN.png (before applyAction) and
step-NNNNN-after.png (after the action + wait-for-idle). The runner
samples Driver.Metrics(bundleID) before writing the trace line and
stamps Step.Metrics with cpu_percent, heap_bytes, total_memory_bytes
so the inspect UI can chart CPU and heap over the run.
* fix(runner,sidecar): measure CPU across step via /proc stat delta
'top -d 0.3 -n 2' measures CPU in a 300ms window that coincides with
the SDK-paused app, always reporting 0%. Switch to reading
/proc/<pid>/stat utime+stime and computing the delta between successive
calls; the natural step cadence gives a 2-5s measurement window that
captures the action response and render cycle. Also moved the sample
to before snapshotStep so the delta starts before the SDK pause.
* feat(web): add Metrics type for per-step cpu and memory
* refactor(web): replace --accent-change with --accent-positive token
* refactor(web): recolor chip-progress as neutral outlined chip
* refactor(web): use neutral border for changed snapshot rows
* feat(web): add MetricsChart panel with HEAP and CPU lanes
SVG-based time-series chart rendering heap bytes and CPU percent per
step across two stacked lanes, with a shared step axis below. Lines are
monochrome; a vertical highlight marks the selected step; per-step hit
rects make any click seek to that step.
* feat(web): revamp ActionList with tag targets, elapsed time, and expandable rows
Render selector-based Tap actions as <tag/> markup, show zero-padded MM:SS.mmm
elapsed time per row, and expand the active row with Position/Content sub-rows
when a full Step is available. Adds formatActionRow/formatElapsed helpers and
covers both with unit tests.
* fix(runner): stop copying Tap selector into action.text
The 'Content' inspect row should show the user-supplied text for
InputText actions and stay empty for Taps. Previously the runner copied
action.On into traceAction.Text for both, so the inspect UI showed the
selector as the tap's 'Content'.
* fix(web): use text-muted for swipe arrow after accent-change removal
* fix(web): snapshot values truncate with ellipsis + title tooltip
Long JSON values were breaking one character per line due to
overflow-wrap:anywhere in a narrow column. Switch to single-line ellipsis
with the full value exposed via the title attribute on hover.
* feat(web): state-before/after columns + metrics chart at bottom
RunDetail now renders a four-column grid:
actions | state-before | state-after | side (exceptions + timeline)
with MetricsChart spanning the bottom row. Each state column shows its
own screenshot (step-NNNNN.png vs step-NNNNN-after.png), snapshot table,
and violations panel. ActionList now receives runStartMillis and the
selected Step so the active row can expand Position/Content sub-rows.
* fix(web): skip zero-value ticks + add exception markers to metrics
HEAP '0B' and CPU '100%' labels overlapped at the lane boundary. Drop
the bottom-of-range tick on both lanes (baseline is implied) and widen
LANE_GAP so the remaining labels have breathing room. Accept an
exceptionStepIndices prop and draw a dashed red vertical line at each
to surface exception spikes directly on the CPU/heap chart.
* fix(web): let action body column shrink below its content
Required minmax(0, 1fr) so the row grid honours the column's min-size of
0 instead of the implicit 'auto', preventing the action-list from
overflowing its parent when the target string is long.
* feat(web): bigger state screenshots + single properties row
Collapse snapshots into a summary chip ('SNAPSHOTS · N violations') so
the screenshot fills its state card. Deduplicate ViolationsPanel —
show it once in a new full-width 'properties' row between the state
cards and the timeline. Drop the right sidebar; exceptions now surface
as dashed markers on the metrics chart with the ExceptionsPanel only
rendering when there are actual exceptions to report.
* feat(web): add minimal Tabs component
Monochrome tab strip with underline-on-active. Used by state-before
and state-after cards to swap between Screenshot, Snapshots, Properties.
Pane scrolls internally so the outer grid stays fixed-height.
* feat(web): fold timeline into MetricsChart as STEPS lane
Adds a thin per-step status row above HEAP showing violated (red),
pending (dim gray) or holds (green-tinted). Extends highlight +
exception markers to span the status lane. Frees a whole row in the
detail grid so the page can fit in 100vh.
* refactor(web): tabbed state cards, drop standalone Timeline panel
State-before/after now use Tabs (Screenshot / Snapshots / Properties,
default Screenshot). Removes the dedicated timeline row; status lane
lives on the metrics chart. Banner is gone from the shell.
* feat(web): lock app shell to 100vh with no page scroll
html/body/#root fill the viewport, body gets overflow:hidden, and the
detail grid uses minmax(0, 1fr) rows so inner panels own their scroll.
Tightens toolbar + panel padding for a denser feel.
* feat(web): arrow-key nav + badges on Tabs (WAI-ARIA tablist)
Roving tabindex, ArrowLeft/Right/Up/Down/Home/End navigation, explicit
aria-selected/aria-controls/id wiring, and support for an optional
badge inside each tab (used for violation counts).
* feat(web): ViolationsPanel supports violationsOnly filter
* feat(web): ActionList arrow-key nav + listbox semantics + smaller font
Promote the list to role=listbox with role=option rows; roving tabindex
lets ArrowUp/Down (and Home/End) seek between steps with focus. Font
size dropped to 11px and padding tightened so long selector-tag labels
fit in the 340px actions column.
* fix(web): useKeyboardNav yields arrow keys to tablist/listbox targets
Previously pressing ArrowRight on a focused tab switched tabs AND
advanced the step. Skip arrow handling when the event target is inside
an element with an arrow-owning ARIA role.
* feat(web): fourth 'Violations' tab + wider actions + shorter metrics
Adds a Violations tab to each state card showing only violated properties
(with count badge on the tab label when > 0). Actions column widened
from 280px to 340px, bottom metrics strip trimmed from 220px to 140px
with tighter lane heights, so the whole page still fits in 100vh with
no scrollbar.
* feat(web): compact RunDetail layout using 1px borders instead of panel padding
* refactor(inspect): simplify MetricsChart to HEAP+CPU with time axis
Drop the STEPS status lane and per-sample circle markers, switch the
x-axis from step indices to mm:ss clock time, trim y-axis ticks to
min/max with compact units, rotate lane labels into the left gutter,
and replace the thin playhead line with a wider dotted red band.
Traces stay grayscale; red appears only on the playhead pattern.
* fix(web): RunList rows no longer stretch to fill viewport height
Tables inherited flex: 1 1 auto from .app-main > * and distributed extra
vertical space across rows. Override with flex: 0 0 auto + align-self.
* misc changes
* fix(web): hoist useState above early return in MetricsChart
Calling useState after an unconditional early return violates React's
Rules of Hooks: the empty-samples branch renders 0 hooks while the
populated branch calls 1. On the initial null->loaded transition of
history the hook count changes and React throws.
* fix(web): subscribe to named SSE event instead of 'message'
Server emits 'event: runs.changed' frames; the WHATWG EventSource spec
dispatches those as events of type 'runs.changed', not 'message'. The
listener registered on 'message' was never fired, so RunList never
auto-refreshed on run create/finish/delete.
* fix(inspect): unsubscribe SSE clients on disconnect
Watcher.Subscribe appended to a slice with no matching removal path,
so every closed EventSource connection leaked its channel. Over a
long-running server the slice grew unbounded and every fs event paid
O(N) iterating dead channels. Add Unsubscribe + defer it in
handleEvents.
Unsubscribe does not close the channel: broadcast snapshots the
slice without holding the mutex, so a concurrent close would race
with its non-blocking send.
* fix(trace): rename resolvedBounds/tapPoint to snake_case
Every other json tag in the trace schema (from_x, duration_millis,
bundle_sha256, etc.) uses snake_case. The two new Action fields
introduced with the inspect UI broke that pattern. Rename them
before the format ships to external consumers.
* chore(web): drop vitest and remove UI tests from CI
No UI tests wanted in web. Removes vitest, jsdom, testing-library
devDeps and the vitest.setup.ts + vite.config.ts test block.
Makefile test-web becomes web-typecheck (typecheck only).
Fixes CI failure where `vitest run` exits 1 with no test files.
* chore(make): dedupe sidecar embed and drop recursive make
Make $(SIDECAR_JAR) the real recipe and $(SIDECAR_EMBED) a file
target, so uatu/install/inspect-dev share one copy step and
sidecar/release-cli just depend on the jar instead of re-invoking make.
|
||
|
pj
|
d0578dbaaa |
fix(runner): warn on malformed screen snapshot (#13)
* fix(runner): warn on malformed screen snapshot screenFromSnapshot swallowed json.Unmarshal errors, so a non-string screen value silently became "" in the step log and trace while the verifier still saw the raw JSON. Return the error and warn at the call site, matching the hierarchy warning pattern. * docs: clarify --avd is optional for uatu test The CLI accepts --avd as an empty-string default (cmd/uatu/main.go:49) and only requires it when no device is connected and multiple AVDs exist (cmd/uatu/android_env.go:63). Docs and examples that showed it as required or always-passed were misleading. |
||
|
pj
|
e62319e916 |
docs: pandoc-based site and v0.1.0 groundwork (#5)
* chore(prose): remove em-dashes from config files * chore(prose): remove em-dashes from android sdk config * docs(spec-api): remove em-dash from README * fix(doctor): reword sidecar-jar error without em-dash * test(sidecar): reword assertion message without em-dash * docs: add CLAUDE.md with project conventions * build: add docs target for pandoc site * docs(site): add pandoc template and stylesheet * docs(site): add pandoc build script * docs(site): add landing pages * docs(manual): add getting-started * docs(manual): add writing-specs * docs(manual): add runs * docs(manual): add cli reference * docs(dev): add design principles * docs(dev): add architecture * ci: deploy docs site to github pages * docs: rewrite README as entry point to docs site |