mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 11:07:10 +00:00
merge origin/master into llm-recording-and-analysis
both sides independently fixed the same three bugs, so each one had to pick a winner rather than keep both implementations. extractor encoding: master's recordableValue in worker.go wins over ours in marshal.go, since master's is pinned by extractor_encoding_test.go and ours had no tests. our error semantics stay: encodeExtractorValue still returns an error instead of nil, so an extractor cannot vanish from the trace silently. apply errors: only the residual generic branch takes master's unconfirmed copy, where the device may have committed the action before the call failed. the finer branches that know nothing was dispatched keep lastAction = nil, and our actionSkipReason taxonomy stays alongside master's held/skippedVerification. selector matching: our matchAttr with matchSelectorKind wins over master's match, since ours also handles idPrefix. matchSelector now calls it, which git did not flag as a conflict and left calling a function our side had deleted. the ltl doc comment takes master's correction: an unbounded eventually that never fires IS violated at run end.
This commit is contained in:
pj committed
commit
6e85cac8b3
130 files changed
+12772
-1617
No files matched your search
+215
-37
@@ -4,22 +4,18 @@ title: CI
|
||||
|
||||
# CI
|
||||
|
||||
`ci.yml` runs on every pull request: it builds, unit-tests, and drives three
|
||||
small web fixtures through headless Chrome (`test/browser/testdata`). It never
|
||||
runs sanderling against a real app.
|
||||
`ci.yml` runs on every pull request and every push to master. The `Check`
|
||||
jobs build, unit-test, and drive three small web fixtures through headless
|
||||
Chrome (`test/browser/testdata`). The `Folio` and `Replay UI` jobs in the same
|
||||
workflow do run sanderling against real apps, on emulators and simulators, and
|
||||
they are what make a run take the better part of an hour.
|
||||
|
||||
Two other workflows do, and both are `workflow_dispatch` only. They boot devices,
|
||||
build apps and take minutes, which is not what you want on every push, and
|
||||
neither is a merge gate.
|
||||
`All checks passed` is the one status check to point branch protection at, and
|
||||
it is what gates a release: `release.yml` cuts one only after a whole ci run
|
||||
went green. See [Releases](#releases) at the bottom.
|
||||
|
||||
## folio
|
||||
|
||||
Actions -> folio -> Run workflow. Inputs pick the legs (`all`, `android`, `ios`,
|
||||
`web`), and override the seed, the step budget and the wall-clock budget. Seed
|
||||
and step budget take `0` to mean "use the calibrated value in the workflow"; the
|
||||
wall-clock budget has no such sentinel and is passed through as written, so
|
||||
every leg gets whatever you type there.
|
||||
|
||||
Each leg builds `examples/folio` for its platform, builds the CLI with only the
|
||||
tags that platform needs (`make sanderling-android` and friends), and runs
|
||||
`examples/folio/sanderling/spec.ts` through `.github/scripts/folio-run.sh`. That
|
||||
@@ -32,8 +28,8 @@ SEED=9 MAX_STEPS=200 .github/scripts/folio-run.sh android
|
||||
|
||||
**web and ios expect the bug.** Folio double-submits a transaction when the
|
||||
submit button is double-tapped, and two properties catch it:
|
||||
`submitMovesBalanceByTypedAmount`, which demands the total balance move by
|
||||
exactly the amount typed, and `submitCommitsOneTransactionPerAction`, which
|
||||
`submitMovesBalanceByAtMostTypedAmount`, which demands the total balance move by
|
||||
no more than the amount typed, and `submitCommitsOneTransactionPerAction`, which
|
||||
demands no more transactions committed over a window than there were submit
|
||||
actions in it. A double tap is one action committing two transactions, so it
|
||||
breaks both.
|
||||
@@ -96,27 +92,83 @@ The wasmJs app is served with `Cross-Origin-Opener-Policy` and
|
||||
cross-origin isolation. Served without them the app loads a blank canvas and
|
||||
every step observes an empty accessibility tree.
|
||||
|
||||
The seeds are calibrated, not guessed. On an M-series mac, web seed 3 convicts at
|
||||
step 185-187 and ios seed 7 at step 97-101, each 3 runs out of 3 and each with a
|
||||
delta of exactly twice the typed amount. Both run a 240-step budget. Keep them
|
||||
pinned: honest evidence is rare, and across 2261 ios steps only one submit tap
|
||||
landing on Home had a single-submit window.
|
||||
The seeds are calibrated, not guessed, and every number here says which host it
|
||||
was measured on, because the hosts do not agree. On an M3 mac driving iOS 26.1
|
||||
simulators, ios seed 7 convicts at step 97-101, 11 runs out of 11 from a cleared
|
||||
install, each on both properties and each with the balance moving by exactly
|
||||
twice the typed amount: 199 typed, 39800 cents moved, one account's transaction
|
||||
count rising by two against a window holding one submit. Web seed 3 convicts at
|
||||
step 185-187 on that mac and at step 192 on the ubuntu runner. Both legs run a
|
||||
240-step budget.
|
||||
|
||||
Android runs seed 9 over 200 steps: its conviction lands at step 178, so a
|
||||
What those numbers assume is a cleared starting state, and that is the only thing
|
||||
that moved them. Measured four ways on one simulator, seed 7 convicts at step 97
|
||||
from a fresh install with clear-state on, at 100 from a fresh install with it
|
||||
off, and at 97 from a dirty container with it on. It walks 240 steps clean
|
||||
exactly once: dirty container, clear-state off, where the app opens already
|
||||
signed in on the previous run's accounts and the walk diverges at step 1. The leg
|
||||
therefore clears state for itself rather than relying on how it was called.
|
||||
|
||||
Do not read a mac number as a statement about CI, but the ios leg does now
|
||||
convict there. It had never done so before 2026-08-16, through every dispatch,
|
||||
and no seed was ever the reason. `submitCommitsOneTransactionPerAction` counted
|
||||
every tap on TxnSubmit toward its window, including taps the app refuses because
|
||||
the amount field is empty, and more than half of a typical window was those.
|
||||
Measured on recorded android runs: 35 taps against a real budget of 16, and 42
|
||||
against 17. A window that wide cannot attribute anything, which is why seed 28
|
||||
reached the bug at the step it convicts at locally and was still not judged.
|
||||
|
||||
`submitCouldCommit` stopped counting them, and the numbers moved a long way:
|
||||
|
||||
leg before after (run 31898888205)
|
||||
ios 240 steps clean, every time convicts step 59, detected 60
|
||||
web step 192 on the ubuntu runner convicts step 185, detected 186
|
||||
android never reached AddTransaction healthy over 200 steps, reached it
|
||||
|
||||
The ios witness at that conviction reads one account's transaction count rising
|
||||
from 0 to 7 against a window holding 6 submits, with `applied: true` on the
|
||||
action and `is_error` unset. Seven transactions from six submits is one double
|
||||
submit, which is the bug the leg exists to find.
|
||||
|
||||
On the calibration mac the same seed now convicts around step 48, twice in a row,
|
||||
where it used to convict at 97-101. Treat both as approximate: the point is that
|
||||
the window is now tight enough to attribute a submit, not that any particular
|
||||
step number is pinned. A run that fails is worth reading before it is worth
|
||||
recalibrating.
|
||||
|
||||
Android runs seed 9 over 200 steps. Its conviction lands around step 178, and a
|
||||
shorter budget would never see the bonus. A full run costs about five minutes.
|
||||
|
||||
Repeating the ios leg by hand is not the same as running it in CI: with
|
||||
`--clear-data=false` a second local run inherits the first one's accounts, so
|
||||
`simctl uninstall` before each repeat or the numbers drift.
|
||||
That step number was measured on a local emulator with animations ON, and the CI
|
||||
job sets `disable-animations: true`, so it does not describe the CI leg. The
|
||||
worry that follows is that zeroing the 700ms Compose fade would stop the leg
|
||||
exercising the cross-fade wait entirely, and the traces say that worry is
|
||||
largely right. The first dispatch, whose run was stuck on one screen, carried 4
|
||||
`transitional` steps in 200. The healthy runs since carry **zero**. So on CI the
|
||||
wait almost never fires, and a leg that is green there is not evidence the wait
|
||||
works. Local runs with animations on are where that gets exercised. Treat the
|
||||
android number as an order of magnitude, not a pin. It is a health gate, so
|
||||
nothing keys on it.
|
||||
|
||||
The ios leg passes `--clear-data=false`, because the job installs a fresh build
|
||||
immediately before the run and a freshly installed app is already clear state.
|
||||
The in-run reinstall is worth avoiding: `simctl uninstall` + `install` followed
|
||||
straight away by the XCTest runner's own launch fails with `app.folio is unknown
|
||||
to FrontBoard` maybe half the time. That used to hang the run outright; the
|
||||
launch RPC is bounded now, so it fails in about 90 seconds with a real error
|
||||
instead, but a failing leg is still a failing leg. The job timeouts are the
|
||||
backstop if it happens anyway.
|
||||
Repeating the ios leg by hand needs nothing special now, because the run clears
|
||||
the app's state itself. It used to: `just ios` installs over the top without
|
||||
uninstalling and folio's signed-in session survives that, so a repeat under the
|
||||
old `--clear-data=false` opened on the previous run's Home screen and diverged at
|
||||
step 1. That is how the leg came to look dead while the app and the seed were
|
||||
both fine, and it is worth recognising: a leg that reports "the double-submit bug
|
||||
was NOT found" from a machine that has been running the app all day is describing
|
||||
the machine.
|
||||
|
||||
The ios leg clears state and passes no `--ios-app-path`, which is deliberate:
|
||||
without an app path the driver wipes the app's data container instead of
|
||||
reinstalling, and the reinstall is the path that races FrontBoard. `simctl
|
||||
uninstall` + `install` followed straight away by the XCTest runner's own launch
|
||||
has failed with `app.folio is unknown to FrontBoard` about half the time on the
|
||||
host that reported it. That race is untouched and still open; the leg simply
|
||||
does not take that path. It did not reproduce here at all, in 20 consecutive
|
||||
reinstall-and-launch cycles on iOS 26.1, 10 of them reinstalling on top of a
|
||||
live app, so any fix for it has to be developed on a host that can still show it
|
||||
failing.
|
||||
|
||||
Only one sanderling run may drive a given simulator at a time. The driver takes
|
||||
an advisory lock on the target's UDID and a second run is refused with the lock
|
||||
@@ -125,16 +177,36 @@ run's automation session bound to a bundle the simulator no longer knows.
|
||||
|
||||
## replay-ui
|
||||
|
||||
Actions -> replay-ui -> Run workflow. This one is dogfooding: it records a trace
|
||||
This one is dogfooding: it records a trace
|
||||
from `test/browser/testdata/throwing` (violations and uncaught exceptions, so
|
||||
every panel has something to render), serves it with `sanderling replay`, and
|
||||
fuzzes that UI with `replay-ui/sanderling/spec.ts`.
|
||||
|
||||
Six of the seven properties there are cross-panel agreements - two panels
|
||||
deriving the same fact by different paths have to say the same thing - so they
|
||||
hold for any trace and need no recalibrating when the fixture changes. The
|
||||
seventh is the stock `noUncaughtExceptions`, which asks nothing of the panels
|
||||
and only fails if the UI throws. Any violation fails the job.
|
||||
Three of the seven properties there are cross-panel agreements - two panels
|
||||
deriving the same fact by different paths have to say the same thing. The other
|
||||
four are a range invariant on the step in the URL, a count of selected rows
|
||||
inside the list, a no-effect property across a tab switch, and the stock
|
||||
`noUncaughtExceptions`, which asks nothing of the panels and only fails if the
|
||||
UI throws. All seven hold for any trace and need no recalibrating when the
|
||||
fixture changes. Any violation fails the job.
|
||||
|
||||
So does a run that judged nothing. Exit 0 says no property returned false, which
|
||||
is not the same as any property having been evaluated: each one declines to
|
||||
judge when the elements it reads are absent, so a run where the trace failed to
|
||||
serve, or where the fuzzer sat on the run list, renders nothing and passes.
|
||||
`.github/scripts/replay-ui-summary.sh` reads the trace and puts a per-property
|
||||
count of judged against declined steps in the job summary. Run it by hand with
|
||||
`GITHUB_STEP_SUMMARY=/dev/stdout .github/scripts/replay-ui-summary.sh runs/dogfood`.
|
||||
|
||||
It fails the job when any of the four properties that need nothing beyond the
|
||||
step page having rendered - `selectedStepIsInRange`, `exactlyOneStepIsSelected`,
|
||||
`stepCountMatchesTheList`, `screenshotShowsTheSelectedStep` - judged nothing at
|
||||
all. The other three are reported and not gated, because a zero on them is a
|
||||
seed getting unlucky rather than a broken leg: `switchingTabsKeepsTheStep` needs
|
||||
a tab switch between consecutive steps, and `badgeCountMatchesThePanel` needs the
|
||||
fuzzer to land on a violating step and open the violations tab in that same
|
||||
step. On the first run measured this way (seed 3, 80 steps) that last one judged
|
||||
nothing at all, so the fixture reaches it far too rarely to be worth gating on.
|
||||
|
||||
## Reading a failure
|
||||
|
||||
@@ -159,3 +231,109 @@ do not raise the step budget blindly - run a seed sweep with the campaign tool
|
||||
(`cmd/internal-tools/campaign`), which exists for exactly this, and pin a seed
|
||||
that finds the bug with room to spare. A leg failing with "a predicate threw" is
|
||||
a different problem entirely and no seed will fix it.
|
||||
|
||||
Sweep in the leg's own configuration, though. The campaign tool and the ios leg
|
||||
now clear state the same way, so a swept seed means what the leg means, but the
|
||||
starting frame is not a detail you can skip checking: while the leg still passed
|
||||
`--clear-data=false`, seed 14 convicted at step 17 in 2 campaign runs out of 2
|
||||
and in 0 leg-shaped runs out of 3. Prefer the earliest conviction on offer over
|
||||
the first one found, too. A run reproduces its trajectory on another host only
|
||||
for as long as every snapshot agrees, and every step of prefix is another chance
|
||||
for it not to: seeds convicting at steps 33, 60, 114, 187 and 189 all turned up
|
||||
within the first 30, so an early one is usually there to be found.
|
||||
|
||||
A short prefix is necessary and not sufficient, though, and ios is the standing
|
||||
counter-example: seed 28 has the shortest prefix on offer, reproduced its walk on
|
||||
the runner exactly, reached the bug at step 32, and still did not convict,
|
||||
because the window the counting invariant had to judge it in was 117 steps wide.
|
||||
Sweeping selects for a seed that reaches the bug. It cannot select for one whose
|
||||
walk also closes the window, so when a property needs a window, check what the
|
||||
window looked like and not only that the conviction happened.
|
||||
|
||||
|
||||
## Releases
|
||||
|
||||
**Every merge to master cuts a patch.** The `Tag`, `Release (npm)` and
|
||||
`Release (cli)` jobs sit in `ci.yml` alongside everything else, waiting on
|
||||
`Checks`, `Folio` and `Replay UI`, so nothing reaches a registry that the
|
||||
emulators and the simulator have not agreed on. `0.1.4` becomes `0.1.5`:
|
||||
published to npm, and to GitHub Releases with the CLI binaries.
|
||||
|
||||
**A milestone consolidates them.** Actions -> ci -> Run workflow, set `promote`
|
||||
to `minor` or `major`, and the patches you have been shipping become `0.2.0`.
|
||||
Leaving `promote` on `none` is an ordinary ci run that publishes nothing, which
|
||||
is what stops a dispatch meant to re-run the tests from cutting a release.
|
||||
|
||||
A promotion runs the whole suite, device legs included. It is the same pipeline
|
||||
either way, and a release that skipped the checks would be the only release
|
||||
nobody checked. Both paths release the commit the run tested rather than
|
||||
whatever master drifted to while it ran. Afterwards the patch line continues
|
||||
from the milestone: the next merge counts off `0.2.0` and cuts `0.2.1`.
|
||||
|
||||
**The tags are the version.** Nothing in the tree holds it:
|
||||
`pkg/spec/package.json` stays at `0.0.0-dev` and CI stamps the real version in
|
||||
before it publishes. So there is no version-bump commit to land on master,
|
||||
nothing to conflict on, and no second record to hold in step with the tags.
|
||||
`.github/scripts/next-version.sh` is the whole rule, and it counts off stable
|
||||
tags only, because `v0.0.1-rc4` is a candidate for `0.0.1` and a patch counted
|
||||
off it would skip the version it was a candidate for. Run it anywhere to see
|
||||
what the next release would be:
|
||||
|
||||
```
|
||||
BUMP=minor .github/scripts/next-version.sh
|
||||
```
|
||||
|
||||
The tag is pushed before anything is published, because npm is the half of a
|
||||
release that cannot be taken back and a tag is the half that can.
|
||||
|
||||
### How far back the notes reach
|
||||
|
||||
GoReleaser builds its changelog from the commits between the previous tag and
|
||||
this one, and works that previous tag out on its own. For a patch that is
|
||||
exactly right. For a milestone it is not: the notes on a `0.2.0` consolidating
|
||||
six patches would describe the one merge that happened to be last.
|
||||
|
||||
So the resolver also emits `previous_tag`, which the release passes as
|
||||
`GORELEASER_PREVIOUS_TAG`: the last release at the level being cut. A minor
|
||||
reaches back to the last `vX.Y.0`, counting a major as one, and a major reaches
|
||||
back to the last `vX.0.0`. The first milestone of its kind has nothing at its own
|
||||
level, so it reaches back to the first release there has ever been. A patch emits
|
||||
nothing, and an empty value leaves GoReleaser on the default that was already
|
||||
right for it.
|
||||
|
||||
The boundary is exclusive, the way a changelog always is: the notes cover what
|
||||
landed *after* that tag. So the one release this shortchanges is the first
|
||||
milestone of its kind, whose notes start after the first release rather than at
|
||||
it. That is one merge, once, and it is not worth a special case.
|
||||
|
||||
### Why the release is not its own workflow
|
||||
|
||||
It reads like it should be. The reason it is not is npm.
|
||||
|
||||
npm publishes over OIDC here, against a trusted publisher configured for
|
||||
`@sanderling/spec`, so CI holds no npm credential at all. That is not a
|
||||
preference: npm disabled classic token creation in November 2025, revoked every
|
||||
classic token on 9 December 2025, and caps a granular token at 90 days. A token
|
||||
in CI would now expire quarterly, which is exactly the failure this replaced.
|
||||
The August 2026 outage was an expired granular token, and npm answers a publish
|
||||
it will not authorise with `404`, so it read as "package does not exist" while
|
||||
`@sanderling/spec` sat in the registry the whole time.
|
||||
|
||||
A package carries exactly one trusted publisher, and npm matches it against the
|
||||
filename of the workflow that *starts* the run. A reusable workflow does not
|
||||
help, because npm sees the caller's name, not the callee's. So every publish has
|
||||
to enter through one file, and since a merge's release has to run inside ci, that
|
||||
file is `ci.yml`.
|
||||
|
||||
Setting it up again, or moving the package, means npmjs.com -> the package ->
|
||||
trusted publisher: repository `priyanshujain/sanderling`, workflow `ci.yml`. Or
|
||||
from a shell, which needs an interactive 2FA challenge:
|
||||
|
||||
```
|
||||
npm trust github @sanderling/spec --file ci.yml --repo priyanshujain/sanderling --allow-publish
|
||||
npm trust list @sanderling/spec
|
||||
```
|
||||
|
||||
The job installs npm 11.5.1 or newer before publishing, because
|
||||
`actions/setup-node` writes an empty `_authToken` line into `.npmrc` and an older
|
||||
npm reads that as "auth is configured" and never asks for an OIDC token.
|
||||
@@ -24,10 +24,11 @@ Nobody writes this test. A manual tester taps submit once, sees the right number
|
||||
|
||||
You do not script the double tap. You state the invariant and let sanderling find the inputs that break it.
|
||||
|
||||
The amount the user types must equal the amount the balance moves:
|
||||
One submit commits one transaction, so the balance cannot move by more than the
|
||||
amount that submit typed:
|
||||
|
||||
```ts
|
||||
const submitMovesBalanceByTypedAmount = always(
|
||||
const submitMovesBalanceByAtMostTypedAmount = always(
|
||||
next(() => {
|
||||
if (route.current !== "home") return true;
|
||||
const action = lastAction.current;
|
||||
@@ -38,16 +39,18 @@ const submitMovesBalanceByTypedAmount = always(
|
||||
if (typed === 0) return true;
|
||||
const before = totalBalance.previous;
|
||||
if (before === null || totalBalance.current === null) return true;
|
||||
return Math.abs(totalBalance.current - before) === typed;
|
||||
return Math.abs(totalBalance.current - before) <= typed;
|
||||
})
|
||||
);
|
||||
```
|
||||
|
||||
`always` checks the formula at every step; `next` lets it compare the step before a submit to the step after. The guards narrow it to the one transition that matters, a submit that lands back on home, and the last line states the rule: the balance moved by exactly the typed amount. Double-submit moves it by twice that, and the formula is false.
|
||||
`always` checks the formula at every step; `next` lets it compare the step before a submit to the step after. The guards narrow it to the one transition that matters, a submit that lands back on home, and the last line states the rule: the balance moved by no more than the typed amount. Double-submit moves it by twice that, and the formula is false.
|
||||
|
||||
The window guard is the difference between a property and a false conviction. `totalBalance.previous` is the last total we read, not the total as of the last transaction, so the two numbers being compared can straddle any number of commits: a real run produced a delta of 13000 against a typed 19600, because the window held a double-submit's two 19600 debits and an unrelated 26200 credit. A delta like that is not evidence about the amount typed into any one submit. Exactly one submit action in the window still catches the bug, because the double tap is a single action.
|
||||
The obvious version of that last line is `=== typed`, and it is the version this spec used to ship. It was wrong. Folio's `createTransaction` runs in a coroutine and Home's total re-renders on the store's own schedule, so a total that has not caught up yet is what a healthy app looks like a frame after a submit, and an equality convicts it. So does a submit the app rejected, and so does a tap that never landed. The bound declines on all three without needing a case for any of them, and it still catches the bug, because twice the typed amount is more than the typed amount. What it gives up is worth naming: a transaction committed for less than the amount typed is a real ledger bug this property no longer sees. It cannot be told apart from a total one frame behind, and a check that fires on both is evidence about neither.
|
||||
|
||||
The null guard is not defensive clutter either. Read a balance you could not parse as `0` and the comparison becomes `0 - 0 === typed`, which is false at every healthy submit. A reading you do not have is not evidence, so the property declines to judge. The real spec guards the same way against a balance too large for exact integer arithmetic.
|
||||
The window guard is what keeps the comparison about one submit. `totalBalance.previous` is the last total we read, not the total as of the last transaction, so the two numbers being compared can straddle any number of commits: a real run produced a delta of 13000 against a typed 19600, because the window held a double-submit's two 19600 debits and an unrelated 26200 credit. A delta over a window like that is not evidence about the amount typed into any one submit, whichever side of the bound it falls on. Exactly one submit action in the window still catches the bug, because the double tap is a single action.
|
||||
|
||||
The null guard is not defensive clutter either, and under a bound its failure mode is the quiet one. Read a balance you could not parse as `0` and the comparison becomes `|0 - 0| <= typed`, which is true at every submit: the property stops judging and never says so. A reading you do not have is not evidence, so it has to decline in the open rather than pass by accident. The real spec guards the same way against a balance too large for exact integer arithmetic.
|
||||
|
||||
The values it reads come from extractors, which pull state out of the UI tree once per step:
|
||||
|
||||
@@ -179,7 +182,7 @@ That is the whole input. Three invariants, a way in, and a weighted sense of whe
|
||||
|
||||
## What the run does
|
||||
|
||||
sanderling launches Folio, logs in, and starts exploring. Most steps are unremarkable: open an account, add a transaction, watch the balance move by exactly what was typed, `submitMovesBalanceByTypedAmount` holds.
|
||||
sanderling launches Folio, logs in, and starts exploring. Most steps are unremarkable: open an account, add a transaction, watch the balance move by the amount that was typed, `submitMovesBalanceByAtMostTypedAmount` holds.
|
||||
|
||||
Then a step lands two taps on submit before the first save settles. Two transactions post. The balance jumps by twice the typed amount. At that step the formula evaluates false and the run records a violation: the step, the screenshot, the offending action, and the residual formula that failed.
|
||||
|
||||
|
||||
+4
-2
@@ -16,7 +16,8 @@ Run a spec against an app for a fixed duration.
|
||||
|---|---|---|
|
||||
| `--spec` | required | Path to the TypeScript spec. |
|
||||
| `--bundle-id` | required | Target app bundle ID (Android: applicationId). |
|
||||
| `--launcher-activity` | resolved | Optional `<pkg>/<activity>` to launch. Overrides default resolution. |
|
||||
| `--device` | optional (android) | Android device serial, as `adb devices` reports it. Required when more than one device is attached. |
|
||||
| `--android-app-path` | optional (android) | Path to the APK. Clear-state reinstalls from it instead of running `pm clear`. |
|
||||
| `--platform` | `android` | Target platform: `android`, `ios`, or `web`. |
|
||||
| `--avd` | optional (android) | Android AVD name to boot if no device is connected. Required only when no device is connected and multiple AVDs exist. |
|
||||
| `--ios-device` | optional (ios) | iOS target: a simulator name/UDID to boot, or a connected device's name, UDID, or CoreDevice id. |
|
||||
@@ -24,6 +25,7 @@ Run a spec against an app for a fixed duration.
|
||||
| `--duration` | `5m` | Total test duration (`30s`, `5m`, `2h`, `1d`). |
|
||||
| `--max-steps` | `0` | Stop after this many steps (`0` = no cap, the duration governs). A step budget is what makes two generators comparable. |
|
||||
| `--exit-on-violation` | `false` | Stop the run at the first property violation and exit `2`. |
|
||||
| `--arm` | optional | Experiment label recorded in the run's metadata. Used by the campaign tool to tell one sweep cell from another. |
|
||||
| `--seed` | `0` | PRNG seed. `0` uses a random seed and records it in `meta.json`. |
|
||||
| `--generator` | `seeded` | Who picks each action: `seeded` (the run's PRNG) or `llm` (a vision model). See [the LLM generator](../spec-language/#llm-generator). |
|
||||
| `--output` | `./runs` | Output directory for traces. |
|
||||
@@ -56,7 +58,7 @@ sanderling doctor [--platform web|android|ios|ios-device|all]
|
||||
| Platform | Checks |
|
||||
|---|---|
|
||||
| `web` | headless Chromium can launch (the bundled CDP surface boots a real browser). |
|
||||
| `android` | `adb` on PATH; `emulator` on PATH or under `ANDROID_HOME`; Java 17+; embedded native sidecar JAR is real. |
|
||||
| `android` | `adb` and `emulator` on PATH, or under `$ANDROID_HOME`, `$ANDROID_SDK_ROOT` or a standard SDK install location; Java 17+; embedded native sidecar JAR is real. |
|
||||
| `ios` | `xcrun` on PATH; `simctl` on PATH. The simulator path drives the native companion with no JVM. |
|
||||
| `ios-device` | the `ios` checks plus `devicectl`; the macOS `usbmuxd` socket; a connected, paired device; App Store Connect signing credentials present. |
|
||||
|
||||
|
||||
@@ -20,6 +20,8 @@ The spec package, in your project:
|
||||
npm install --save-dev @sanderling/spec
|
||||
```
|
||||
|
||||
Both come from the same release tag, and the CLI bundles the package's TypeScript sources when it evaluates your spec, so upgrade them together. Pre-releases are published under npm's `next` tag; `npm install @sanderling/spec` gives you the current stable one.
|
||||
|
||||
## Check your environment
|
||||
|
||||
```sh
|
||||
@@ -28,9 +30,9 @@ sanderling doctor
|
||||
|
||||
`doctor` reports what the target platform needs and what is missing:
|
||||
|
||||
- **Android**: `adb` on your PATH, and an emulator (API 30 or newer) or a connected device.
|
||||
- **iOS**: Xcode 16 or newer, with a simulator. For a connected iPhone, run `sanderling doctor --platform ios-device`.
|
||||
- **Web**: Chrome.
|
||||
- **Android**: `adb` and `emulator`, on your PATH or under the Android SDK; Java 17 or newer; a real (not placeholder) sidecar JAR in the binary.
|
||||
- **iOS**: `xcrun` and `simctl`. For a connected iPhone, run `sanderling doctor --platform ios-device`, which also wants `devicectl`, a paired device, and signing credentials.
|
||||
- **Web**: a Chromium that launches headless.
|
||||
|
||||
## Write a spec
|
||||
|
||||
@@ -44,7 +46,7 @@ export const properties = { noUncaughtExceptions };
|
||||
export const actionsRoot = defaultActions;
|
||||
```
|
||||
|
||||
This taps, types, scrolls, and swipes at random, and fails the moment your app throws an uncaught exception. From here you add extractors to read your screens, properties that state what your app guarantees, and actions that drive its real flows. The [case study](../case-study/) walks a complete spec, and the [spec language reference](../spec-language/) lists every primitive.
|
||||
This taps, types, double-taps, scrolls, and swipes at random. Check the property matches your platform before you trust a green run: `noUncaughtExceptions` reads exceptions the web runtime captures in the page, so it fires on `--platform web` and holds unconditionally on Android and iOS. On Android use `noLogcatErrors` instead, which fails on any error-level log line and so catches an uncaught throwable. iOS has neither today, so an iOS run is only worth as much as the properties you write yourself. From here you add extractors to read your screens, properties that state what your app guarantees, and actions that drive its real flows. The [case study](../case-study/) walks a complete spec, and the [spec language reference](../spec-language/) lists every primitive.
|
||||
|
||||
## Run it
|
||||
|
||||
|
||||
+4
-2
@@ -13,7 +13,7 @@ A run is not a unit test. The closer picture is: boot a fuzzer for an hour and s
|
||||
```
|
||||
sanderling test --spec spec.ts --bundle-id com.example.app --duration 30m
|
||||
│
|
||||
├── launch the app under test (pass --clear-data to wipe app data first)
|
||||
├── launch the app under test (wipes app data first unless --clear-data=false)
|
||||
├── boot the sidecar (or connect to Chrome on web)
|
||||
├── bundle the spec, load it into the JS runtime
|
||||
│
|
||||
@@ -64,4 +64,6 @@ A run ends when:
|
||||
- `--duration` elapses, or
|
||||
- the process is interrupted (Ctrl+C).
|
||||
|
||||
Additional conditions (`--max-steps`, `--exit-on-violation`, hard crash handling) land in the [v0.1.0 milestone](https://github.com/priyanshujain/sanderling/milestone/1).
|
||||
- `--max-steps` is reached, or `--exit-on-violation` was passed and a property was violated.
|
||||
|
||||
Hard crash handling lands in the [v0.1.0 milestone](https://github.com/priyanshujain/sanderling/milestone/1).
|
||||
@@ -29,7 +29,7 @@ Every extractor callback receives a `State`:
|
||||
interface State {
|
||||
ax: AccessibilityTree;
|
||||
snapshots: Record<string, unknown>;
|
||||
lastAction: Action | null;
|
||||
lastAction: (Action & { applied: true | null }) | null;
|
||||
logs: readonly LogEntry[];
|
||||
exceptions: readonly ExceptionRecord[];
|
||||
time: number; // ms since run start
|
||||
@@ -40,11 +40,13 @@ interface State {
|
||||
|---|---|
|
||||
| `ax` | Live UI hierarchy for this step |
|
||||
| `snapshots` | Key-value data pushed by the app SDK (empty if SDK not integrated) |
|
||||
| `lastAction` | The action dispatched in the previous step, or `null` on the first step |
|
||||
| `lastAction` | The action dispatched in the previous step, or `null` on the first step and on any step that dispatched nothing |
|
||||
| `logs` | Log entries collected since the previous step |
|
||||
| `exceptions` | Uncaught exceptions or `Sanderling.reportError()` calls since the previous step |
|
||||
| `exceptions` | Uncaught exceptions and unhandled promise rejections captured in the page. Web only: nothing fills this on Android or iOS, where it is always empty |
|
||||
| `time` | Milliseconds elapsed since the run started |
|
||||
|
||||
`lastAction.applied` is `true` when the runner saw the dispatch succeed and `null` when the apply call failed with the action possibly already delivered: an RPC deadline can fire after the tap reached the app, and nothing can find out afterwards. So there are three states, not two. `state.lastAction === null` means no action ran; `applied === null` means one ran whose fate is unknown. A property that attributes an effect to the action ("this submit must move the balance by the typed amount") has to decline unless `applied` is `true`, or a timeout convicts a healthy app. A property that counts what the app COULD have done should include it: an unconfirmed submit belongs in an upper bound on how many submits a window holds.
|
||||
|
||||
## Selectors
|
||||
|
||||
Selectors are passed to `ax.find()`, `ax.findAll()`, and element-scoped `.find()` / `.findAll()`. A tree-level lookup scans the whole hierarchy, the root element included; an element-scoped one scans that element's descendants. Both selector forms scan the same set, so `ax.find("id:page")` and `ax.find({ id: "page" })` return the same element.
|
||||
@@ -347,9 +349,9 @@ import { defaultActions, doubleTaps } from "@sanderling/spec/defaults";
|
||||
import { noUncaughtExceptions, noLogcatErrors } from "@sanderling/spec/defaults/properties";
|
||||
```
|
||||
|
||||
`defaultActions` is a ready-made weighted tree of the built-in generators: taps and typing at weight 100, scrolls 50, swipes 25, double taps 10. Use it as a baseline pool or as one entry in your own tree.
|
||||
`defaultActions` is a ready-made weighted tree of five of the built-in generators: taps and typing at weight 100, scrolls 50, swipes 25, double taps 10. `longPresses`, `pressKeys` and `waitOnce` are not in it; weight them in yourself if you want them. Use it as a baseline pool or as one entry in your own tree.
|
||||
|
||||
| Property | Fails when |
|
||||
|---|---|
|
||||
| `noUncaughtExceptions` | An uncaught exception or `Sanderling.reportError()` call is captured |
|
||||
| `noUncaughtExceptions` | The page captured an uncaught exception or an unhandled rejection (web only; holds on Android and iOS, where `state.exceptions` is never populated) |
|
||||
| `noLogcatErrors` | Logcat emits any error-level (`E`) lines since the previous step (Android only; holds elsewhere) |
|
||||
Reference in new issue
Block a user