mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 11:07:10 +00:00
* ci(folio): run gradle on jdk 21 for the metro plugin the metro gradle plugin folio builds with publishes org.gradle.jvm.version 21 and java 21 class files, so every leg failed at the folio build on a 17 runtime. local builds pass on jdk 25, which is why only ci saw it. * fix(build): clean pkg/spec/dist, not the dead spec-api path * chore: point stale spec-api comments at pkg/spec * fix(spec): publish src so an installed package carries the runtime entries * fix(testrun): alias the installed spec package so one module graph loads * fix(spec): export Direction, ScrollAction and LongPressAction from the entry * docs(spec): cut the package readme to a description and doc links * docs: say how the cli and spec package versions relate * fix(verifier): report whether the last action was confirmed applied Both hosts get applied: true when the runner saw the dispatch succeed and applied: null when it could not, so an unconfirmed action stops arriving at the spec as no action at all. * fix(runner): an apply error leaves the action's fate unknown, not undone A deadline that fires after the tap was dispatched leaves the effect committed. Reporting nil made the spec see an effect with no action to cause it, which is how the counting property convicts a healthy app. * fix(release): stage the sidecar jar at the renamed embed path * test(replay-ui): trace fixtures for the vacuity counts one real green run, one run that rendered nothing, one that judges every property at least once. * ci(replay-ui): count the steps each property judged the exit code says no property returned false; it does not say any property was ever evaluated. this reads the trace and reports judged vs declined per property, and fails when the step page never rendered. * test(replay-ui): cover the summary script from make test * ci(replay-ui): summarise through the vacuity script * docs(ci): explain the replay-ui judged/declined counts * fix(verifier): encode element-valued extractors into the trace An ax element exports with its find/findAll host functions attached, and json.Marshal refuses the whole value over them: json: unsupported type: func(goja.FunctionCall) goja.Value. The encoding failed, curr stayed nil, and the goja hosts (ios, android) recorded null for every element-valued extractor in both the per-step diff and the violation witness. Apply the web host's sanitize rule before marshaling, so one rule encodes an element on both hosts. * test(verifier): pin element encoding to one rule on both hosts * test(runner): assert an element reaches trace.jsonl and its witness * feat(spec): give state.lastAction an applied field Three states, not two: no action is a null lastAction, applied: true is an action the runner confirmed, applied: null is one it dispatched and never learned the fate of. * fix(folio): do not attribute an effect to an unconfirmed action submitChangesBalanceByTypedAmount and createdAccountHasNonZeroBalance both convict by pinning an effect on the last action, so both decline unless the runner saw it applied. The fixtures now say which fate they mean. * test(folio): an unconfirmed submit belongs in the window The count is an upper bound on the submits a window holds, so the tap that may have landed counts and committedTransactionsExceedSubmits has nothing to convict on. * test(runner): a tap that lands under a failed apply is not a double submit Drives the real folio counting predicates through the runner against a device that commits the tap and then times out. The double-submit case is the control: without it a green proves only that the property never fired. * test(verifier): pin the three lastAction states on both hosts The web page is handed the same applied field the goja object exposes, so a property cannot read one thing on native and another on web. * docs(spec-language): document the three lastAction states
This commit is contained in:
34 files changed
+1410
-158
No files matched your search
@@ -136,6 +136,24 @@ hold for any trace and need no recalibrating when the fixture changes. The
|
||||
seventh is the stock `noUncaughtExceptions`, which asks nothing of the panels
|
||||
and only fails if the UI throws. Any violation fails the job.
|
||||
|
||||
So does a run that judged nothing. Exit 0 says no property returned false, which
|
||||
is not the same as any property having been evaluated: each one declines to
|
||||
judge when the elements it reads are absent, so a run where the trace failed to
|
||||
serve, or where the fuzzer sat on the run list, renders nothing and passes.
|
||||
`.github/scripts/replay-ui-summary.sh` reads the trace and puts a per-property
|
||||
count of judged against declined steps in the job summary. Run it by hand with
|
||||
`GITHUB_STEP_SUMMARY=/dev/stdout .github/scripts/replay-ui-summary.sh runs/dogfood`.
|
||||
|
||||
It fails the job when any of the four properties that need nothing beyond the
|
||||
step page having rendered - `selectedStepIsInRange`, `exactlyOneStepIsSelected`,
|
||||
`stepCountMatchesTheList`, `screenshotShowsTheSelectedStep` - judged nothing at
|
||||
all. The other three are reported and not gated, because a zero on them is a
|
||||
seed getting unlucky rather than a broken leg: `switchingTabsKeepsTheStep` needs
|
||||
a tab switch between consecutive steps, and `badgeCountMatchesThePanel` needs the
|
||||
fuzzer to land on a violating step and open the violations tab in that same
|
||||
step. On the first run measured this way (seed 3, 80 steps) that last one judged
|
||||
nothing at all, so the fixture reaches it far too rarely to be worth gating on.
|
||||
|
||||
## Reading a failure
|
||||
|
||||
Both workflows upload their run directories as artifacts, and write the step
|
||||
|
||||
@@ -20,6 +20,8 @@ The spec package, in your project:
|
||||
npm install --save-dev @sanderling/spec
|
||||
```
|
||||
|
||||
Both come from the same release tag, and the CLI bundles the package's TypeScript sources when it evaluates your spec, so upgrade them together. Pre-releases are published under npm's `next` tag; `npm install @sanderling/spec` gives you the current stable one.
|
||||
|
||||
## Check your environment
|
||||
|
||||
```sh
|
||||
|
||||
@@ -29,7 +29,7 @@ Every extractor callback receives a `State`:
|
||||
interface State {
|
||||
ax: AccessibilityTree;
|
||||
snapshots: Record<string, unknown>;
|
||||
lastAction: Action | null;
|
||||
lastAction: (Action & { applied: true | null }) | null;
|
||||
logs: readonly LogEntry[];
|
||||
exceptions: readonly ExceptionRecord[];
|
||||
time: number; // ms since run start
|
||||
@@ -40,11 +40,13 @@ interface State {
|
||||
|---|---|
|
||||
| `ax` | Live UI hierarchy for this step |
|
||||
| `snapshots` | Key-value data pushed by the app SDK (empty if SDK not integrated) |
|
||||
| `lastAction` | The action dispatched in the previous step, or `null` on the first step |
|
||||
| `lastAction` | The action dispatched in the previous step, or `null` on the first step and on any step that dispatched nothing |
|
||||
| `logs` | Log entries collected since the previous step |
|
||||
| `exceptions` | Uncaught exceptions or `Sanderling.reportError()` calls since the previous step |
|
||||
| `time` | Milliseconds elapsed since the run started |
|
||||
|
||||
`lastAction.applied` is `true` when the runner saw the dispatch succeed and `null` when the apply call failed with the action possibly already delivered: an RPC deadline can fire after the tap reached the app, and nothing can find out afterwards. So there are three states, not two. `state.lastAction === null` means no action ran; `applied === null` means one ran whose fate is unknown. A property that attributes an effect to the action ("this submit must move the balance by the typed amount") has to decline unless `applied` is `true`, or a timeout convicts a healthy app. A property that counts what the app COULD have done should include it: an unconfirmed submit belongs in an upper bound on how many submits a window holds.
|
||||
|
||||
## Selectors
|
||||
|
||||
Selectors are passed to `ax.find()`, `ax.findAll()`, and element-scoped `.find()` / `.findAll()`.
|
||||
|
||||
Reference in new issue
Block a user