mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
Merge origin/master into llm-recording-and-analysis
Brings in #73, which landed web fact-parity work overlapping this branch: selector-tagged ax handles, aria-disabled in `enabled`, shadow-DOM traversal and a `scrollable`/`editable` dump, plus a third folio property and two new testTags on HomeScreen. Six files conflicted. The option-carrying ones took the union of both sides' fields, so `--label-source` and `--exit-on-violation` both reach the pipeline. In web-runtime.ts both sides changed how an element is described: master gave `elementHandle` a selector to tag handles with, this branch gave it raw attribute names and a field hint. Both survive, and `enabled` now answers through master's isEnabled while `editable` stays. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
This commit is contained in:
commit
d987526e47
73 files changed
+6639
-457
No files matched your search
+67
-17
@@ -33,45 +33,95 @@ const submitMovesBalanceByTypedAmount = always(
|
||||
const action = lastAction.current;
|
||||
if (action?.kind !== "Tap" && action?.kind !== "DoubleTap") return true;
|
||||
if (!JSON.stringify(action.on ?? "").includes("TxnSubmit")) return true;
|
||||
if (submitsInWindow.current !== 1) return true;
|
||||
const typed = parseTypedAmount(txnAmountField.previous?.text);
|
||||
if (typed === 0) return true;
|
||||
return Math.abs(totalBalance.current - (totalBalance.previous ?? 0)) === typed;
|
||||
const before = totalBalance.previous;
|
||||
if (before === null || totalBalance.current === null) return true;
|
||||
return Math.abs(totalBalance.current - before) === typed;
|
||||
})
|
||||
);
|
||||
```
|
||||
|
||||
`always` checks the formula at every step; `next` lets it compare the step before a submit to the step after. The guards narrow it to the one transition that matters, a submit that lands back on home, and the last line states the rule: the balance moved by exactly the typed amount. Double-submit moves it by twice that, and the formula is false.
|
||||
|
||||
The window guard is the difference between a property and a false conviction. `totalBalance.previous` is the last total we read, not the total as of the last transaction, so the two numbers being compared can straddle any number of commits: a real run produced a delta of 13000 against a typed 19600, because the window held a double-submit's two 19600 debits and an unrelated 26200 credit. A delta like that is not evidence about the amount typed into any one submit. Exactly one submit action in the window still catches the bug, because the double tap is a single action.
|
||||
|
||||
The null guard is not defensive clutter either. Read a balance you could not parse as `0` and the comparison becomes `0 - 0 === typed`, which is false at every healthy submit. A reading you do not have is not evidence, so the property declines to judge. The real spec guards the same way against a balance too large for exact integer arithmetic.
|
||||
|
||||
The values it reads come from extractors, which pull state out of the UI tree once per step:
|
||||
|
||||
```ts
|
||||
const route = extract<string | null>("route", s => {
|
||||
if (s.ax.find({ testTag: "AddTransactionScreen" })) return "add-transaction";
|
||||
if (s.ax.find({ testTag: "HomeScreen" })) return "home";
|
||||
// ...other screens
|
||||
return null;
|
||||
});
|
||||
const SCREENS = {
|
||||
login: "LoginScreen",
|
||||
"add-account": "AddAccountScreen",
|
||||
"add-transaction": "AddTransactionScreen",
|
||||
ledger: "LedgerScreen",
|
||||
home: "HomeScreen",
|
||||
} as const;
|
||||
type Route = keyof typeof SCREENS;
|
||||
|
||||
const totalBalance = extract("totalBalance", s =>
|
||||
s.ax.findAll([{ testTag: "HomeScreen" }, { testTag: "AccountCard" }])
|
||||
.reduce((sum, c) => sum + parseDollarCents(c.find({ testTag: "AccountBalance" })?.text), 0));
|
||||
// The screen this frame shows, or null when it does not show exactly one.
|
||||
// Android's hierarchy dump carries the outgoing and the incoming screen
|
||||
// together on better than one frame in five, and such a frame is a navigation
|
||||
// transition: evidence about neither screen. Ranking the markers and returning
|
||||
// the first one found is how a spec convicts itself on a half-drawn screen.
|
||||
const routeOf = (s: State): Route | null => {
|
||||
let shown: Route | null = null;
|
||||
for (const [name, tag] of Object.entries(SCREENS) as [Route, string][]) {
|
||||
if (!s.ax.find({ testTag: tag })) continue;
|
||||
if (shown !== null) return null;
|
||||
shown = name;
|
||||
}
|
||||
return shown;
|
||||
};
|
||||
const route = extract<Route | null>("route", routeOf);
|
||||
|
||||
// Home's own TOTAL BALANCE node, which the app computes over every account.
|
||||
// Summing the AccountCard balances reads only the cards laid out inside the
|
||||
// viewport, and a card clipped at the bottom edge looks exactly like money
|
||||
// moving. Off Home there is nothing to read, so the last total we did read is
|
||||
// carried forward and `previous` and `current` stay on the same scale.
|
||||
let lastHomeTotal: number | null = null;
|
||||
const totalBalance = extract<number | null>("totalBalance", s => {
|
||||
if (routeOf(s) !== "home") return lastHomeTotal;
|
||||
const total = parseDollarCents(
|
||||
s.ax.find([{ testTag: "HomeScreen" }, { testTag: "TotalBalance" }])?.text);
|
||||
if (total === null) return null; // unreadable is unknown; the carrier keeps its value
|
||||
lastHomeTotal = total;
|
||||
return total;
|
||||
});
|
||||
```
|
||||
|
||||
Every Folio screen and control carries a `testTag`. Compose exposes it as the resource-id on Android and the accessibility identifier on iOS, so one selector resolves on both. `extract` runs against the live tree each step; properties and actions read `.current` and `.previous`, never the raw state.
|
||||
Every Folio screen and control carries a `testTag`. Compose exposes it as the resource-id on Android and the accessibility identifier on iOS, so one selector resolves on both. `extract` runs against the live tree each step; properties and actions read `.current` and `.previous`, never the raw state. An extractor may not read another extractor's handle, which is why `totalBalance` calls `routeOf` rather than `route.current`.
|
||||
|
||||
A second property states what new accounts must look like: a freshly created account starts at zero.
|
||||
A second property states what new accounts must look like: a freshly created account starts at zero. The work is in naming the account it judges.
|
||||
|
||||
```ts
|
||||
const newAccountBalanceIsZero = always(
|
||||
next(() => {
|
||||
const before = new Set((accounts.previous ?? []).map(a => a.name));
|
||||
return accounts.current
|
||||
.filter(a => !before.has(a.name))
|
||||
.every(a => a.balance === 0);
|
||||
if (route.current !== "home") return true;
|
||||
if (!isAddAccountSubmitTap(lastAction.current)) return true;
|
||||
const typed = accountNameField.previous?.text?.trim();
|
||||
const before = accounts.previous ?? null;
|
||||
const after = accounts.current;
|
||||
if (!typed || before === null || after === null) return true;
|
||||
// The only card attributable to a creation is the one named what the fuzzer
|
||||
// typed, on the step its submit landed. Anything else that turned up is a
|
||||
// card that scrolled into view, not an account that came into existence.
|
||||
const matches = after.filter(a => a.name.endsWith(typed));
|
||||
if (matches.length !== 1) return true;
|
||||
const created = matches[0];
|
||||
if (before.some(a => a.name === created.name)) return true;
|
||||
return created.balance === null || created.balance === 0;
|
||||
})
|
||||
);
|
||||
```
|
||||
|
||||
Diffing the two account lists and judging whatever is new is the version that reads better and does not work: Home lists the accounts that fit the viewport, so a card that scrolls in is indistinguishable from an account that was just created. That version convicted this property on android over a Travel account holding $24,112.00.
|
||||
|
||||
A third property, `submitCommitsOneTransactionPerAction`, states the same bug without arithmetic: over any window, no more transactions may be committed than there were submit actions. It needs no amount and no float comparison, so it survives a window of any width, and it does most of the detecting in practice.
|
||||
|
||||
## Reaching the screens that matter
|
||||
|
||||
A fuzzer that pokes at random never logs in, and never reaches a transaction form. The spec gives sanderling enough to drive the real flows, no more.
|
||||
@@ -125,7 +175,7 @@ export const actionsRoot = weighted(
|
||||
);
|
||||
```
|
||||
|
||||
That is the whole input. Two invariants, a way in, and a weighted sense of where to spend time. Nothing here names the bug.
|
||||
That is the whole input. Three invariants, a way in, and a weighted sense of where to spend time. Nothing here names the bug.
|
||||
|
||||
## What the run does
|
||||
|
||||
|
||||
+4
-2
@@ -22,11 +22,15 @@ Run a spec against an app for a fixed duration.
|
||||
| `--ios-device` | optional (ios) | iOS target: a simulator name/UDID to boot, or a connected device's name, UDID, or CoreDevice id. |
|
||||
| `--ios-app-path` | optional (ios) | Path to the `.app` bundle for clear-state reinstall (simulator via `simctl`, device via `devicectl`). |
|
||||
| `--duration` | `5m` | Total test duration (`30s`, `5m`, `2h`, `1d`). |
|
||||
| `--max-steps` | `0` | Stop after this many steps (`0` = no cap, the duration governs). A step budget is what makes two generators comparable. |
|
||||
| `--exit-on-violation` | `false` | Stop the run at the first property violation and exit `2`. |
|
||||
| `--seed` | `0` | PRNG seed. `0` uses a random seed and records it in `meta.json`. |
|
||||
| `--generator` | `seeded` | Who picks each action: `seeded` (the run's PRNG) or `llm` (a vision model). See [the LLM generator](../spec-language/#llm-generator). |
|
||||
| `--output` | `./runs` | Output directory for traces. |
|
||||
| `--clear-data` | `true` | Clear app data before launching so the run starts from a fresh install. Pass `--clear-data=false` to resume prior state. |
|
||||
|
||||
Exit codes: `0` the run finished (violations, if any, are in the summary), `2` the run stopped on a violation under `--exit-on-violation`, `1` something went wrong. CI reads the difference between `2` and `1` to tell a found bug from a broken harness.
|
||||
|
||||
## `sanderling replay [run-or-runs-dir]`
|
||||
|
||||
Serve a local web UI for browsing traces. The positional argument is optional and may point at either a runs directory (the parent of many runs) or a single run directory (auto-detected by the presence of `meta.json`). Defaults to `./runs`.
|
||||
@@ -63,7 +67,5 @@ Print the CLI version.
|
||||
## Flags coming in v0.1.0
|
||||
|
||||
- `--permissions` to pre-set OS-level permissions (for example `--permissions location=allow,notifications=deny`).
|
||||
- `--max-steps` hard cap on step count.
|
||||
- `--exit-on-violation` stop the run on the first property violation.
|
||||
|
||||
Tracked in the [v0.1.0 milestone](https://github.com/priyanshujain/sanderling/milestone/1).
|
||||
@@ -80,6 +80,8 @@ The loop runs until the duration you set elapses. Every step is written to a tra
|
||||
|
||||
Kotlin Multiplatform apps need nothing special: the Android build is tested through the Android driver and the iOS build through the iOS driver.
|
||||
|
||||
Canvas-rendered web apps are the exception. On web, `InputText` types through the DevTools Protocol's `Input.insertText`, which needs a focused DOM element to deliver to, and a bare `<canvas>` is not one: the call produces no events at all and the app sees nothing typed. The driver cannot work around that, because there is no event target. Compose Multiplatform for Web is fine because it keeps a hidden `<input>` IME proxy in its shadow root, a real focusable element that receives the text. A canvas app that exposes no such element is tap-fuzzable but not text-fuzzable.
|
||||
|
||||
## Reading this manual
|
||||
|
||||
- [Case study: Folio](../case-study/) follows sanderling finding a real bug in the example app, and shows how its spec is written.
|
||||
|
||||
Reference in new issue
Block a user