mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
docs(manual): rewrite writing specs as a folio tutorial
This commit is contained in:
pj committed
1 parent
105b4c03b8
commit
d65728eec3
1 file changed
+292
-219
+292
-219
@@ -4,280 +4,353 @@ title: Writing specs
|
||||
|
||||
# Writing specs
|
||||
|
||||
A spec has three parts: extractors, properties, and actions.
|
||||
This page teaches spec writing by building one. The subject is folio, the example app from [getting started](./getting-started/). We start from an empty file and end at the finished spec that ships in the repo at `examples/folio/sanderling/spec.ts`. Each section adds one concept.
|
||||
|
||||
```ts
|
||||
import { extract, always, now, actions, weighted, Tap, taps, swipes } from "@sanderling/spec";
|
||||
## The app
|
||||
|
||||
const loggedIn = extract((s) => !!s.ax.find("id:home-tab-bar"));
|
||||
folio is a small personal ledger:
|
||||
|
||||
export const properties = {
|
||||
cartNeverNegative: always(() => cartCount.current >= 0),
|
||||
};
|
||||
- **Login** asks for email and password. The demo account is `[email protected]` / `ledger123`.
|
||||
- **Home** lists account cards, each with a name and balance.
|
||||
- Tapping a card opens the **Ledger** for that account, listing its transactions.
|
||||
- From the ledger, **Add transaction** opens a form: amount, type (credit or debit), note.
|
||||
- From home, **Add account** opens a form: account name.
|
||||
|
||||
export const actionsRoot = weighted(
|
||||
[10, taps],
|
||||
[2, swipes],
|
||||
);
|
||||
```
|
||||
Every screen and control in folio carries a `testTag`. In Compose this is `Modifier.testTag("LoginScreen")`, which sanderling reads as the resource-id on Android and the accessibility identifier on iOS. Tags are the most reliable way to find elements, and folio's spec uses them throughout:
|
||||
|
||||
The Go runner calls into the JS runtime each step. Extractors re-read the current state. Properties re-evaluate with their residual formulas. The action generator returns a tree, and one leaf is sampled by weight and dispatched.
|
||||
|
||||
## The `State` object
|
||||
|
||||
What extractors receive:
|
||||
|
||||
```ts
|
||||
interface State {
|
||||
ax: AccessibilityTree;
|
||||
snapshots: Record<string, unknown>;
|
||||
lastAction: Action | null;
|
||||
logs: readonly LogEntry[];
|
||||
exceptions: readonly ExceptionRecord[];
|
||||
time: number; // ms since run start
|
||||
}
|
||||
```
|
||||
|
||||
`ax` is the live UI hierarchy. `snapshots` carries any key-value data pushed by the app SDK. `logs` and `exceptions` contain entries collected since the previous step.
|
||||
|
||||
## Extractors
|
||||
|
||||
`extract()` wraps a getter that runs against every new state. The returned object exposes `.current` (this step's value) and `.previous` (last step's, or `undefined` on the first step).
|
||||
|
||||
```ts
|
||||
const loggedIn = extract((s) => !!s.ax.find("id:home-tab-bar"));
|
||||
const balance = extract<number>((s) => s.snapshots["account.balance"] as number ?? 0);
|
||||
|
||||
// Inside a property or action:
|
||||
loggedIn.current // boolean
|
||||
loggedIn.previous // boolean | undefined
|
||||
```
|
||||
|
||||
Extractors are cheap. Prefer one extractor per concept and reuse it across properties and action generators.
|
||||
|
||||
## Finding elements
|
||||
|
||||
`ax.find(selector)` returns the first matching `AccessibilityElement`, or `undefined`. `ax.findAll(selector)` returns all matches. Both are available on the tree root and on any element (scoped to its subtree).
|
||||
|
||||
**String selectors:**
|
||||
|
||||
| Form | Match rule |
|
||||
| Tag | What it marks |
|
||||
|---|---|
|
||||
| `id:<value>` | Exact match on resource-id, or suffix after `:id/` (Android) |
|
||||
| `text:<value>` | Substring match on text content |
|
||||
| `desc:<value>` | Exact match on accessibility description, or starts-with for iOS merged labels |
|
||||
| `descPrefix:<prefix>` | Starts-with on accessibility description |
|
||||
| `<attr>:<value>` | Substring match on any raw attribute by name |
|
||||
| `LoginScreen`, `LoginEmail`, `LoginPassword`, `LoginSubmit` | login screen and its controls |
|
||||
| `HomeScreen`, `AccountCard`, `AccountName`, `AccountBalance`, `AddAccountButton` | home screen |
|
||||
| `AddAccountScreen`, `AccountNameField`, `AddAccountSubmit` | add-account form |
|
||||
| `LedgerScreen`, `AddTransactionButton` | ledger |
|
||||
| `AddTransactionScreen`, `TxnAmountField`, `TxnSubmit` | add-transaction form |
|
||||
|
||||
**Object selectors** (AND of all given attributes):
|
||||
## Start with the defaults
|
||||
|
||||
The smallest spec that does something:
|
||||
|
||||
```ts
|
||||
s.ax.find({ accessibilityText: "LoginScreen" })
|
||||
s.ax.find({ accessibilityText: "login_email" })
|
||||
import { defaultActions } from "@sanderling/spec/defaults";
|
||||
import { noUncaughtExceptions } from "@sanderling/spec/defaults/properties";
|
||||
|
||||
export const properties = { noUncaughtExceptions };
|
||||
export const actionsRoot = defaultActions;
|
||||
```
|
||||
|
||||
**Path queries** (global only):
|
||||
A spec has two exports. `properties` is a map of named rules that must hold at every step. `actionsRoot` is the pool of actions the explorer may take.
|
||||
|
||||
Both come from the standard library here. `noUncaughtExceptions` fails the moment the app throws an exception nobody caught. `defaultActions` is a weighted mix of the built-in generators: random taps, typed text, scrolls, swipes, and double taps on whatever is on screen. There is also `noLogcatErrors`, which fails on error-level logcat lines; it applies on Android and silently holds elsewhere.
|
||||
|
||||
Run this spec and watch what happens. The explorer taps the login fields, types garbage into them, taps Sign in, and gets rejected. It never reaches the rest of the app, because random typing does not produce valid credentials. Already useful (a crash on the login screen would be caught), but the app behind the login stays untested.
|
||||
|
||||
The explorer needs to be taught how to get past the gate.
|
||||
|
||||
## Reading the screen: extractors
|
||||
|
||||
Before the spec can act on the app's state, it has to read it. That is what extractors do.
|
||||
|
||||
```ts
|
||||
s.ax.find("id:HomeScreen > descPrefix:account_card:")
|
||||
import { extract } from "@sanderling/spec";
|
||||
|
||||
const loggedIn = extract("loggedIn", s => s.ax.find({ testTag: "LoginScreen" }) == null);
|
||||
```
|
||||
|
||||
Each segment is matched within the subtree of the previous match.
|
||||
`extract` takes a callback that runs once per step, against that step's state. `s.ax` is the UI tree: every element on screen with its text, position, and state. `find` returns the first element matching a selector, or `undefined`. Here the selector is an object: match any element whose `testTag` is `LoginScreen`. If no such element exists, we are logged in.
|
||||
|
||||
**Cross-platform aliases** are resolved automatically. `label` and `accessibilityLabel` both resolve to `accessibilityText`; `content-desc` and `accessibilityText` are interchangeable; `identifier` and `accessibilityIdentifier` resolve to `resource-id`.
|
||||
The string name is optional but worth setting. Named extractors show up by name in the replay UI, so you can watch their values change step by step.
|
||||
|
||||
See the [Spec language reference](./spec-language/) for the complete selector grammar and per-platform field availability.
|
||||
`extract` returns a handle with two fields:
|
||||
|
||||
## Properties
|
||||
- `loggedIn.current` is the value at this step.
|
||||
- `loggedIn.previous` is the value at the step before, or `undefined` on the first step.
|
||||
|
||||
Properties are named LTL formulas exported from the spec. The verifier evaluates each one every step and fails the run when a formula is violated.
|
||||
One rule to remember: the state argument `s` exists only inside the `extract` callback. Properties and action generators never see the state directly. They read extractor handles. This keeps every read of the screen in one place, visible in replay, and evaluated exactly once per step.
|
||||
|
||||
## Getting past login
|
||||
|
||||
To log in, the spec needs the login controls. Elements are state too, so they come from extractors. Passing an array of selectors matches a path: find `LoginScreen`, then find `LoginEmail` inside it.
|
||||
|
||||
```ts
|
||||
export const properties = {
|
||||
balanceNeverNegative: always(() => balance.current >= 0),
|
||||
loginReachable: eventually(() => loggedIn.current).within(30, "seconds"),
|
||||
};
|
||||
const loginEmailField = extract("loginEmailField", s =>
|
||||
s.ax.find([{ testTag: "LoginScreen" }, { testTag: "LoginEmail" }]));
|
||||
const loginPasswordField = extract("loginPasswordField", s =>
|
||||
s.ax.find([{ testTag: "LoginScreen" }, { testTag: "LoginPassword" }]));
|
||||
const loginSubmit = extract("loginSubmit", s =>
|
||||
s.ax.find([{ testTag: "LoginScreen" }, { testTag: "LoginSubmit" }]));
|
||||
```
|
||||
|
||||
**Operators:**
|
||||
|
||||
- `always(f)` - `f` must hold at every step.
|
||||
- `eventually(f).within(n, unit)` - `f` must hold at some step within `n` milliseconds, seconds, or steps.
|
||||
- `now(f)` - evaluates `f` at the current step (used for implication antecedents).
|
||||
- `next(f)` - evaluates `f` at the next step.
|
||||
|
||||
**Combinators** - available on any formula:
|
||||
Now the action. `actions` wraps a callback that returns a list of actions the explorer may take right now. Returning an empty list means "nothing to do here".
|
||||
|
||||
```ts
|
||||
now(() => loggedIn.current).implies(now(() => cartCount.current !== undefined))
|
||||
formulaA.and(formulaB)
|
||||
formulaA.or(formulaB)
|
||||
formulaA.not()
|
||||
```
|
||||
import { InputText, Tap, actions } from "@sanderling/spec";
|
||||
|
||||
`implies`, `and`, `or`, and `not` compose freely.
|
||||
const DEMO_EMAIL = "[email protected]";
|
||||
const DEMO_PASSWORD = "ledger123";
|
||||
|
||||
## Actions
|
||||
|
||||
Action generators return a list of actions to perform. The runner samples one from the weighted tree and dispatches it through the driver.
|
||||
|
||||
**Built-in generators** (pass directly to `weighted`):
|
||||
|
||||
- `taps` - autonomous random taps on clickable elements.
|
||||
- `swipes` - autonomous random swipe gestures.
|
||||
- `waitOnce` - idles one step.
|
||||
- `pressKeys` - presses a random supported key.
|
||||
|
||||
**Action constructors:**
|
||||
|
||||
```ts
|
||||
Tap({ on: element }) // tap an element or selector string
|
||||
InputText({ into: element, text: "hello" }) // clear and type into a field
|
||||
Swipe({ from: elementOrPoint, to: elementOrPoint, durationMillis?: number })
|
||||
PressKey({ key: "back" | "home" | "enter" | "tab" | "up" | "down" | "left" | "right" })
|
||||
Wait({ durationMillis: number })
|
||||
```
|
||||
|
||||
**Samplers** - cycle over a fixed list:
|
||||
|
||||
```ts
|
||||
const names = from(["Checking", "Savings", "Travel"]);
|
||||
names.generate() // picks from the list
|
||||
```
|
||||
|
||||
**Custom generators:**
|
||||
|
||||
```ts
|
||||
const doLogin = actions(() => {
|
||||
if (loggedIn.current) return [];
|
||||
const emailField = loginEmail.current;
|
||||
const submit = loginSubmit.current;
|
||||
if (!emailField || !submit) return [];
|
||||
return [InputText({ into: emailField, text: "[email protected]" }), Tap({ on: submit })];
|
||||
});
|
||||
```
|
||||
|
||||
**Weighted trees:**
|
||||
|
||||
```ts
|
||||
export const actionsRoot = weighted(
|
||||
[100, dismissOnboarding],
|
||||
[50, doLogin],
|
||||
[10, taps],
|
||||
[2, swipes],
|
||||
[1, weighted(
|
||||
[3, openDeepLink("app://home")],
|
||||
[1, openDeepLink("app://settings")],
|
||||
)],
|
||||
);
|
||||
```
|
||||
|
||||
Weights are relative within each tree. Nested trees get their own local budget.
|
||||
|
||||
## Default properties
|
||||
|
||||
`@sanderling/spec/defaults/properties` exports ready-made properties:
|
||||
|
||||
```ts
|
||||
import { noUncaughtExceptions, noLogcatErrors } from "@sanderling/spec/defaults/properties";
|
||||
|
||||
export const properties = {
|
||||
noUncaughtExceptions, // fails if the app throws an uncaught exception
|
||||
noLogcatErrors, // android-only; reads logcat, no-ops on ios/web
|
||||
};
|
||||
```
|
||||
|
||||
`noLogcatErrors` reads from logcat and only applies on Android. Including it in a spec that targets iOS or web is harmless; it silently holds.
|
||||
|
||||
## Pattern: preconditions
|
||||
|
||||
sanderling has no setup phase. Preconditions are action generators with high weight that self-disable once the condition is satisfied.
|
||||
|
||||
```ts
|
||||
const onLoginScreen = extract((s) => !!s.ax.find("id:login-form"));
|
||||
const loginEmailField = extract((s) => s.ax.find("id:email-field"));
|
||||
const loginSubmit = extract((s) => s.ax.find("id:sign-in-button"));
|
||||
|
||||
const doLogin = actions(() => {
|
||||
if (!onLoginScreen.current) return [];
|
||||
const email = loginEmailField.current;
|
||||
const submit = loginSubmit.current;
|
||||
if (!email || !submit) return [];
|
||||
return [
|
||||
InputText({ into: email, text: "[email protected]" }),
|
||||
Tap({ on: submit }),
|
||||
];
|
||||
});
|
||||
```
|
||||
|
||||
Stack these for onboarding, consent dialogs, and cold-start flows:
|
||||
|
||||
```ts
|
||||
export const actionsRoot = weighted(
|
||||
[100, dismissOnboarding],
|
||||
[50, doLogin],
|
||||
[10, taps],
|
||||
[2, swipes],
|
||||
);
|
||||
```
|
||||
|
||||
Once `onLoginScreen.current` is false, `doLogin` returns `[]` and drops out of the eligible set automatically.
|
||||
|
||||
## Pattern: setup export
|
||||
|
||||
Preconditions that drive the app from a fresh state into the surface you actually want to fuzz (login, onboarding, permission grants, seed data) can be exported as `setup` instead of mixing into `actionsRoot`. The runner tries `setup` first; if it yields no action, it falls through to `actionsRoot`. State regressing back across the precondition (logout under fuzz) automatically re-engages setup.
|
||||
|
||||
```ts
|
||||
const login = actions(() => {
|
||||
if (loggedIn.current) return [];
|
||||
const email = loginEmailField.current;
|
||||
const pwd = loginPasswordField.current;
|
||||
if (email && !email.text) return [InputText({ into: email, text: DEMO_EMAIL })];
|
||||
if (pwd && !pwd.text) return [InputText({ into: pwd, text: DEMO_PASSWORD })];
|
||||
const submit = loginSubmit.current;
|
||||
if (!email || !submit) return [];
|
||||
return [InputText({ into: email, text: "[email protected]" }), Tap({ on: submit })];
|
||||
return submit ? [Tap({ on: submit })] : [];
|
||||
});
|
||||
|
||||
export const setup = login;
|
||||
export const actionsRoot = weighted([60, browse], [40, edit]);
|
||||
```
|
||||
|
||||
`setup` is just an `ActionGenerator`; compose with `actions`, `weighted`, or `whenRoute` exactly like the main pool. Works identically across Android, iOS, and web.
|
||||
The generator reads what is on screen and offers the next sensible step: fill the email if it is empty, then the password, then tap submit. It decides by reading the fields, not by tracking what it did before. Generators must work this way. They are pure functions of the current state, called fresh every step, with no memory of their own. If the app throws the user back to a half-filled login form, this generator picks up from whatever the form actually contains.
|
||||
|
||||
## Pattern: conditional properties
|
||||
The `setup` export gives the generator priority. Each step, the runner asks `setup` first; only when it returns an empty list does the main `actionsRoot` pool get used. Once logged in, `loggedIn.current` is true, `login` yields nothing, and exploration proceeds. If a stray action ever logs the user out mid-run, `loggedIn.current` flips back and `setup` takes over again. No retry logic needed.
|
||||
|
||||
Gate a property so it only applies when a precondition holds:
|
||||
Run the spec now and the explorer logs in within a few steps, then fuzzes everything behind the login.
|
||||
|
||||
## Knowing where you are: routes
|
||||
|
||||
Random taps will wander into the add-account and add-transaction forms, but rarely complete them. To exercise the flows that matter, the spec needs deliberate actions, and those need to know which screen is showing.
|
||||
|
||||
folio tags each screen, so a route extractor is one chain of lookups:
|
||||
|
||||
```ts
|
||||
export const properties = {
|
||||
cartPersistsWhenLoggedIn: always(
|
||||
now(() => loggedIn.current).implies(now(() => cartCount.current !== undefined)),
|
||||
),
|
||||
};
|
||||
const route = extract<string | null>("route", s => {
|
||||
if (s.ax.find({ testTag: "LoginScreen" })) return "login";
|
||||
if (s.ax.find({ testTag: "AddAccountScreen" })) return "add-account";
|
||||
if (s.ax.find({ testTag: "AddTransactionScreen" })) return "add-transaction";
|
||||
if (s.ax.find({ testTag: "LedgerScreen" })) return "ledger";
|
||||
if (s.ax.find({ testTag: "HomeScreen" })) return "home";
|
||||
return null;
|
||||
});
|
||||
```
|
||||
|
||||
## Pattern: step-to-step invariants
|
||||
|
||||
Use `next()` to express invariants that span two consecutive steps:
|
||||
`whenRoute` builds a generator that is only eligible on the given routes. Here is the account-creation flow:
|
||||
|
||||
```ts
|
||||
import { from, weighted, whenRoute } from "@sanderling/spec";
|
||||
|
||||
const addAccountButton = extract("addAccountButton", s =>
|
||||
s.ax.find([{ testTag: "HomeScreen" }, { testTag: "AddAccountButton" }]));
|
||||
const accountNameField = extract("accountNameField", s =>
|
||||
s.ax.find([{ testTag: "AddAccountScreen" }, { testTag: "AccountNameField" }]));
|
||||
const addAccountSubmit = extract("addAccountSubmit", s =>
|
||||
s.ax.find([{ testTag: "AddAccountScreen" }, { testTag: "AddAccountSubmit" }]));
|
||||
|
||||
const accountNames = from(["Checking", "Savings", "Travel", "Emergency Fund", "Investments"]);
|
||||
|
||||
const addAccount = whenRoute(route, ["home", "add-account"], () => {
|
||||
if (route.current === "home") {
|
||||
const btn = addAccountButton.current;
|
||||
return btn ? [Tap({ on: btn })] : [];
|
||||
}
|
||||
const field = accountNameField.current;
|
||||
const submit = addAccountSubmit.current;
|
||||
const opts = [];
|
||||
if (field) opts.push(InputText({ into: field, text: accountNames.generate() }));
|
||||
if (submit) opts.push(Tap({ on: submit }));
|
||||
return opts;
|
||||
});
|
||||
```
|
||||
|
||||
On home it offers one action: tap the add-account button. On the form it offers two: type a name, or tap submit. Everything a generator returns is eligible, and the runner samples one. That sampling is deliberate. Sometimes the explorer types then submits; sometimes it submits an empty form. Both are things real users do, and the second is exactly the kind of input that finds validation bugs.
|
||||
|
||||
`from(...)` is a sampler: `.generate()` picks from the list. For numbers there is `integers().between(lo, hi)`, used the same way in the transaction flow:
|
||||
|
||||
```ts
|
||||
import { integers } from "@sanderling/spec";
|
||||
|
||||
const addTxnButton = extract("addTxnButton", s =>
|
||||
s.ax.find([{ testTag: "LedgerScreen" }, { testTag: "AddTransactionButton" }]));
|
||||
const txnAmountField = extract("txnAmountField", s =>
|
||||
s.ax.find([{ testTag: "AddTransactionScreen" }, { testTag: "TxnAmountField" }]));
|
||||
const txnSubmit = extract("txnSubmit", s =>
|
||||
s.ax.find([{ testTag: "AddTransactionScreen" }, { testTag: "TxnSubmit" }]));
|
||||
const accountCards = extract("accountCards", s =>
|
||||
s.ax.findAll([{ testTag: "HomeScreen" }, { testTag: "AccountCard" }]));
|
||||
|
||||
const amounts = integers().between(1, 500);
|
||||
|
||||
const addTxn = whenRoute(route, ["home", "ledger", "add-transaction"], () => {
|
||||
if (route.current === "home") {
|
||||
const cards = accountCards.current;
|
||||
if (cards.length === 0) return [];
|
||||
return [Tap({ on: from(cards).generate() })];
|
||||
}
|
||||
if (route.current === "ledger") {
|
||||
const btn = addTxnButton.current;
|
||||
return btn ? [Tap({ on: btn })] : [];
|
||||
}
|
||||
const field = txnAmountField.current;
|
||||
const submit = txnSubmit.current;
|
||||
const opts = [];
|
||||
if (field) opts.push(InputText({ into: field, text: String(amounts.generate()) }));
|
||||
if (submit) opts.push(Tap({ on: submit }));
|
||||
return opts;
|
||||
});
|
||||
```
|
||||
|
||||
This one spans three screens: pick an account card on home, tap the add button on the ledger, fill and submit the form. The explorer chains these into complete flows on its own, because after each action the route changes and the generator offers the next link.
|
||||
|
||||
## The first property
|
||||
|
||||
The spec can now drive the app. Time to state what must be true.
|
||||
|
||||
A good property describes an outcome the app guarantees, regardless of how the user got there. folio guarantees that a freshly created account starts at zero. To check it, the spec needs the list of accounts as data:
|
||||
|
||||
```ts
|
||||
interface Account {
|
||||
name: string;
|
||||
balance: number;
|
||||
}
|
||||
|
||||
function parseDollarCents(text: string | undefined): number {
|
||||
if (!text) return 0;
|
||||
const sign = text.startsWith("-") ? -1 : 1;
|
||||
const digits = text.replace(/[^0-9]/g, "");
|
||||
return digits ? sign * parseInt(digits, 10) : 0;
|
||||
}
|
||||
|
||||
const accounts = extract<Account[]>("accounts", s =>
|
||||
s.ax.findAll([{ testTag: "HomeScreen" }, { testTag: "AccountCard" }]).map(card => ({
|
||||
name: card.find({ testTag: "AccountName" })?.text ?? "",
|
||||
balance: parseDollarCents(card.find({ testTag: "AccountBalance" })?.text),
|
||||
})));
|
||||
```
|
||||
|
||||
`findAll` returns every match. Each card element supports `.find` scoped to its own subtree, so name and balance come from inside the card. The balance on screen is text like `"$5.00"`; a plain helper function parses it to cents. Specs are ordinary TypeScript, so helpers like this are fine anywhere.
|
||||
|
||||
Now the property:
|
||||
|
||||
```ts
|
||||
import { always, next } from "@sanderling/spec";
|
||||
|
||||
const newAccountBalanceIsZero = always(
|
||||
next(() => {
|
||||
const prev = accounts.previous ?? [];
|
||||
const curr = accounts.current;
|
||||
if (prev.length === 0 || curr.length === 0) return true;
|
||||
const prevIds = new Set(prev.map((a) => a.id));
|
||||
return curr.filter((a) => !prevIds.has(a.id)).every((a) => a.balance === 0);
|
||||
}),
|
||||
const prevNames = new Set(prev.map(a => a.name));
|
||||
return curr.filter(a => !prevNames.has(a.name)).every(a => a.balance === 0);
|
||||
})
|
||||
);
|
||||
|
||||
export const properties = {
|
||||
noUncaughtExceptions,
|
||||
newAccountBalanceIsZero,
|
||||
};
|
||||
```
|
||||
|
||||
Read it inside out. The callback compares two consecutive steps: any account present now but absent before is new, and every new account must show a zero balance. `next` makes the formula span a step boundary, so `.previous` and `.current` are both defined inside it. `always` applies the check at every step of the run.
|
||||
|
||||
The early return matters. When the explorer navigates away from home, the card list goes empty, and when it comes back, every account looks "new". Without the guard, the property would fire on ordinary navigation. Writing a property is mostly this: stating the rule, then excluding the states where the rule does not apply.
|
||||
|
||||
## The second property
|
||||
|
||||
The deepest flow in folio is adding a transaction, so it deserves the strongest property: submitting a transaction must move the total balance by exactly the typed amount.
|
||||
|
||||
The total balance needs care. Only home shows all account cards, so the sum is only fresh there. On other screens the spec carries the last home value forward, keeping `.previous` and `.current` comparable across screen changes:
|
||||
|
||||
```ts
|
||||
let lastHomeTotal = 0;
|
||||
const totalBalance = extract("totalBalance", s => {
|
||||
const cards = s.ax.findAll([{ testTag: "HomeScreen" }, { testTag: "AccountCard" }]);
|
||||
if (cards.length === 0) return lastHomeTotal;
|
||||
lastHomeTotal = cards.reduce(
|
||||
(sum, c) => sum + parseDollarCents(c.find({ testTag: "AccountBalance" })?.text), 0);
|
||||
return lastHomeTotal;
|
||||
});
|
||||
|
||||
const lastAction = extract("lastAction", s => s.lastAction);
|
||||
```
|
||||
|
||||
`s.lastAction` is the action the explorer performed in the previous step. The property uses it to recognize "the user just tapped Submit":
|
||||
|
||||
```ts
|
||||
function parseTypedAmount(text: string | undefined | null): number {
|
||||
if (!text) return 0;
|
||||
const trimmed = text.trim().replace(/,/g, "").replace(/^[+-]/, "");
|
||||
if (!/^\d+(\.\d{1,2})?$/.test(trimmed)) return 0;
|
||||
const [whole, frac = ""] = trimmed.split(".");
|
||||
return parseInt(whole, 10) * 100 + parseInt((frac + "00").slice(0, 2), 10);
|
||||
}
|
||||
|
||||
const submitMovesBalanceByTypedAmount = always(
|
||||
next(() => {
|
||||
if (route.current !== "home") return true;
|
||||
const action = lastAction.current;
|
||||
if (!action || (action.kind !== "Tap" && action.kind !== "DoubleTap")) return true;
|
||||
if (!JSON.stringify(action.on ?? "").includes("TxnSubmit")) return true;
|
||||
const typed = parseTypedAmount(txnAmountField.previous?.text);
|
||||
if (typed === 0) return true;
|
||||
const moved = Math.abs(totalBalance.current - (totalBalance.previous ?? 0));
|
||||
return moved === typed;
|
||||
})
|
||||
);
|
||||
```
|
||||
|
||||
The first four lines are guards: only check when the last action was a tap on the transaction submit button and the app landed back on home. After the guards, the rule is one line: the balance moved by exactly what was typed.
|
||||
|
||||
This property catches a real bug in folio. The transaction form does not disable its submit button while saving, so a fast double tap lands two transactions. The balance moves by twice the typed amount, `moved === typed` is false, and the violation is recorded with the screenshot and the action that caused it.
|
||||
|
||||
The shipped spec factors these checks into pure functions in `predicates.ts` and unit-tests them separately. The property stays a thin wrapper. Worth copying once your predicates grow past a few lines: a property with a bug in it reports violations that are not there, or misses the ones that are.
|
||||
|
||||
## Weights
|
||||
|
||||
The last piece is the action mix. `weighted` builds a tree of generators; weights are relative within the tree.
|
||||
|
||||
```ts
|
||||
import { weighted } from "@sanderling/spec";
|
||||
import { defaultActions, doubleTaps } from "@sanderling/spec/defaults";
|
||||
|
||||
export const actionsRoot = weighted(
|
||||
[25, addAccount],
|
||||
[45, addTxn],
|
||||
[5, doubleTaps],
|
||||
[25, defaultActions],
|
||||
);
|
||||
```
|
||||
|
||||
Weights declare testing intent:
|
||||
|
||||
- `addTxn` gets the most weight because the transaction chain is the deepest flow and both balance properties watch it.
|
||||
- `addAccount` stays in the mix because `newAccountBalanceIsZero` needs fresh accounts to check.
|
||||
- `doubleTaps` gets explicit weight because rapid double submission is a failure mode these forms must survive. This is the line that flushes out the double-submit bug.
|
||||
- `defaultActions` keeps a quarter of the budget on random exploration, so the explorer still wanders everywhere and types edge-case values into every field. Without it, the spec only tests the flows you thought of, which defeats the point.
|
||||
|
||||
A generator that returns an empty list drops out of the sampling for that step, so weights only ever distribute across what is currently possible.
|
||||
|
||||
## The finished spec
|
||||
|
||||
The complete file is `examples/folio/sanderling/spec.ts`, about 190 lines including comments. Run it:
|
||||
|
||||
```sh
|
||||
cd examples/folio
|
||||
just test # android
|
||||
just test-ios # ios
|
||||
```
|
||||
|
||||
Then `sanderling replay` and press `.` to jump to the first violation.
|
||||
|
||||
## Finding properties for your own app
|
||||
|
||||
Properties follow a few recurring shapes. When staring at a new app, look for:
|
||||
|
||||
- **Conserved quantities.** Money, item counts, scores. State the arithmetic: a transfer leaves the total unchanged, a deletion drops the count by one.
|
||||
- **Initial states.** New account, empty cart, fresh document. State what they must contain.
|
||||
- **Idempotence.** Submitting once and submitting twice must have the same effect. Give `doubleTaps` weight and let the property judge.
|
||||
- **Reachability.** `eventually(() => loggedIn.current).within(30, "seconds")` says login must succeed within bounds. Useful as a smoke check that setup works.
|
||||
- **Never-states.** No uncaught exceptions, no error toasts, no negative balances. Cheap to write, always on.
|
||||
|
||||
Start with the defaults plus one property. A spec with one sharp property and good action coverage beats a spec with ten vague ones.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
**Accessing `state` outside of `extract`.** The `state` argument exists only inside the `extract()` callback. Use extractors and `.current` everywhere else.
|
||||
**Accessing state outside `extract`.** The `s` argument exists only inside the `extract()` callback. Everywhere else, read `.current` and `.previous` on extractor handles.
|
||||
|
||||
**Positional taps.** `Tap({ on: { x: 100, y: 200 } })` breaks on any layout change. Always prefer an `ax.find("id:...")` reference.
|
||||
**Positional taps.** `Tap({ on: { x: 100, y: 200 } })` breaks on any layout change. Find the element instead.
|
||||
|
||||
**Unbounded `eventually`.** Without `.within(...)`, `eventually` never fails within a finite run. Almost always you want a bound.
|
||||
**Unbounded `eventually`.** Without `.within(...)`, `eventually` can never fail within a finite run. Always bound it.
|
||||
|
||||
**`Wait()` inside generators.** Waiting for a condition belongs in an extractor guard, not inside a generator.
|
||||
**Memory in generators.** Generators must derive everything from the current state. Counters, flags, and retry logic inside a generator make its behavior depend on history, which breaks reproducibility and breaks recovery when the app moves unexpectedly.
|
||||
|
||||
**Retry logic inside generators.** Generators must be pure. Given the same state they produce the same actions. Retry is the runner's responsibility.
|
||||
**Properties without guards.** A rule that is only meaningful on one screen must check the route first. Unguarded properties fire on navigation noise and train you to ignore violations.
|
||||
Reference in new issue
Block a user