sanderling/docs/development/ci.md

---
title: CI
---

# CI

`ci.yml` runs on every pull request: it builds, unit-tests, and drives three
small web fixtures through headless Chrome (`test/browser/testdata`). It never
runs sanderling against a real app.

Two other workflows do, and both are `workflow_dispatch` only. They boot devices,
build apps and take minutes, which is not what you want on every push, and
neither is a merge gate.

## folio

Actions -> folio -> Run workflow. Inputs pick the legs (`all`, `android`, `ios`,
`web`), and override the seed, the step budget and the wall-clock budget. Seed
and step budget take `0` to mean "use the calibrated value in the workflow"; the
wall-clock budget has no such sentinel and is passed through as written, so
every leg gets whatever you type there.

Each leg builds `examples/folio` for its platform, builds the CLI with only the
tags that platform needs (`make sanderling-android` and friends), and runs
`examples/folio/sanderling/spec.ts` through `.github/scripts/folio-run.sh`. That
script is plain bash so you can reproduce a job locally, with that leg's pinned
numbers:

```
SEED=9 MAX_STEPS=200 .github/scripts/folio-run.sh android
```

**web and ios expect the bug.** Folio double-submits a transaction when the
submit button is double-tapped, and two properties catch it:
`submitMovesBalanceByTypedAmount`, which demands the total balance move by
exactly the amount typed, and `submitCommitsOneTransactionPerAction`, which
demands no more transactions committed over a window than there were submit
actions in it. A double tap is one action committing two transactions, so it
breaks both.

The runs pass `--exit-on-violation`, which exits 2 when the run recorded a
violation and 1 when something went wrong. Telling those apart is the whole
point of the exit code: a job that only knew "non-zero" could not tell a working
fuzzer from a broken emulator.

Exit 2 on its own is not a conviction, though, so the script reads the trace
before it decides:

| what the trace says | job |
|---|---|
| a violation of one of the two properties above, with no `is_error` on its witness | green |
| a violation whose witness carries `is_error` | red: a predicate threw, and a thrown predicate is recorded as a violation like any other |
| a violation of any other property (`newAccountBalanceIsZero` fires on one android seed) | red: a real finding, but not the one this leg gates on |
| no violation and exit 0 | red: the fuzzer stopped finding a bug that is still there |
| no trace at all, on exit 0 or 2 | red: the run recorded nothing, so there is no verdict to read |
| exit 1, or any other code | red: the harness broke, and the code propagates |

The first two rows are why the check is worth the code it takes. A `TypeError`
in `predicates.ts` and a fuzzer that no longer reaches the bug both used to
print "found the submit bug" and exit 0, and they need opposite responses: one
is a spec to fix, the other is a seed to recalibrate. A thrown predicate fails
android as well, where a conviction is otherwise only a bonus, because a spec
that stopped running is not evidence about the app.

The balance property only judges a window holding exactly one submit action.
Without that rule the recorded delta covers every transaction since the last Home
visit, and the property convicts on arithmetic it cannot attribute: an early
version of this gate went green on a witness whose delta was 3.16x the typed
amount. Honest windows are rare, so the counting invariant carries most of the
detection: it needs no amount and survives a wide window.

**android is a health gate**, not a conviction gate, because it convicts in four
runs out of five rather than five. The leg asserts that the run stayed healthy
and reached the transaction screen, and reports the conviction it usually gets as
a bonus. A gate that fails one run in five would be useless here: its failure
message, "the double-submit bug was NOT found", is indistinguishable from the
regression the gate exists to catch.

It used to be two runs in five. The android backend now waits out a route
cross-fade before it snapshots, the way the ios companion and the chrome driver
do, because a dump holding two screens at once makes the runner refuse to act on
it and a quarter of android steps therefore applied no action at all. The number
of those wasted steps varied run to run, so the same seed never walked the same
trajectory. With the wait, four runs of one seed produced byte-identical decision
sequences over 179 steps.

The fifth diverged for the remaining reason: a route can settle before its
content composes, so the tree holds one screen and almost nothing in it, and the
fuzzer acts on a screen that is still filling in. That happened once in about a
thousand steps. Catching it needs structural stability polling on every snapshot,
which costs roughly 2.8s per mutating step and was removed for that reason, so it
is the open item between android and a real conviction gate.

The wasmJs app is served with `Cross-Origin-Opener-Policy` and
`Cross-Origin-Embedder-Policy` headers, because its sqlite worker needs
cross-origin isolation. Served without them the app loads a blank canvas and
every step observes an empty accessibility tree.

The seeds are calibrated, not guessed, and every number here says which host it
was measured on, because the hosts do not agree. On an M3 mac driving iOS 26.1
simulators, ios seed 7 convicts at step 97-101, 11 runs out of 11 from a cleared
install, each on both properties and each with the balance moving by exactly
twice the typed amount: 199 typed, 39800 cents moved, one account's transaction
count rising by two against a window holding one submit. Web seed 3 convicts at
step 185-187 on that mac and at step 192 on the ubuntu runner. Both legs run a
240-step budget.

What those numbers assume is a cleared starting state, and that is the only thing
that moved them. Measured four ways on one simulator, seed 7 convicts at step 97
from a fresh install with clear-state on, at 100 from a fresh install with it
off, and at 97 from a dirty container with it on. It walks 240 steps clean
exactly once: dirty container, clear-state off, where the app opens already
signed in on the previous run's accounts and the walk diverges at step 1. The leg
therefore clears state for itself rather than relying on how it was called.

Do not read a mac number as a statement about CI. **The ios leg does not
currently convict on the runner at all**, and no seed fixes that. Seed 7 and seed
28 were both dispatched against macos-15 and both ran 240 steps clean, from the
state the mac convicts from.

Seed 7 diverges: the two walks agree action for action through step 48, where a
double-tapped submit lands, and there the mac's next snapshot showed Home while
the runner's still showed the transaction screen. Past that they are unrelated
walks.

Seed 28 is the informative one, because it did not diverge. It double-tapped
Submit on the runner at step 32, which is exactly where it convicts on the mac 8
runs out of 8. The counting invariant still could not judge it, and the trace
says why. `submitCommitsOneTransactionPerAction` only evaluates when a Home
reading arrives, and that run went from step 19 to step 136 without once
returning Home:

    step  19  null             -> {Checking: 0}              submits 0
    step 136  {Checking: 0}    -> {Checking: 15}             submits 37
    step 164  {Checking: 15}   -> {Checking: 19}             submits 7
    step 222  {..., Travel: 0} -> {Checking: 25, Travel: 0}  submits 13
    step 239  {Checking: 25}   -> {Checking: 26, Travel: 0}  submits 1

A rise of 15 against a window of 37 is not a violation, and neither is 4 against
7, 6 against 13, or 1 against 1. The property is sound; it needs a window holding
roughly one submit before it can convict, and whether the walk closes the window
soon after a double tap is timing dependent.

So reaching the bug is necessary and not sufficient. Across 2411 swept steps only
14 double-tapped Submit at all, and only one of those landed in a window the
counting invariant could judge. Until the property can attribute a submit without
waiting for Home, treat an ios pass as evidence and an ios failure as unproven.
The transaction rows carry a `LedgerRow` test tag on the ledger screen, which the
walk visits far more often than Home, so a count that does not depend on Home is
available; it needs per-account attribution, since the ledger shows one account
where Home shows all of them.

Android runs seed 9 over 200 steps: its conviction lands at step 178, so a
shorter budget would never see the bonus. A full run costs about five minutes.

Repeating the ios leg by hand needs nothing special now, because the run clears
the app's state itself. It used to: `just ios` installs over the top without
uninstalling and folio's signed-in session survives that, so a repeat under the
old `--clear-data=false` opened on the previous run's Home screen and diverged at
step 1. That is how the leg came to look dead while the app and the seed were
both fine, and it is worth recognising: a leg that reports "the double-submit bug
was NOT found" from a machine that has been running the app all day is describing
the machine.

The ios leg clears state and passes no `--ios-app-path`, which is deliberate:
without an app path the driver wipes the app's data container instead of
reinstalling, and the reinstall is the path that races FrontBoard. `simctl
uninstall` + `install` followed straight away by the XCTest runner's own launch
has failed with `app.folio is unknown to FrontBoard` about half the time on the
host that reported it. That race is untouched and still open; the leg simply
does not take that path. It did not reproduce here at all, in 20 consecutive
reinstall-and-launch cycles on iOS 26.1, 10 of them reinstalling on top of a
live app, so any fix for it has to be developed on a host that can still show it
failing.

Only one sanderling run may drive a given simulator at a time. The driver takes
an advisory lock on the target's UDID and a second run is refused with the lock
path in the message, because two runs interleaving app lifecycle leave the first
run's automation session bound to a bundle the simulator no longer knows.

## replay-ui

Actions -> replay-ui -> Run workflow. This one is dogfooding: it records a trace
from `test/browser/testdata/throwing` (violations and uncaught exceptions, so
every panel has something to render), serves it with `sanderling replay`, and
fuzzes that UI with `replay-ui/sanderling/spec.ts`.

Six of the seven properties there are cross-panel agreements - two panels
deriving the same fact by different paths have to say the same thing - so they
hold for any trace and need no recalibrating when the fixture changes. The
seventh is the stock `noUncaughtExceptions`, which asks nothing of the panels
and only fails if the UI throws. Any violation fails the job.

So does a run that judged nothing. Exit 0 says no property returned false, which
is not the same as any property having been evaluated: each one declines to
judge when the elements it reads are absent, so a run where the trace failed to
serve, or where the fuzzer sat on the run list, renders nothing and passes.
`.github/scripts/replay-ui-summary.sh` reads the trace and puts a per-property
count of judged against declined steps in the job summary. Run it by hand with
`GITHUB_STEP_SUMMARY=/dev/stdout .github/scripts/replay-ui-summary.sh runs/dogfood`.

It fails the job when any of the four properties that need nothing beyond the
step page having rendered - `selectedStepIsInRange`, `exactlyOneStepIsSelected`,
`stepCountMatchesTheList`, `screenshotShowsTheSelectedStep` - judged nothing at
all. The other three are reported and not gated, because a zero on them is a
seed getting unlucky rather than a broken leg: `switchingTabsKeepsTheStep` needs
a tab switch between consecutive steps, and `badgeCountMatchesThePanel` needs the
fuzzer to land on a violating step and open the violations tab in that same
step. On the first run measured this way (seed 3, 80 steps) that last one judged
nothing at all, so the fixture reaches it far too rarely to be worth gating on.

## Reading a failure

Both workflows upload their run directories as artifacts, and write the step
count, seed and violations to the job summary. To replay a failure:

```
gh run download <run-id> -n folio-android
sanderling replay <the downloaded runs directory>
```

That is the same UI the replay-ui workflow fuzzes. Open the step the summary
named, and the Violations tab shows the witness: the property, the reason, and
the extractor values at the step that caused it.

## When a device leg flakes

Expecting a violation from a single seed is timing-sensitive, most of all on an
emulator. Calibration reduces that; it does not remove it. If a platform starts
failing across repeated dispatches with "the double-submit bug was NOT found",
do not raise the step budget blindly - run a seed sweep with the campaign tool
(`cmd/internal-tools/campaign`), which exists for exactly this, and pin a seed
that finds the bug with room to spare. A leg failing with "a predicate threw" is
a different problem entirely and no seed will fix it.

Sweep in the leg's own configuration, though. The campaign tool and the ios leg
now clear state the same way, so a swept seed means what the leg means, but the
starting frame is not a detail you can skip checking: while the leg still passed
`--clear-data=false`, seed 14 convicted at step 17 in 2 campaign runs out of 2
and in 0 leg-shaped runs out of 3. Prefer the earliest conviction on offer over
the first one found, too. A run reproduces its trajectory on another host only
for as long as every snapshot agrees, and every step of prefix is another chance
for it not to: seeds convicting at steps 33, 60, 114, 187 and 189 all turned up
within the first 30, so an early one is usually there to be found.

A short prefix is necessary and not sufficient, though, and ios is the standing
counter-example: seed 28 has the shortest prefix on offer, reproduced its walk on
the runner exactly, reached the bug at step 32, and still did not convict,
because the window the counting invariant had to judge it in was 117 steps wide.
Sweeping selects for a seed that reaches the bug. It cannot select for one whose
walk also closes the window, so when a property needs a window, check what the
window looked like and not only that the conviction happened.