mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
117 lines
5.4 KiB
Markdown
117 lines
5.4 KiB
Markdown
---
|
|
title: CI
|
|
---
|
|
|
|
# CI
|
|
|
|
`ci.yml` runs on every pull request: it builds, unit-tests, and drives two small
|
|
web fixtures through headless Chrome. It never runs sanderling against a real
|
|
app.
|
|
|
|
Two other workflows do, and both are `workflow_dispatch` only. They boot devices,
|
|
build apps and take minutes, which is not what you want on every push, and
|
|
neither is a merge gate.
|
|
|
|
## folio
|
|
|
|
Actions -> folio -> Run workflow. Inputs pick the legs (`all`, `android`, `ios`,
|
|
`web`), and override the seed, the wall-clock budget, and the step budget; `0`
|
|
means "use the calibrated value in the workflow".
|
|
|
|
Each leg builds `examples/folio` for its platform, builds the CLI with only the
|
|
tags that platform needs (`make sanderling-android` and friends), and runs
|
|
`examples/folio/sanderling/spec.ts` through `.github/scripts/folio-run.sh`. That
|
|
script is plain bash so you can reproduce a job locally:
|
|
|
|
```
|
|
SEED=3 MAX_STEPS=240 .github/scripts/folio-run.sh android
|
|
```
|
|
|
|
**Every leg expects the bug.** Folio double-submits a transaction when the
|
|
submit button is double-tapped, and `submitMovesBalanceByTypedAmount` is the
|
|
property that catches it. The runs pass `--exit-on-violation`, so:
|
|
|
|
| exit | meaning | job |
|
|
|---|---|---|
|
|
| 2 | the run found a violation | green |
|
|
| 0 | the run finished clean | red: the fuzzer stopped finding a bug that is still there |
|
|
| 1 | something went wrong | red: the harness broke |
|
|
|
|
Distinguishing 0 from 1 is the whole point of the exit code: a job that only
|
|
knew "non-zero" could not tell a working fuzzer from a broken emulator.
|
|
|
|
On web the property fires for a structural reason worth knowing when you read a
|
|
witness. `AddTransactionViewModel.submit()` inserts the transaction and pops one
|
|
level, so a submit tap lands on Ledger; reaching Home takes two pops, which takes
|
|
two inserts. A submit tap that lands on Home therefore is the double submit. Over
|
|
451 single taps on the submit button, none reached Home.
|
|
|
|
The witness's arithmetic is noisier than that reasoning. `totalBalance` only
|
|
refreshes on a Home step, so the recorded delta covers every transaction since
|
|
the last Home visit, and only some witnesses read as a clean 2x the typed amount.
|
|
The violation is still real; the number beside it just needs that context.
|
|
|
|
The wasmJs app is served with `Cross-Origin-Opener-Policy` and
|
|
`Cross-Origin-Embedder-Policy` headers, because its sqlite worker needs
|
|
cross-origin isolation. Served without them the app loads a blank canvas and
|
|
every step observes an empty accessibility tree.
|
|
|
|
The seeds in the workflow are calibrated, not guessed. On an M-series mac,
|
|
android seed 3 finds the bug at step 95-116 (seeds 1, 2 and 4 run 120 steps
|
|
clean), ios seed 1 finds it at step 130-134, and web seed 1 finds it at step
|
|
105-109, each reproducing 3 runs out of 3. All three run against a 240-step budget. Keep the
|
|
web seed pinned: 9 of 12 random seeds found the bug within 200 steps, so an
|
|
unpinned one would flake.
|
|
|
|
Repeating the ios leg by hand is not the same as running it in CI: with
|
|
`--clear-data=false` a second local run inherits the first one's accounts, so
|
|
`simctl uninstall` before each repeat or the numbers drift.
|
|
|
|
The ios leg passes `--clear-data=false`, because the job installs a fresh build
|
|
immediately before the run and a freshly installed app is already clear state.
|
|
The in-run reinstall is worth avoiding: `simctl uninstall` + `install` followed
|
|
straight away by the XCTest runner's own launch fails with `app.folio is unknown
|
|
to FrontBoard` maybe half the time. That used to hang the run outright; the
|
|
launch RPC is bounded now, so it fails in about 90 seconds with a real error
|
|
instead, but a failing leg is still a failing leg. The job timeouts are the
|
|
backstop if it happens anyway.
|
|
|
|
Only one sanderling run may drive a given simulator at a time. The driver takes
|
|
an advisory lock on the target's UDID and a second run is refused with the lock
|
|
path in the message, because two runs interleaving app lifecycle leave the first
|
|
run's automation session bound to a bundle the simulator no longer knows.
|
|
|
|
## replay-ui
|
|
|
|
Actions -> replay-ui -> Run workflow. This one is dogfooding: it records a trace
|
|
from `test/browser/testdata/throwing` (violations and uncaught exceptions, so
|
|
every panel has something to render), serves it with `sanderling replay`, and
|
|
fuzzes that UI with `replay-ui/sanderling/spec.ts`.
|
|
|
|
Every property there is a cross-panel agreement - two panels deriving the same
|
|
fact by different paths have to say the same thing - so it holds for any trace
|
|
and needs no recalibrating when the fixture changes. Any violation fails the job.
|
|
|
|
## Reading a failure
|
|
|
|
Both workflows upload their run directories as artifacts, and write the step
|
|
count, seed and violations to the job summary. To replay a failure:
|
|
|
|
```
|
|
gh run download <run-id> -n folio-android
|
|
sanderling replay <the downloaded runs directory>
|
|
```
|
|
|
|
That is the same UI the replay-ui workflow fuzzes. Open the step the summary
|
|
named, and the Violations tab shows the witness: the property, the reason, and
|
|
the extractor values at the step that caused it.
|
|
|
|
## When a device leg flakes
|
|
|
|
Expecting a violation from a single seed is timing-sensitive, most of all on an
|
|
emulator. Calibration reduces that; it does not remove it. If a platform starts
|
|
failing across repeated dispatches with exit 0, do not raise the step budget
|
|
blindly - run a seed sweep with the campaign tool
|
|
(`cmd/internal-tools/campaign`), which exists for exactly this, and pin a seed
|
|
that finds the bug with room to spare.
|