From 788c68c77787a31e83d1c3acb52908eb4f807192 Mon Sep 17 00:00:00 2001 From: PJ Date: Sat, 15 Aug 2026 18:52:59 +0530 Subject: [PATCH] docs(ci): record what the ios calibration assumes and where it was measured --- docs/development/ci.md | 70 ++++++++++++++++++++++++++++++++---------- 1 file changed, 54 insertions(+), 16 deletions(-) diff --git a/docs/development/ci.md b/docs/development/ci.md index e82641c..4d75b08 100644 --- a/docs/development/ci.md +++ b/docs/development/ci.md @@ -96,27 +96,55 @@ The wasmJs app is served with `Cross-Origin-Opener-Policy` and cross-origin isolation. Served without them the app loads a blank canvas and every step observes an empty accessibility tree. -The seeds are calibrated, not guessed. On an M-series mac, web seed 3 convicts at -step 185-187 and ios seed 7 at step 97-101, each 3 runs out of 3 and each with a -delta of exactly twice the typed amount. Both run a 240-step budget. Keep them -pinned: honest evidence is rare, and across 2261 ios steps only one submit tap -landing on Home had a single-submit window. +The seeds are calibrated, not guessed, and every number here says which host it +was measured on, because the hosts do not agree. On an M3 mac driving iOS 26.1 +simulators, ios seed 7 convicts at step 97-101, 11 runs out of 11 from a cleared +install, each on both properties and each with the balance moving by exactly +twice the typed amount: 199 typed, 39800 cents moved, one account's transaction +count rising by two against a window holding one submit. Web seed 3 convicts at +step 185-187 on that mac and at step 192 on the ubuntu runner. Both legs run a +240-step budget. + +What those numbers assume is a cleared starting state, and that is the only thing +that moved them. Measured four ways on one simulator, seed 7 convicts at step 97 +from a fresh install with clear-state on, at 100 from a fresh install with it +off, and at 97 from a dirty container with it on. It walks 240 steps clean +exactly once: dirty container, clear-state off, where the app opens already +signed in on the previous run's accounts and the walk diverges at step 1. The leg +therefore clears state for itself rather than relying on how it was called. + +Do not read a mac number as a statement about CI, though. Seed 7 ran 240 steps +clean on the macos-15 runner from a state the mac convicts from. The two walks +agree action for action through step 48, where a double-tapped submit lands: the +mac's next snapshot showed Home, the runner's still showed the transaction +screen, and past that they are unrelated random walks. An arbitrary walk convicts +about one run in ten, so this leg is only as pinned as the runner's timing lets +it be. Honest evidence is scarce either way: across 2411 swept steps only 14 +double-tapped Submit at all, and only one of those landed in a window the +counting invariant could judge. Android runs seed 9 over 200 steps: its conviction lands at step 178, so a shorter budget would never see the bonus. A full run costs about five minutes. -Repeating the ios leg by hand is not the same as running it in CI: with -`--clear-data=false` a second local run inherits the first one's accounts, so -`simctl uninstall` before each repeat or the numbers drift. +Repeating the ios leg by hand needs nothing special now, because the run clears +the app's state itself. It used to: `just ios` installs over the top without +uninstalling and folio's signed-in session survives that, so a repeat under the +old `--clear-data=false` opened on the previous run's Home screen and diverged at +step 1. That is how the leg came to look dead while the app and the seed were +both fine, and it is worth recognising: a leg that reports "the double-submit bug +was NOT found" from a machine that has been running the app all day is describing +the machine. -The ios leg passes `--clear-data=false`, because the job installs a fresh build -immediately before the run and a freshly installed app is already clear state. -The in-run reinstall is worth avoiding: `simctl uninstall` + `install` followed -straight away by the XCTest runner's own launch fails with `app.folio is unknown -to FrontBoard` maybe half the time. That used to hang the run outright; the -launch RPC is bounded now, so it fails in about 90 seconds with a real error -instead, but a failing leg is still a failing leg. The job timeouts are the -backstop if it happens anyway. +The ios leg clears state and passes no `--ios-app-path`, which is deliberate: +without an app path the driver wipes the app's data container instead of +reinstalling, and the reinstall is the path that races FrontBoard. `simctl +uninstall` + `install` followed straight away by the XCTest runner's own launch +has failed with `app.folio is unknown to FrontBoard` about half the time on the +host that reported it. That race is untouched and still open; the leg simply +does not take that path. It did not reproduce here at all, in 20 consecutive +reinstall-and-launch cycles on iOS 26.1, 10 of them reinstalling on top of a +live app, so any fix for it has to be developed on a host that can still show it +failing. Only one sanderling run may drive a given simulator at a time. The driver takes an advisory lock on the target's UDID and a second run is refused with the lock @@ -177,3 +205,13 @@ do not raise the step budget blindly - run a seed sweep with the campaign tool (`cmd/internal-tools/campaign`), which exists for exactly this, and pin a seed that finds the bug with room to spare. A leg failing with "a predicate threw" is a different problem entirely and no seed will fix it. + +Sweep in the leg's own configuration, though. The campaign tool and the ios leg +now clear state the same way, so a swept seed means what the leg means, but the +starting frame is not a detail you can skip checking: while the leg still passed +`--clear-data=false`, seed 14 convicted at step 17 in 2 campaign runs out of 2 +and in 0 leg-shaped runs out of 3. Prefer the earliest conviction on offer over +the first one found, too. A run reproduces its trajectory on another host only +for as long as every snapshot agrees, and every step of prefix is another chance +for it not to: seeds convicting at steps 33, 60, 114, 187 and 189 all turned up +within the first 30, so an early one is usually there to be found.