pj/sanderling
1
0 Fork 0
mirror of https://github.com/priyanshujain/sanderling.git synced 2026-10-02 11:07:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

ci(folio): make web an expect-the-bug leg

Browse Source 
the web runtime can observe the double submit now, so the health gate
understates it. seed 1 finds it at step 109, 3 runs out of 3.
This commit is contained in:
pj committed 2026-08-13 22:41:24 +05:30
1 parent 94084d9992
commit 7cd3bc7749
2 files changed
+8 -33
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/scripts/folio-run.sh
+2 -20
View File
@@ -5,10 +5,8 @@
#
# SEED=3 MAX_STEPS=240 .github/scripts/folio-run.sh android
#
# android and ios expect exit 2: folio's double-submit bug is still there, and a
# run that no longer finds it is a regression in the fuzzer, not a pass. web
# expects a clean run that reached the transaction screen; it cannot observe the
# submit bug (see docs/development/ci.md).
# Every platform expects exit 2: folio's double-submit bug is still there, and a
# run that no longer finds it is a regression in the fuzzer, not a pass.
set -uo pipefail
platform="${1:?usage: folio-run.sh android|ios|web}"
@@ -98,22 +96,6 @@ violated=$(grep -ho '"violations":\[[^]]*\]' "$run_dir/trace.jsonl" 2>/dev/null
[ -n "$violated" ] && echo "- $violated"
} >> "$summary"
if [ "$platform" = "web" ]; then
# The web leg is a health gate: the same spec has to log in and drive the app
# as far as the transaction screen. Reaching it is the evidence; the submit
# property cannot fire here.
if [ "$code" -ne 0 ]; then
echo "folio/web: expected a clean run, got exit $code" >&2
exit 1
fi
if ! grep -q '"AddTransactionScreen"' "$run_dir/trace.jsonl"; then
echo "folio/web: the run never reached AddTransactionScreen, so it never got past login" >&2
exit 1
fi
echo "folio/web: clean run over $steps steps, reached the transaction screen"
exit 0
fi
case "$code" in
2) echo "folio/$platform: found the submit bug in $steps steps"; exit 0 ;;
0) echo "folio/$platform: the run finished clean; the double-submit bug was NOT found in $steps steps (seed $seed)" >&2; exit 1 ;;
.github/workflows/folio.yml
+6 -13
View File
@@ -4,18 +4,11 @@ name: folio
# browser, builds the folio app for that platform, and runs
# examples/folio/sanderling/spec.ts against it.
#
# android and ios are expect-the-bug jobs: folio double-submits a transaction on
# a double tap, so the run is supposed to end with exit 2. Exit 0 means the
# fuzzer stopped finding a bug that is still there; exit 1 means the harness
# broke. The two are worth telling apart, which is why --exit-on-violation exits
# 2 and not 1.
#
# web is a health gate instead: the same spec drives the wasmJs build through
# login and into the transaction flow, but it cannot observe the double submit.
# The property keys off state.lastAction, which the web runtime does not report,
# and off the action's selector, which the web picker does not carry (it emits
# coordinates). Both are fixable, neither is a small fix; see
# docs/development/ci.md.
# All three are expect-the-bug jobs: folio double-submits a transaction on a
# double tap, so the run is supposed to end with exit 2. Exit 0 means the fuzzer
# stopped finding a bug that is still there; exit 1 means the harness broke. The
# two are worth telling apart, which is why --exit-on-violation exits 2 and not
# 1.
on:
workflow_dispatch:
@@ -252,7 +245,7 @@ jobs:
run: .github/scripts/folio-run.sh web
env:
SEED: ${{ inputs.seed != '0' && inputs.seed || '1' }}
MAX_STEPS: ${{ inputs.max-steps != '0' && inputs.max-steps || '200' }}
MAX_STEPS: ${{ inputs.max-steps != '0' && inputs.max-steps || '240' }}
DURATION: ${{ inputs.duration }}
- name: Upload the run