mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
docs(ci): explain the replay-ui judged/declined counts
This commit is contained in:
1 parent
184a96939e
commit
0dd13d0480
1 file changed
+18
@@ -136,6 +136,24 @@ hold for any trace and need no recalibrating when the fixture changes. The
|
|||||||
seventh is the stock `noUncaughtExceptions`, which asks nothing of the panels
|
seventh is the stock `noUncaughtExceptions`, which asks nothing of the panels
|
||||||
and only fails if the UI throws. Any violation fails the job.
|
and only fails if the UI throws. Any violation fails the job.
|
||||||
|
|
||||||
|
So does a run that judged nothing. Exit 0 says no property returned false, which
|
||||||
|
is not the same as any property having been evaluated: each one declines to
|
||||||
|
judge when the elements it reads are absent, so a run where the trace failed to
|
||||||
|
serve, or where the fuzzer sat on the run list, renders nothing and passes.
|
||||||
|
`.github/scripts/replay-ui-summary.sh` reads the trace and puts a per-property
|
||||||
|
count of judged against declined steps in the job summary. Run it by hand with
|
||||||
|
`GITHUB_STEP_SUMMARY=/dev/stdout .github/scripts/replay-ui-summary.sh runs/dogfood`.
|
||||||
|
|
||||||
|
It fails the job when any of the four properties that need nothing beyond the
|
||||||
|
step page having rendered - `selectedStepIsInRange`, `exactlyOneStepIsSelected`,
|
||||||
|
`stepCountMatchesTheList`, `screenshotShowsTheSelectedStep` - judged nothing at
|
||||||
|
all. The other three are reported and not gated, because a zero on them is a
|
||||||
|
seed getting unlucky rather than a broken leg: `switchingTabsKeepsTheStep` needs
|
||||||
|
a tab switch between consecutive steps, and `badgeCountMatchesThePanel` needs the
|
||||||
|
fuzzer to land on a violating step and open the violations tab in that same
|
||||||
|
step. On the first run measured this way (seed 3, 80 steps) that last one judged
|
||||||
|
nothing at all, so the fixture reaches it far too rarely to be worth gating on.
|
||||||
|
|
||||||
## Reading a failure
|
## Reading a failure
|
||||||
|
|
||||||
Both workflows upload their run directories as artifacts, and write the step
|
Both workflows upload their run directories as artifacts, and write the step
|
||||||
|
|||||||
Reference in new issue
Block a user