mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-04 20:17:09 +00:00
fix(testrun): the refusal asks whether the generator drove, not whether anything did
A dead provider against folio exited 0 on a real emulator: the login setup dispatched three actions before the generator was consulted, so DispatchedActions was 3 and the gate never fired while the generator drove the app zero times across 83 steps. Any spec with a login setup was immune, which is the normal case. Summary counts generator actions separately and the refusal reads that. NoActionsDispatchedError becomes NoGeneratorActionsError, because a run that dispatched three login taps was lying in the old name.
This commit is contained in:
1 parent
5a36b583ea
commit
58168a6a3e
6 files changed
+274
-29
No files matched your search
@@ -25,7 +25,10 @@ complete and useful answer.
|
||||
- **1** means the harness broke. A bad target gives
|
||||
`error: launch app: page load error net::ERR_UNSAFE_PORT` and exit 1, and
|
||||
writes no run directory at all, because the trace is created after the launch
|
||||
succeeds.
|
||||
succeeds. A run that finished also exits 1 when it holds no verdict to report:
|
||||
a spec with no properties, a run no step of which reached the verifier, or one
|
||||
whose action generator never drove the app (section 5). Those do leave a full
|
||||
run directory behind.
|
||||
|
||||
Anything other than 0 and 2 means the run did not complete, and a missing
|
||||
`trace.jsonl` under a 0 or a 2 means there is nothing to judge rather than
|
||||
@@ -182,7 +185,15 @@ seconds a step for a run that is driving something real.
|
||||
**It spent its budget on one action.** Count `next_action` by kind and selector.
|
||||
A run whose actions are one selector explored nothing, whatever its step count.
|
||||
|
||||
None of these change the exit code. All of them change what the run proves,
|
||||
**The generator never drove the app.** A model run whose every call fails still
|
||||
dispatches the actions its spec's setup produced, so a login-fronted spec leaves
|
||||
a trace and a summary that read like a run that acted. This one the run refuses
|
||||
itself: exit 1 and `N step(s) ran and the action generator drove the app in none
|
||||
of them`, followed by the per reason tally of what never reached the app.
|
||||
`llm-calls.jsonl` carries the cause, one record per step, and the trace's
|
||||
`action_skipped` names it on each step that produced nothing.
|
||||
|
||||
None of the others change the exit code. All of them change what the run proves,
|
||||
which is nothing.
|
||||
|
||||
## 6. `skipped_verification`, `transitional`, and the judged count
|
||||
|
||||
Reference in new issue
Block a user