mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
f02a65772b11662d2d4ad8dbbc87cd3b2a561673
59
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 pj
|
6e551e9559 |
fix(spec): a hint selector names the ladder both producers derive
hintText and placeholderValue are the accessible-name ladder, derived from the live element, and compiling them to [placeholder="..."] made them name the wrong field or none at all. A field labelled by an aria-label or a bound <label> carries no placeholder, so it resolved against the dump on the goja host and reached nothing here; one carrying both answered to its placeholder here where the dump answers to its aria-label, which lands a find on an element nobody named. Both keys read the same fieldHint elementHandle and the hierarchy dump (internal/driver/chrome/driver.go) derive the fact with, so a selector cannot name a field this host calls something else. An empty hint names nothing rather than everything that is no field: both producers write the fact only where the ladder answered. placeholder stays the attribute the markup writes, which is what the dump carries under that name too, so a field whose hint is something else still answers to it on both hosts. |
||
|
pj
|
48425e113f |
fix(hierarchy): bounds is a raw attribute, not a cross-platform key
Every native dump writes the rectangle out as a string under bounds, and no DOM element carries an attribute of that name, so the key resolved against the dump and matched nothing on web on every page there is. It is accepted, so no unknown-key error said so, and no mapping can be invented for it: there is no DOM fact to map it to. Off the accepted list the web runtime raises the unknown-key error instead of matching nothing in silence, and the key still resolves wherever a producer writes it, through the escape hatch every other raw attribute already uses: a key some element carries is a key that can match, on both sides. |
||
|
pj
|
43970686d7 |
fix(spec): an editable or scrollable selector names what this host derives
Both facts are derived from the live element rather than written by the markup, and matching them as attributes built [editable="true"], which no page carries. Both resolve against the dump on the goja host, so a spec naming a field or a scroll container that way found it there and no element at all here, with no unknown-key error to say so. Each reads the same function the fact is derived with, so a selector cannot name an element this host calls something else: the handle, the picker's target list and the editable selector all go through isEditable, and scrollable reads the overflow test collectTargets reads. scrollable false names nothing rather than every element that does not scroll: both producers state the fact only where it holds, the way an element that is no field at all answers to neither value of secure. |
||
|
pj
|
c4fbe79016 |
fix(spec): resolve the identifier, label and class aliases against the DOM
identifier, accessibilityIdentifier, accessibilityText and elementType are the names ios writes four facts under, and internal/hierarchy aliases each onto the key the other producers write. This table listed none of them, so each fell through to a raw attribute lookup and built [accessibilityIdentifier="summary_card"], which no element carries. Every one of them resolved against the dump on the goja host and named nothing here. The keys are accepted, so no unknown-key error fires, and a property over the element that was never found passes having checked nothing. |
||
|
pj
|
2cbd60d70b |
test(spec): pin className and class on the same elements
this host answers both names against the live DOM and internal/hierarchy now aliases the second onto the first, so a name dropped from the table here would match nothing on web while the dump still answers it. |
||
|
pj
|
c16cb88d38 |
test(spec): pin text against the key beside it in either order
object keys iterate in insertion order, so the order the author wrote them in decided what a compound selector meant. the innermost rule is pinned over the whole selector's matches: a row whose badge carries the class and the text both is dropped, one whose badge carries the text alone is kept, and a state key is anded before either. |
||
|
pj
|
4768b8048b |
fix(spec): and text with the keys written beside it
a compound object selector dropped text and matched on the other keys
alone, so {testTag: "Row", text: "Alice"} selected every row carrying
the tag where internal/hierarchy selects the one row the author named.
matching more than the spec said is silent: the find lands on a row
nobody wrote and every property over it still passes.
text is answered against the element the way the boolean states are,
since css cannot ask what an element's text says and the xpath that can
cannot ask about the rest, and the innermost rule now holds over what
the whole selector matched, where internal/hierarchy holds it. a
text-only selector still compiles to the same innermost xpath.
|
||
|
pj
|
84baf18ee9 |
fix(spec): keep a selector out of the head subtree
the head renders nothing, so the hierarchy dump drops it and so does the enumeration the picker walks, but a selector still resolved into it: a whole-page findAll answered with <head> and <title> here and with neither on the goja host, which is a divergence the moment a state selector asks a question every element has an answer to. |
||
|
pj
|
5404909e12 |
test(chrome): compare checked, selected and focused across both producers
the target enumeration carries none of the three, so they reach a spec through the ax handle alone, and a selector naming one of them resolves against that same reading. the shadow fixture holds the focused control inside its shadow root, where document.activeElement names the mount element and only a producer that descends finds the field. |
||
|
pj
|
d177726c4e |
fix(spec): keep a tag selector valid beside another key
a multi-key object selector concatenates its parts into one compound,
and a type selector is valid only at the head of one, so {id, tag} built
'[id="amount"]input' and querySelectorAll threw. whether a spec got an
exception or an element depended on the order its author wrote the keys
in.
|
||
|
pj
|
f7135b81d8 |
fix(spec): a boolean state selector names what the live element reports
clickable, enabled, focused, checked and selected are derived from the element rather than written by the markup, so matching them as raw attributes built [clickable="true"] and reached nothing: the keys are accepted, no unknown-key error fires, and the worked example in docs/manual/spec-language.md found no element on web and passed having checked nothing. Each key is answered by the same function elementHandle derives the fact with, since no CSS says what any of them says: :focus names the shadow host of a focused field as well, :checked misses a checked custom element and answers for a selected option besides, and [checked] is the state the page loaded with rather than the one the user left it in. |
||
|
pj
|
e2f9864201 |
fix(spec): keep a secure selector valid beside another key
a multi-key object selector concatenates its parts into one compound,
and a type selector is valid only at the head of one, so {id, secure}
built '[id="pwd"]input[type="password"]' and querySelectorAll threw.
|
||
|
pj
|
9a920444b3 |
test(chrome): compare the secure fact across both producers
it is the fourth fact the dump and the web runtime derive independently, and the one that decides whether a typed value is written into the shared record. three-valued, so the fixture guard requires all three states rather than both polarities. |
||
|
pj
|
b0895ef179 |
fix(spec): a secure selector names the password field on web
secure is derived from the field type, not written by the markup, so matching it as a raw attribute reached nothing: the key is accepted, no unknown-key error fires, and find answered undefined on web for the field it answers with on ios. false is every editable field that is not a password entry, since an element that is no field reports null and answers to neither value. |
||
|
pj
|
454988fbc8 |
fix(trace): an action names the generator that produced it
The setup exclusion landed for the model arm only, because only a model pick stamped a source. A seeded run returned setup's action through the same entry with no marker, so its denominator still counted the login while the model arm's did not, and the two are compared. serializeAction names setup and seeded on the wire, so both arms are counted by one rule. An already-recorded trace names nothing and keeps exactly the count it was reported with; unattributed_actions counts those steps so the old denominator cannot pass as the new one. TraceVersion is deliberately unbumped: oracle-reduction refuses a differing version, and a bump would make all 169 recorded runs unreplayable. |
||
|
pj
|
66fd5bce5d |
fix(runner): a secure field's value does not reach state.lastAction either
folio extracts lastAction, and extractor values are persisted as extractor_changes, so the password still reached the run directory through the spec after the three render sites were closed. The wrap sits in the runner rather than in lastActionFields because the hosts hold the next step's tree, not the one the action was chosen against: a field that stops being secure between the two would publish what the trace withheld. Live and replay now agree byte for byte. |
||
|
pj
|
b1e95739ad |
feat(hierarchy): an element reports whether it masks what is typed into it
ios reads it off SecureTextField, which the companion already sent and nothing read; web reads input[type=password]. Android cannot: the native tree mapper drops the password attribute before the sidecar sees it, so the fact is three-valued and null there rather than a false that would read as "not secure". |
||
|
pj
|
ab4c42601d |
fix(web-runtime): focus follows the caret to the field it types into
Mirrors the driver, so the two hosts agree about focus on a Compose page. The harness inherits custom properties down the parent chain the way CSS does, so an implementation matching the inline style attribute fails. |
||
|
pj
|
04327e63b6 |
test(chrome): the handle and the enumeration agree on editable too
The helper compared clickable alone, so the inherited-contenteditable bug was caught by unit test only and never in a real browser. |
||
|
pj
|
bd01a76d70 |
fix(web-runtime): a handle answers editable for itself, not its container
isContentEditable is inherited, so every span inside a contenteditable div called itself typeable. collectTargets and the chrome dump both require the element itself to match; the handle was the one that did not. |
||
|
pj
|
6706461b91 |
fix(web-runtime): focus descends into the shadow root here too
The Go driver already descends the boundary; the V8 host did not, so the two enumerations disagreed about focus on any shadow-mounted app. The harness now answers activeElement the way a real root does: a root names a node of its own tree, so only the shadow root itself names the field. |
||
|
pj
|
dc2da35e2c | feat(folio-web): predicates for counting commits against submit actions | ||
|
pj
|
6e85cac8b3 |
merge origin/master into llm-recording-and-analysis
both sides independently fixed the same three bugs, so each one had to pick a winner rather than keep both implementations. extractor encoding: master's recordableValue in worker.go wins over ours in marshal.go, since master's is pinned by extractor_encoding_test.go and ours had no tests. our error semantics stay: encodeExtractorValue still returns an error instead of nil, so an extractor cannot vanish from the trace silently. apply errors: only the residual generic branch takes master's unconfirmed copy, where the device may have committed the action before the call failed. the finer branches that know nothing was dispatched keep lastAction = nil, and our actionSkipReason taxonomy stays alongside master's held/skippedVerification. selector matching: our matchAttr with matchSelectorKind wins over master's match, since ours also handles idPrefix. matchSelector now calls it, which git did not flag as a conflict and left calling a function our side had deleted. the ltl doc comment takes master's correction: an unbounded eventually that never fires IS violated at run end. |
||
|
pj
|
944d1e9ad9 |
feat(chrome): read the page's exceptions and navigations, and hold the picker state across them
a page navigation replaces the runtime, so the seeded picker restarted the seed's stream at its first draw on every reload and a trace could not tell a reload from a generator repeating itself. the driver now drains the main-frame navigations it saw, reports the page's buffered uncaught errors so state.exceptions is the page's list on the goja host too, and carries the picker's draw position out of v8 and back in around each decision. |
||
|
pj
|
05c8523762 |
fix(chrome): emit every markup attribute and read checked and selected off the property
the dump emitted a fixed standard attribute set, so a spec reading data-cents or data-account-id saw undefined on the goja host and nothing at all in the trace. it now keys every attribute by the name the markup writes, derived keys overwriting. checked and selected come from the dom property rather than whatever a component left on the object, which is also what the page-side element handle now reports, so a ticked box reads as ticked instead of reporting its starting state forever. |
||
|
pj
|
ca667b1b77 |
fix(selectors): resolve text to the innermost match and scan the root in both forms
an element's text is its whole subtree's text on web and on ios, so every ancestor of a matching element matched too, up to the root. a match a descendant also makes is now dropped, in internal/hierarchy, in the chrome xpath translation and in the page-side web runtime, so all three resolvers name the same element. a raw attribute now matches on a substring (exact for true/false) the way the docs describe, and tree-level FindBySelector considers the root, so ax.find("id:page") and ax.find({id: "page"}) agree.
|
||
|
pj
|
da3cc5f68c |
feat(driver): add escape to the pressKey surface
escape is a key a spec has real use for and no platform could send it. android maps it to KEYCODE_ESCAPE, the ios companion to HID usage 41 and the in-simulator runner to XCUIKeyboardKey.escape, and the Key union accepts it so it can be written at all. |
||
|
pj
|
9fb121e9d0 |
correctness fixes from the first real folio dispatch, and spec authoring skills (#82)
* docs: add the apache 2.0 license text
package.json has declared Apache-2.0 since the first release and .goreleaser.yaml
globs LICENSE* into the archives, so that glob has been matching nothing. npm
only picks up a license from the package directory, hence the copy under
pkg/spec.
* fix(sidecar): close the soft keyboard after typing on android
* test(sidecar): pin the guarded ime dismissal
* fix(sidecar): treat a failed ime probe as no keyboard open
* ci(folio): let the ios leg clear state for itself
* ci(folio): drop the stale frontboard note from the ios job
* docs(ci): record what the ios calibration assumes and where it was measured
* fix(ios): replace the session when a launch blows its bound
a launch the simulator refuses is never reported: xctest records it as a test
failure the runner cannot see, then holds the session's main thread for about
four minutes on a diagnostic chain. so the only signal is the expired bound,
and every later call queues behind the same wedge. restart the session once and
launch again, bounded so the launch path stays inside testrun's backstop.
* test(ios): cover the session replacement a wedged launch needs
* fix(ios): share one deadline across the restart and the second launch
the recovery a blown bound triggers now costs at most launchRecoveryTimeout
whatever it spends it on, so the launch path tops out at 150s and testrun's
three minute backstop stays a backstop.
* test(ios): the restart a blown launch triggers has to be bounded
* docs(ci): the ios leg does not convict on the runner, and a seed cannot fix it
seed 28 reproduced its walk on macos-15 and reached the bug at the step it
convicts at locally. it still could not be judged: the run did not return
home between step 19 and step 136, so the counting invariant saw a rise of
15 against a window of 37 submits.
* docs(sidecar): record why the stale ime flag stays out of reach
* test(folio): add commonTest source sets to core and shared
* fix(folio): reject amounts parseCents cannot represent
* fix(folio): cap a transaction at one million dollars
* test(spec): give the fake dom a real tree and a walking querySelectorAll
* style(sidecar): make ktlint clean, formatting only
ktlint -F over every kotlin file except DriverBackend.kt, then hand
fixes where the reflow read worse and for the long lines ktlint cannot
break. No behaviour changes.
DriverBackend.kt is left untouched to avoid a conflict with concurrent
work; its three over-long lines still fail fmt-kotlin.
* fix(spec): deepQueryAll returns matches in document order
* ci(folio): say what the android gate found, not why
The gate proves only that AddTransactionScreen is absent from the trace.
Claiming the run never got past login was an inference it cannot make: the
run that produced it had logged in and was stuck on the new account screen.
Name the routes the trace does record instead.
* test(runner): a relaunch must not convict the submit counting property
* test(browser): compare ax.find across both hosts on one page
* test(spec): name the shadow match in the grammar both hosts parse
* ci: move every action off the node20 runtime
checkout v4->v7, setup-go v5->v7, setup-node v4->v7, setup-java v4->v5,
upload-artifact v4->v7, cache v4->v6, upload-pages-artifact v3->v5,
deploy-pages v4->v5, setup-chrome v1->v2, setup-android v3->v4,
goreleaser-action v6->v7. setup-bun and android-emulator-runner are
already node24; buf-setup-action stays on its deliberate SHA pin.
setup-chrome v2 resolves stable from Chrome for Testing rather than the
official installer, so the ci.yml comment about the action's default no
longer held.
* feat(verifier): report a relaunch on state.lastAction
* test(verifier): pin the relaunch field on both hosts
* fix(runner): keep the action the app was relaunched after
* test: pin that a nested undefined does not survive the wire
* fix(folio): uninstall before installing in just ios
folio's signed-in session lives in the data container, which an install
over the top keeps, so a local run started right after just ios opened on
the previous run's Home screen and diverged at step 1. On CI's fresh
simulator the uninstall is a no-op, so the ios leg is unchanged.
* style(sidecar): bring the last three lines under the line limit
* fix(ios): the runner must not answer ok for a launch that failed
XCTest records a refused launch as a test issue that never throws, so the
companion returned ok for an app that never started. Check the state the
app actually reached and report the refusal instead.
* test(ios): a refusal the runner names costs no session restart
The session restart is for a launch that never answers. A launch that
reports the app's state has already said what a fresh session would.
* fix(folio): attribute a created account by its whole key, not a suffix
createdAccountHasNonZeroBalance matched the created card with endsWith, so
an older account whose name ends with the typed one ("Emergency Fund" for a
typed "Fund") was judged instead whenever the new card was clipped out of
the reading. Build both keys the card can carry, the plain name and web's
initials + name, and compare them whole.
* fix(hierarchy): object selectors resolve by the same rule as string ones
* test(verifier): both ax.find selector forms resolve the same element
* test(browser): the cross-host fixture uses the object selector form
* fix(replay-ui): wrap the tab strip so its last tabs stay clickable
* test(replay-ui): drive the fuzzer onto a violating step with a panel
* feat(folio): judge a submit against the account's own balance
The counting invariant can only close its window on Home, and the iOS run
in #78 went 117 steps between two Home readings: 37 submits against a rise
of 15 transactions is no evidence about the double tap sitting inside it.
The ledger and the add-transaction screen both show the account's own
balance, and an accepted submit pops back to the ledger, so a window
bounded by those readings holds one action.
The bound is an upper one: a balance that has not moved is a commit still
in flight, a rejected submit or a tap that never landed, and none of those
is a violation. Moving by more than the one submit in the window typed is.
* test(runner): an overlay dismissal must not convict the counting property
* docs(runner): point the guard comment at the renamed test
* docs(replay-ui): name the viewport the tab overflow was measured at
* feat(folio): close the submit window on the account's own screens
submitCommitsOneTransactionPerAction now states its rule over two windows:
the Home counts it already compared, and the account balance the ledger and
the add-transaction screen redraw on nearly every frame of the transaction
flow. Same rule, and the second window is usually one action wide.
* fix(sidecar): reach the adb server the environment names
buildDadb hardcoded localhost:5037, so a serial-addressed device always
resolved through this machine's adb server and ADB_SERVER_SOCKET was
ignored. Read the endpoint the way the adb CLI does instead.
Fixes #79
* test(sidecar): pin the adb server endpoint parsing
* fix(sidecar): close a keyboard standing in the snapshot
A tap on a text field raises the keyboard and nothing closed it, so the
tree the picker chooses from was missing every app node underneath it,
the submit control included. Close it before the read rather than after
the tap: the picker only ever sees snapshots, and the keyboard is still
on its way up when the tap returns.
Fixes #78
* test(sidecar): pin the tree-guarded keyboard dismissal
* chore(make): a target that runs folio's unit tests
* chore(folio): a just recipe for the unit tests
* ci: run folio's unit tests on every pr
* ci: switch to jdk 21 only for the folio step
* docs(sidecar): put the measured read cost in the dismissal bound
* fix(folio): decline the two demanding properties across a relaunch
The runner now keeps lastAction and marks it relaunched: true where it used
to report nothing at all, so the two properties that demand an effect judge
a step whose process may have died before the write landed.
submitChangesBalanceByTypedAmount and createdAccountHasNonZeroBalance both
decline there. The counting bound does not: a relaunch cannot manufacture a
transaction, and the submit is counted, so declining would throw away the
detection the runner fix restored.
* docs(folio): say why the merged card key cannot be made injective
Folio rejects a duplicate account name, so the twin the drop rule guards
against is two names the web key cannot tell apart, not two accounts
sharing a name. State what closing the rest would cost and what the tree
would have to carry to close it properly.
* test(folio): pin that two accounts can render the same card text
The proof behind the comment: "Travel1" holding 25 transactions and
"Travel12" holding 5 merge to the same string, so no identity key read off
a web card can tell them apart.
* fix(sidecar): bound the diagnostic adb reads
adbOutput and readLogcat read to EOF and then waited with no timeout, so
a wedged adb held the step for as long as it liked; one stall over a
remote adb server measured ~100s. The bound has to sit on the read, not
on waitFor: a wedged adb never reaches EOF, so a bounded waitFor after
the read is a line that never runs.
* test(sidecar): pin the bound on a wedged adb read
* fix(sidecar): an unreadable animation count is not idle
Defaulting the count to zero made a dumpsys that said nothing mean
nothing is animating, so a degraded link broke out of the settle early
and handed the runner a frame caught mid-animation. Unknown now waits,
inside the deadline waitForIdle already holds.
* test(sidecar): unknown animation state must not read as idle
* fix(folio): stop spending the submit budget on taps the app refused
The window is an upper bound on the transactions an interval could hold, and
a bound inflated by taps that commit nothing is a bound the app can never
exceed: #78 read a rise of 15 transactions against 37 submits. TxnSubmit is
clickable(enabled = amount.isNotBlank()) and parseCents refuses anything its
regex misses, so a tap whose landing frame shows a refused amount cannot have
committed. Over four recorded android runs that is 19, 11, 25 and 25 of 35,
26, 42 and 42 submit taps.
A relaunch is excepted: a fresh process draws an empty field whatever was
submitted.
* feat(folio): read the amount field into every submit window
Each of the three windows asks whether the tap could have committed, off the
field as the landing frame shows it.
* fix(sidecar): a foreground read that fails degrades the typing guard
An unreadable dumpsys passed a null owner to typeChunks, which switches
the mid-type focus guard off outright and lets the rest of the string
spray into whatever holds the foreground. Fall back to the launched
bundle instead: the guard stays armed, typing still happens, and the
degradation is said out loud rather than assumed away.
* test(sidecar): pin the degraded typing guard both ways
* test(runner): answer Snapshot and Hierarchy off one tree in the fakes
* feat(runner): skip a step whose tree changed between two reads
* test(runner): cover the reread's cost to the existing snapshot rules
* fix(ios): clear app state before the automation session attaches
New performs the clear-state reset, so the uninstall and reinstall no
longer land underneath a live XCTest session that is already bound to
the app. Launch refuses a clear-state request the driver was not built
for rather than reinstalling under its own session.
* fix(ios): the device path clears before its runner session too
* fix(testrun): thread clear-data into the ios drivers
* test(runner): a skipped step must not swallow the action before it
* fix(runner): hold the action back on a step nothing verified
* refactor(runner): drop the empty branch from the hold path
* docs(runner): describe both modes of the composing test driver
* fix(sidecar): erase a field by selecting it, not one delete per character
maestro's eraseText sends one delete per character through its
instrumentation, measured 29.6 ms/char on the API 34 emulator. The
4096-character string the corpus types cost ~121s to clear, a fifth of a
20 minute run spent on one step, and it recurred every time that field
was typed into again.
Select the content and delete the selection instead: two key events at
any length, measured 0.15s to 1.16s for 4096 characters across API 34,
35 and 36. The result is read back off the tree, and a field that is not
empty, or that the tree cannot report on, is finished off per character
in batches rather than assumed clear.
Fixes #80
* test(sidecar): pin the constant-cost erase and its residue check
* fix(sidecar): find the erased field by class, past the keyboard's own focus
The check that decides whether the select-all worked looked for an
"editable" attribute maestro's tree does not carry, so it answered
"cannot tell" every time and every erase paid the per-character
fallback. Worse, an open keyboard puts a second focused node in the
tree, one of the IME's own keys, carrying no text: taking the first
focused node would read a field still holding 4096 characters as empty,
which is the one answer that stops the erase early.
Match the text field by class instead. Measured against the real
backend, 4096 characters now clear in 385ms on API 34, 409ms on API 35
and 870ms on API 36, verified empty, where the fallback took ~4s.
* test(sidecar): use the tree the device really returns
* docs(ci): the android step number describes a local emulator, not ci
the leg disables animations and the number was measured with them on. the
first real dispatch carries 4 transitional steps over 200, so the cross-fade
wait does still fire in ci, just far less often.
* fix(android): say what the sdk lookup checked, not just to set ANDROID_HOME
* fix(doctor): resolve adb and emulator the way a run does
* docs(cli): the android doctor checks are not path-only
* fix(testrun): preflight resolves adb through the sdk, not just PATH
* docs(skills): add a spec review skill and the skills index
* fix(testrun): report a sidecar that dies at startup as the exit it was
* test(testrun): cover the sidecar shutdown path after an early exit
* docs(skills): add a property patterns catalogue skill
* fix(folio): bound the total-balance move instead of demanding it exactly
The write finishes before AddTransactionViewModel navigates, but nothing
establishes that Home's total has re-rendered before the frame is read, and
an equality convicts a healthy app for a total one frame behind. A delta of
zero is exactly the shape nine of the eleven measured android false
convictions had. 2x still exceeds x, so all four recorded convictions
survive, checked against the traces.
The trade is real: a balance that moves by LESS than the amount typed is no
longer judged anywhere in this spec.
* docs(folio): say what property 2 demands now that it is a bound
* docs(skills): add a spec authoring skill
covers hooks, extractors, selectors, properties, actions and the order to write them in, with a complete sample spec that typechecks against the real export surface.
* docs(skills): ground the property patterns catalogue in the merged specs
* docs(skills): name the selector keys that still substring match
* docs(skills): add a setup skill for adopting sanderling
* docs(skills): add a run triage skill
* docs(skills): point the setup skill at its siblings
* docs(manual): correct the flags the cli reference gets wrong
--launcher-activity does not exist in cmd/sanderling/main.go. --device,
--android-app-path and --arm do and were undocumented. runs.md still listed
--max-steps and --exit-on-violation as unshipped, and described --clear-data
as opt-in when the default is already true, contradicting itself ten lines on.
* test(folio): pin that a commit stays in the window until Home reads it
The interaction that keeps a stale Home card list from ever banking counts
the budget has already forgotten: a submit lands on the ledger, so the
reading that resets the window is a whole action later and the submit is
still in it. Characterization, not a regression: no code changed and it
cannot go red first.
* docs(folio): record why a banked card reading can be trusted as current
The freshness rule rests on the app popping one entry back to the ledger,
not on anything the frame carries, so the assumption and the measurements
behind it belong next to it.
* refactor(folio): name the balance property for the bound it asserts
it stopped being an equality and became |delta| <= typed, so the old name
demanded more than the property does. renamed with the ci gate's
GATED_PROPERTIES in the same commit so the gate never sees a name it does
not know.
* fix(android): a refused uninstall must not pass for clear-state
adb uninstall answers Failure [DELETE_FAILED_INTERNAL_ERROR] both when the package was never installed and when it refuses to remove one, so the failure text cannot say which happened and the old code installed over the top either way, keeping the data clear-state was asked to drop. Ask pm path instead, and fall back to pm clear when the app is still there.
* fix(ios): a failed simctl uninstall must fail the reinstall
simctl install over an installed app carries its data container across, so discarding the uninstall error reported a clear-state that never happened. Uninstalling an app that is not installed exits 0 on a booted simulator, so every failure here is a real one.
* fix(ios): a failed devicectl uninstall must fail the reinstall
same hole as the simulator path: devicectl install over an app keeps its data, and the discarded uninstall error hid it. Uninstalling a bundle id that is not installed exits 0 with 'App uninstalled.' on a paired iPhone, so a failure here is always real.
* docs(android): say why the uninstall text cannot be read
* test(android): name the uninstall failure for what it says, not why
* docs(ci): the ios leg convicts on the runner now, and why it did not before
* docs(ci): the cross-fade wait does not fire on ci, say so
* fix(runner): a bounded hold puts the swallow back one step later
the hold carries one action; letting the runner act again while the verifier
is still skipped overwrites it, so the carried action reaches no spec. hold
for as long as the verifier is skipped, and settle on a held step so the
reread pair is not tighter than the window the detector was measured over.
* test(runner): pin what the two reads are compared on
structuralShape excluding text and bounds is the decision separating this
feature from a run that verifies nothing, and only prose held it. adding
either field back now turns a case red.
* fix(ci): close shell injection into the npm publish job
A refname is attacker-controlled and git permits backtick, $, (, ; and |
in it. Three sites substituted it into a run: block, and NODE_AUTH_TOKEN
sat at job level, so a pushed tag ran arbitrary commands with the publish
credential in reach.
The tag now goes through env:, is validated against an anchored version
pattern before anything consumes it, and reaches the other jobs as a job
output. The token is scoped to the publish step. release-npm declares
contents: read instead of inheriting the repo default.
* fix(ios): recognise every shape a blown launch bound arrives in
The runner transport reports a blown budget two ways, its own comment says
so: the context's error once cancellation has landed, and the connection's
i/o timeout when the deadline armed from that context fires first. The
legacy transport reports it as a gRPC status. errors.Is against
context.DeadlineExceeded only matches the first, so the session restart
never fired for the other two and a wedged session stayed wedged.
* test(ios): drive the launch recovery with what the transports return
The wedged-session fake answered with ctx.Err() raw, which is the one
shape the guard already matched. The recovery now runs against the error
each transport really produces for the same expiry, taken from a runner
and a legacy companion that never answer.
* test(runner): pin both guard writes to what the spec reads
deleting lastAction.Relaunched or lastAction.Applied left the whole suite
green, so the only producer of the two fields every spec-side guard reads
had nothing holding it. both now assert the value out of the trace.
* fix(testrun): a run that judged nothing is not a green run
every step skipped means no property ever evaluated, so no violations is the
absence of a verdict rather than a clean one. the hold makes that reachable
now, so the run says it instead of exiting 0.
* fix(ios): stop the app before clearing its state
Launch terminated and then cleared; the clear moved to construction and
left nothing stopping the app first. The container wipe deletes files a
live app still holds open, and the CI ios leg passes no app path so the
wipe is the path it takes. simctl stops it, since the clear now runs
before any automation session exists. On a device the uninstall that is
its only clear takes the running app with it.
* test(ios): pin the stop that has to precede a clear
The ordering probe now records the stop, and a scripted xcrun holds what
reaches the tool: terminate before get_app_container, with the previous
run's files gone after. A simctl terminate that finds nothing to stop
still leaves the clear a success.
* docs(spec): an unbounded eventually is violated at run end
* docs(skills): an unreached eventually convicts at run end
* docs(skills): noUncaughtExceptions only fires on web
* fix(ios): a device clear-state that cannot happen must fail
--clear-data on a physical device with no --ios-app-path warned and then
ran anyway, so the run started on the previous run's data while the flag
said it started clean. There is no data-container wipe on a device, so
there is nothing to fall back to.
* test(ios): a device clear-state without an app path ends the run
* docs(skills): the stock properties each cover one platform
* docs(ci): the balance property demands a bound, not an equality
* fix(ios): the clear-state guard checks the bundle that was cleared
A bool only said that something was cleared, so Launch(ctx, otherBundle,
clearState=true) passed the guard and reported a reset that had reached a
different app. Record what was cleared and compare against the bundle
being launched.
* test(ios): a clear-state launch for an uncleared bundle is refused
* docs(manual): the flagship property is a bound, and say what that costs
* fix(ios): one address picker for every bring-up
bringUpRunner reads the picker from a field, and NewDevice only ever set
the device one, so a device driver that reached bringUpRunner would call
nil. The two fields held the same function; keeping one leaves no path
that can be wired without it.
* test(ios): a device driver can bring a runner up
* docs(skills): both shipped balance forms are bounds now
* docs(skills): name the balance predicate that still exists
* docs(skills): quote the doctor the binary actually prints
* docs(skills): screen= is the chrome driver's url, web only
* docs(skills): substring selector matching is native only
* docs(skills): web selectors are exact, native ones are substrings
* fix(ci): a run that wrote no trace is not evidence about folio
run_dir is empty when the run produced no output directory, and the
fallback made trace ./trace.jsonl. A stray trace in the working directory
was then read as this run's, so a run that wrote nothing reported 'found
the submit bug' and exited 0, defeating the missing-trace check below it.
* fix(ci): fail folio when a gated property is not in the spec
Nothing tied GATED_PROPERTIES to the spec it gates. Renaming a property
left the classifier matching nothing: ios and web blamed the spec for
finding a different bug, and android silently reclassified a real
conviction as 'judging health only' and stayed green.
replay-ui-summary.sh already makes this check for its own list. The spec
path becomes SPEC-overridable the same way, so the check is testable.
* test(ci): cover the folio classifier's verdicts
21 cases through a stubbed sanderling: every exit path, the drift check,
a missing trace, a zero-byte trace, an empty glob and a truncated line.
Asserts the flags that reached the binary, not just the exit code.
Invoked as bash -eo pipefail -c, which is what a run: block does. Running
folio-run.sh itself under -e would kill it at the first non-zero
sanderling test, which is the exit code it exists to read.
* docs(skills): defaultActions bundles five of the eight generators
* test(ios): name the picker test for what it covers
* docs(skills): three of the replay-ui properties are cross-panel
* docs(manual): state.exceptions is web only and reportError does not exist
* fix(folio): the bound carries no unconfirmed-submit guard
deleting confirmedApplied here broke 0 of 355 tests: under a bound a submit
that may not have landed moves the balance by 0, which the bound already
permits, so the guard could only ever drop the double commit it exists to
catch. the relaunch guard stays for a reason the bound does not cover, and
both tests now assert a verdict that changes when their guard does.
* docs(ci): three of the replay-ui properties are cross-panel
* docs(manual): the starter property only fires on web
* test(folio): judge the conjunct on the landings a real run produces
three of the 18 frames the recorded ios run drove it down, each with the
second commit the bound is there to catch. neutering the comparison reddens
it: a second commit on 357900 went unjudged.
* test(folio): the walk drives the composition the spec runs
countSubmitsInWindow never saw an amountText here, so every walk test counted
submits the app must have refused. with the field passed, a refused submit no
longer buys a later double tap an alibi: without it the window reads 3, not 1.
* docs(folio): say which double submit the conjunct can see, and which it cannot
the home landing is the counting invariant's, three of three in the recorded
ios run; this one gets the interleaving whose second pop is cancelled. it is
still the only judge on the 18 ledger landings that run produced.
* docs(folio): the narrow window is not where the detection comes from
the double taps land on home, so the counting form convicts them; what turned
0 convictions into 4 on the recorded ios run is submitCouldCommit, which drops
the windows at those three steps from 5/4/7 to 2/1/2.
* docs(skills): folio drives three platforms from one spec
* fix(folio): an amount over the app's cap spends no window budget
the corpus reaches TxnSubmit with 999999999999999999999, AMOUNT_REGEX takes it
and AddTransactionViewModel refuses it against MAX_TRANSACTION_AMOUNT_CENTS, so
counting it was budget a double submit could hide behind.
* docs(folio): say which form judged one step, not which node was read once
* docs: a bound still needs the relaunch guard, and eventually does convict
* fix(sidecar): the hierarchy rpc serves the tree the snapshot reads
the runner compares the two per step, but snapshot settles and closes a
keyboard while hierarchy was a bare contentDescriptor. measured on emulator
-5556 (api 34) with an ime open: 489 nodes against the snapshot's 134. both
now come off snapshotTree under the same lock; the reread still costs ~75ms
when no keyboard is up.
* docs(runner): say what makes the two reads comparable
the reread's comment claimed the round trip was the only interval between
them; what it left out is that the two rpcs have to read the same way, which
the repo's own android backend did not do.
* refactor(runner): name the settle predicate for what it means
* ci: add a headless-chrome composite action
The setup-chrome / apparmor sysctl / launch-check trio is copied across
three jobs. The old comment described setup-chrome v1 semantics: under v2
stable is the default and the alternative is Chrome for Testing latest,
not a dev Chromium, so it is restated for what the pin actually does.
* ci(examples): add the folio setup actions
folio-app holds the per-platform toolchain and app build, so a caller
guards one step instead of eight. folio-simulator boots the simulator,
installs folio and leaves the app stopped.
* ci(examples): add the replay-ui fixture action
Records a trace and serves it with sanderling replay. The step page URL
is a composite output rather than GITHUB_ENV, so it is scoped to the one
step that drives it.
* ci(examples): one dispatch workflow for every example
folio.yml and replay-ui.yml ran the same operation: build sanderling for
a platform, bring a target up, run a spec against it, classify the trace,
upload the run. They are now one matrix over four examples, each naming
its own runner.
The job is named for what it fuzzes. 'dogfood' named why we run it, not
what runs, the same error as a diagnostic that reports a motivation
instead of an observation.
The matrix is computed by a plan job because jobs.<id>.if cannot read the
matrix context, so a static matrix has no way to leave a leg out. Seeds,
budgets, timeouts, runners and artifact names are unchanged.
* ci: reuse the headless-chrome action in the browser job
Same three steps the examples workflow needs, and the comment explaining
the AppArmor sysctl now lives in one place.
* ci: move the folio jdk step to setup-java v5
The only setup-java left on v4; every other one moved.
* ci: pin third-party actions to commit shas
buf-setup-action was already pinned with a comment saying why; the other
five rode mutable major tags, so a tag move is an unreviewed change to
what runs. Each major currently resolves to the release named in the
comment, so this freezes today's behaviour rather than changing it.
actions/* stay on major tags: they are first-party to the runner.
* ci(replay-ui): name the run directory for what it fuzzes
runs/dogfood and the '### replay-ui dogfood' heading carried the same
naming error as the job name: dogfooding is why the run exists, not what
it fuzzes.
* docs(driver): state the log level scale on LogEntry
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* docs(sidecar): name the device the node counts came off
* fix(chrome): keep a log entry the level scale cannot rank
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* fix(chrome): record console levels on the logcat scale
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* docs(ci): say why upload-pages-artifact needs no include-hidden-files
v3 to v5 crossed v4's change to exclude dot-files. build/site has none,
so nothing was dropped, and the underscore directory is not hidden.
* test(browser): drive a console error through to the spec
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* docs(ioscompanion): name the vacuity behind the empty log slice
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: run the folio classifier's test in make test-ci-scripts
* fix(chrome): keep the message of an object console argument
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(browser): cover console.error with an error object
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* feat(spec): let the runner install state.logs in the page
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* feat(verifier): encode state.logs for the web host
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* feat(chrome): install the step's logs in the page
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(ci): pin three real folio traces from run 31902501859
ios convicted on submitCommitsOneTransactionPerAction, web on both gated
properties, android ran its full 200 steps healthy. Every step is kept;
of each step only step, violations, witnesses and residuals survive.
hierarchy is replaced by the quoted "...Screen" resource ids it held, in
order. It cannot just be dropped: it is 95% of the bytes and also the
only place the android route gate's grep can match, so dropping it flips
that leg from healthy to 'never reached'. 8.4MB to 59KB.
* test(ci): drive the classifier over the real traces
Four cases on real data: each leg's real verdict, plus the android trace
cut before it reached the transaction screen, which is what proves the
route gate reads a real hierarchy dump.
Also corrects the hand-written fixtures. They set is_error to false on a
plain violation; internal/trace/writer.go tags that field omitempty, so a
real trace omits it entirely. Harmless to the classifier, but a fixture
that does not look like reality is the thing that hides drift.
* test(runner): teach the web fakes to take the step's logs
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* fix(runner): install the step's logs before the page extracts
On web every extractor reading is replaced by the one the page computed,
and the page answered logs: [], so noLogcatErrors counted an empty array
however full of errors the console was.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(runner): cover the logs reaching the page and failing to
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(spec): cover the host pushing state.logs into the page
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(browser): drive console.error through to a fired property
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* fix(runner): report a log fetch the driver could not make
The comment claimed the failure was warned about; nothing warned, so a
device whose log fetch failed every step held noLogcatErrors on evidence
nobody collected.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* test(runner): cover the silently dropped log fetch
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* fix(ci): read the route the spec reports, not the hierarchy dump
The android health gate grepped the trace for "AddTransactionScreen",
which occurs in exactly one place: the hierarchy dump, as a resource-id.
That is a debug artifact standing in for a fact the spec already reports,
and it was wrong in both directions. Against the real 8.4MB trace with
hierarchy stripped, the old gate failed a healthy 200-step run; against a
trace carrying the marker on a transition frame without the route ever
being reported, it passed and called it healthy.
It reads extractor_changes.route now, whose values come from SCREENS in
the spec, so the gate and the app agree on what being on a screen means.
routeOf answers null on a frame showing two screens, which is exactly the
frame the marker was matching.
The drift check grows to cover both new names: extract("route") and the
SCREENS key. The fixtures are re-derived keeping the route entry and no
hierarchy at all; the full artifacts and the fixtures give byte-identical
verdicts, which is what proves the coupling is gone.
Script and fixtures move together: either alone leaves the suite red.
* ci: check that the workflow references resolve
actionlint reads a local action's inputs but never checks its path
exists: uses: ./.github/actions/typo lints clean and fails only when the
job runs. Covers composite action paths, make targets including the ones
the examples matrix builds from $SANDERLING, and the scripts a run: step
invokes plus their executable bit.
Fails when it parses fewer references out of a file than that file
mentions, because a checker that matches nothing reports a safety it
never looked for.
* ci: lint the workflows on every pr
The workflows that fuzz the examples are dispatch-only, and GitHub will
not dispatch a workflow that is not on the default branch, so their first
real run is after merge. actionlint and the reference checker are the
only things that can fail before that.
actionlint is pinned by commit, and its tool version is pinned too so a
new release cannot change what CI enforces.
* ci: collapse the four workflows into one
Nine jobs written out one by one, each with its own steps and its own
calibrated seed and budget as literals. Triggers are pull requests, master
and v* tags, and a dispatch with no inputs.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: inline the two composite actions with one caller each
Both existed to give the matrix a per-target hook. folio-app and
headless-chrome stay: three and three callers.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: check that no run: block interpolates an expression
A ${{ }} lands in the script text before bash reads the line, and
actionlint only flags the contexts it already knows are attacker
controlled. Nothing enforced the rule the workflow follows. Also drops the
matrix table lookup, which has no table to read now.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci(folio): name the run, not the fuzzer, in the clean-run message
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* docs: point at the workflow that holds the release secrets now
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: name every job Category (variant), and gate the lot on one check
Follows the convention in antithesishq/bombadil: the display name is what
groups a run in the Actions UI, so Check (tests), Check (browser),
Check (workflows), Folio (android), Folio (ios), Folio (web), Replay UI,
Release and Docs. Every job carries a name, so none of them falls back to
its kebab-case id.
All checks passed needs all nine and runs with if: always(), so branch
protection has one check to point at and a skipped job cannot read as a
pass. Release and docs now gate on startsWith(github.ref, 'refs/tags/v')
alongside master, which is the form the trigger filter already uses.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: split the release job back in two
Collapsing them left the npm publish steps in a job holding contents:
write, because GoReleaser needs it, so npm ci ran its dependency lifecycle
scripts with a write-capable GITHUB_TOKEN in reach of the same job as a
live NPM_TOKEN. Release (npm) is back on contents: read and Release (cli)
keeps contents: write, which is what they each had before.
Each validates the tag from its own copy of the pattern rather than
waiting on a job that exists only to pass a string. Release (cli) is tags
only: there is no CLI to cut on a merge.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: run folio on pull requests
folio was skipped on pull requests, so ios, android and web only ever ran
after a merge. The three legs are 3 to 19 minutes and run in parallel, and
a superseded pull request run already cancels itself.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
* ci: draw each group as its own box in the run graph
The run graph boxes jobs together when they share the same dependencies and
the same dependents. All ten jobs fed only all-checks-passed, so all ten drew
as one pile. A gate per group gives each group a dependent that is exactly
that group.
Release and docs now need the checks, which they should have all along: npm
publish and the pages deploy ran on a merge without waiting for the test job.
Folio stays unblocked so a 20 minute leg does not wait on a 3 minute one.
Claude-Session: https://claude.ai/code/session_01ShuAy8q8ZfPi8KHxwc8JpQ
|
||
|
pj
|
11f72a722a |
follow-ups from the pr #73 review (#77)
* ci(folio): run gradle on jdk 21 for the metro plugin the metro gradle plugin folio builds with publishes org.gradle.jvm.version 21 and java 21 class files, so every leg failed at the folio build on a 17 runtime. local builds pass on jdk 25, which is why only ci saw it. * fix(build): clean pkg/spec/dist, not the dead spec-api path * chore: point stale spec-api comments at pkg/spec * fix(spec): publish src so an installed package carries the runtime entries * fix(testrun): alias the installed spec package so one module graph loads * fix(spec): export Direction, ScrollAction and LongPressAction from the entry * docs(spec): cut the package readme to a description and doc links * docs: say how the cli and spec package versions relate * fix(verifier): report whether the last action was confirmed applied Both hosts get applied: true when the runner saw the dispatch succeed and applied: null when it could not, so an unconfirmed action stops arriving at the spec as no action at all. * fix(runner): an apply error leaves the action's fate unknown, not undone A deadline that fires after the tap was dispatched leaves the effect committed. Reporting nil made the spec see an effect with no action to cause it, which is how the counting property convicts a healthy app. * fix(release): stage the sidecar jar at the renamed embed path * test(replay-ui): trace fixtures for the vacuity counts one real green run, one run that rendered nothing, one that judges every property at least once. * ci(replay-ui): count the steps each property judged the exit code says no property returned false; it does not say any property was ever evaluated. this reads the trace and reports judged vs declined per property, and fails when the step page never rendered. * test(replay-ui): cover the summary script from make test * ci(replay-ui): summarise through the vacuity script * docs(ci): explain the replay-ui judged/declined counts * fix(verifier): encode element-valued extractors into the trace An ax element exports with its find/findAll host functions attached, and json.Marshal refuses the whole value over them: json: unsupported type: func(goja.FunctionCall) goja.Value. The encoding failed, curr stayed nil, and the goja hosts (ios, android) recorded null for every element-valued extractor in both the per-step diff and the violation witness. Apply the web host's sanitize rule before marshaling, so one rule encodes an element on both hosts. * test(verifier): pin element encoding to one rule on both hosts * test(runner): assert an element reaches trace.jsonl and its witness * feat(spec): give state.lastAction an applied field Three states, not two: no action is a null lastAction, applied: true is an action the runner confirmed, applied: null is one it dispatched and never learned the fate of. * fix(folio): do not attribute an effect to an unconfirmed action submitChangesBalanceByTypedAmount and createdAccountHasNonZeroBalance both convict by pinning an effect on the last action, so both decline unless the runner saw it applied. The fixtures now say which fate they mean. * test(folio): an unconfirmed submit belongs in the window The count is an upper bound on the submits a window holds, so the tap that may have landed counts and committedTransactionsExceedSubmits has nothing to convict on. * test(runner): a tap that lands under a failed apply is not a double submit Drives the real folio counting predicates through the runner against a device that commits the tap and then times out. The double-submit case is the control: without it a green proves only that the property never fired. * test(verifier): pin the three lastAction states on both hosts The web page is handed the same applied field the goja object exposes, so a property cannot read one thing on native and another on web. * docs(spec-language): document the three lastAction states |
||
|
pj
|
03af6abe72 |
fix(web-runtime): hold the V8 host to the same naming gate
elementHandle stamped the query selector on every result the same way, so the merge carried the sibling collision onto web for authored ax targets. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX |
||
|
pj
|
7066b23a22 |
fix(web-runtime): answer clickable for an element reached through ax
The handle hardcoded true, so every text node and container a spec reached through state.ax claimed to be a tap target while the enumeration and the hierarchy dump both resolved it through the tappable selector. The parity test now compares the handle against the enumeration element by element in a real browser, which is where the three answers have to agree. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX |
||
|
pj
|
d987526e47 |
Merge origin/master into llm-recording-and-analysis
Brings in #73, which landed web fact-parity work overlapping this branch: selector-tagged ax handles, aria-disabled in `enabled`, shadow-DOM traversal and a `scrollable`/`editable` dump, plus a third folio property and two new testTags on HomeScreen. Six files conflicted. The option-carrying ones took the union of both sides' fields, so `--label-source` and `--exit-on-violation` both reach the pipeline. In web-runtime.ts both sides changed how an element is described: master gave `elementHandle` a selector to tag handles with, this branch gave it raw attribute names and a field hint. Both survive, and `enabled` now answers through master's isEnabled while `editable` stays. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX |
||
|
pj
|
b02e86b2e3 |
ci: dispatch workflows for folio and the replay ui (#73)
* feat(runner): stop the step loop at the first violation on request
* feat(testrun): report violations as a typed error under exit-on-violation
* feat(cli): add --exit-on-violation and exit 2 when it fires
* docs(cli): document --exit-on-violation, --max-steps, and exit codes
* fix(web): enumerate and query across shadow roots in both producers
* test(chrome): compare both producers on a shadow-dom parity page
* test(browser): drive a canvas-under-shadow-root fixture end to end
* fix(web): select the focused field inside a shadow root before typing
* fix(web): report the pathname as the screen when there is no hash route
* fix(web): settle on dom quiescence instead of returning at body ready
* feat(replay-ui): add data-testid hooks the dogfood spec drives
* feat(replay-ui): add the dogfood spec sanderling runs against the replay ui
* fix(replay-ui): scope the screenshot property to the named state panel
* chore(make): add per-platform sanderling build targets
* ci: add dispatch workflows for folio and the replay ui
* docs: describe the dispatch workflows and how to read a failure
* ci(folio): give the ios leg its jdk, android sdk, just, and a clean app start
* refactor(web): use max for the settle budget
* ci: pin calibrated seeds, skip the flaky ios reinstall, bound every job
* ci: authenticate and pin the buf setup step
the anonymous release download hit the shared runner ip rate limit and
failed the job with 'socket hang up' after three retries.
* docs: record that canvas apps need a dom proxy to be text-fuzzable
* fix(ios): bound lifecycle rpcs and claim the target device
a launch the simulator rejects sent the xctest session into a recovery
chain that answered minutes late or never, and the rpc had no deadline,
so the run hung with no trace and no error. also take a per-udid flock:
a second run's reinstall lands under the first's live automation session
and wedges it.
* docs(ci): correct the ios hang wording and note the device lock
* refactor(verifier): derive the lastAction shape from one field list
both hosts must show a spec the same lastAction. one ordered list now
feeds the goja object and the json the web host installs, so they
cannot drift.
* fix(web): install lastAction in the page before extractors read it
state.lastAction was hardcoded null on web, so every property reading it
was silently vacuous: a correct property passed without ever firing.
* fix(web): carry element identity on actions and fix findAll on paths
an action's target was coordinates only, so a property matching on which
element was acted upon could never fire. ax.findAll([a,b]) also returned
nothing on web.
* fix(chrome): wait out a route transition before sampling facts
the tree stays byte-identical and quiet across a cross-fade, so both the
quiet timer and the unchanged-tree escape called it settled mid-flight
and extractors read two screens at once.
* fix: bound the pre-run app launch
launch happens before the runner starts, so --duration never covered it
and a wedged driver hung with no trace and no error.
* fix(folio): read balances from merged cards and treat unreadable as unknown
compose for web merges the whole accountcard subtree, so the balance
child never exists there and every card parsed as 0. the property then
compared 0 to 0 and fired on any submit, which is a false positive
generator. unknown is now null and null is vacuously true.
* test(folio): cover merged-card parsing and unknown balances
* ci(folio): make web an expect-the-bug leg
the web runtime can observe the double submit now, so the health gate
understates it. seed 1 finds it at step 109, 3 runs out of 3.
* docs(ci): explain why a submit tap landing on home is the bug
* fix(ios): read a StaticText's label as its text
AXValue was the only source for text, but a StaticText carries its
string in AXLabel, so nothing on screen had .text on ios: a spec reading
it saw everything on android and nothing here.
* docs(ci): correct the calibrated step ranges
* fix(folio): stop convicting on arithmetic float64 cannot hold
past 2^53 cents the gap between representable values is 128, so a real
1600-cent move reads back as something else and the equality is false
for a healthy submit as readily as a double one. also match parseCents:
a sign or an oversized amount is rejected, not read as an amount.
* test(folio): pin the safe-integer guard and its boundary
* docs: stop teaching the zero-default that caused a false alarm
* docs: write down the silent-vacuity failure modes
* feat(folio): tag the home total and the card transaction count
the total was the only untagged node on the screen, so the spec had to
sum cards and a clipped card broke the sum.
* fix(folio): read the app's own total and refuse contaminated windows
summing cards went null when one was clipped, and the null poisoned the
carrier for the rest of the run. the balance window also spanned every
transaction since the last home visit, so the property convicted on
deltas it could not attribute: the old web witness was 3.16x the typed
amount, not 2x.
* test(folio): pin the window rules and the count invariant
* fix(folio): never read a frame that shows two screens
android dumps a cross-fade with both screens in the tree. the route said
add-transaction while an unscoped find said home, so the oracle took a
half-rendered total as fresh and convicted on a tap that committed
nothing. one function now decides the route and returns null when the
frame is ambiguous.
* test(folio): cover transition frames, card readings and creation
* fix(folio): only disambiguate counts that came from merged text
the equal-length digit rule exists because web merges the card and an
account named -1 makes '12' ambiguous. a dedicated count node has
nothing to disambiguate, so applying it there threw away real evidence.
* ci(folio): pin the recalibrated seeds and drop android to a health gate
web 3 and ios 7 convict 3 runs out of 3 with an exactly 2x witness.
android convicts 2 in 5 because the same seed does not walk the same
trajectory there, so it proves the app runs instead.
* docs(ci): describe the two properties and why android cannot convict
* fix(android): wait out a route cross-fade before snapshotting
the dump could hold two screens at once, and the runner refuses to act
on such a tree, so a quarter of android steps applied no action and the
count varied per run: the same seed never walked the same trajectory.
the ios companion and the chrome driver already do this.
* ci(folio): let the android leg run far enough to see its conviction
* docs: only the repo owner merges
* ci(folio): a thrown predicate is not a conviction
exit 2 means the run recorded a violation, and a predicate that throws
is recorded as one too. so was newAccountBalanceIsZero, an unrelated
property in the same spec. the gate read the exit code and went green
with detection dead.
* ci: install idb-companion from its tap and stop interpolating inputs
idb-companion is not in homebrew-core, so the ios leg died before it
built anything. replay-ui expanded dispatch inputs into the shell.
* docs: correct the snippets and numbers that drifted from the code
* test(sidecar): pin that a slow read counts toward the stability streak
* fix(web): read the page's extractors only on steps that count
the page advances the spec's carriers when it evaluates, but the runner
applied the result only on non-transitional steps. a discarded step
moved the window forward anyway, so the next accepted pair bracketed two
transactions while counting one submit, and convicted a healthy app.
extractor errors now fail the run instead of leaving goja's values in
current against v8's in previous.
* fix(chrome): anchor the transition deadline when the dom goes quiet
it was anchored at script start, so a page that churned past the window
reached the check already expired and returned mid cross-fade. the
driver now publishes the idle timeout it needs, since the caller's 1s
could never spend the 800ms window.
* fix(web): fail on a partial extractor override
same mixed-producer hazard as the install error: some extractors hold
the page's value and the rest hold goja's, and a property comparing
across that split fires on a healthy app.
* docs: six of seven, the seventh is the stock property
* fix(folio): drop a name two cards answer to
homeTxnCountsOf keyed on the account name and let the last card win, so
two accounts the fuzzer named the same collapsed into one entry. a
reading that saw one Travel card and a later one that saw both then
subtracted two different accounts' counts, and
submitCommitsOneTransactionPerAction convicted a healthy app of
double-submitting. it is a gated property in folio-run.sh, so that reads
as "found the submit bug" over a card scrolling into view.
same rule createdAccountHasNonZeroBalance already applies: a name
nothing can attribute is no evidence. counted over every card, since an
unreadable twin spoils the identity too.
* perf(folio): read each frame once
every extractor asked routeOf, and routeOf does five ax.find calls. on
web each find walks the document and every shadow root beneath it, so
the spec cost 110 tree walks a step; homeCards was parsed four times
over. now 5 and once.
keyed on the identity of the state object because both hosts build a new
one per step and hand that one object to every getter, so it cannot
outlive its frame. holding the reference is what keeps that true rather
than likely.
* fix(web): keep an undefined reading's index through JSON
json has no undefined, so an extractor whose getter returned one had its
whole index dropped by JSON.stringify. that index then kept goja's
dump-derived value while its neighbours held the page's, and a property
comparing previous to current across the split fires on a healthy app.
folio has nine on(route, tag) extractors, so this was most extractors on
most steps.
each reading is wrapped in a {value} envelope: the drop now happens
inside the entry, and an absent value means the getter returned
undefined, which is what the goja host records for the same getter. a
json null would instead claim it returned null and x.current ===
undefined would answer differently on the two hosts.
* feat(verifier): report the registered extractor count
the web path needs it to check the page sent one reading per extractor.
* fix(runner): fail when the page reports fewer readings than extractors
the comment here already claimed a partial override was fatal. it was
not: the skipped check only catches indices outside the extractor list,
so a page reporting values for some extractors and not others left the
rest holding goja's reading of the dump with nothing said.
* test(browser): drive an undefined reading through the whole web path
four layers carry it: the page's envelope, the driver's unwrap, the
runner's count check and the verifier's decode. each has a unit test and
only a run proves they compose. goes red both ways, decoding an absent
value as null and dropping the envelope.
* fix(web): offer the aria roles a user activates
only role=button was in the tappable set, so link, checkbox, radio,
switch, tab, option, the menuitems and treeitem were invisible to the
enumeration however plain the control looked. the replay ui builds its
step rows as <li role="option">, and the spec dogfooding it had to
hand-write an action to reach them because no default verb could see a
single row.
both producers build the set from the same role list, since the parity
test compares them element by element.
* test(browser): tap a role-based control end to end
every control on the page is an <li role="option">, the shape the
replay ui gives its step rows, and the spec carries no action of its
own: the property firing is the evidence the default enumeration offered
a tap on one.
* fix(web): read aria-disabled as disabled
the enabled fact came off the disabled property, which only real form
controls have. it reads undefined on the role-based controls the
tappable set now covers, so every one of them looked enabled however
plainly it was marked otherwise, and the fuzzer would spend actions on
inert ones.
both producers answer the same two ways, and the parity fixture carries
a disabled row so the comparison covers it: reverting one side alone
names the element and the fact.
* docs(replay-ui): the enumeration reaches step rows now
the comment said role="option" is not in the tappable selector set,
which stopped being true a few commits ago. selectAStep stays, for the
reason the tab weight below it stays: one row among the page's clickable
elements is a thin chance, and both step-facing properties go vacuous on
a run that never selects one.
* test(runner): bound the last-action test by steps, not wall clock
100ms of wall clock against an assertion that two steps ran fatals under
load with "the web path never installed it", which reads as a
regression. every sibling test in the package uses a long duration and
MaxSteps.
* ci: run the kotlin tests in make test
RouteTransitionTest and the stability poll cover the android settle and
nothing in ci ran them. :sidecar:test needs no android sdk, checked by
running it with ANDROID_HOME pointed at nothing.
* fix(sidecar): measure the stability streak as observed quiet
parameterising pollUntilStable also moved the clock to the start of the
read that opened a run of identical snapshots, so a read's own duration
counted as quiet. the pre-existing caller polls a real uiautomator dump:
at 400ms a read, 750ms of required quiet became 250ms of observed quiet
and the poll settled in two reads instead of four.
the parameters stay, the semantics go back.
* test(sidecar): pin the transition cap by driving it
it asserted 1500 >= 700 + 300, two constants, which can only fail if
someone edits a constant. it now drives awaitSettledTree against a fade
that lands after 700ms and asserts it hands back the settled tree before
the cap. cut the cap to 1000 and it goes red.
* ci: pin buf-setup-action to a commit
it takes a token now, so a floating tag is a token handed to whatever
that tag moves to. note v1 there is a branch, not a tag, so the ref
lookup that resolves it is matching-refs/heads/v1.
* ci: declare least-privilege permissions
none of the three declared any, so each got the repository default.
release.yml and docs.yml already do this. all three only check out,
build, test and upload artifacts.
* ci: fail fast when a server never comes up
the readiness loops fell through silently after 30 tries, so a server
that never started surfaced as an opaque driver failure minutes later.
each now says what did not answer and on which port.
* ci(folio): a missing trace is not a verdict
with no trace the android gate ran its grep against ./trace.jsonl and
reported "never reached AddTransactionScreen, so it never got past
login", which is not what happened. the web and ios branches had the
same misdiagnosis on exit 0.
same class, one line up: the classifier's own failure was swallowed, so
with the evidence reader dead the gate printed a healthy run and exited
0.
* ci(replay-ui): skip a run directory with no trace
the summarise step is if: always(), and under github's bash -eo pipefail
an unmatched glob stays literal, the redirect fails, pipefail carries it
into the assignment and -e kills the step. so a failed fuzz run went red
twice, once for the real reason.
|
||
|
pj
|
349a0644ac |
test(spec): guard the step unit on the authoring surface
Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX |
||
|
pj
|
037a803f0e |
fix(spec): key web attrs by the names the markup writes
attrs was spread from element.dataset, whose DOMStringMap keys are camelCase, so a spec reading attrs["data-cents"] the way every native host reports it read undefined. In folio-web that left ledgerTxnCount and ledgerBalance permanently zero: someTransactionExists could never be satisfied, balanceMatchesTransaction Delta could never fire, and totalBalanceMatchesAccounts compared 0 to 0 and passed vacuously. Three properties reported nothing because the harness was blind, not because the application was correct. The handle also fills hintText and editable now, so an authored InputText on web names its field the way the same action names it on Android instead of rendering as Type "12.34" into "". Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX |
||
|
pj
|
0a71e05588 |
feat(spec): refuse a multi-value generator while enumerating
integers, strings, emails and edgeCaseText read the same rng from() does, so under the model policy an authored InputText typed the same value on every step while the seeded arm varied it. That is a silently different experiment, not just a silently different action space. Single-valued spans are exempt, because both policies then get the same value: between(7,7), a zero-length string, and a one-entry corpus. length(4,4) is still refused, since the length is pinned but each character is drawn from 62. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX |
||
|
pj
|
c76ba4b497 |
feat(spec): refuse a multi-item authored sampler while enumerating
from().generate() draws from the picker's rng, which exists only inside walkActions. The model policy enumerates authored leaves outside that walk, so the sampler silently yielded its first item on every step: measured over 30 draws the seeded arm reached three targets in roughly equal proportion and the model was offered only the first. The two policies had different action spaces and nothing said so. A single-item sampler short-circuits before the rng, so both policies get the same value and it is not refused. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX |
||
|
pj
|
8a98915118 | test(chrome): drive the live-page parity test through both selector forms | ||
|
pj
|
cb880edf44 |
fix(spec): match a merged label by its leading name on web too
The native desc rule accepts the label or the label at the head of an iOS merged label; both web translators compared the whole string, so the same selector matched natively and missed on web. The live-page parity test caught it. |
||
|
pj
|
b8bb44cf61 |
test(spec): pin the unknown-key diagnostic to one text
The two runtimes each claimed to raise the other's message and nothing checked it. Both now render the committed text for the committed key. |
||
|
pj
|
269574706d |
feat(spec): reject an unknown object-selector key in the web runtime
Same rule and the same message as the native side: a key no element can carry throws instead of matching nothing. The accepted list is one list, committed as a fixture both suites assert, so a spec cannot be accepted by one runtime and rejected by the other. |
||
|
pj
|
c77b78d63b |
test(chrome): compare both selector matchers over one live page
Selector matching is written once per runtime: internal/hierarchy over the dump, web-runtime.ts over the DOM. Nothing made the two agree, and a selector that resolves on one and not the other is silent, since an empty match yields no action and the run still passes. |
||
|
pj
|
bf3f230b1b |
fix(spec): read the injected seed per call
Binding it at module scope bound it to whenever the module was first imported, so a test file that imported the runtime before setting SANDERLING_SEED froze the seed at zero for every file after it. The bundler still replaces the expression with a literal. |
||
|
pj
|
a8444cc34f |
feat(spec): match idPrefix in the web runtime
The DOM has no package prefix, so the native rule reduces to [id^=]. Both prefix kinds now go through the one key table, which drops the separate descPrefix branch that string and object selectors each carried. |
||
|
pj
|
5ad2b39e6f |
fix(spec): carry the container on an authored scroll
serializeAction sent the container's own point as both endpoints, so an
authored Scroll({in, direction}) reached the driver as a drag from a point to
itself and did nothing, on the seeded arm. The wire now carries the selector
and leaves the drag to the runner, which sizes it from the container's bounds
and has always had tested support for it that nothing could produce.
No rng runs in the serializer, which lowers an already-drawn action, so the
draw stream does not move. Builtin scrolls compute both endpoints and their
bytes are unchanged.
Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX
|
||
|
pj
|
76dce1a75e |
experiment instrumentation: step budgets, arm labels, campaign runner (#72)
* feat(cli): add --max-steps for step-bounded runs runner.Options.MaxSteps already worked but was unreachable from the command line. A step budget is what makes two generators comparable: one making a model call per step and one drawing from a PRNG are not comparable per second. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * feat(trace): record arm membership and host in meta.json meta.json recorded the seed but not which picker ran, how it was configured, what budget it was given, or which machine produced it. A directory of runs cannot be attributed to an experiment cell without those, which makes any factorial computed from such a directory unanalysable after the fact. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * feat(cli): add --arm and populate run meta from it Model and instructions are recorded only when the LLM picker is the one that will actually run, so a spec declaring generator = llm() that is run under the seeded picker does not label its trace with a model it never called. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * feat(campaign): sweep seeds for one experiment cell campaign.json lists the seeds a sweep intended to run and is written before the first run, so a host that dropped runs shows up as missing seeds rather than as a smaller sample. Seed 0 is rejected: sanderling test reads it as "derive a seed from the clock", which is why conformance/gates.sh controls nothing today. Each run contributes one runs.jsonl line carrying steps to first violation by origin step, the step that armed the failed obligation, so the survival analysis never reopens a trace. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(runner): no silent generator fallback, and llm on web --generator llm against a spec declaring no generator = llm(...) logged a warning and ran the seeded picker. For a comparison campaign that is silent arm corruption: the run completes, the directory looks correct, and the wrong policy drove it. It is now fatal. pickSources also returned the V8 source for both action and extractor on web before it looked at the generator, so the llm policy was unreachable there. The two axes are now independent: the driver picks the extractor source, the flag picks the action source, and llmSource composes with either because the runner populates the candidate list and screenshot on every platform. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(chrome): make the hierarchy dump agree with the web runtime Three facts differed between the dump the goja host reads and the DOM the V8 host reads, so the two enumerated different candidates on one page. scrollable was never emitted, and worker.go reads exactly that attribute while targets.ts requires it for scrolls, so the goja host could not offer a single web scroll. clickable tested el.onclick, which React assigns to its root container for event delegation, making the whole viewport a tap target here and in no other enumeration. Both now resolve through the selector sets in pkg/spec/src/web-runtime.ts. The dump also rooted at body while collectTargets walks querySelectorAll("*"), so the goja host never saw html, where page-level scrolling lives. It now roots at documentElement and skips the head subtree, which is all zero-bounds and would otherwise carry script and title text into the trace. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(conformance): give the gate reproducible seeds SEED defaulted to 0 and sanderling test reads --seed 0 as "derive a seed from the clock", so the tunable controlled nothing and a gate failure could not be re-run. SEEDS now takes one explicit non-zero seed per run, recorded in the results table so a failing row names its stream. The five runs stay on five different streams: a gate that scored one path five times would catch less than one that scores five. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(chrome): emit editable as a plain boolean editable was emitted as `isEditable || null`, and an absent field sends internal/hierarchy into the native fallback, which reads any class name containing "EditText" as an Android text widget. On web that is just a CSS class, so a page styling a div with it was editable to the goja host and not to the web runtime, and the model policy could be offered typing into a div. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(spec): leave the head subtree out of the web target walk collectTargets walked querySelectorAll("*") while the hierarchy dump skips head, so the two hosts enumerated different element sets on every page with a <head>. No candidate changes: builtinCandidates pushes only for targets acceptsTarget admits, and head elements have no positive bounds, so the list the draw ranges over is untouched. What changes is that targetIndex now means the same thing on both hosts. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * test(chrome): compare the facts both hosts derive from one DOM The existing parity harness hand-authors the facts on both sides, so it proves that given identical facts both hosts select identical candidates, and says nothing about the two code paths that derive those facts from a real page. Four divergences lived in that blind spot and it passed throughout. This drives one real page and compares clickable, enabled, editable, scrollable and positiveBounds element by element, plus the element sets themselves, which is what catches a host that omits html or includes head. Reverting any of the four fixes makes it fail naming the element and the fact. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * chore(make): run the browser packages one at a time Both launch Chrome and launching two at once has failed with "Launch: context canceled". Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * style: remove every em-dash and en-dash Eighteen occurrences across fourteen files. Each sentence was repunctuated to suit what the dash was doing rather than swapped for a hyphen, which produces comma splices. The minus sign in folio-web's ledger is a minus sign and stays. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(chrome): honor the caller context in Launch Launch and clearState ran against d.tabCtx, so a target that accepts the connection and never answers wedged the process past its own --duration and through SIGTERM, needing SIGKILL. Unattended that is a campaign worker lost for the rest of the sweep with no diagnostic. The browser is still allocated against d.tabCtx first, because chromedp starts Chrome under whichever context calls Run first and allocating under a caller deadline would kill the browser when Launch returns. Everything after allocation goes through runCtx. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * fix(sidecarassets): publish the extracted jar through a rename Extract wrote a 96 MB jar with a plain WriteFile into a temp path every sanderling process on the host shares. On a cold host several concurrent workers all miss the checksum and all write the same path, and O_TRUNC lets one spawn a JVM against another's half-written archive. A fresh experiment host is exactly a cold host. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * feat(campaign): kill a run that outlives --run-timeout A wedged run holds its worker for the rest of the sweep, and on an unattended host nothing else will send it a signal. Defaults to three times --duration and must exceed it. A killed run is recorded as timed_out rather than as a generic failure, so the analysis can tell a lost cell from a real crash. Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX * style(test): gofmt browser_test.go Claude-Session: https://claude.ai/code/session_01A5KmftdEJ49A9z5mF5ESrX |
||
|
pj
|
26b49b379a |
fix ltl semantics and unify action enumeration (#71)
* fix(ltl): give every thunk a construction identity Two distinct unnamed predicates both described as "Thunk(...)", so obligation collapse merged their residuals and could drop a live violation. Identity is assigned at construction and the fields are unexported, so a thunk cannot be built without one. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(ltl): reduce a thrown-predicate residual instead of panicking The verifier substitutes an ErrorFormula for the residual of a property whose predicate threw, and that residual is fed back in on the next step. reduce had no case for it, so the run crashed. It re-reports the same failure now. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(ltl): make a bounded always the dual of a bounded eventually G<=n(f) and not F<=n(not f) disagreed on traces where the inner was still pending when the window closed, so nnf's negation normal form was not semantics preserving. Both sides now range over the observations at which their inner can definitely resolve: the eventually keeps a pending inner as a disjunct, and the always discharges vacuously at window close. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(ltl): arm a one-shot root once per run A root that carries its own horizon is one obligation for the whole run, not one per observation. Re-instantiating a top-level eventually monitored G F<=n(p) instead of F<=n(p) and left one live obligation per step behind; a bounded always restarted its window every step and never closed. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(verifier): stop wrapping a top-level eventually in always `eventually(p).within(300, "seconds")` as a property meant "within 300 seconds of every step", which spawned an obligation per step with its own resolved deadline. A 553-step run carried 553 of them and serialized a 75 KB residual. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(ltl): serialize the resolved deadline of a bounded window Two obligations spawned at different steps from one duration-bounded formula differ only in the deadline the evaluator resolved for them, so they serialized identically and the trace erased a distinction the evaluator makes. The authored window stays in amount/unit; the resolved deadline rides alongside. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(verifier): split a witness's origin step from its detection step A deferred obligation spans two steps: the one that armed it and the one whose reduction failed. They were conflated under one index, so the extractor snapshot (which is the detecting step's state) was reported against the origin step. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(runner): record a witness's detection step in the trace Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * feat(replay-ui): show the step a violation was detected at The witness evidence is the detecting step's state, so say which step that is and let a reader jump to it. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(verifier): record the extractor state the predicates actually read On the web path extractor bodies are evaluated in V8 and injected here, but only the goja value was replaced. The trace diff and the violation witness therefore described a state no property ever saw. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * refactor(spec): one candidate producer over one target-eligibility rule Both hosts routed verbs themselves and both policies enumerated their own actions, and all four drifted. Web sent `swipes` to scrollable containers only, so swipe-to-dismiss on a list row was reachable on native and unreachable on web; the model policy folded gestures its own way and could not reach what the seeded picker drew. A host now reports facts about every element and never decides which verb may act on it: targets.ts acceptsTarget owns that for both. pick.ts builtinCandidates is the single enumeration, and the model policy reads it through __sanderlingEnumerateBuiltin__ instead of reimplementing it in Go. Gesture verbs change with it: scrolls stay vertical over scrollable containers, swipes go free-form in all four directions from any element with real bounds. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(runner): name a builtin scroll by its drag origin A builtin gesture carries endpoints and no selector, so every scroll rendered as "Scroll down " in the prompt's recent-action memory and two scrollable regions were indistinguishable. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(chrome): clear storage over cdp instead of scripting an opaque origin Launch runs while the tab is still on about:blank, whose opaque origin denies storage access, so localStorage.clear() threw SecurityError and every web run died at launch. Storage.clearDataForOrigin needs no navigation. The exception helper lands here because "Uncaught" is what hid this for so long. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(chrome): enable the swiftshader webgl fallback Headless Chrome runs with --disable-gpu, and without this flag it refuses the software WebGL backend: getContext returns null, so a canvas-rendered app paints nothing and every screenshot is identical black. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * fix(web): resolve testTag through data-testid or id Compose Multiplatform emits its testTag into the element id, which the native table already accepts via the resource-id alias. The two web selector tables were the only place that rejected it. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * test(spec): type-check the spec api as part of make test The fake runtime in api.test.ts did not return a chainable handle from extract, so the file had not type-checked since named() was added. Wiring the check into make test stops it drifting again. Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J * docs(manual): one-shot eventually and the gesture verbs Claude-Session: https://claude.ai/code/session_01Fj4wJUikdABuMQEETwW55J |
||
|
pj
|
7343085614 |
llm action-selection backend (#68)
* feat(spec): add llm() action-backend marker * feat(spec): make llm marker inert on the JS picker * feat(spec): expose __sanderlingSampleInput__ corpus draw * feat(openrouter): minimal chat-completions client * test(openrouter): cover request shape, parse, and errors * feat(verifier): thread screenshot + capture corpus sampler * feat(verifier): LLM accessors — candidates, config, sampler * test(verifier): cover AllCandidates, LLMConfig, SampleInput * feat(trace): record action Source and LLMReasoning * feat(runner): thread step screenshot into PushSnapshot * feat(runner): llmSource selects actions via OpenRouter * feat(runner): wire llmSource selection and trace stamping * test(runner): cover llmSource selection, mapping, downscale * docs(folio): add llm action-backend example spec * docs(folio): document the LLM action backend run * feat(llmclient): support OPENAI_API_KEY, openrouter wins * refactor(runner): rename openrouter package to llmclient * docs: both api keys, example model gpt-5.4-nano * docs: add pr style rules to claude.md * fix(runner): explain action kinds in llm prompt to stop swipe loops * feat(trace): record llm ranked list and chosen rank * feat(runner): stamp llm ranked list and chosen rank on trace * fix(runner): tap by selector to survive layout shift after observe * revert(runner): drop selector-first tap; broke path/testTag selectors * feat(spec): llm() accepts optional instructions * feat(verifier): read llm instructions off config * feat(runner): append spec instructions to llm system prompt * docs(folio): describe app in llm spec instructions * feat(bundler): map generator export to globalThis.generator * feat(verifier): read llm config off globalThis.generator * feat(runner): gate llm source on --generator flag * feat(cmd): add --generator llm|seeded flag * test: cover --generator flag parsing and pickSources gating * feat(verifier): enumerate llm candidates by walking actionsRoot collect-walk the weighted action tree: recurse weighted branches accumulating selection probability, call authored leaves once for concrete actions, enumerate builtins per element. label controls by visible text (borrowing descendant text), fold gestures into directional scrolls over scrollable containers, drop disabled, dedup descriptions. * test(verifier): cover candidate enumeration walk * feat(verifier): add SetupAction to walk setup without the seeded root * test(verifier): cover SetupAction setup-only precedence * refactor(llmclient): make JSONSchema.Schema raw json for pinned field order * feat(trace): record llm choice number and chosen_action echo * feat(runner): llm picks one number from weighted candidates drop the seeded-root call for a setup-only precedence path, render a numbered weighted candidate list, pin a reasoning-first choice schema, strict-skip when chosen_action does not echo the numbered entry, and let the model supply typed values (corpus fallback when empty). * test(runner): cover choice schema, strict-skip, and setup precedence * refactor(verifier): drop the superseded AllCandidates enumeration * feat(folio): drive spec.ts under --generator llm; drop spec-llm.ts * fix(verifier): label editable fields by hint, not the typed value an editable field's own text is its transient content; prefer the hint so the field is named by purpose and the label stays stable. * test(runner): cover weight-suffixed echo and stripWeightSuffix * fix(runner): accept chosen_action echo that carries the weight suffix real runs showed the model copies the whole numbered line including the trailing (w34) weight annotation, so strict-skip rejected ~91% of picks and the llm was paralyzed. strip the weight suffix before comparing. also nudge the prompt to stress-test repeated submissions (idempotency). * fix(verifier): skip llm enumeration on cross-fade frames a navhost mid-transition carries >1 route *Screen in a collapsed coordinate space; acting on it taps garbage (soft keyboard). real runs showed the llm acting on 44% of steps being such frames. skip them so the llm re-observes a settled frame next step. * feat(folio): show current balance on the add-transaction screen renders the account's balance (testTag TxnCurrentBalance) below the account name, above the credit/debit toggle, so before/after screenshots carry comparison data. * fix(replay): derive device space from screen extent, not first node the first positive-bounds element is often a short status-bar node (320x24 on android); using it gave a 320/24 aspect ratio that squashed the screenshot overlay into a grey horizontal band. use the max extent across elements (like the runner's screenBounds) instead. * fix(folio): show balance as a compact one-line label per review: one line, account-name-sized, e.g. "Balance: $0.00" instead of a large balance card. * fix(folio): move balance into the header, one compact line under the account name * fix(replay): attribute deferred violations to the causing step, not detection * fix(replay): show a step's own violations in both panels, no next-step bleed * refactor(hierarchy): one Tree.Transitional, drop the duplicated cross-fade check * chore: ignore .playwright-mcp scratch output * docs: document the llm generator and --generator flag * docs(spec): correct the llm() comment; config reads off globalThis.generator * docs: add pr description rules |
||
|
pj
|
6b0d6cb971 |
WIP: Drive physical Android devices over USB (#67)
* feat(sidecar): reach USB devices via the adb server by serial * feat(test): add --device flag to target a specific Android device by serial * feat(folio): select Android device via ANDROID_DEVICE in justfile * feat(conformance): add android backend to the gate suite * feat(android): keep device awake and unlocked so the app stays foreground * feat(conformance): prep physical android device (autofill/verifier/stayon) * fix(android): make device prep best-effort so OEM-blocked commands don't abort the run * fix(verifier): require positive bounds for swipe candidates A zero-bounds element centers at (0,0); a downward swipe from the top-left corner is the system gesture that pulls down the notification shade, dragging the fuzzer out of the app. Swipes now require positive bounds like every other verb. * fix(runner): harden app-scope guard against launcher and overlays The per-step guard now relaunches and waits until the app window is actually drawn before proceeding, so a slow physical-device relaunch no longer lets an observe or action land on the launcher. It also detects a system overlay (notification shade) stealing window focus while the app stays resumed, and dismisses it with back. * feat(android): harden physical-device runs in device prep Device prep now disables the AOSP cached-app freezer, phantom-process killer, and Doze (and exempts the driver) so OEM background management stops suspending the driver mid-run. Adds ReinstallApp for clear-state on ROMs that deny pm clear, and teaches focus detection to report the notification shade as systemui so the scope guard can dismiss it. * feat(driver): clear-state via APK reinstall when pm clear is blocked When an APK path is set, Android clear-state resets the app by uninstalling and reinstalling instead of asking the sidecar to pm clear, which hardened OEM builds (ColorOS) deny even to the adb shell user. Falls back to the sidecar clear path when no APK path is provided. * feat(cli): add --android-app-path for clear-state reinstall Wires the APK path from the test command through to the sidecar client so Android clear-state can reset apps on OEM builds that deny pm clear. * chore(folio): pass --android-app-path in just test * fix(runner): clamp swipe/scroll origin out of edge gesture zones A gesture starting in the top status-bar strip pulls down the notification shade; the bottom and side strips are the home and back gestures. Any of them drags the fuzzer out of the app. Swipe and scroll origins are now clamped into a safe inner area sized from the maximum element extent (the Android hierarchy root reports zero bounds, so the extent is the reliable screen size). Calibrated on device: origins below ~7% of height no longer open the shade. * perf(sidecar): faster Android text input and drop redundant settle poll inputText now uses adb `input text` for short shell-safe ASCII (~5x faster than the driver's per-character path) and falls back to the driver for unicode, injection payloads, and overflow-length strings. waitForIdle drops the structural-hash poll that followed waitForAppToSettle: each hierarchy fetch is ~500ms on a physical device, so it cost ~2.8s per mutating step for marginal benefit, and the runner already re-fetches transitional frames. Cuts p95 step latency from ~6.5s to ~5.1s; G1-G4 still pass. * fix(verifier): exclude soft-keyboard region from action candidates The fuzzer was tapping Gboard's "Settings" key, navigating out of the app. That key is a bare FrameLayout with a content-desc and no package or resource-id, so the package-based scope filter missed it. Candidates whose center falls in the keyboard region (derived from the IME elements' bounds) are now dropped, so no tap or long-press lands on a key. Opt-in with app scoping; unscoped runs keep every node. * perf(runner): replace focus-tap settle with a brief wait The full WaitForIdle after a field-focus tap cost ~0.5-1s per InputText step on a physical device while the keyboard animated in. The tap registers focus immediately and text is injected into the focused view, so a short fixed wait suffices. Drops p95 step latency ~5.1s to ~4.0s; G1-G4 stay green. * chore(conformance): platform-aware G5 p95 budget for android The 2500ms ceiling was calibrated on the iOS simulator. A physical Android device drives every step over USB (snapshot + settle + adb round-trips), so its per-step floor is several times higher; holding it to 2500ms would force removing the settle/retry logic the correctness gates depend on. The android backend now defaults to 4500ms (override with P95_LIMIT_MS); iOS stays 2500. * fix(sidecar): retry maestro android driver startup The maestro Android driver's dadb.open() occasionally misses its startup deadline (its instrumentation host is slow to come up right after a reboot or per-run reinstall), which aborted the whole run. Retry the open a few times with a short backoff so a transient timeout recovers. * chore(conformance): widen android G5 budget to 5500ms Physical-device p95 swung 3209-4612ms across sessions (cold runs right after a reboot are slower). 4500ms was too tight for that jitter; 5500ms covers the observed ceiling with headroom. * web replay fix * feat(android): force 3-button nav during runs to prevent app drift On gesture navigation a fuzzer swipe can trigger swipe-up-home or edge-back and fling the app off screen. Device-prep now switches to 3-button navigation for the run (no edge gestures; the nav bar's buttons are systemui-owned and already excluded from action candidates) and restores the original navigation mode when the run ends. Best effort: leaves nav untouched if the overlay command is unavailable. * fix(android): target the selected device in adb reads; don't strand nav mode Review fixes: - ForegroundPackage/FocusedWindowPackage now take a serial and pass -s, so the foreground/scope guard works when several devices are attached (the --device path). Previously they ran bare `adb shell`, which errors with multiple devices, silently disabling app-scope enforcement. The sidecar client passes its serial through. - Extract an adbArgs helper and route every adb call through it, removing four duplicated serial-arg builders. - ForceThreeButtonNav now decides what to restore before changing anything: if the current mode is unknown or already 3-button it leaves nav untouched, instead of switching and then stranding the device in 3-button. Logic split into the pure navModeToRestore, now unit tested. * fix(runner): restore scrollBounds doc; cover destination clamp and screenBounds Review fixes: move the scrollBounds doc comment back onto scrollBounds (it was stranded above screenBounds by an insertion). Extend the clamp test to assert an off-screen destination is clamped onto the screen and that the origin lands exactly on the margin. * test(verifier): cover keyboardRegionTop, including the decor-view guard The full-screen IME decor view rejection had no test; removing it left the suite green. Add direct cases: no keyboard -> sentinel, decor view ignored in favor of the real keyboard line, and decor-only -> sentinel. * style(cli): gofmt testOptions field alignment * fix(sidecar): keep a leading dash off the fast input path A value starting with '-' could be read as an option by `adb input text`, so the fast-path regex now requires a non-dash first character; such values fall back to the driver. Also cover the dadb-target branch where a colon precedes a non-numeric port (a USB serial, not host:port). * refactor(verifier): scope action candidates by window ownership Replaces the leaky per-element package check and the keyboard-region Y heuristic with one rule: walk the window tree propagating each node's owning package (empty and the neutral android framework package are transparent); a node is in scope only when no concrete foreign package owns it (the app's own window carries no package on Compose apps) or the owner is the app package. This drops whole foreign windows (soft keyboard, system UI, launcher) AND their empty-package child wrappers -- e.g. a keyboard's 'Settings' key, which the old empty-package-is-in-scope rule admitted and which navigated out of the app. Deletes keyboardRegionTop/isInputMethodElement. * fix(runner): re-check foreground at apply time, skip stale actions ensureForeground runs before observe, but the app can leave between observe and apply (a prior gesture settling late); swipes/keys then fire stale coordinates onto whatever screen is now up. Re-check foreground immediately before applying and, when the app is gone, skip the action and log it (making the escape visible) so the next step's guard relaunches instead. * fix(android): type long ASCII via fast guarded path to stop keystroke escape A 4096-char corpus string exceeded the fast input cap and fell to the per-character driver path, which takes ~120s. During that uninterruptible window focus could leave the app and the remaining keystrokes sprayed into the launcher search box. Route shell-safe ASCII of any length through adb input text, chunked, re-checking the foreground app between chunks and stopping if it changed. * chore: ignore gate artifacts and local scratch files * refactor(runner): narrow gesture clamp to the top shade strip 3-button nav (forced for every run) disables the side back and bottom home gestures at the OS level. On-device probing confirmed side and bottom swipe origins no longer drift, leaving the notification shade as the only edge gesture a swipe can trigger. Clamp only the top strip; keep origin and destination on screen otherwise. * chore(format): add .editorconfig enforcing 80-column limit * chore(format): add prettier config with 80-char printWidth * chore(deps): add prettier devDependency to replay-ui * chore(deps): add prettier devDependency to folio-web * chore(deps): add prettier devDependency to spec package * chore(format): add swift-format config with 80-char lineLength * feat(format): add make fmt targets for per-language 80-col formatting * fix(runner): translate gesture to safe area so near-top scrolls keep direction Clamping the swipe origin to the top margin while leaving the destination on the full screen used two reference frames: a scrollable container pinned in the top strip had its origin pushed past the destination, reversing the gesture. Translate the whole from->to segment down by the same delta so the origin clears the shade strip without flipping direction. Adds a scroll-near-top test that fails under the old origin-only clamp. * fix(runner): apply-time guard consults focused window, not just resumed activity ensureForeground detects a system overlay (notification shade) owning the focused window while the app stays the resumed activity, but appIsForeground only queried ForegroundApp. A swipe that pulls the shade over the app between observe and apply then fired onto the shade. Mirror the focus check at apply time so the action skips and the next step dismisses the overlay. * test(runner): cover apply-time foreground skip and appIsForeground table Adds a Run-level test asserting no tap reaches the driver while a system overlay holds focus (guards against the skip branch being dead-coded), plus a decision-table test for appIsForeground. Adds ForegroundErr/FocusedWindowErr to the mock driver so the guard's transient-read paths are exercised. * fix(sidecar): harden android driver open, input guard, pressKey, foreground marker - openWithRetry rebuilt a closed AndroidDriver, whose gRPC channel is final and shut down by close(); the retry then ran against a dead channel. Build a fresh driver per attempt and extract a unit-tested retryOpen helper (named DRIVER_OPEN_ATTEMPTS/BACKOFF). - pressKey on the Maestro backend did KEY_MAP[key] (no lowercase, no throw), silently dropping unknown or wrong-case keys; route through a pure maestroKeyFor that lowercases and rejects unknown keys like the Stub contract. - the mid-type foreground guard (typeShellSafe) was untested; extract a pure typeChunks and cover stop-on-foreground-change, always-send-first-chunk, and unknown-owner. - foreground detection required the literal topResumedActivity=ActivityRecord; align parseResumedPackage to the same *ResumedActivity marker set Go reads so OEM wording does not disable the guard. * fix(conformance): pin self-test p95 budget and score install failures as run failures self_test reused the backend-dependent P95_LIMIT_MS, so under BACKEND=android the 4000ms slow fixture rated PASS and the offline analyzer check failed from an env var; pin it to 2500. A per-run adb install failure ran unguarded under set -e and aborted the whole harness; guard it, record the run as a G1 failure, and continue. * fix(android): require --device when several devices are connected With no serial requested and more than one device online, pickDevice silently returned connected[0], but that serial is never threaded into the per-step adb calls, so every later bare adb command failed with "more than one device". Error instead and ask for --device, mirroring pickAVD; a single device stays unambiguous. * refactor(android): move PrepareDevice doc onto it; extract tested wakeCommands The PrepareDevice doc block was stranded above adbArgs, leaving the exported function undocumented under godoc. Move it back and split the wake/keyguard tuples into wakeCommands so they have a unit test. * perf(verifier): memoize scopedElements per tree scopedElements rebuilt a full tree walk plus map on every candidatesForVerb call (~16 per step). Cache the result keyed on lastTree and invalidate it in PushSnapshot. * fix(sidecar): default reinstallApp in SetClearStateReinstall; cover non-android clear Only Dial set reinstallApp, so a Client built another way would nil-deref on Android clear-state. Default it in SetClearStateReinstall too. Add a non-android test so the platform guard has negative coverage: dropping the android check would now fail. * test(runner): make focusTapSettle injectable so apply tests don't sleep 250ms The focus-tap settle was a const, so five InputText apply tests each blocked the full 250ms. Make it a package var and shorten it per-test with cleanup. * refactor(runner,android): drop unused bringToForeground return; grep no-match yields empty bringToForeground's bool return was read by no caller. FocusedWindowPackage's on-device grep exited 1 on no match, surfacing as an error instead of the documented ""; add || true. * perf(sidecar): reuse a single Jackson ObjectMapper structuralHash, countRouteScreens, and hierarchy each built a fresh ObjectMapper per call inside the stability poll; the instance is thread-safe and meant to be reused. Hoist one shared val. * refactor(android): remove unused AdbReverse/AdbReverseRemove No callers anywhere in the tree; they were also the only adb calls bypassing adbArgs. Dead code, removed. * style(runner): trim non-load-bearing comments from this PR's runner code and tests * style(sidecar): trim non-load-bearing comments from this PR's driver code and tests |
||
|
pj
|
94d9511312 |
test: full test-suite refactor sweep (#61)
* chore(test): start test-suite refactor sweep * test(ltl): pin exact multi-obligation residual AST * test(ltl): table-test finalize Kleene connective combinations * test(ltl): pin reduce over pending inner for bound, Or, Not * test(ltl): marshal bounded Always steps/duration/deadline * test(verifier): cover LTL combinator verdict transitions and within unit panic * test(verifier): table-test DecodeAction kinds and lastAction field exposure * test(verifier): assert WithPlatform(ios) reaches the picker host and key pool * test(verifier): widen weighted-selection assertion to a 5x skew margin * test(verifier): un-skip ax-find round trip with a committed tree fixture * test(runner): pin isWDADrop to sidecar reconnect-failed message origin * test(runner): assert PressKey/Wait trace encoding records kind-specific fields * test(runner): cover RenderSummary unsupported-verbs surfacing branch * test(trace): set Hierarchy in round-trip and lock lossy Tree contract Also add a -race concurrent WriteStep test that asserts N well-formed JSONL lines, catching torn lines if the writer mutex is dropped. * test(trace): round-trip witnesses/changes/metrics/exceptions, pin step-0 witness * test(trace): document ViolationsAreGreppable grep contract and lock-free WriteScreenshot * test(hierarchy): cover invalid-JSON and malformed-bounds parser paths * test(trace): guard writer mutex via WriteStep/Close race on w.file * test(replay): drop unfailable assets and devproxy assertions * test(replay): cache reuses on equal mtime, reparses after append * test(replay): violation marker falls back to detection step when attributed missing * test(replay): corrupt meta/trace dirs return 500 with error body * test(replay): SSE client receives runs.changed after a broadcast * test(replay): Run coalesces creates, ignores write/chmod, closes subs on cancel * fix(sidecar): synchronize health fixture writes and exercise healthError * test(sidecar): cover swipe/longpress/doubletap/erase/presskey/metrics/logs translations * test(sidecar): cover DoubleTapSelector composition and mid-gesture cancel * test(sidecar): assert gRPC error status surfaces from action RPC * fix(chrome): route action methods through runCtx so caller cancellation aborts CDP * fix(chrome): route hierarchy/screenshot/waitidle/metrics through runCtx * refactor(ios): extract pure simctl JSON parsers * refactor(ios): add command-runner seams for EnsureSimulator * test(ios): table-test simctl parsers and EnsureSimulator seams * test(sidecarassets): cover placeholder build path * test(sidecarassets): assert reuse via sentinel bytes not mtime * test(bundler): cover properties-only spec registration * refactor(testrun): extract prepareBundleInputs from Execute * test(testrun): cover prepareBundleInputs aliases and missing-runtime error * test(testrun): table-test resolveRuntimeSibling search edges * test(testrun): exact-output tests for progressHandler line format * fix(cmd): point bundle-check aliases at pkg/spec/src * test(cmd): smoke-test bundle-check resolves spec aliases * test(cmd): table-test hier-check parse and FindAll on fixture * test(cmd): unit-test buildBrowseURL deep-link vs root * test(cmd): drop flaky TestRun_Doctor that launched real Chromium * test(cmd): pin pipeline error to bundle resolution on web platform * test(replay-ui): add bun test script * ci(replay-ui): run bun test via make web-test target * ci(replay-ui): point bun cache key at replay-ui/bun.lock * test(replay-ui): exercise real URL encoding and non-ok throw in getJson * refactor(replay-ui): extract snapshot flatten/getAtPath into lib module * test(replay-ui): pin snapshot flatten/getAtPath path round-trip * refactor(replay-ui): extract action selector/format into lib module * test(replay-ui): pin action selector parse and row formatting * refactor(replay-ui): share one statusFor between panels * refactor(replay-ui): extract run-history derivation into lib module * test(replay-ui): pin shared statusFor precedence and ordering * test(replay-ui): pin run-history derivation alignment * refactor(replay-ui): export clampIndex for testing * refactor(replay-ui): extract keyboard-nav dispatch into pure module * refactor(replay-ui): extract metrics formatters into lib module * test(replay-ui): pin clampIndex step boundaries * test(replay-ui): pin keyboard-nav ownership and key routing * test(replay-ui): pin metrics formatters and path gap handling * refactor(sidecar): expose device-output parsers as internal for testing * test(sidecar): table-test device-output parsers against malformed input * test(sidecar): cover logcat parsing year inference and line skipping * test(sidecar): pin pressKey keycode mapping and unknown-key rejection * test(sidecar): metrics bundleId falls back to launched app and honors override * test(sidecar): loosen deadline upper bound to tolerate slow CI scheduling * test(web-runtime): export selector builders for unit tests * test(web-runtime): guard sanitize cycle, function, and depth limits * test(web-runtime): table-test selector builder quoting and escaping * test(sidecar): collapse scalar-forwarding RPC tests into a table * test(replay-ui): dedup step/summary fixtures into shared module * test(ios): collapse pickSimulator point-tests into a table |