mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 11:07:10 +00:00
ci: dispatch workflows for folio and the replay ui (#73)
* feat(runner): stop the step loop at the first violation on request
* feat(testrun): report violations as a typed error under exit-on-violation
* feat(cli): add --exit-on-violation and exit 2 when it fires
* docs(cli): document --exit-on-violation, --max-steps, and exit codes
* fix(web): enumerate and query across shadow roots in both producers
* test(chrome): compare both producers on a shadow-dom parity page
* test(browser): drive a canvas-under-shadow-root fixture end to end
* fix(web): select the focused field inside a shadow root before typing
* fix(web): report the pathname as the screen when there is no hash route
* fix(web): settle on dom quiescence instead of returning at body ready
* feat(replay-ui): add data-testid hooks the dogfood spec drives
* feat(replay-ui): add the dogfood spec sanderling runs against the replay ui
* fix(replay-ui): scope the screenshot property to the named state panel
* chore(make): add per-platform sanderling build targets
* ci: add dispatch workflows for folio and the replay ui
* docs: describe the dispatch workflows and how to read a failure
* ci(folio): give the ios leg its jdk, android sdk, just, and a clean app start
* refactor(web): use max for the settle budget
* ci: pin calibrated seeds, skip the flaky ios reinstall, bound every job
* ci: authenticate and pin the buf setup step
the anonymous release download hit the shared runner ip rate limit and
failed the job with 'socket hang up' after three retries.
* docs: record that canvas apps need a dom proxy to be text-fuzzable
* fix(ios): bound lifecycle rpcs and claim the target device
a launch the simulator rejects sent the xctest session into a recovery
chain that answered minutes late or never, and the rpc had no deadline,
so the run hung with no trace and no error. also take a per-udid flock:
a second run's reinstall lands under the first's live automation session
and wedges it.
* docs(ci): correct the ios hang wording and note the device lock
* refactor(verifier): derive the lastAction shape from one field list
both hosts must show a spec the same lastAction. one ordered list now
feeds the goja object and the json the web host installs, so they
cannot drift.
* fix(web): install lastAction in the page before extractors read it
state.lastAction was hardcoded null on web, so every property reading it
was silently vacuous: a correct property passed without ever firing.
* fix(web): carry element identity on actions and fix findAll on paths
an action's target was coordinates only, so a property matching on which
element was acted upon could never fire. ax.findAll([a,b]) also returned
nothing on web.
* fix(chrome): wait out a route transition before sampling facts
the tree stays byte-identical and quiet across a cross-fade, so both the
quiet timer and the unchanged-tree escape called it settled mid-flight
and extractors read two screens at once.
* fix: bound the pre-run app launch
launch happens before the runner starts, so --duration never covered it
and a wedged driver hung with no trace and no error.
* fix(folio): read balances from merged cards and treat unreadable as unknown
compose for web merges the whole accountcard subtree, so the balance
child never exists there and every card parsed as 0. the property then
compared 0 to 0 and fired on any submit, which is a false positive
generator. unknown is now null and null is vacuously true.
* test(folio): cover merged-card parsing and unknown balances
* ci(folio): make web an expect-the-bug leg
the web runtime can observe the double submit now, so the health gate
understates it. seed 1 finds it at step 109, 3 runs out of 3.
* docs(ci): explain why a submit tap landing on home is the bug
* fix(ios): read a StaticText's label as its text
AXValue was the only source for text, but a StaticText carries its
string in AXLabel, so nothing on screen had .text on ios: a spec reading
it saw everything on android and nothing here.
* docs(ci): correct the calibrated step ranges
* fix(folio): stop convicting on arithmetic float64 cannot hold
past 2^53 cents the gap between representable values is 128, so a real
1600-cent move reads back as something else and the equality is false
for a healthy submit as readily as a double one. also match parseCents:
a sign or an oversized amount is rejected, not read as an amount.
* test(folio): pin the safe-integer guard and its boundary
* docs: stop teaching the zero-default that caused a false alarm
* docs: write down the silent-vacuity failure modes
* feat(folio): tag the home total and the card transaction count
the total was the only untagged node on the screen, so the spec had to
sum cards and a clipped card broke the sum.
* fix(folio): read the app's own total and refuse contaminated windows
summing cards went null when one was clipped, and the null poisoned the
carrier for the rest of the run. the balance window also spanned every
transaction since the last home visit, so the property convicted on
deltas it could not attribute: the old web witness was 3.16x the typed
amount, not 2x.
* test(folio): pin the window rules and the count invariant
* fix(folio): never read a frame that shows two screens
android dumps a cross-fade with both screens in the tree. the route said
add-transaction while an unscoped find said home, so the oracle took a
half-rendered total as fresh and convicted on a tap that committed
nothing. one function now decides the route and returns null when the
frame is ambiguous.
* test(folio): cover transition frames, card readings and creation
* fix(folio): only disambiguate counts that came from merged text
the equal-length digit rule exists because web merges the card and an
account named -1 makes '12' ambiguous. a dedicated count node has
nothing to disambiguate, so applying it there threw away real evidence.
* ci(folio): pin the recalibrated seeds and drop android to a health gate
web 3 and ios 7 convict 3 runs out of 3 with an exactly 2x witness.
android convicts 2 in 5 because the same seed does not walk the same
trajectory there, so it proves the app runs instead.
* docs(ci): describe the two properties and why android cannot convict
* fix(android): wait out a route cross-fade before snapshotting
the dump could hold two screens at once, and the runner refuses to act
on such a tree, so a quarter of android steps applied no action and the
count varied per run: the same seed never walked the same trajectory.
the ios companion and the chrome driver already do this.
* ci(folio): let the android leg run far enough to see its conviction
* docs: only the repo owner merges
* ci(folio): a thrown predicate is not a conviction
exit 2 means the run recorded a violation, and a predicate that throws
is recorded as one too. so was newAccountBalanceIsZero, an unrelated
property in the same spec. the gate read the exit code and went green
with detection dead.
* ci: install idb-companion from its tap and stop interpolating inputs
idb-companion is not in homebrew-core, so the ios leg died before it
built anything. replay-ui expanded dispatch inputs into the shell.
* docs: correct the snippets and numbers that drifted from the code
* test(sidecar): pin that a slow read counts toward the stability streak
* fix(web): read the page's extractors only on steps that count
the page advances the spec's carriers when it evaluates, but the runner
applied the result only on non-transitional steps. a discarded step
moved the window forward anyway, so the next accepted pair bracketed two
transactions while counting one submit, and convicted a healthy app.
extractor errors now fail the run instead of leaving goja's values in
current against v8's in previous.
* fix(chrome): anchor the transition deadline when the dom goes quiet
it was anchored at script start, so a page that churned past the window
reached the check already expired and returned mid cross-fade. the
driver now publishes the idle timeout it needs, since the caller's 1s
could never spend the 800ms window.
* fix(web): fail on a partial extractor override
same mixed-producer hazard as the install error: some extractors hold
the page's value and the rest hold goja's, and a property comparing
across that split fires on a healthy app.
* docs: six of seven, the seventh is the stock property
* fix(folio): drop a name two cards answer to
homeTxnCountsOf keyed on the account name and let the last card win, so
two accounts the fuzzer named the same collapsed into one entry. a
reading that saw one Travel card and a later one that saw both then
subtracted two different accounts' counts, and
submitCommitsOneTransactionPerAction convicted a healthy app of
double-submitting. it is a gated property in folio-run.sh, so that reads
as "found the submit bug" over a card scrolling into view.
same rule createdAccountHasNonZeroBalance already applies: a name
nothing can attribute is no evidence. counted over every card, since an
unreadable twin spoils the identity too.
* perf(folio): read each frame once
every extractor asked routeOf, and routeOf does five ax.find calls. on
web each find walks the document and every shadow root beneath it, so
the spec cost 110 tree walks a step; homeCards was parsed four times
over. now 5 and once.
keyed on the identity of the state object because both hosts build a new
one per step and hand that one object to every getter, so it cannot
outlive its frame. holding the reference is what keeps that true rather
than likely.
* fix(web): keep an undefined reading's index through JSON
json has no undefined, so an extractor whose getter returned one had its
whole index dropped by JSON.stringify. that index then kept goja's
dump-derived value while its neighbours held the page's, and a property
comparing previous to current across the split fires on a healthy app.
folio has nine on(route, tag) extractors, so this was most extractors on
most steps.
each reading is wrapped in a {value} envelope: the drop now happens
inside the entry, and an absent value means the getter returned
undefined, which is what the goja host records for the same getter. a
json null would instead claim it returned null and x.current ===
undefined would answer differently on the two hosts.
* feat(verifier): report the registered extractor count
the web path needs it to check the page sent one reading per extractor.
* fix(runner): fail when the page reports fewer readings than extractors
the comment here already claimed a partial override was fatal. it was
not: the skipped check only catches indices outside the extractor list,
so a page reporting values for some extractors and not others left the
rest holding goja's reading of the dump with nothing said.
* test(browser): drive an undefined reading through the whole web path
four layers carry it: the page's envelope, the driver's unwrap, the
runner's count check and the verifier's decode. each has a unit test and
only a run proves they compose. goes red both ways, decoding an absent
value as null and dropping the envelope.
* fix(web): offer the aria roles a user activates
only role=button was in the tappable set, so link, checkbox, radio,
switch, tab, option, the menuitems and treeitem were invisible to the
enumeration however plain the control looked. the replay ui builds its
step rows as <li role="option">, and the spec dogfooding it had to
hand-write an action to reach them because no default verb could see a
single row.
both producers build the set from the same role list, since the parity
test compares them element by element.
* test(browser): tap a role-based control end to end
every control on the page is an <li role="option">, the shape the
replay ui gives its step rows, and the spec carries no action of its
own: the property firing is the evidence the default enumeration offered
a tap on one.
* fix(web): read aria-disabled as disabled
the enabled fact came off the disabled property, which only real form
controls have. it reads undefined on the role-based controls the
tappable set now covers, so every one of them looked enabled however
plainly it was marked otherwise, and the fuzzer would spend actions on
inert ones.
both producers answer the same two ways, and the parity fixture carries
a disabled row so the comparison covers it: reverting one side alone
names the element and the fact.
* docs(replay-ui): the enumeration reaches step rows now
the comment said role="option" is not in the tappable selector set,
which stopped being true a few commits ago. selectAStep stays, for the
reason the tab weight below it stays: one row among the page's clickable
elements is a thin chance, and both step-facing properties go vacuous on
a run that never selects one.
* test(runner): bound the last-action test by steps, not wall clock
100ms of wall clock against an assertion that two steps ran fatals under
load with "the web path never installed it", which reads as a
regression. every sibling test in the package uses a long duration and
MaxSteps.
* ci: run the kotlin tests in make test
RouteTransitionTest and the stability poll cover the android settle and
nothing in ci ran them. :sidecar:test needs no android sdk, checked by
running it with ANDROID_HOME pointed at nothing.
* fix(sidecar): measure the stability streak as observed quiet
parameterising pollUntilStable also moved the clock to the start of the
read that opened a run of identical snapshots, so a read's own duration
counted as quiet. the pre-existing caller polls a real uiautomator dump:
at 400ms a read, 750ms of required quiet became 250ms of observed quiet
and the poll settled in two reads instead of four.
the parameters stay, the semantics go back.
* test(sidecar): pin the transition cap by driving it
it asserted 1500 >= 700 + 300, two constants, which can only fail if
someone edits a constant. it now drives awaitSettledTree against a fade
that lands after 700ms and asserts it hands back the settled tree before
the cap. cut the cap to 1000 and it goes red.
* ci: pin buf-setup-action to a commit
it takes a token now, so a floating tag is a token handed to whatever
that tag moves to. note v1 there is a branch, not a tag, so the ref
lookup that resolves it is matching-refs/heads/v1.
* ci: declare least-privilege permissions
none of the three declared any, so each got the repository default.
release.yml and docs.yml already do this. all three only check out,
build, test and upload artifacts.
* ci: fail fast when a server never comes up
the readiness loops fell through silently after 30 tries, so a server
that never started surfaced as an opaque driver failure minutes later.
each now says what did not answer and on which port.
* ci(folio): a missing trace is not a verdict
with no trace the android gate ran its grep against ./trace.jsonl and
reported "never reached AddTransactionScreen, so it never got past
login", which is not what happened. the web and ios branches had the
same misdiagnosis on exit 0.
same class, one line up: the classifier's own failure was swallowed, so
with the evidence reader dead the gate printed a healthy run and exited
0.
* ci(replay-ui): skip a run directory with no trace
the summarise step is if: always(), and under github's bash -eo pipefail
an unmatched glob stays literal, the redirect fails, pipefail carries it
into the assignment and -e kills the step. so a failed fuzz run went red
twice, once for the real reason.
This commit is contained in:
70 files changed
+6656
-430
No files matched your search
@@ -256,12 +256,7 @@ func (d *Driver) InputText(callerCtx context.Context, text string) error {
|
||||
defer cancel()
|
||||
return chromedp.Run(runCtx,
|
||||
chromedp.ActionFunc(func(ctx context.Context) error {
|
||||
// Select any existing content so InsertText replaces rather than appends.
|
||||
if err := chromedp.Evaluate(`
|
||||
(function() {
|
||||
const el = document.activeElement;
|
||||
if (el && typeof el.select === 'function') el.select();
|
||||
})()`, nil).Do(ctx); err != nil {
|
||||
if err := selectFocusedText(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
return input.InsertText(text).Do(ctx)
|
||||
@@ -269,6 +264,28 @@ func (d *Driver) InputText(callerCtx context.Context, text string) error {
|
||||
)
|
||||
}
|
||||
|
||||
// selectAllScript selects everything in the focused field so the InsertText
|
||||
// that follows replaces rather than appends.
|
||||
//
|
||||
// document.activeElement stops at a shadow boundary: it names the HOST, not the
|
||||
// focused node inside. Compose for Web focuses a hidden <input> inside the
|
||||
// shadow root it mounts, so the host answer has no select() and the selection
|
||||
// never happened - every InputText appended to the last one, and a fuzzer that
|
||||
// types into the same field twice built up garbage it could never clear.
|
||||
// Descending activeElement through each shadow root finds the real field.
|
||||
const selectAllScript = `
|
||||
(function() {
|
||||
let el = document.activeElement;
|
||||
while (el && el.shadowRoot && el.shadowRoot.activeElement) {
|
||||
el = el.shadowRoot.activeElement;
|
||||
}
|
||||
if (el && typeof el.select === 'function') el.select();
|
||||
})()`
|
||||
|
||||
func selectFocusedText(ctx context.Context) error {
|
||||
return chromedp.Evaluate(selectAllScript, nil).Do(ctx)
|
||||
}
|
||||
|
||||
// ReplacesTextOnInput reports that InputText replaces existing content via
|
||||
// select-all, so the runner skips its pre-erase.
|
||||
func (d *Driver) ReplacesTextOnInput() bool {
|
||||
@@ -282,11 +299,7 @@ func (d *Driver) EraseText(callerCtx context.Context, _ int) error {
|
||||
defer cancel()
|
||||
return chromedp.Run(runCtx,
|
||||
chromedp.ActionFunc(func(ctx context.Context) error {
|
||||
if err := chromedp.Evaluate(`
|
||||
(function() {
|
||||
const el = document.activeElement;
|
||||
if (el && typeof el.select === 'function') el.select();
|
||||
})()`, nil).Do(ctx); err != nil {
|
||||
if err := selectFocusedText(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
return input.InsertText("").Do(ctx)
|
||||
@@ -368,7 +381,11 @@ func (d *Driver) Hierarchy(ctx context.Context) (string, error) {
|
||||
defer cancel()
|
||||
script := `
|
||||
(function() {
|
||||
const route = window.location.hash.replace(/^#/, '').split('?')[0] || '/';
|
||||
// Hash first (a HashRouter names the screen there), then the pathname, which
|
||||
// is where a path-routed SPA keeps it. Reporting '/' for every step of a
|
||||
// BrowserRouter app made every screen look like the same screen.
|
||||
const route = window.location.hash.replace(/^#/, '').split('?')[0] ||
|
||||
window.location.pathname || '/';
|
||||
// clickable and editable are resolved through the SAME selector sets
|
||||
// pkg/spec/src/web-runtime.ts uses, so the goja host (which reads this dump)
|
||||
// and the V8 host (which reads the DOM directly) cannot mean different things
|
||||
@@ -376,6 +393,14 @@ func (d *Driver) Hierarchy(ctx context.Context) (string, error) {
|
||||
// root a full-viewport tap target here and nowhere else.
|
||||
const NON_TEXT_INPUT_TYPES =
|
||||
['button','submit','checkbox','radio','range','color','file','image','reset'];
|
||||
// The disabled property belongs to real form controls only, so it reads
|
||||
// undefined on the role-based controls the tappable set now covers, and every
|
||||
// one of them looked enabled however plainly it was marked otherwise.
|
||||
// isEnabled in pkg/spec/src/web-runtime.ts answers the same two ways.
|
||||
function isEnabled(el) {
|
||||
if (el.disabled) return false;
|
||||
return el.getAttribute('aria-disabled') !== 'true';
|
||||
}
|
||||
function isEditableElement(el) {
|
||||
if (el.isContentEditable) return true;
|
||||
const tag = el.tagName.toLowerCase();
|
||||
@@ -383,10 +408,28 @@ func (d *Driver) Hierarchy(ctx context.Context) (string, error) {
|
||||
if (tag === 'input') return !NON_TEXT_INPUT_TYPES.includes((el.type || '').toLowerCase());
|
||||
return false;
|
||||
}
|
||||
const clickableSet = new Set(document.querySelectorAll(
|
||||
'a, button, input, select, textarea, [role="button"], [onclick]'));
|
||||
const editableSet = new Set(Array.from(
|
||||
document.querySelectorAll('input, textarea, [contenteditable]')).filter(isEditableElement));
|
||||
// Shadow roots are part of the page a user sees, so they are part of the page
|
||||
// we enumerate. Compose for Web mounts its canvas AND its accessibility tree
|
||||
// inside a shadow root on the mount element, so a light-DOM-only walk reports
|
||||
// four nodes for a whole app and offers no action on any of them.
|
||||
function deepQuery(sel) {
|
||||
const out = [];
|
||||
const visit = (root) => {
|
||||
for (const el of root.querySelectorAll(sel)) out.push(el);
|
||||
for (const el of root.querySelectorAll('*')) if (el.shadowRoot) visit(el.shadowRoot);
|
||||
};
|
||||
visit(document);
|
||||
return out;
|
||||
}
|
||||
const TAPPABLE_ROLES = [
|
||||
'button', 'link', 'checkbox', 'radio', 'switch', 'tab', 'option',
|
||||
'menuitem', 'menuitemcheckbox', 'menuitemradio', 'treeitem'];
|
||||
const clickableSet = new Set(deepQuery(
|
||||
'a, button, input, select, textarea, ' +
|
||||
TAPPABLE_ROLES.map(role => '[role="' + role + '"]').join(', ') +
|
||||
', [onclick]'));
|
||||
const editableSet = new Set(deepQuery(
|
||||
'input, textarea, [contenteditable]').filter(isEditableElement));
|
||||
function buildTree(el, isRoot) {
|
||||
const rect = el.getBoundingClientRect();
|
||||
const attrs = {};
|
||||
@@ -414,6 +457,14 @@ func (d *Driver) Hierarchy(ctx context.Context) (string, error) {
|
||||
const isClickable = clickableSet.has(el);
|
||||
const isEditable = editableSet.has(el);
|
||||
const children = [];
|
||||
// Shadow content first, then light children: the shadow tree is what the
|
||||
// host actually renders, and targetElements in web-runtime.ts walks the same
|
||||
// order, which is the order the two enumerations are compared in.
|
||||
if (el.shadowRoot) {
|
||||
for (const child of el.shadowRoot.children) {
|
||||
children.push(buildTree(child, false));
|
||||
}
|
||||
}
|
||||
for (const child of el.children) {
|
||||
if (child.tagName === 'HEAD') continue;
|
||||
children.push(buildTree(child, false));
|
||||
@@ -422,7 +473,7 @@ func (d *Driver) Hierarchy(ctx context.Context) (string, error) {
|
||||
attributes: attrs,
|
||||
children: children,
|
||||
clickable: isClickable || null,
|
||||
enabled: (!el.disabled) || null,
|
||||
enabled: isEnabled(el) || null,
|
||||
focused: document.activeElement === el || null,
|
||||
checked: el.checked || null,
|
||||
selected: el.selected || null,
|
||||
@@ -498,10 +549,138 @@ func (d *Driver) RecentLogs(_ context.Context, since time.Time, minLevel string)
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func (d *Driver) WaitForIdle(ctx context.Context, _ time.Duration) error {
|
||||
// domQuietPeriod is how long the DOM must stop changing before the page counts
|
||||
// as settled. Compose for Web syncs its accessibility DOM off the frame loop:
|
||||
// measured at ~136 ms behind an InputText on the folio wasm build, so waiting
|
||||
// for frames alone (~16 ms each) returns while the app still reports the old
|
||||
// text, and the next step types into a field it believes is still empty.
|
||||
const domQuietPeriod = 150 * time.Millisecond
|
||||
|
||||
// transitionSettlePeriod is how much longer the settle waits for a route
|
||||
// transition to finish once the DOM has gone quiet. A canvas app's cross-fade
|
||||
// is invisible to a mutation observer: Compose splices the incoming screen's
|
||||
// accessibility nodes in when the animation STARTS and removes the outgoing
|
||||
// screen's when it ends, and nothing in between touches the DOM, so the tree
|
||||
// sits byte-identical (and quiet) with both routes live for the whole
|
||||
// animation. Settling on quiet alone returns there, and the next step then
|
||||
// verifies a tree that names the screen the app is leaving: on the folio wasm
|
||||
// build a submit that landed on Home was recorded as still being on the
|
||||
// transaction screen, so a property gated on where the action landed read the
|
||||
// wrong route and went vacuous. The wait is bounded so a page that genuinely
|
||||
// shows two *Screen ids at rest costs this much per step and no more.
|
||||
const transitionSettlePeriod = 800 * time.Millisecond
|
||||
|
||||
// settleReturnMargin is what WaitForIdle holds back from the caller's timeout,
|
||||
// so returning late by our own doing surfaces as a settled page rather than a
|
||||
// context cancellation.
|
||||
const settleReturnMargin = 100 * time.Millisecond
|
||||
|
||||
// settleScanMargin covers the in-page work the two waits do not themselves
|
||||
// account for: liveScreens() walks the document and every shadow root on each
|
||||
// 16 ms poll, and the whole script costs one CDP round trip.
|
||||
const settleScanMargin = 250 * time.Millisecond
|
||||
|
||||
// MinIdleTimeout is the shortest timeout WaitForIdle can be handed and still
|
||||
// spend the waits it is built from: the DOM quiet period, the route-transition
|
||||
// window that only opens once that quiet period has elapsed, and the second
|
||||
// quiet period the transition's own closing mutation starts. A caller that
|
||||
// passes less caps the settle below its own budget, and the step then samples a
|
||||
// page that is still mid-transition - which is the exact failure the transition
|
||||
// wait exists to prevent. internal/runner raises a shorter caller timeout to
|
||||
// this value.
|
||||
func (d *Driver) MinIdleTimeout() time.Duration {
|
||||
return 2*domQuietPeriod + transitionSettlePeriod +
|
||||
settleScanMargin + settleReturnMargin
|
||||
}
|
||||
|
||||
func (d *Driver) WaitForIdle(ctx context.Context, timeout time.Duration) error {
|
||||
runCtx, cancel := d.runCtx(ctx)
|
||||
defer cancel()
|
||||
return chromedp.Run(runCtx, chromedp.WaitReady("body", chromedp.ByQuery))
|
||||
// Leave the caller's deadline some room: returning late by our own doing
|
||||
// would surface as a context cancellation instead of a settled page.
|
||||
budget := max(timeout-settleReturnMargin, domQuietPeriod)
|
||||
script := fmt.Sprintf(settleScript,
|
||||
domQuietPeriod.Milliseconds(),
|
||||
budget.Milliseconds(),
|
||||
transitionSettlePeriod.Milliseconds(),
|
||||
)
|
||||
return chromedp.Run(runCtx,
|
||||
chromedp.WaitReady("body", chromedp.ByQuery),
|
||||
chromedp.Evaluate(script, nil, awaitPromise),
|
||||
)
|
||||
}
|
||||
|
||||
// liveScreensFunction defines liveScreens(), the page-side count of live ids
|
||||
// ending in "Screen". More than one is a route transition in flight: the same
|
||||
// rule the tree parser applies (Transitional in internal/hierarchy), so the
|
||||
// driver and the runner agree on what a settled route looks like. It descends
|
||||
// shadow roots because a canvas app keeps its whole accessibility tree inside
|
||||
// one.
|
||||
const liveScreensFunction = `
|
||||
const liveScreens = () => {
|
||||
let count = 0;
|
||||
const visit = (root) => {
|
||||
count += root.querySelectorAll('[id$="Screen"]').length;
|
||||
for (const element of root.querySelectorAll('*')) {
|
||||
if (element.shadowRoot) visit(element.shadowRoot);
|
||||
}
|
||||
};
|
||||
visit(document);
|
||||
return count;
|
||||
};`
|
||||
|
||||
// settleScript resolves once the document has gone quiet for %d ms and is not
|
||||
// mid route transition, or after %d ms whatever happens; the transition wait
|
||||
// itself gives up after %d ms. Shadow roots get their own observer: a canvas
|
||||
// app keeps its whole accessibility tree inside one, and mutations there do not
|
||||
// reach an observer on the document.
|
||||
//
|
||||
// The transition window opens when the quiet period ends, not when the script
|
||||
// starts. Anchored at the start it is already spent by the time the check can
|
||||
// first run on any page that keeps mutating for longer than the window, so the
|
||||
// wait resolves immediately with both routes still live - the mid-transition
|
||||
// return this whole wait exists to prevent. Each mutation reopens it, and the
|
||||
// budget above bounds the total either way.
|
||||
const settleScript = `
|
||||
new Promise(resolve => {
|
||||
const quietMillis = %d, budgetMillis = %d, transitionMillis = %d;
|
||||
const observers = [];
|
||||
let transitionDeadline = 0;
|
||||
let timer = null;
|
||||
const finish = () => {
|
||||
clearTimeout(timer);
|
||||
for (const observer of observers) observer.disconnect();
|
||||
resolve();
|
||||
};
|
||||
` + liveScreensFunction + `
|
||||
const quiet = () => {
|
||||
if (transitionDeadline === 0) transitionDeadline = Date.now() + transitionMillis;
|
||||
if (liveScreens() > 1 && Date.now() < transitionDeadline) {
|
||||
timer = setTimeout(quiet, 16);
|
||||
return;
|
||||
}
|
||||
finish();
|
||||
};
|
||||
const restart = () => {
|
||||
clearTimeout(timer);
|
||||
transitionDeadline = 0;
|
||||
timer = setTimeout(quiet, quietMillis);
|
||||
};
|
||||
const watch = (root) => {
|
||||
const observer = new MutationObserver(restart);
|
||||
observer.observe(root, {subtree: true, childList: true, attributes: true, characterData: true});
|
||||
observers.push(observer);
|
||||
for (const element of root.querySelectorAll('*')) {
|
||||
if (element.shadowRoot) watch(element.shadowRoot);
|
||||
}
|
||||
};
|
||||
watch(document);
|
||||
setTimeout(finish, budgetMillis);
|
||||
restart();
|
||||
})`
|
||||
|
||||
func awaitPromise(params *runtime.EvaluateParams) *runtime.EvaluateParams {
|
||||
return params.WithAwaitPromise(true)
|
||||
}
|
||||
|
||||
func (d *Driver) Health(_ context.Context) (driver.Health, error) {
|
||||
@@ -596,12 +775,21 @@ func (d *Driver) InstallBundle(ctx context.Context, source []byte) error {
|
||||
|
||||
// EvaluateExtractors invokes the bundle-installed extractor table and returns
|
||||
// each extractor's JSON-encoded current value keyed by its registration index.
|
||||
//
|
||||
// The read waits out a route transition first, bounded by
|
||||
// transitionSettlePeriod. The hierarchy fetch already re-fetches a transitional
|
||||
// tree (fetchSyncedState in internal/runner); without the same rule here the
|
||||
// two halves of one step describe different moments, and the spec's own
|
||||
// extractors are the half that loses: on the folio wasm build the extractors
|
||||
// sampled mid cross-fade and reported the route the app was leaving, so a
|
||||
// property gated on where the action landed skipped the only step that action
|
||||
// could be judged on.
|
||||
func (d *Driver) EvaluateExtractors(ctx context.Context) (map[int]json.RawMessage, error) {
|
||||
const script = `JSON.stringify(window.__sanderlingExtractors__ ? window.__sanderlingExtractors__() : {})`
|
||||
script := fmt.Sprintf(extractorScript, transitionSettlePeriod.Milliseconds())
|
||||
var encoded string
|
||||
runCtx, cancel := d.runCtx(ctx)
|
||||
defer cancel()
|
||||
if err := chromedp.Run(runCtx, chromedp.Evaluate(script, &encoded)); err != nil {
|
||||
if err := chromedp.Run(runCtx, chromedp.Evaluate(script, &encoded, awaitPromise)); err != nil {
|
||||
return nil, fmt.Errorf("evaluate extractors: %w", err)
|
||||
}
|
||||
if encoded == "" || encoded == "{}" {
|
||||
@@ -612,16 +800,90 @@ func (d *Driver) EvaluateExtractors(ctx context.Context) (map[int]json.RawMessag
|
||||
return nil, fmt.Errorf("decode extractor map: %w", err)
|
||||
}
|
||||
result := make(map[int]json.RawMessage, len(stringMap))
|
||||
for key, value := range stringMap {
|
||||
for key, entry := range stringMap {
|
||||
index, err := strconv.Atoi(key)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("non-integer extractor key %q", key)
|
||||
}
|
||||
result[index] = value
|
||||
reading, err := extractorReading(entry)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("extractor %d: %w", index, err)
|
||||
}
|
||||
result[index] = reading
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// extractorReading unwraps one entry of the page's extractor table. The page
|
||||
// wraps every reading in a {"value": ...} envelope (evaluateExtractors in
|
||||
// pkg/spec/src/web-runtime.ts) because JSON has no undefined: an absent `value`
|
||||
// is the getter returning undefined, and returning it as an empty payload is
|
||||
// what makes the goja host record undefined too. Reading it as JSON null would
|
||||
// claim the getter returned null, so `x.current === undefined` would answer one
|
||||
// thing on native and another on web.
|
||||
func extractorReading(entry json.RawMessage) (json.RawMessage, error) {
|
||||
var envelope struct {
|
||||
Value json.RawMessage `json:"value"`
|
||||
}
|
||||
if err := json.Unmarshal(entry, &envelope); err != nil {
|
||||
return nil, fmt.Errorf(
|
||||
"reading %s is not a {\"value\"} envelope; the page and the host are "+
|
||||
"running different bundles: %w", entry, err)
|
||||
}
|
||||
return envelope.Value, nil
|
||||
}
|
||||
|
||||
// SetLastAction installs the previous step's action as state.lastAction inside
|
||||
// the page runtime. The page cannot derive it: only the runner knows which
|
||||
// action was actually applied. Without this call every web state.lastAction is
|
||||
// null, so a property gated on what the last action did is vacuously true and
|
||||
// reports a green run while checking nothing.
|
||||
//
|
||||
// The call is deliberately unguarded. A `setter && setter(...)` form evaluates
|
||||
// to undefined on a page whose runtime does not define the setter, and chromedp
|
||||
// reports that as success, so "the page cannot accept lastAction" would be
|
||||
// indistinguishable from "installed". That page is reachable: a run resolving
|
||||
// its web runtime from an older published @sanderling/spec would silently no-op
|
||||
// every step. Unguarded, the missing global throws and the run fails loudly.
|
||||
func (d *Driver) SetLastAction(ctx context.Context, encoded json.RawMessage) error {
|
||||
payload := strings.TrimSpace(string(encoded))
|
||||
if payload == "" {
|
||||
payload = "null"
|
||||
}
|
||||
script := fmt.Sprintf(`window.__sanderlingSetLastAction__(%s)`, payload)
|
||||
runCtx, cancel := d.runCtx(ctx)
|
||||
defer cancel()
|
||||
if err := chromedp.Run(runCtx, chromedp.Evaluate(script, nil)); err != nil {
|
||||
return fmt.Errorf("set last action: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// extractorScript resolves the extractor table once the page is not mid route
|
||||
// transition, giving up on that wait after %d ms.
|
||||
//
|
||||
// A missing table rejects rather than reporting {}, for the same reason
|
||||
// SetLastAction no longer guards its call: an empty override map is what a
|
||||
// spec with no extractors returns, so the guarded form made "this page has no
|
||||
// sanderling runtime" read as a normal step whose properties then ran on
|
||||
// goja's dump-derived values instead of the page's.
|
||||
const extractorScript = `
|
||||
new Promise((resolve, reject) => {
|
||||
const deadline = Date.now() + %d;` + liveScreensFunction + `
|
||||
const read = () => {
|
||||
if (liveScreens() > 1 && Date.now() < deadline) {
|
||||
setTimeout(read, 16);
|
||||
return;
|
||||
}
|
||||
if (typeof window.__sanderlingExtractors__ !== "function") {
|
||||
reject(new Error("__sanderlingExtractors__ is not installed in the page"));
|
||||
return;
|
||||
}
|
||||
resolve(JSON.stringify(window.__sanderlingExtractors__()));
|
||||
};
|
||||
read();
|
||||
})`
|
||||
|
||||
// NextActionFromV8 invokes the bundle-installed action generator and returns
|
||||
// the resulting Action JSON. Returns an empty json.RawMessage when the
|
||||
// generator declines to act this tick.
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -474,3 +475,493 @@ func TestLaunch_KeepsBrowserAliveAfterCallerContextEnds(t *testing.T) {
|
||||
t.Fatalf("second Launch after the first caller context ended: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestInputText_ReplacesTextInsideAShadowRoot pins ReplacesTextOnInput's promise
|
||||
// on the shape a canvas app actually has. Compose for Web draws its text fields
|
||||
// on a canvas and routes typing through a hidden <input> INSIDE the shadow root
|
||||
// it mounts, and document.activeElement stops at a shadow boundary: it names the
|
||||
// host. The select-all therefore ran against a <div> with no select(), every
|
||||
// InputText appended to the last, and a fuzzer typing twice into one field built
|
||||
// up text it could never clear (observed on the folio wasm build as
|
||||
// "0.0000001" -> "0.0000001\t-1").
|
||||
func TestInputText_ReplacesTextInsideAShadowRoot(t *testing.T) {
|
||||
const page = `<body><div id="app"></div><script>
|
||||
const root = document.getElementById("app").attachShadow({mode: "open"});
|
||||
root.innerHTML = ` + "`" + `
|
||||
<style>
|
||||
#surface { position: absolute; left: 0; top: 0; }
|
||||
#a11y { position: absolute; left: 0; top: 0; pointer-events: none; }
|
||||
#proxy { position: absolute; left: -9999px; }
|
||||
</style>
|
||||
<canvas id="surface" width="300" height="200"></canvas>
|
||||
<div id="a11y"><div id="field">-</div></div>
|
||||
<input id="proxy" type="text">` + "`" + `;
|
||||
const proxy = root.getElementById("proxy");
|
||||
const field = root.getElementById("field");
|
||||
// The canvas owns the pointer (the a11y overlay is pointer-events: none)
|
||||
// and hands focus to the proxy, exactly as a canvas app does.
|
||||
root.getElementById("surface").addEventListener("click", function () { proxy.focus(); });
|
||||
proxy.addEventListener("input", function () { field.textContent = proxy.value; });
|
||||
</script></body>`
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte(page))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL, false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
if err := d.Tap(ctx, 40, 40); err != nil {
|
||||
t.Fatalf("Tap: %v", err)
|
||||
}
|
||||
if err := d.InputText(ctx, "alpha"); err != nil {
|
||||
t.Fatalf("InputText: %v", err)
|
||||
}
|
||||
if err := d.InputText(ctx, "beta"); err != nil {
|
||||
t.Fatalf("InputText: %v", err)
|
||||
}
|
||||
|
||||
var shown string
|
||||
script := `document.getElementById("app").shadowRoot.getElementById("field").textContent`
|
||||
if err := chromedp.Run(d.tabCtx, chromedp.Evaluate(script, &shown)); err != nil {
|
||||
t.Fatalf("read field: %v", err)
|
||||
}
|
||||
if shown != "beta" {
|
||||
t.Errorf("field holds %q, want %q; the second InputText appended instead of replacing", shown, "beta")
|
||||
}
|
||||
|
||||
if err := d.EraseText(ctx, len("beta")); err != nil {
|
||||
t.Fatalf("EraseText: %v", err)
|
||||
}
|
||||
if err := chromedp.Run(d.tabCtx, chromedp.Evaluate(script, &shown)); err != nil {
|
||||
t.Fatalf("read field: %v", err)
|
||||
}
|
||||
if shown != "" {
|
||||
t.Errorf("field holds %q after EraseText, want empty", shown)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHierarchy_ScreenFallsBackToThePathname pins the route the goja host reads
|
||||
// off the dump. Reading location.hash alone reported "/" on every step of a
|
||||
// path-routed SPA (react-router's BrowserRouter, which the replay UI itself
|
||||
// uses), so every screen looked like the same screen and no route-scoped
|
||||
// property or action could tell them apart.
|
||||
func TestHierarchy_ScreenFallsBackToThePathname(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte(`<body><div id="app">app</div></body>`))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
for _, testCase := range []struct {
|
||||
name string
|
||||
path string
|
||||
want string
|
||||
}{
|
||||
{"path-routed", "/runs/20260101-120000/steps/7", "/runs/20260101-120000/steps/7"},
|
||||
{"hash wins when present", "/runs/1#/detail", "/detail"},
|
||||
{"root", "/", "/"},
|
||||
} {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL+testCase.path, false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
dump, err := d.Hierarchy(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("Hierarchy: %v", err)
|
||||
}
|
||||
var root struct {
|
||||
Attributes map[string]string `json:"attributes"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(dump), &root); err != nil {
|
||||
t.Fatalf("unmarshal hierarchy: %v", err)
|
||||
}
|
||||
if got := root.Attributes["sanderling-screen"]; got != testCase.want {
|
||||
t.Errorf("sanderling-screen: got %q, want %q", got, testCase.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestWaitForIdle_WaitsForWorkTheActionKickedOff pins the settle the runner
|
||||
// relies on between acting and observing. WaitForIdle used to return the moment
|
||||
// <body> existed, which is true before the app has reacted at all: measured on
|
||||
// the folio wasm build, Compose's accessibility DOM lands ~136 ms after an
|
||||
// InputText, so the next step read the pre-action text and typed into a field
|
||||
// it believed was still empty.
|
||||
func TestWaitForIdle_WaitsForWorkTheActionKickedOff(t *testing.T) {
|
||||
const page = `<body><div id="app"></div><script>
|
||||
const root = document.getElementById("app").attachShadow({mode: "open"});
|
||||
root.innerHTML = '<button id="go" style="width:200px;height:80px">go</button><div id="out">pending</div>';
|
||||
root.getElementById("go").addEventListener("click", function () {
|
||||
setTimeout(function () { root.getElementById("out").textContent = "settled"; }, 100);
|
||||
});
|
||||
</script></body>`
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte(page))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL, false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
if err := d.Tap(ctx, 40, 40); err != nil {
|
||||
t.Fatalf("Tap: %v", err)
|
||||
}
|
||||
if err := d.WaitForIdle(ctx, time.Second); err != nil {
|
||||
t.Fatalf("WaitForIdle: %v", err)
|
||||
}
|
||||
|
||||
var shown string
|
||||
script := `document.getElementById("app").shadowRoot.getElementById("out").textContent`
|
||||
if err := chromedp.Run(d.tabCtx, chromedp.Evaluate(script, &shown)); err != nil {
|
||||
t.Fatalf("read: %v", err)
|
||||
}
|
||||
if shown != "settled" {
|
||||
t.Errorf("observed %q; WaitForIdle returned before the tap's own work landed", shown)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWaitForIdle_ReturnsOnABusyPage is the other half: a page that never stops
|
||||
// mutating (an animation, a polling widget) must not hold the step loop open.
|
||||
func TestWaitForIdle_ReturnsOnABusyPage(t *testing.T) {
|
||||
const page = `<body><div id="tick">0</div><script>
|
||||
let n = 0;
|
||||
setInterval(function () { document.getElementById("tick").textContent = String(++n); }, 15);
|
||||
</script></body>`
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte(page))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL, false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
start := time.Now()
|
||||
if err := d.WaitForIdle(ctx, time.Second); err != nil {
|
||||
t.Fatalf("WaitForIdle: %v", err)
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed > 2*time.Second {
|
||||
t.Errorf("WaitForIdle took %s on a busy page; it must return inside its budget", elapsed)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWaitForIdle_WaitsOutARouteTransition covers the settle case a mutation
|
||||
// observer cannot see. A canvas app splices the incoming screen's
|
||||
// accessibility nodes in when its cross-fade STARTS and drops the outgoing
|
||||
// screen's when it ends; between those two mutations the DOM is quiet with both
|
||||
// routes live. Returning there hands the next step a tree naming the screen the
|
||||
// app is leaving, which on the folio wasm build recorded a submit that had
|
||||
// landed on Home as still being on the transaction screen: the route gate of an
|
||||
// action-gated property then skipped the very step the action landed on.
|
||||
//
|
||||
// The page keeps mutating for 300 ms after the route splice, and the settle
|
||||
// runs on the timeout production hands it (MinIdleTimeout). Both details are
|
||||
// load-bearing. A quiet page reaches the transition check immediately, so it
|
||||
// passes whether the transition window is anchored at the script start or at
|
||||
// the end of the quiet period; churn is what pushes the check past a
|
||||
// start-anchored deadline, which then finishes at once with two live screens.
|
||||
// And a caller timeout below MinIdleTimeout cuts the whole settle off before
|
||||
// the transition window can be spent, which is the same bug from the other end.
|
||||
func TestWaitForIdle_WaitsOutARouteTransition(t *testing.T) {
|
||||
const page = `<body><div id="app"></div><script>
|
||||
const root = document.getElementById("app").attachShadow({mode: "open"});
|
||||
root.innerHTML = '<button id="go" style="width:200px;height:80px">go</button>' +
|
||||
'<div id="LedgerScreen">ledger</div><div id="spinner">0</div>';
|
||||
root.getElementById("go").addEventListener("click", function () {
|
||||
const incoming = document.createElement("div");
|
||||
incoming.id = "HomeScreen";
|
||||
incoming.textContent = "home";
|
||||
root.appendChild(incoming);
|
||||
let frame = 0;
|
||||
const churn = setInterval(function () {
|
||||
root.getElementById("spinner").textContent = String(++frame);
|
||||
}, 30);
|
||||
setTimeout(function () { clearInterval(churn); }, 300);
|
||||
setTimeout(function () { root.getElementById("LedgerScreen").remove(); }, 1000);
|
||||
});
|
||||
</script></body>`
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte(page))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL, false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
if err := d.Tap(ctx, 40, 40); err != nil {
|
||||
t.Fatalf("Tap: %v", err)
|
||||
}
|
||||
if err := d.WaitForIdle(ctx, d.MinIdleTimeout()); err != nil {
|
||||
t.Fatalf("WaitForIdle: %v", err)
|
||||
}
|
||||
|
||||
var live []string
|
||||
script := `Array.from(document.getElementById("app").shadowRoot
|
||||
.querySelectorAll('[id$="Screen"]')).map(e => e.id)`
|
||||
if err := chromedp.Run(d.tabCtx, chromedp.Evaluate(script, &live)); err != nil {
|
||||
t.Fatalf("read: %v", err)
|
||||
}
|
||||
if len(live) != 1 || live[0] != "HomeScreen" {
|
||||
t.Errorf("live screens after the settle = %v, want [HomeScreen]; WaitForIdle "+
|
||||
"returned mid-transition, so the next step verifies the outgoing route", live)
|
||||
}
|
||||
}
|
||||
|
||||
// TestWaitForIdle_BoundsTheTransitionWait is the other half of the transition
|
||||
// wait: a page that shows two *Screen ids at rest is not mid-transition, it
|
||||
// just matches the heuristic, and it must cost one bounded wait rather than the
|
||||
// whole step budget on every step.
|
||||
func TestWaitForIdle_BoundsTheTransitionWait(t *testing.T) {
|
||||
const page = `<body><div id="HomeScreen">home</div><div id="LedgerScreen">ledger</div></body>`
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte(page))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL, false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
start := time.Now()
|
||||
if err := d.WaitForIdle(ctx, 10*time.Second); err != nil {
|
||||
t.Fatalf("WaitForIdle: %v", err)
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed > transitionSettlePeriod+time.Second {
|
||||
t.Errorf("WaitForIdle took %s on a page with two resting screens; the "+
|
||||
"transition wait must be bounded by %s", elapsed, transitionSettlePeriod)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEvaluateExtractors_WaitsOutARouteTransition covers the other sampler. A
|
||||
// step reads the page twice: the hierarchy dump (which re-fetches while the
|
||||
// tree looks transitional) and the spec's own extractors in V8. Sampling the
|
||||
// extractors mid cross-fade reports the route the app is leaving, and an
|
||||
// action-gated property then skips the one step its action can be judged on:
|
||||
// on the folio wasm build a double-submit that landed on Home was extracted as
|
||||
// still being on the transaction screen.
|
||||
func TestEvaluateExtractors_WaitsOutARouteTransition(t *testing.T) {
|
||||
const page = `<body><div id="app"></div><script>
|
||||
const root = document.getElementById("app").attachShadow({mode: "open"});
|
||||
root.innerHTML = '<div id="LedgerScreen">ledger</div>';
|
||||
window.__sanderlingExtractors__ = function () {
|
||||
return {0: {value: Array.from(root.querySelectorAll('[id$="Screen"]')).map(e => e.id).join(",")}};
|
||||
};
|
||||
window.startTransition = function () {
|
||||
const incoming = document.createElement("div");
|
||||
incoming.id = "HomeScreen";
|
||||
root.appendChild(incoming);
|
||||
setTimeout(function () { root.getElementById("LedgerScreen").remove(); }, 400);
|
||||
};
|
||||
</script></body>`
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte(page))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL, false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
if err := chromedp.Run(d.tabCtx, chromedp.Evaluate(`window.startTransition()`, nil)); err != nil {
|
||||
t.Fatalf("start transition: %v", err)
|
||||
}
|
||||
|
||||
values, err := d.EvaluateExtractors(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("EvaluateExtractors: %v", err)
|
||||
}
|
||||
if got := string(values[0]); got != `"HomeScreen"` {
|
||||
t.Errorf("extractor read %s, want \"HomeScreen\"; the extractors sampled "+
|
||||
"mid-transition, so the spec sees the route the app is leaving", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSetLastAction_ReportsAPageThatCannotTakeIt covers the install the whole
|
||||
// web path's action-gated properties hang off. A page without the setter is
|
||||
// reachable: internal/testrun resolves the web runtime from
|
||||
// node_modules/@sanderling/spec when no sibling checkout is present, and an
|
||||
// older published runtime does not define it. Guarded as
|
||||
// `setter && setter(...)`, that page returns undefined and chromedp reports
|
||||
// success, so every step silently no-ops and every property gated on the last
|
||||
// action goes vacuously true - a green run that checked nothing.
|
||||
func TestSetLastAction_ReportsAPageThatCannotTakeIt(t *testing.T) {
|
||||
const withSetter = `<body><script>
|
||||
window.__lastActionSeen = null;
|
||||
window.__sanderlingSetLastAction__ = function (value) { window.__lastActionSeen = value; };
|
||||
</script></body>`
|
||||
const withoutSetter = `<body><div id="app">no sanderling runtime here</div></body>`
|
||||
pages := map[string]string{"/with": withSetter, "/without": withoutSetter}
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte(pages[r.URL.Path]))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
if err := d.Launch(ctx, server.URL+"/with", false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
action := json.RawMessage(`{"kind":"Tap","on":"id:TxnSubmit"}`)
|
||||
if err := d.SetLastAction(ctx, action); err != nil {
|
||||
t.Fatalf("SetLastAction on a page that defines the setter: %v", err)
|
||||
}
|
||||
var seen map[string]string
|
||||
if err := chromedp.Run(d.tabCtx,
|
||||
chromedp.Evaluate(`window.__lastActionSeen`, &seen)); err != nil {
|
||||
t.Fatalf("read installed action: %v", err)
|
||||
}
|
||||
if seen["on"] != "id:TxnSubmit" {
|
||||
t.Errorf("the page received %v, want the action the runner applied", seen)
|
||||
}
|
||||
|
||||
if err := d.Launch(ctx, server.URL+"/without", false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
if err := d.SetLastAction(ctx, action); err == nil {
|
||||
t.Error("SetLastAction reported success on a page with no setter; " +
|
||||
"a runtime that cannot take lastAction is indistinguishable from one that did")
|
||||
}
|
||||
}
|
||||
|
||||
// TestEvaluateExtractors_ReportsAMissingTable is the same failure on the other
|
||||
// sampler. An empty override map is what a spec with no extractors returns, so
|
||||
// treating a missing table as {} makes "this page has no sanderling runtime"
|
||||
// read as an ordinary step - and the verifier then judges the run on goja's
|
||||
// dump-derived values while believing they came from the page.
|
||||
func TestEvaluateExtractors_ReportsAMissingTable(t *testing.T) {
|
||||
const page = `<body><div id="app">no sanderling runtime here</div></body>`
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte(page))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL, false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
values, err := d.EvaluateExtractors(ctx)
|
||||
if err == nil {
|
||||
t.Errorf("EvaluateExtractors returned %v and no error on a page with no "+
|
||||
"extractor table; a page that cannot be read must not read as empty", values)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEvaluateExtractors_KeepsUndefinedApartFromNull covers the wire the page's
|
||||
// readings cross. JSON has no undefined, so the web runtime wraps each reading
|
||||
// in a {value} envelope: written straight into the table, an extractor that
|
||||
// returned undefined lost its whole index to JSON.stringify and the host kept
|
||||
// goja's dump-derived reading for it while the rest held the page's. An absent
|
||||
// value has to arrive as an empty payload, which is what makes the verifier
|
||||
// record undefined (the value the native host records for the same getter);
|
||||
// arriving as JSON null would claim the getter returned null.
|
||||
func TestEvaluateExtractors_KeepsUndefinedApartFromNull(t *testing.T) {
|
||||
const page = `<body><script>
|
||||
window.__sanderlingExtractors__ = function () {
|
||||
return {0: {}, 1: {value: null}, 2: {value: {balance: 7}}};
|
||||
};
|
||||
</script></body>`
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte(page))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL, false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
|
||||
values, err := d.EvaluateExtractors(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("EvaluateExtractors: %v", err)
|
||||
}
|
||||
if len(values) != 3 {
|
||||
t.Fatalf("the page reported 3 readings, %d survived the wire: %v", len(values), values)
|
||||
}
|
||||
if got := values[0]; len(got) != 0 {
|
||||
t.Errorf("the undefined reading arrived as %s, want an empty payload; "+
|
||||
"the verifier records anything else as a value the getter never returned", got)
|
||||
}
|
||||
if got := string(values[1]); got != "null" {
|
||||
t.Errorf("the null reading arrived as %s, want null", got)
|
||||
}
|
||||
if got := string(values[2]); got != `{"balance":7}` {
|
||||
t.Errorf("the object reading arrived as %s, want {\"balance\":7}", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestEvaluateExtractors_RejectsAnUnenvelopedReading is the loud failure a page
|
||||
// running an older @sanderling/spec produces. Its readings are bare values, and
|
||||
// a bare value is indistinguishable from a reading whose getter returned that
|
||||
// value, so accepting them silently puts the two engines on different bundles.
|
||||
func TestEvaluateExtractors_RejectsAnUnenvelopedReading(t *testing.T) {
|
||||
const page = `<body><script>
|
||||
window.__sanderlingExtractors__ = function () { return {0: "home"}; };
|
||||
</script></body>`
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = w.Write([]byte(page))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL, false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
|
||||
values, err := d.EvaluateExtractors(ctx)
|
||||
if err == nil {
|
||||
t.Fatalf("EvaluateExtractors accepted %v from a page whose readings are not "+
|
||||
"enveloped; the page and the host are running different bundles", values)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "different bundles") {
|
||||
t.Errorf("EvaluateExtractors failed with %q, want it to name the bundle mismatch", err)
|
||||
}
|
||||
}
|
||||
@@ -60,30 +60,41 @@ type factRow struct {
|
||||
facts elementFacts
|
||||
}
|
||||
|
||||
// parityPages are the pages both producers are compared on. Each one must
|
||||
// exercise every fact both ways on its own (requireBothPolarities), so adding a
|
||||
// page never weakens the comparison. fact-parity-shadow.html is shaped like a
|
||||
// Compose for Web app: the whole UI lives inside a shadow root, which both
|
||||
// producers have to descend into or they enumerate one node for an entire app.
|
||||
var parityPages = []string{"fact-parity.html", "fact-parity-shadow.html"}
|
||||
|
||||
func TestHierarchy_DerivesTheSameFactsAsTheWebRuntime(t *testing.T) {
|
||||
server := httptest.NewServer(http.FileServer(http.Dir("testdata")))
|
||||
defer server.Close()
|
||||
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL+"/fact-parity.html", false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
for _, page := range parityPages {
|
||||
t.Run(page, func(t *testing.T) {
|
||||
d := New()
|
||||
defer d.Terminate(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
if err := d.Launch(ctx, server.URL+"/"+page, false, nil); err != nil {
|
||||
t.Fatalf("Launch: %v", err)
|
||||
}
|
||||
|
||||
dump, err := d.Hierarchy(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("Hierarchy: %v", err)
|
||||
}
|
||||
fromDump := factsFromHierarchyDump(t, dump)
|
||||
fromWebRuntime := factsFromWebRuntime(ctx, t, d)
|
||||
dump, err := d.Hierarchy(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("Hierarchy: %v", err)
|
||||
}
|
||||
fromDump := factsFromHierarchyDump(t, dump)
|
||||
fromWebRuntime := factsFromWebRuntime(ctx, t, d)
|
||||
|
||||
requireEveryElementNamed(t, "the hierarchy dump", fromDump)
|
||||
requireEveryElementNamed(t, "the web runtime", fromWebRuntime)
|
||||
requireBothPolarities(t, fromWebRuntime)
|
||||
compareEnumeratedElements(t, fromDump, fromWebRuntime)
|
||||
compareDerivedFacts(t, fromDump, fromWebRuntime)
|
||||
requireEveryElementNamed(t, "the hierarchy dump", fromDump)
|
||||
requireEveryElementNamed(t, "the web runtime", fromWebRuntime)
|
||||
requireBothPolarities(t, fromWebRuntime)
|
||||
compareEnumeratedElements(t, fromDump, fromWebRuntime)
|
||||
compareDerivedFacts(t, fromDump, fromWebRuntime)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// factsFromHierarchyDump reads the dump the way the goja host does: parse it,
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
<!doctype html>
|
||||
<html id="page">
|
||||
<head id="page-head">
|
||||
<meta id="page-charset" charset="utf-8" />
|
||||
<title id="page-title">fact parity: shadow dom</title>
|
||||
</head>
|
||||
<body id="page-body">
|
||||
<!-- Shaped like Compose for Web: one mount element owning a shadow root
|
||||
that holds the canvas plus an accessibility overlay whose nodes are
|
||||
pointer-events: none, so taps fall through to the canvas. A producer
|
||||
that stops at the shadow boundary reports this whole app as one node. -->
|
||||
<div id="host">
|
||||
<div id="host-light-child">light child, no slot to render into</div>
|
||||
</div>
|
||||
<script id="page-script">
|
||||
const root = document.getElementById("host").attachShadow({ mode: "open" });
|
||||
root.innerHTML = `
|
||||
<style id="shadow-style">
|
||||
#shadow-surface { position: absolute; inset: 0; }
|
||||
#shadow-overlay { position: absolute; inset: 0; pointer-events: none; }
|
||||
#shadow-scroller { width: 320px; height: 120px; overflow: auto; }
|
||||
#shadow-scroller-content { height: 600px; }
|
||||
</style>
|
||||
<canvas id="shadow-surface" width="420" height="640"></canvas>
|
||||
<div id="shadow-overlay">
|
||||
<button id="shadow-save">save</button>
|
||||
<button id="shadow-cancel" disabled>cancel</button>
|
||||
<input id="shadow-amount" type="text" value="10" />
|
||||
<div id="shadow-plain">plain</div>
|
||||
</div>
|
||||
<div id="shadow-scroller"><div id="shadow-scroller-content"></div></div>`;
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -56,6 +56,23 @@
|
||||
<textarea id="notes"></textarea>
|
||||
<div id="bio" contenteditable="true">bio</div>
|
||||
<div id="menu" role="button">menu</div>
|
||||
<!-- One element per ARIA role both producers resolve as clickable. A role
|
||||
covered on one side only makes that control reachable for one host,
|
||||
which is how the whole set is kept honest: role="option" is the shape
|
||||
the replay UI gives its step rows. -->
|
||||
<div id="role-link" role="link">link</div>
|
||||
<div id="role-checkbox" role="checkbox">checkbox</div>
|
||||
<div id="role-radio" role="radio">radio</div>
|
||||
<div id="role-switch" role="switch">switch</div>
|
||||
<div id="role-tab" role="tab">tab</div>
|
||||
<div id="role-menuitem" role="menuitem">menu item</div>
|
||||
<div id="role-menuitemcheckbox" role="menuitemcheckbox">menu item checkbox</div>
|
||||
<div id="role-menuitemradio" role="menuitemradio">menu item radio</div>
|
||||
<div id="role-treeitem" role="treeitem">tree item</div>
|
||||
<ul id="role-listbox" role="listbox">
|
||||
<li id="role-option" role="option">option</li>
|
||||
<li id="role-option-disabled" role="option" aria-disabled="true">disabled option</li>
|
||||
</ul>
|
||||
<div id="attribute-click" onclick="void 0">attribute click</div>
|
||||
<div id="delegating-root">delegating root</div>
|
||||
<div id="plain">plain</div>
|
||||
|
||||
@@ -109,6 +109,13 @@ func NewDevice(ctx context.Context, options DeviceOptions) (*Driver, error) {
|
||||
d.restart = d.respawnDevice
|
||||
d.processContext, d.processCancel = context.WithCancel(ctx)
|
||||
|
||||
lock, err := acquireDeviceLock(d.udid)
|
||||
if err != nil {
|
||||
d.processCancel()
|
||||
return nil, err
|
||||
}
|
||||
d.deviceLock = lock
|
||||
|
||||
if err := d.bringUpDevice(ctx); err != nil {
|
||||
d.Close()
|
||||
return nil, err
|
||||
|
||||
@@ -42,6 +42,17 @@ const runnerStartupTimeout = 120 * time.Second
|
||||
// before it is killed. A variable so the kill-escalation test can shrink it.
|
||||
var shutdownGrace = 15 * time.Second
|
||||
|
||||
// launchTimeout bounds a single app lifecycle RPC. The runner serves lifecycle
|
||||
// inside its XCTest session, and a launch the simulator rejects sends that
|
||||
// session down a recovery chain (a 120s accessibility wait, a spindump, then an
|
||||
// idle wait) that answers minutes late or never. Callers reach Launch with an
|
||||
// undeadlined context, since it runs before the run's duration clock starts, so
|
||||
// the bound has to come from here or a wedged session hangs the run with no
|
||||
// trace, no error, and no end. Kept under runnerStartupTimeout: launching an
|
||||
// app inside a live session must cost less than cold-starting that session.
|
||||
// A variable so the timeout test can shrink it.
|
||||
var launchTimeout = 90 * time.Second
|
||||
|
||||
// longPressHoldMilliseconds is how long LongPress holds the finger down.
|
||||
const longPressHoldMilliseconds = 600
|
||||
|
||||
@@ -141,6 +152,34 @@ type Driver struct {
|
||||
// the moment startup finishes.
|
||||
processContext context.Context
|
||||
processCancel context.CancelFunc
|
||||
|
||||
// deviceLock is the exclusive claim on the target, held for the driver's
|
||||
// whole life and released by Close.
|
||||
deviceLock io.Closer
|
||||
}
|
||||
|
||||
// acquireDeviceLock takes an exclusive advisory lock on the target so only one
|
||||
// run drives it at a time. Two runs on one device interleave app lifecycle: the
|
||||
// second run's uninstall and reinstall land under the first's live automation
|
||||
// session, leaving its app proxies bound to a bundle the simulator no longer
|
||||
// knows, and every later snapshot and launch on that session stalls. Failing
|
||||
// fast beats recovering silently, since the other run owns the device and would
|
||||
// be corrupted either way. The lock lives on the file descriptor, so a crashed
|
||||
// run's claim is released by the kernel and never strands the device.
|
||||
func acquireDeviceLock(udid string) (io.Closer, error) {
|
||||
path := filepath.Join(os.TempDir(), "sanderling-ios-"+udid+".lock")
|
||||
file, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0o644)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open device lock %s: %w", path, err)
|
||||
}
|
||||
if err := syscall.Flock(int(file.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil {
|
||||
file.Close()
|
||||
return nil, fmt.Errorf(
|
||||
"ios target %s is already driven by another sanderling run (lock %s); "+
|
||||
"wait for that run to finish or point this one at a different device with --ios-device",
|
||||
udid, path)
|
||||
}
|
||||
return file, nil
|
||||
}
|
||||
|
||||
// New extracts the embedded companion, spawns it against the configured
|
||||
@@ -199,7 +238,15 @@ func New(ctx context.Context, options Options) (*Driver, error) {
|
||||
driverInstance.grantPaste = driverInstance.grantPasteboardAccess
|
||||
driverInstance.processContext, driverInstance.processCancel = context.WithCancel(ctx)
|
||||
|
||||
lock, err := acquireDeviceLock(driverInstance.udid)
|
||||
if err != nil {
|
||||
driverInstance.processCancel()
|
||||
return nil, err
|
||||
}
|
||||
driverInstance.deviceLock = lock
|
||||
|
||||
if err := driverInstance.bringUp(ctx); err != nil {
|
||||
driverInstance.Close()
|
||||
return nil, err
|
||||
}
|
||||
if driverInstance.hybrid {
|
||||
@@ -408,7 +455,9 @@ func (d *Driver) Launch(ctx context.Context, bundleID string, clearState bool, e
|
||||
|
||||
// Terminate first so the launch is a clean cold start regardless of the
|
||||
// app's prior state. A not-running app is not an error here.
|
||||
_ = d.withRecovery(ctx, func() error { return d.lifecycleCompanion().Terminate(ctx, d.bundleID) })
|
||||
_ = d.lifecycleCall(ctx, func(callCtx context.Context, companion transport.Companion) error {
|
||||
return companion.Terminate(callCtx, d.bundleID)
|
||||
})
|
||||
|
||||
if clearState {
|
||||
if err := d.clearAppState(ctx); err != nil {
|
||||
@@ -428,14 +477,26 @@ func (d *Driver) Launch(ctx context.Context, bundleID string, clearState bool, e
|
||||
}
|
||||
}
|
||||
|
||||
if err := d.withRecovery(ctx, func() error {
|
||||
return d.lifecycleCompanion().Launch(ctx, d.bundleID, true)
|
||||
if err := d.lifecycleCall(ctx, func(callCtx context.Context, companion transport.Companion) error {
|
||||
return companion.Launch(callCtx, d.bundleID, true)
|
||||
}); err != nil {
|
||||
return fmt.Errorf("launch %s: %w", d.bundleID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// lifecycleCall runs an app lifecycle RPC against lifecycleCompanion under a
|
||||
// launchTimeout-bounded context, with the usual one-restart recovery. The
|
||||
// companion is resolved inside the retry so a restart's replacement client
|
||||
// serves the second attempt.
|
||||
func (d *Driver) lifecycleCall(ctx context.Context, call func(context.Context, transport.Companion) error) error {
|
||||
boundedCtx, cancel := context.WithTimeout(ctx, launchTimeout)
|
||||
defer cancel()
|
||||
return d.withRecovery(boundedCtx, func() error {
|
||||
return call(boundedCtx, d.lifecycleCompanion())
|
||||
})
|
||||
}
|
||||
|
||||
// lifecycleCompanion is the transport that owns app launch and terminate: the
|
||||
// in-simulator runner when the hybrid is active, otherwise the legacy
|
||||
// companion. Lifecycle performed outside the runner's automation session
|
||||
@@ -528,7 +589,9 @@ func (d *Driver) resetDataContainer(ctx context.Context) error {
|
||||
}
|
||||
|
||||
func (d *Driver) Terminate(ctx context.Context) error {
|
||||
return d.withRecovery(ctx, func() error { return d.lifecycleCompanion().Terminate(ctx, d.bundleID) })
|
||||
return d.lifecycleCall(ctx, func(callCtx context.Context, companion transport.Companion) error {
|
||||
return companion.Terminate(callCtx, d.bundleID)
|
||||
})
|
||||
}
|
||||
|
||||
func (d *Driver) Tap(ctx context.Context, x, y int) error {
|
||||
@@ -999,6 +1062,10 @@ func (d *Driver) Close() {
|
||||
if d.processCancel != nil {
|
||||
d.processCancel()
|
||||
}
|
||||
if d.deviceLock != nil {
|
||||
_ = d.deviceLock.Close()
|
||||
d.deviceLock = nil
|
||||
}
|
||||
}
|
||||
|
||||
// stopTunnel closes the in-process usbmux forwarder on the device path. Closing
|
||||
|
||||
@@ -851,3 +851,175 @@ func (s *sequencedDumpCompanion) AccessibilityInfo(context.Context) (string, err
|
||||
*s.reads++
|
||||
return s.dumps[index], nil
|
||||
}
|
||||
|
||||
// wedgedLifecycleCompanion never answers a lifecycle RPC, standing in for a
|
||||
// runner whose XCTest session is stuck inside a rejected launch.
|
||||
type wedgedLifecycleCompanion struct {
|
||||
fakeCompanion
|
||||
release chan struct{}
|
||||
}
|
||||
|
||||
func (w *wedgedLifecycleCompanion) block(ctx context.Context) error {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-w.release:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func (w *wedgedLifecycleCompanion) Launch(ctx context.Context, _ string, _ bool) error {
|
||||
return w.block(ctx)
|
||||
}
|
||||
|
||||
func (w *wedgedLifecycleCompanion) Terminate(ctx context.Context, _ string) error {
|
||||
return w.block(ctx)
|
||||
}
|
||||
|
||||
// TestLaunchBoundsWedgedLifecycleRPC proves the launch path carries its own
|
||||
// deadline. Callers hand Launch an undeadlined context, so without one a runner
|
||||
// that never answers hangs the run forever with nothing printed.
|
||||
func TestLaunchBoundsWedgedLifecycleRPC(t *testing.T) {
|
||||
previous := launchTimeout
|
||||
launchTimeout = 100 * time.Millisecond
|
||||
defer func() { launchTimeout = previous }()
|
||||
|
||||
companion := &wedgedLifecycleCompanion{release: make(chan struct{})}
|
||||
defer close(companion.release)
|
||||
d := newTestDriver(companion)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- d.Launch(context.Background(), "com.example.app", false, nil) }()
|
||||
|
||||
select {
|
||||
case err := <-done:
|
||||
if err == nil {
|
||||
t.Fatal("wedged launch returned nil; a stuck runner must surface an error")
|
||||
}
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("err = %v, want a deadline-exceeded error", err)
|
||||
}
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("Launch never returned: the lifecycle RPC is unbounded, so a stuck runner hangs the run forever")
|
||||
}
|
||||
}
|
||||
|
||||
// TestTerminateBoundsWedgedLifecycleRPC covers the same bound on the standalone
|
||||
// terminate, which the runner calls mid-run on an equally stuck session.
|
||||
func TestTerminateBoundsWedgedLifecycleRPC(t *testing.T) {
|
||||
previous := launchTimeout
|
||||
launchTimeout = 100 * time.Millisecond
|
||||
defer func() { launchTimeout = previous }()
|
||||
|
||||
companion := &wedgedLifecycleCompanion{release: make(chan struct{})}
|
||||
defer close(companion.release)
|
||||
d := newTestDriver(companion)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- d.Terminate(context.Background()) }()
|
||||
|
||||
select {
|
||||
case err := <-done:
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("err = %v, want a deadline-exceeded error", err)
|
||||
}
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("Terminate never returned: the lifecycle RPC is unbounded")
|
||||
}
|
||||
}
|
||||
|
||||
// TestLaunchLeavesATighterCallerDeadlineAlone confirms the bound narrows the
|
||||
// caller's context and never widens it, so a caller that wants to give up
|
||||
// sooner still does.
|
||||
func TestLaunchLeavesATighterCallerDeadlineAlone(t *testing.T) {
|
||||
previous := launchTimeout
|
||||
launchTimeout = 30 * time.Second
|
||||
defer func() { launchTimeout = previous }()
|
||||
|
||||
companion := &wedgedLifecycleCompanion{release: make(chan struct{})}
|
||||
defer close(companion.release)
|
||||
d := newTestDriver(companion)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
start := time.Now()
|
||||
if err := d.Launch(ctx, "com.example.app", false, nil); !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("err = %v, want a deadline-exceeded error", err)
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed > 5*time.Second {
|
||||
t.Fatalf("Launch took %v; the driver's bound overrode the caller's tighter deadline", elapsed)
|
||||
}
|
||||
}
|
||||
|
||||
// newLockTestOptions builds New options that dial a seamed companion, so the
|
||||
// device-lock tests exercise New without spawning anything.
|
||||
func newLockTestOptions(t *testing.T, udid string) Options {
|
||||
t.Helper()
|
||||
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { listener.Close() })
|
||||
go func() {
|
||||
for {
|
||||
connection, acceptErr := listener.Accept()
|
||||
if acceptErr != nil {
|
||||
return
|
||||
}
|
||||
_ = connection.Close()
|
||||
}
|
||||
}()
|
||||
return Options{
|
||||
UniqueDeviceIdentifier: udid,
|
||||
pickAddress: func() (string, error) { return listener.Addr().String(), nil },
|
||||
spawnChild: func(context.Context, string) (*exec.Cmd, error) { return &exec.Cmd{}, nil },
|
||||
dialCompanion: func(string) (transport.Companion, error) {
|
||||
return &fakeCompanion{accessibilityJSON: "[]"}, nil
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewRejectsConcurrentRunOnSameDevice proves a second run cannot claim a
|
||||
// device the first is driving. Two runs interleave app lifecycle on one
|
||||
// simulator: the second's reinstall lands under the first's automation session
|
||||
// and wedges it. Failing fast names the contended device; the claim is released
|
||||
// on Close so the next run is not locked out.
|
||||
func TestNewRejectsConcurrentRunOnSameDevice(t *testing.T) {
|
||||
t.Setenv("SANDERLING_SIMULATOR_COMPANION", "legacy")
|
||||
udid := "LOCK-TEST-" + t.Name()
|
||||
|
||||
first, err := New(context.Background(), newLockTestOptions(t, udid))
|
||||
if err != nil {
|
||||
t.Fatalf("first New: %v", err)
|
||||
}
|
||||
|
||||
_, err = New(context.Background(), newLockTestOptions(t, udid))
|
||||
if err == nil {
|
||||
t.Fatal("second run claimed a device the first still drives; concurrent runs corrupt each other's session")
|
||||
}
|
||||
if !strings.Contains(err.Error(), udid) {
|
||||
t.Fatalf("err = %v, want it to name the contended device %s", err, udid)
|
||||
}
|
||||
|
||||
first.Close()
|
||||
third, err := New(context.Background(), newLockTestOptions(t, udid))
|
||||
if err != nil {
|
||||
t.Fatalf("device stayed locked after Close: %v", err)
|
||||
}
|
||||
third.Close()
|
||||
}
|
||||
|
||||
// TestAcquireDeviceLockKeepsDistinctDevicesIndependent guards against a lock
|
||||
// path that ignores the udid and serializes unrelated runs.
|
||||
func TestAcquireDeviceLockKeepsDistinctDevicesIndependent(t *testing.T) {
|
||||
first, err := acquireDeviceLock("LOCK-TEST-DEVICE-A")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer first.Close()
|
||||
second, err := acquireDeviceLock("LOCK-TEST-DEVICE-B")
|
||||
if err != nil {
|
||||
t.Fatalf("a second device was refused while another was locked: %v", err)
|
||||
}
|
||||
defer second.Close()
|
||||
}
|
||||
@@ -129,6 +129,9 @@ func mapElement(element *rawElement) (treeNode, bool) {
|
||||
attributes["hintText"] = label
|
||||
} else if label != "" {
|
||||
attributes["accessibilityText"] = label
|
||||
if labelIsDisplayedText(element.Type) {
|
||||
attributes["text"] = label
|
||||
}
|
||||
}
|
||||
|
||||
enabled := element.Enabled
|
||||
@@ -150,6 +153,22 @@ func isEditable(elementType string) bool {
|
||||
return elementType == "TextArea" || elementType == "TextField"
|
||||
}
|
||||
|
||||
// labelIsDisplayedText reports whether an element type's AXLabel is the string
|
||||
// drawn on screen rather than an accessibility annotation about it. Only
|
||||
// StaticText qualifies: a text element's label IS what it renders, so it
|
||||
// belongs in `text`, matching a TextView on Android and a text node on web.
|
||||
//
|
||||
// Buttons and images are deliberately excluded even though a titled button's
|
||||
// label is also its visible title. The snapshot cannot tell that button apart
|
||||
// from an icon-only one whose label exists purely for VoiceOver, nor from a
|
||||
// container whose label is a comma-joined reading of its children ("CH,
|
||||
// Checking, $0.00, 0 transactions"). Inventing `text` for those would put
|
||||
// strings in `text` that no user can read, and would diverge from Android,
|
||||
// which leaves `text` empty and reports a contentDescription as `description`.
|
||||
func labelIsDisplayedText(elementType string) bool {
|
||||
return elementType == "StaticText"
|
||||
}
|
||||
|
||||
func stringValue(pointer *string) string {
|
||||
if pointer == nil {
|
||||
return ""
|
||||
|
||||
@@ -132,6 +132,79 @@ func TestNonEmptyValueMapsToText(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestStaticTextLabelMapsToText(t *testing.T) {
|
||||
dump := `[{"type":"StaticText","frame":{"x":0,"y":0,"width":50,"height":18},
|
||||
"AXLabel":"$239.00","AXValue":null,"enabled":true}]`
|
||||
element := parseSingle(t, dump)
|
||||
// A StaticText renders its label, so a spec reading .text must see it.
|
||||
if element.Text != "$239.00" {
|
||||
t.Fatalf("text = %q, want $239.00", element.Text)
|
||||
}
|
||||
// The raw label stays available: desc:/label:/content-desc: selectors and
|
||||
// the settle hash read it on the iOS path.
|
||||
if element.Description != "$239.00" {
|
||||
t.Fatalf("description = %q, want $239.00", element.Description)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNonTextLabelStaysDescriptionOnly(t *testing.T) {
|
||||
// An icon-only button's label is a VoiceOver annotation, and a container's
|
||||
// is a comma-joined reading of its children. Neither is on screen, so
|
||||
// neither may become text; Android reports both as description too.
|
||||
cases := []struct{ elementType, label string }{
|
||||
{"Button", "Log out"},
|
||||
{"Image", "Avatar"},
|
||||
{"Other", "CH, Checking, $0.00, 0 transactions"},
|
||||
}
|
||||
for _, testCase := range cases {
|
||||
dump := `[{"type":"` + testCase.elementType + `","frame":{"x":0,"y":0,"width":10,"height":10},
|
||||
"AXLabel":"` + testCase.label + `","AXValue":null,"enabled":true}]`
|
||||
element := parseSingle(t, dump)
|
||||
if element.Text != "" {
|
||||
t.Errorf("%s text = %q, want empty", testCase.elementType, element.Text)
|
||||
}
|
||||
if element.Description != testCase.label {
|
||||
t.Errorf("%s description = %q, want %q", testCase.elementType, element.Description, testCase.label)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Mirrors the folio Home screen as the companion actually dumps it: each
|
||||
// AccountCard is a Button carrying a merged VoiceOver label, with the name and
|
||||
// balance as StaticText siblings that the flat tree re-parents by containment.
|
||||
// Reading a card's balance is what folio's totalBalance extractor does, and it
|
||||
// read nothing on iOS until StaticText labels became text.
|
||||
func TestGoldenHomeCardBalancesAreReadableAsText(t *testing.T) {
|
||||
tree := mapAndParse(t, readDump(t, "home-cards-describe.json"), 402, 874)
|
||||
|
||||
cards := tree.FindAllNodes("id:HomeScreen > id:AccountCard")
|
||||
if len(cards) != 2 {
|
||||
t.Fatalf("account cards = %d, want 2", len(cards))
|
||||
}
|
||||
want := []struct{ name, balance string }{{"Checking", "$12.34"}, {"Savings", "$500.00"}}
|
||||
for i, card := range cards {
|
||||
balance := card.Find("id:AccountBalance")
|
||||
if balance == nil {
|
||||
t.Fatalf("card %d: id:AccountBalance did not resolve", i)
|
||||
}
|
||||
if balance.Text != want[i].balance {
|
||||
t.Errorf("card %d: balance text = %q, want %q", i, balance.Text, want[i].balance)
|
||||
}
|
||||
name := card.Find("id:AccountName")
|
||||
if name == nil {
|
||||
t.Fatalf("card %d: id:AccountName did not resolve", i)
|
||||
}
|
||||
if name.Text != want[i].name {
|
||||
t.Errorf("card %d: name text = %q, want %q", i, name.Text, want[i].name)
|
||||
}
|
||||
// The card's own merged label is not visible text; the web fallback in
|
||||
// the folio spec parses cardText and must not see a VoiceOver reading.
|
||||
if card.Element.Text != "" {
|
||||
t.Errorf("card %d: card text = %q, want empty", i, card.Element.Text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEditableAndClickableFlags(t *testing.T) {
|
||||
cases := []struct {
|
||||
elementType string
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
[
|
||||
{"type":"Application","frame":{"x":0,"y":0,"width":402,"height":874},"enabled":true,"AXLabel":"Folio","AXValue":null,"AXUniqueId":null},
|
||||
{"type":"Other","frame":{"x":0,"y":62,"width":402,"height":778},"enabled":true,"AXLabel":null,"AXValue":null,"AXUniqueId":"HomeScreen"},
|
||||
{"type":"StaticText","frame":{"x":20,"y":76,"width":97,"height":24},"enabled":true,"AXLabel":"Accounts","AXValue":null,"AXUniqueId":null},
|
||||
{"type":"Button","frame":{"x":340,"y":71,"width":48,"height":48},"enabled":true,"AXLabel":"Log out","AXValue":null,"AXUniqueId":"LogoutButton"},
|
||||
{"type":"Button","frame":{"x":20,"y":130,"width":362,"height":72},"enabled":true,"AXLabel":"CH, Checking, $12.34, 0 transactions","AXValue":null,"AXUniqueId":"AccountCard"},
|
||||
{"type":"StaticText","frame":{"x":90,"y":150,"width":74,"height":18},"enabled":true,"AXLabel":"Checking","AXValue":null,"AXUniqueId":"AccountName"},
|
||||
{"type":"StaticText","frame":{"x":313,"y":157,"width":53,"height":18},"enabled":true,"AXLabel":"$12.34","AXValue":null,"AXUniqueId":"AccountBalance"},
|
||||
{"type":"Button","frame":{"x":20,"y":210,"width":362,"height":72},"enabled":true,"AXLabel":"SA, Savings, $500.00, 2 transactions","AXValue":null,"AXUniqueId":"AccountCard"},
|
||||
{"type":"StaticText","frame":{"x":90,"y":230,"width":66,"height":18},"enabled":true,"AXLabel":"Savings","AXValue":null,"AXUniqueId":"AccountName"},
|
||||
{"type":"StaticText","frame":{"x":306,"y":237,"width":60,"height":18},"enabled":true,"AXLabel":"$500.00","AXValue":null,"AXUniqueId":"AccountBalance"},
|
||||
{"type":"StaticText","frame":{"x":20,"y":716,"width":106,"height":14},"enabled":true,"AXLabel":"TOTAL BALANCE","AXValue":null,"AXUniqueId":null},
|
||||
{"type":"StaticText","frame":{"x":20,"y":731,"width":85,"height":33},"enabled":true,"AXLabel":"$512.34","AXValue":null,"AXUniqueId":null},
|
||||
{"type":"Button","frame":{"x":20,"y":777,"width":362,"height":48},"enabled":true,"AXLabel":"+ Add account","AXValue":null,"AXUniqueId":"AddAccountButton"},
|
||||
{"type":"StaticText","frame":{"x":141,"y":792,"width":121,"height":18},"enabled":true,"AXLabel":"+ Add account","AXValue":null,"AXUniqueId":null}
|
||||
]
|
||||
+81
-19
@@ -31,6 +31,11 @@ type Options struct {
|
||||
// positive value stops the loop once that many steps have run.
|
||||
MaxSteps int
|
||||
|
||||
// StopOnViolation ends the step loop as soon as a step records a
|
||||
// violation, so a run that exists to find one bug stops at the evidence
|
||||
// instead of spending the rest of its budget past it.
|
||||
StopOnViolation bool
|
||||
|
||||
BundleID string
|
||||
Driver driver.DeviceDriver
|
||||
Verifier *verifier.Verifier
|
||||
@@ -69,6 +74,7 @@ func Run(ctx context.Context, options Options) (Summary, error) {
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
options.IdleTimeout = resolveIdleTimeout(options)
|
||||
|
||||
// Gate on the app actually being on top before acting, so the first
|
||||
// action never fires against a leftover screen or a system dialog. Done
|
||||
@@ -82,6 +88,7 @@ func Run(ctx context.Context, options Options) (Summary, error) {
|
||||
if err != nil {
|
||||
return Summary{}, err
|
||||
}
|
||||
_, pageExtractors := extractorSource.(webSource)
|
||||
|
||||
summary := Summary{StartTime: time.Now()}
|
||||
deadline := summary.StartTime.Add(options.Duration)
|
||||
@@ -117,9 +124,8 @@ func Run(ctx context.Context, options Options) (Summary, error) {
|
||||
var logs []verifier.LogEntry
|
||||
|
||||
// gctx is bound to the errgroup so a returned error (or outer
|
||||
// cancellation) propagates to siblings - notably the V8 extractor
|
||||
// goroutine, whose CDP round-trip can otherwise outrun the step
|
||||
// budget on a hung tab.
|
||||
// cancellation) propagates to every sibling read rather than leaving
|
||||
// one blocked on a hung device.
|
||||
g, gctx := errgroup.WithContext(ctx)
|
||||
si := stepIndex
|
||||
// fetchSyncedState issues a single Snapshot RPC so hierarchy and
|
||||
@@ -138,16 +144,6 @@ func Run(ctx context.Context, options Options) (Summary, error) {
|
||||
logs = collectLogs(gctx, options.Driver, logSince)
|
||||
return nil
|
||||
})
|
||||
var v8Overrides map[int]json.RawMessage
|
||||
g.Go(func() error {
|
||||
overrides, err := extractorSource.ExtractorOverrides(gctx)
|
||||
if err != nil {
|
||||
logger.Warn("v8 extractor evaluation failed", "step", si, "err", err)
|
||||
return nil
|
||||
}
|
||||
v8Overrides = overrides
|
||||
return nil
|
||||
})
|
||||
// All goroutines write to local variables and return nil, so the Wait
|
||||
// error is always nil; ignored intentionally.
|
||||
_ = g.Wait()
|
||||
@@ -190,6 +186,29 @@ func Run(ctx context.Context, options Options) (Summary, error) {
|
||||
var witnesses map[string]trace.Witness
|
||||
skippedVerification := false
|
||||
if !transitional {
|
||||
// The page-side extractors evaluate only on steps the verifier will
|
||||
// accept, which is why this read waits for the tree instead of
|
||||
// racing it. A spec's extractor getters carry state across steps
|
||||
// (folio's last-seen Home total, its submit counters) and that state
|
||||
// advances every time they run: evaluating them on a step whose
|
||||
// values are then thrown away leaves the page one window ahead of
|
||||
// the verifier, so the next accepted pair brackets two committed
|
||||
// transactions while having counted one submit, and the property
|
||||
// convicts a healthy app. It costs the latency the read used to hide
|
||||
// behind the hierarchy fetch; the fetch is what decides whether this
|
||||
// step counts at all, so it has to go first.
|
||||
//
|
||||
// lastAction is the same value PushSnapshot hands the goja state
|
||||
// below: the two engines evaluate this step against one action.
|
||||
v8Overrides, overridesErr := extractorSource.ExtractorOverrides(ctx, lastAction)
|
||||
if overridesErr != nil {
|
||||
// Not a warning. Without the page's values this step's
|
||||
// extractors keep goja's dump-derived readings while the
|
||||
// previous step holds the page's, and a delta property then
|
||||
// compares two producers and fires on an app that did nothing
|
||||
// wrong.
|
||||
return summary, fmt.Errorf("step %d extractor overrides: %w", stepIndex, overridesErr)
|
||||
}
|
||||
if err := options.Verifier.PushSnapshot(verifier.SnapshotInput{
|
||||
Tree: tree,
|
||||
ScreenshotPNG: screenshotPNG,
|
||||
@@ -201,13 +220,26 @@ func Run(ctx context.Context, options Options) (Summary, error) {
|
||||
}); err != nil {
|
||||
return summary, fmt.Errorf("step %d push: %w", stepIndex, err)
|
||||
}
|
||||
// Every failure below leaves some extractors holding the page's
|
||||
// value and the rest holding goja's reading of the dump, and a
|
||||
// property comparing previous to current across that split fires
|
||||
// on a healthy app. Each also means the two engines loaded
|
||||
// different bundles, which nothing downstream can reconcile.
|
||||
if pageExtractors && len(v8Overrides) != options.Verifier.ExtractorCount() {
|
||||
return summary, fmt.Errorf(
|
||||
"step %d: the page reported values for %d of the spec's %d extractors; "+
|
||||
"the page and the host are running different bundles",
|
||||
stepIndex, len(v8Overrides), options.Verifier.ExtractorCount())
|
||||
}
|
||||
skipped, overrideErr := options.Verifier.OverrideExtractorValues(v8Overrides)
|
||||
if overrideErr != nil {
|
||||
logger.Warn("v8 override apply failed", "step", stepIndex, "err", overrideErr)
|
||||
return summary, fmt.Errorf("step %d apply extractor overrides: %w", stepIndex, overrideErr)
|
||||
}
|
||||
if skipped > 0 {
|
||||
logger.Warn("v8 override skipped out-of-range entries",
|
||||
"step", stepIndex, "skipped", skipped, "have", len(v8Overrides))
|
||||
return summary, fmt.Errorf(
|
||||
"step %d: %d of %d extractor overrides fell outside the spec's extractor list; "+
|
||||
"the page and the host are running different bundles",
|
||||
stepIndex, skipped, len(v8Overrides))
|
||||
}
|
||||
options.Verifier.EvaluateProperties()
|
||||
violations = options.Verifier.NewlyViolatedProperties()
|
||||
@@ -296,6 +328,12 @@ func Run(ctx context.Context, options Options) (Summary, error) {
|
||||
summary.Steps = stepIndex
|
||||
if len(violations) > 0 {
|
||||
summary.Violations = append(summary.Violations, violationRecords(violations, witnesses, stepIndex)...)
|
||||
// The step is already written, so the trace ends on the state that
|
||||
// produced the violation. Finalize below still runs, so pending
|
||||
// liveness obligations are reported alongside it.
|
||||
if options.StopOnViolation {
|
||||
break
|
||||
}
|
||||
}
|
||||
// Wait actions are themselves a settling: skip the idle poll. Actions
|
||||
// that mutate the UI fall through to WaitForIdle so the next step's
|
||||
@@ -370,12 +408,36 @@ func validate(options Options) error {
|
||||
if options.Duration <= 0 {
|
||||
return errors.New("runner: Duration must be positive")
|
||||
}
|
||||
if options.IdleTimeout <= 0 {
|
||||
options.IdleTimeout = 2 * time.Second
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// defaultIdleTimeout is the settle budget a caller that names none gets.
|
||||
const defaultIdleTimeout = 2 * time.Second
|
||||
|
||||
// idleTimeoutFloor is a driver that knows how long its own settle can take.
|
||||
// Declared here rather than in the driver package (like lastActionInstaller in
|
||||
// source.go) so the mobile drivers stay untouched.
|
||||
type idleTimeoutFloor interface {
|
||||
MinIdleTimeout() time.Duration
|
||||
}
|
||||
|
||||
// resolveIdleTimeout settles the per-step settle budget: the caller's value,
|
||||
// defaulted when unset, and raised to whatever the driver says its own settle
|
||||
// needs. The chrome driver's settle waits for the DOM to go quiet and only then
|
||||
// opens its route-transition window; handed less than their sum it is cut off
|
||||
// mid-transition, and the step samples the screen the app is leaving. A driver
|
||||
// that reports no floor keeps the caller's value exactly.
|
||||
func resolveIdleTimeout(options Options) time.Duration {
|
||||
timeout := options.IdleTimeout
|
||||
if timeout <= 0 {
|
||||
timeout = defaultIdleTimeout
|
||||
}
|
||||
if floor, ok := options.Driver.(idleTimeoutFloor); ok {
|
||||
timeout = max(timeout, floor.MinIdleTimeout())
|
||||
}
|
||||
return timeout
|
||||
}
|
||||
|
||||
// ensureForeground keeps the app under test in the foreground. When the driver
|
||||
// can report the foreground app and it no longer matches the bundle under test,
|
||||
// the app is relaunched. Returns true when a relaunch happened so the caller
|
||||
|
||||
@@ -1976,3 +1976,99 @@ func TestRunner_SkipsActionWhenOverlayStealsFocusAtApplyTime(t *testing.T) {
|
||||
t.Error("apply-time guard failed: a tap fired while a system overlay held focus")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunner_StopOnViolationEndsAtTheFirstViolation pins the gate CI runs on:
|
||||
// a step budget of 8 against a spec that only violates on the third step must
|
||||
// end on step 3 and write nothing after it, so the trace's last state is the
|
||||
// one that produced the violation.
|
||||
func TestRunner_StopOnViolationEndsAtTheFirstViolation(t *testing.T) {
|
||||
const thirdStepViolationSpec = `
|
||||
import { actions, always, extract } from "@sanderling/spec";
|
||||
let observed = 0;
|
||||
const tick = extract(() => ++observed);
|
||||
globalThis.properties = {
|
||||
staysUnderThree: always(() => tick.current < 3),
|
||||
};
|
||||
globalThis.actions = actions(() => []);
|
||||
`
|
||||
state := newHarnessWithSpec(t, thirdStepViolationSpec)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
summary, err := Run(ctx, Options{
|
||||
Duration: time.Hour,
|
||||
IdleTimeout: 20 * time.Millisecond,
|
||||
MaxSteps: 8,
|
||||
StopOnViolation: true,
|
||||
Driver: state.mock,
|
||||
Verifier: state.verifier,
|
||||
TraceWriter: state.writer,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if !containsProperty(summary.Violations, "staysUnderThree") {
|
||||
t.Fatalf("expected staysUnderThree to fire, got %v", summary.Violations)
|
||||
}
|
||||
if summary.Steps != 3 {
|
||||
t.Errorf("steps: got %d, want 3 (the run must stop at the violating step, not run the 8-step budget)",
|
||||
summary.Steps)
|
||||
}
|
||||
for _, step := range traceStepIndices(t, state.writer.Directory()) {
|
||||
if step > summary.Steps {
|
||||
t.Errorf("trace kept stepping after the violation: found step %d past step %d",
|
||||
step, summary.Steps)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunner_WithoutStopOnViolationRunsTheWholeBudget is the other half: the
|
||||
// default must stay a full-budget fuzz run, so turning the flag on is the only
|
||||
// thing that shortens a run.
|
||||
func TestRunner_WithoutStopOnViolationRunsTheWholeBudget(t *testing.T) {
|
||||
state := newHarnessWithSpec(t, violationSpec)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
summary, err := Run(ctx, Options{
|
||||
Duration: time.Hour,
|
||||
IdleTimeout: 20 * time.Millisecond,
|
||||
MaxSteps: 4,
|
||||
Driver: state.mock,
|
||||
Verifier: state.verifier,
|
||||
TraceWriter: state.writer,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if summary.Steps != 4 {
|
||||
t.Errorf("steps: got %d, want 4; a violation must not shorten a default run", summary.Steps)
|
||||
}
|
||||
}
|
||||
|
||||
// traceStepIndices reads every step index the trace recorded, so a test can
|
||||
// assert on what the run actually wrote rather than on the summary alone.
|
||||
func traceStepIndices(t *testing.T, directory string) []int {
|
||||
t.Helper()
|
||||
file, err := os.Open(filepath.Join(directory, "trace.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer file.Close()
|
||||
var steps []int
|
||||
scanner := bufio.NewScanner(file)
|
||||
scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024)
|
||||
for scanner.Scan() {
|
||||
var line struct {
|
||||
Step int `json:"step"`
|
||||
}
|
||||
if err := json.Unmarshal(scanner.Bytes(), &line); err != nil {
|
||||
t.Fatalf("trace line decode: %v", err)
|
||||
}
|
||||
steps = append(steps, line.Step)
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
t.Fatalf("scan trace: %v", err)
|
||||
}
|
||||
return steps
|
||||
}
|
||||
@@ -23,8 +23,23 @@ type ActionSource interface {
|
||||
// ExtractorSource yields per-step extractor overrides the runner applies after
|
||||
// PushSnapshot. The mobile path has none (returns nil); the web path returns the
|
||||
// values its extractors computed in V8 against the real DOM.
|
||||
//
|
||||
// lastAction is the action the previous step actually applied, the same value
|
||||
// PushSnapshot hands the goja state. The web path has to install it in the page
|
||||
// before its extractors run: a spec extractor reading state.lastAction runs in
|
||||
// V8 there, and V8 has no way to know what the runner dispatched.
|
||||
type ExtractorSource interface {
|
||||
ExtractorOverrides(ctx context.Context) (map[int]json.RawMessage, error)
|
||||
ExtractorOverrides(
|
||||
ctx context.Context,
|
||||
lastAction *verifier.Action,
|
||||
) (map[int]json.RawMessage, error)
|
||||
}
|
||||
|
||||
// lastActionInstaller is the web driver's channel for the previous step's
|
||||
// action. It is declared here rather than folded into driver.WebDriver so the
|
||||
// mobile drivers stay untouched; every web driver must implement it.
|
||||
type lastActionInstaller interface {
|
||||
SetLastAction(ctx context.Context, encoded json.RawMessage) error
|
||||
}
|
||||
|
||||
// gojaSource drives both action selection and (trivially) extractor overrides
|
||||
@@ -38,7 +53,10 @@ func (s gojaSource) NextAction(context.Context) (verifier.Action, error) {
|
||||
return s.verifier.NextAction()
|
||||
}
|
||||
|
||||
func (gojaSource) ExtractorOverrides(context.Context) (map[int]json.RawMessage, error) {
|
||||
func (gojaSource) ExtractorOverrides(
|
||||
context.Context,
|
||||
*verifier.Action,
|
||||
) (map[int]json.RawMessage, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
@@ -59,7 +77,24 @@ func (s webSource) NextAction(ctx context.Context) (verifier.Action, error) {
|
||||
return verifier.DecodeAction(raw)
|
||||
}
|
||||
|
||||
func (s webSource) ExtractorOverrides(ctx context.Context) (map[int]json.RawMessage, error) {
|
||||
// ExtractorOverrides installs the previous step's action in the page, then
|
||||
// reads back what the spec's extractors computed against the live DOM. The
|
||||
// install is not best-effort: a web driver that cannot take it leaves
|
||||
// state.lastAction null in V8, which silently turns every action-gated
|
||||
// property vacuously true, so it is reported as an error instead.
|
||||
func (s webSource) ExtractorOverrides(
|
||||
ctx context.Context,
|
||||
lastAction *verifier.Action,
|
||||
) (map[int]json.RawMessage, error) {
|
||||
installer, ok := s.web.(lastActionInstaller)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf(
|
||||
"web driver %T cannot install state.lastAction; every property gated "+
|
||||
"on the last action would be vacuously true", s.web)
|
||||
}
|
||||
if err := installer.SetLastAction(ctx, verifier.EncodeLastAction(lastAction)); err != nil {
|
||||
return nil, fmt.Errorf("install last action: %w", err)
|
||||
}
|
||||
return s.web.EvaluateExtractors(ctx)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
package runner
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/priyanshujain/sanderling/internal/driver"
|
||||
mockdriver "github.com/priyanshujain/sanderling/internal/driver/mock"
|
||||
"github.com/priyanshujain/sanderling/internal/trace"
|
||||
)
|
||||
|
||||
// carrierSpec registers one extractor whose value the page supplies. It stands
|
||||
// in for every spec whose getters carry state across steps (folio's last-seen
|
||||
// Home total, its submit counters): what matters is that ASKING the page for
|
||||
// the value is what advances it.
|
||||
const carrierSpec = `
|
||||
import { actions, extract } from "@sanderling/spec";
|
||||
const carrier = extract("carrier", () => 0);
|
||||
globalThis.properties = {};
|
||||
globalThis.actions = actions(() => []);
|
||||
`
|
||||
|
||||
// carrierWebDriver is a web target that alternates between a cross-fading
|
||||
// hierarchy (which the runner discards as transitional) and a settled one, and
|
||||
// whose page-side extractor advances a counter on every evaluation - exactly
|
||||
// what a spec-authored carrier does in V8.
|
||||
type carrierWebDriver struct {
|
||||
*mockdriver.Driver
|
||||
transitional bool
|
||||
snapshots int
|
||||
reads int
|
||||
}
|
||||
|
||||
func (d *carrierWebDriver) Snapshot(ctx context.Context) (string, driver.Image, error) {
|
||||
_, image, err := d.Driver.Snapshot(ctx)
|
||||
d.snapshots++
|
||||
if !d.transitional {
|
||||
return `{"attributes":{"resource-id":"HomeScreen"},"children":[]}`, image, err
|
||||
}
|
||||
// A genuine cross-fade: two live routes, and a tree that keeps changing
|
||||
// between retries so the runner spends its whole retry budget on it.
|
||||
return fmt.Sprintf(`{"attributes":{"resource-id":"root"},"children":[
|
||||
{"attributes":{"resource-id":"HomeScreen","text":"frame-%d"},"children":[]},
|
||||
{"attributes":{"resource-id":"LedgerScreen"},"children":[]}
|
||||
]}`, d.snapshots), image, err
|
||||
}
|
||||
|
||||
func (d *carrierWebDriver) InstallBundle(context.Context, []byte) error { return nil }
|
||||
|
||||
func (d *carrierWebDriver) EvaluateExtractors(context.Context) (map[int]json.RawMessage, error) {
|
||||
d.reads++
|
||||
return map[int]json.RawMessage{0: json.RawMessage(strconv.Itoa(d.reads))}, nil
|
||||
}
|
||||
|
||||
// NextActionFromV8 runs once per step, after the hierarchy fetch, so flipping
|
||||
// here makes every other step a cross-fade.
|
||||
func (d *carrierWebDriver) NextActionFromV8(context.Context) (json.RawMessage, error) {
|
||||
d.transitional = !d.transitional
|
||||
return json.RawMessage(`{"kind":"Tap","x":5,"y":5}`), nil
|
||||
}
|
||||
|
||||
func (d *carrierWebDriver) SetLastAction(context.Context, json.RawMessage) error { return nil }
|
||||
|
||||
// TestRunner_TransitionalStepNeverAdvancesThePageCarrier pins the ordering the
|
||||
// web path depends on. The page-side extractors must run only on steps the
|
||||
// verifier accepts: their getters advance spec state every time they evaluate,
|
||||
// so evaluating them on a step whose values are then discarded leaves the page
|
||||
// one window ahead of the verifier. The next accepted pair then brackets two
|
||||
// committed transactions while having counted one submit, and the property
|
||||
// convicts an app that did nothing wrong.
|
||||
func TestRunner_TransitionalStepNeverAdvancesThePageCarrier(t *testing.T) {
|
||||
state := newHarnessWithSpec(t, carrierSpec)
|
||||
web := &carrierWebDriver{Driver: state.mock}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
summary, err := Run(ctx, Options{
|
||||
Duration: 30 * time.Second,
|
||||
IdleTimeout: 20 * time.Millisecond,
|
||||
MaxSteps: 5,
|
||||
Driver: web,
|
||||
Verifier: state.verifier,
|
||||
TraceWriter: state.writer,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if summary.Steps != 5 {
|
||||
t.Fatalf("steps = %d, want 5", summary.Steps)
|
||||
}
|
||||
|
||||
type traceLine struct {
|
||||
Step int `json:"step"`
|
||||
Transitional bool `json:"transitional"`
|
||||
ExtractorChanges map[string]trace.ExtractorChange `json:"extractor_changes"`
|
||||
}
|
||||
body, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
verified, transitional := 0, 0
|
||||
previous := 0
|
||||
for _, raw := range bytes.Split(bytes.TrimSpace(body), []byte("\n")) {
|
||||
var line traceLine
|
||||
if err := json.Unmarshal(raw, &line); err != nil {
|
||||
t.Fatalf("decode trace line: %v", err)
|
||||
}
|
||||
if line.Transitional {
|
||||
transitional++
|
||||
continue
|
||||
}
|
||||
verified++
|
||||
change, ok := line.ExtractorChanges["carrier"]
|
||||
if !ok {
|
||||
t.Fatalf("step %d: no carrier value reached the verifier", line.Step)
|
||||
}
|
||||
current, convErr := strconv.Atoi(string(change.Curr))
|
||||
if convErr != nil {
|
||||
t.Fatalf("step %d: carrier value %s: %v", line.Step, change.Curr, convErr)
|
||||
}
|
||||
if current != previous+1 {
|
||||
t.Errorf("step %d: carrier went %d -> %d; the page advanced it on a "+
|
||||
"step the verifier discarded, so the verifier's window is wider "+
|
||||
"than the one the spec counted actions over",
|
||||
line.Step, previous, current)
|
||||
}
|
||||
previous = current
|
||||
}
|
||||
if verified == 0 || transitional == 0 {
|
||||
t.Fatalf("need both kinds of step to prove anything: %d verified, %d transitional",
|
||||
verified, transitional)
|
||||
}
|
||||
if web.reads != verified {
|
||||
t.Errorf("the page evaluated its extractors %d time(s) across %d verified step(s); "+
|
||||
"every evaluation the verifier does not use still advances spec state",
|
||||
web.reads, verified)
|
||||
}
|
||||
}
|
||||
|
||||
// installFailsWebDriver is a web target whose page cannot take the runner's
|
||||
// lastAction: an older published @sanderling/spec runtime, a bundle that never
|
||||
// installed, a tab that navigated away from it.
|
||||
type installFailsWebDriver struct {
|
||||
*mockdriver.Driver
|
||||
}
|
||||
|
||||
func (d *installFailsWebDriver) InstallBundle(context.Context, []byte) error { return nil }
|
||||
|
||||
func (d *installFailsWebDriver) EvaluateExtractors(context.Context) (map[int]json.RawMessage, error) {
|
||||
return map[int]json.RawMessage{0: json.RawMessage(`1`)}, nil
|
||||
}
|
||||
|
||||
func (d *installFailsWebDriver) NextActionFromV8(context.Context) (json.RawMessage, error) {
|
||||
return json.RawMessage(`{"kind":"Tap","x":5,"y":5}`), nil
|
||||
}
|
||||
|
||||
func (d *installFailsWebDriver) SetLastAction(context.Context, json.RawMessage) error {
|
||||
return errors.New("__sanderlingSetLastAction__ is not a function")
|
||||
}
|
||||
|
||||
// TestRunner_LastActionInstallFailureFailsTheRun covers the other half of the
|
||||
// same trust boundary. A run that cannot install lastAction in the page cannot
|
||||
// apply the page's extractor values either, so the step keeps goja's
|
||||
// dump-derived readings while the step before it holds the page's, and a delta
|
||||
// property compares two producers and fires. Downgraded to a warning that is a
|
||||
// green run reporting a violation nobody can reproduce.
|
||||
func TestRunner_LastActionInstallFailureFailsTheRun(t *testing.T) {
|
||||
state := newHarnessWithSpec(t, carrierSpec)
|
||||
web := &installFailsWebDriver{Driver: state.mock}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
_, err := Run(ctx, Options{
|
||||
Duration: 2 * time.Second,
|
||||
IdleTimeout: 20 * time.Millisecond,
|
||||
MaxSteps: 3,
|
||||
Driver: web,
|
||||
Verifier: state.verifier,
|
||||
TraceWriter: state.writer,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("Run succeeded with a page that cannot take lastAction; the run " +
|
||||
"reported green while its extractor values came from two engines")
|
||||
}
|
||||
if !bytes.Contains([]byte(err.Error()), []byte("install last action")) {
|
||||
t.Errorf("Run error = %v, want it to name the failed lastAction install", err)
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -44,6 +45,8 @@ func (d *webMockDriver) NextActionFromV8(context.Context) (json.RawMessage, erro
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (d *webMockDriver) SetLastAction(context.Context, json.RawMessage) error { return nil }
|
||||
|
||||
// TestRunner_TraceRecordsTheValueTheVerdictUsed fails if the trace and the
|
||||
// verdict disagree about an extractor. A witness is only an explanation of a
|
||||
// violation if it holds the state the violated property was evaluated against.
|
||||
@@ -116,3 +119,48 @@ func TestRunner_TraceRecordsTheValueTheVerdictUsed(t *testing.T) {
|
||||
t.Error("no witness reached the trace; nothing was compared")
|
||||
}
|
||||
}
|
||||
|
||||
// splitTableSpec registers two extractors whose goja bodies both answer "goja".
|
||||
// The page below reports only the first, so index 1 keeps goja's dump-derived
|
||||
// reading while index 0 holds the page's.
|
||||
const splitTableSpec = `
|
||||
import { actions, extract } from "@sanderling/spec";
|
||||
extract("first", () => "goja");
|
||||
extract("second", () => "goja");
|
||||
globalThis.properties = {};
|
||||
globalThis.actions = actions(() => []);
|
||||
`
|
||||
|
||||
// TestRunner_PartialExtractorTableIsFatal pins the failure the runner used to
|
||||
// let through. JSON.stringify drops an undefined-valued key, so a page whose
|
||||
// extractors are mostly undefined off their own screen reported a table with
|
||||
// holes in it, and the run completed with half the extractors reading from V8
|
||||
// and half from goja. A delta property spanning that split convicts an app that
|
||||
// did nothing wrong, which is worse than a crash: it is a green report of a bug
|
||||
// that is not there, or a red one for a bug nobody can reproduce.
|
||||
func TestRunner_PartialExtractorTableIsFatal(t *testing.T) {
|
||||
state := newHarnessWithSpec(t, splitTableSpec)
|
||||
web := &webMockDriver{
|
||||
Driver: state.mock,
|
||||
overrides: map[int]json.RawMessage{0: json.RawMessage(`"v8"`)},
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
_, err := Run(ctx, Options{
|
||||
Duration: time.Hour,
|
||||
IdleTimeout: 20 * time.Millisecond,
|
||||
MaxSteps: 2,
|
||||
Driver: web,
|
||||
Verifier: state.verifier,
|
||||
TraceWriter: state.writer,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("the run completed on a page that reported 1 of 2 extractors; " +
|
||||
"the second extractor silently kept goja's value")
|
||||
}
|
||||
const want = "the page reported values for 1 of the spec's 2 extractors"
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("Run failed with %q, want it to name the split: %q", err, want)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package runner
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
mockdriver "github.com/priyanshujain/sanderling/internal/driver/mock"
|
||||
)
|
||||
|
||||
// On web the spec's extractors run in the page, so state.lastAction has to be
|
||||
// installed there by the runner. It used to be hardcoded null in
|
||||
// pkg/spec/src/web-runtime.ts, which made every property gated on the last
|
||||
// action (folio's submitMovesBalanceByTypedAmount, for one) vacuously true on
|
||||
// web: no failure, no warning, just a green run that proved nothing.
|
||||
|
||||
const lastActionSpec = `
|
||||
import { actions } from "@sanderling/spec";
|
||||
globalThis.actions = actions(() => []);
|
||||
globalThis.properties = {};
|
||||
`
|
||||
|
||||
// tappingWebDriver is a web target whose V8 picker always taps one named
|
||||
// control, so the runner has a real applied action to report on the next step.
|
||||
type tappingWebDriver struct {
|
||||
*mockdriver.Driver
|
||||
installed []string
|
||||
}
|
||||
|
||||
func (d *tappingWebDriver) InstallBundle(context.Context, []byte) error { return nil }
|
||||
|
||||
func (d *tappingWebDriver) EvaluateExtractors(context.Context) (map[int]json.RawMessage, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (d *tappingWebDriver) NextActionFromV8(context.Context) (json.RawMessage, error) {
|
||||
return json.RawMessage(`{"kind":"Tap","x":12,"y":34,"selector":"id:TxnSubmit"}`), nil
|
||||
}
|
||||
|
||||
func (d *tappingWebDriver) SetLastAction(_ context.Context, encoded json.RawMessage) error {
|
||||
d.installed = append(d.installed, string(encoded))
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestRunner_WebInstallsLastActionInThePage(t *testing.T) {
|
||||
state := newHarnessWithSpec(t, lastActionSpec)
|
||||
web := &tappingWebDriver{Driver: state.mock}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
if _, err := Run(ctx, Options{
|
||||
Duration: time.Hour,
|
||||
IdleTimeout: 20 * time.Millisecond,
|
||||
MaxSteps: 3,
|
||||
Driver: web,
|
||||
Verifier: state.verifier,
|
||||
TraceWriter: state.writer,
|
||||
}); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
if len(web.installed) < 2 {
|
||||
t.Fatalf("the page was handed lastAction %d time(s); the web path never installed it",
|
||||
len(web.installed))
|
||||
}
|
||||
// Step 1 has no previous action, exactly as the goja host reports it.
|
||||
if web.installed[0] != "null" {
|
||||
t.Errorf("step 1 installed %s, want null", web.installed[0])
|
||||
}
|
||||
// Every later step carries what the runner actually applied. The shape is
|
||||
// the goja host's (internal/verifier/marshal.go lastActionFields), pinned
|
||||
// against it by TestLastAction_WebJSONMatchesTheGojaObject.
|
||||
const want = `{"kind":"Tap","on":"id:TxnSubmit"}`
|
||||
if web.installed[1] != want {
|
||||
t.Errorf("step 2 installed %s, want %s", web.installed[1], want)
|
||||
}
|
||||
}
|
||||
+59
-11
@@ -20,6 +20,26 @@ import (
|
||||
|
||||
const sidecarStartupTimeout = 30 * time.Second
|
||||
|
||||
// launchTimeout bounds the pre-run app launch. It happens before the runner
|
||||
// starts, so --duration does not cover it, and Execute's context is the bare
|
||||
// signal-aware root with no deadline of its own: a driver wedged here would
|
||||
// hang the run forever having printed nothing and written no trace. Generous
|
||||
// enough to sit above every driver's own launch bound (the iOS clear-state path
|
||||
// reinstalls the app first) so a driver-level error is what a user usually
|
||||
// sees, and this stays the backstop. A variable so the timeout test can shrink
|
||||
// it.
|
||||
var launchTimeout = 3 * time.Minute
|
||||
|
||||
// launchApp starts the app under test under a bounded context.
|
||||
func launchApp(ctx context.Context, activeDriver driver.DeviceDriver, options Options) error {
|
||||
launchCtx, cancel := context.WithTimeout(ctx, launchTimeout)
|
||||
defer cancel()
|
||||
if err := activeDriver.Launch(launchCtx, options.BundleID, options.ClearData, nil); err != nil {
|
||||
return fmt.Errorf("launch app: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Options are the parameters for a single test pipeline run.
|
||||
type Options struct {
|
||||
Spec string
|
||||
@@ -38,6 +58,10 @@ type Options struct {
|
||||
// Arm labels the experiment cell this run belongs to and is recorded in
|
||||
// meta.json so a directory of runs can be attributed to a cell.
|
||||
Arm string
|
||||
// ExitOnViolation stops the run at the first violation and reports the
|
||||
// recorded violations as a ViolationsError, so a caller (CI) can tell
|
||||
// "the run found the bug" from "the run finished clean".
|
||||
ExitOnViolation bool
|
||||
// Generator selects the action picker: "llm" or the default seeded picker.
|
||||
Generator string
|
||||
|
||||
@@ -141,8 +165,8 @@ func Execute(ctx context.Context, options Options, stdout io.Writer) error {
|
||||
}
|
||||
defer cleanup()
|
||||
|
||||
if err := activeDriver.Launch(ctx, options.BundleID, options.ClearData, nil); err != nil {
|
||||
return fmt.Errorf("launch app: %w", err)
|
||||
if err := launchApp(ctx, activeDriver, options); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if web, ok := activeDriver.(driver.WebDriver); ok && len(webBundle.JavaScript) > 0 {
|
||||
@@ -189,15 +213,16 @@ func Execute(ctx context.Context, options Options, stdout io.Writer) error {
|
||||
fmt.Fprintf(stdout, "running for %s (seed=%d)\n", options.Duration, seed)
|
||||
}
|
||||
summary, err := runner.Run(ctx, runner.Options{
|
||||
Duration: options.Duration,
|
||||
MaxSteps: options.MaxSteps,
|
||||
IdleTimeout: 1 * time.Second,
|
||||
BundleID: options.BundleID,
|
||||
Driver: activeDriver,
|
||||
Verifier: verifierInstance,
|
||||
TraceWriter: traceWriter,
|
||||
Logger: newProgressLogger(stdout),
|
||||
Generator: options.Generator,
|
||||
Duration: options.Duration,
|
||||
MaxSteps: options.MaxSteps,
|
||||
IdleTimeout: 1 * time.Second,
|
||||
BundleID: options.BundleID,
|
||||
Driver: activeDriver,
|
||||
Verifier: verifierInstance,
|
||||
TraceWriter: traceWriter,
|
||||
Logger: newProgressLogger(stdout),
|
||||
Generator: options.Generator,
|
||||
StopOnViolation: options.ExitOnViolation,
|
||||
})
|
||||
|
||||
terminateCtx, terminateCancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
@@ -210,9 +235,32 @@ func Execute(ctx context.Context, options Options, stdout io.Writer) error {
|
||||
|
||||
fmt.Fprintf(stdout, "\nelapsed: %s\n", summary.EndTime.Sub(summary.StartTime).Round(time.Millisecond))
|
||||
runner.RenderSummary(stdout, summary, options.Platform)
|
||||
return runOutcome(options, summary)
|
||||
}
|
||||
|
||||
// runOutcome turns a finished run into the pipeline's result. Without
|
||||
// --exit-on-violation a run that found violations is still a successful run
|
||||
// (the summary reports them), which is the behaviour every existing caller
|
||||
// depends on.
|
||||
func runOutcome(options Options, summary runner.Summary) error {
|
||||
if options.ExitOnViolation && len(summary.Violations) > 0 {
|
||||
return ViolationsError{Count: len(summary.Violations)}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ViolationsError reports a run that recorded violations under
|
||||
// --exit-on-violation. It is deliberately distinct from every other error the
|
||||
// pipeline returns: those mean the harness broke, this one means the run did
|
||||
// its job and found something.
|
||||
type ViolationsError struct {
|
||||
Count int
|
||||
}
|
||||
|
||||
func (e ViolationsError) Error() string {
|
||||
return fmt.Sprintf("%d violation record(s)", e.Count)
|
||||
}
|
||||
|
||||
// bundleInputs holds the pre-driver assembly: alias map, seed, esbuild defines,
|
||||
// and the resolved spec-API/goja-runtime paths the bundler consumes.
|
||||
type bundleInputs struct {
|
||||
|
||||
@@ -1,12 +1,16 @@
|
||||
package testrun
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/priyanshujain/sanderling/internal/driver"
|
||||
"github.com/priyanshujain/sanderling/internal/runner"
|
||||
"github.com/priyanshujain/sanderling/internal/verifier"
|
||||
)
|
||||
|
||||
@@ -231,3 +235,72 @@ func TestBuildRunMeta_OmitsModelWhenSpecDeclaresNoLLMGenerator(t *testing.T) {
|
||||
t.Errorf("model recorded without a spec-declared llm generator: %q", meta.Model)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunOutcome_ReportsViolationsOnlyUnderTheFlag pins the CI contract: the
|
||||
// typed error is what makes `sanderling test` exit 2, and it must appear only
|
||||
// when the caller asked for it. A run that finds violations without the flag
|
||||
// stays a successful run, which is what every existing invocation expects.
|
||||
func TestRunOutcome_ReportsViolationsOnlyUnderTheFlag(t *testing.T) {
|
||||
violated := runner.Summary{
|
||||
Steps: 7,
|
||||
Violations: []runner.ViolationRecord{{StepIndex: 3, Properties: []string{"balanceMoves"}}},
|
||||
}
|
||||
clean := runner.Summary{Steps: 7}
|
||||
|
||||
if err := runOutcome(Options{}, violated); err != nil {
|
||||
t.Errorf("without --exit-on-violation a violated run must succeed, got %v", err)
|
||||
}
|
||||
if err := runOutcome(Options{ExitOnViolation: true}, clean); err != nil {
|
||||
t.Errorf("a clean run must succeed under --exit-on-violation, got %v", err)
|
||||
}
|
||||
|
||||
err := runOutcome(Options{ExitOnViolation: true}, violated)
|
||||
var violations ViolationsError
|
||||
if !errors.As(err, &violations) {
|
||||
t.Fatalf("expected a ViolationsError, got %v", err)
|
||||
}
|
||||
if violations.Count != 1 {
|
||||
t.Errorf("count: got %d, want 1", violations.Count)
|
||||
}
|
||||
}
|
||||
|
||||
// wedgedLaunchDriver never returns from Launch, standing in for a driver whose
|
||||
// device-side session is stuck.
|
||||
type wedgedLaunchDriver struct {
|
||||
driver.DeviceDriver
|
||||
release chan struct{}
|
||||
}
|
||||
|
||||
func (w *wedgedLaunchDriver) Launch(ctx context.Context, _ string, _ bool, _ map[string]string) error {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-w.release:
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// TestLaunchAppBoundsWedgedDriver proves the pre-run launch carries a deadline.
|
||||
// It runs before the runner starts, so --duration does not cover it and
|
||||
// Execute's root context has no deadline: unbounded, a wedged driver hangs the
|
||||
// run forever with no trace directory and no error.
|
||||
func TestLaunchAppBoundsWedgedDriver(t *testing.T) {
|
||||
previous := launchTimeout
|
||||
launchTimeout = 100 * time.Millisecond
|
||||
defer func() { launchTimeout = previous }()
|
||||
|
||||
wedged := &wedgedLaunchDriver{release: make(chan struct{})}
|
||||
defer close(wedged.release)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- launchApp(context.Background(), wedged, Options{BundleID: "com.example.app"}) }()
|
||||
|
||||
select {
|
||||
case err := <-done:
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("err = %v, want a deadline-exceeded error", err)
|
||||
}
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("launchApp never returned: the pre-run launch is unbounded, so a wedged driver hangs the run forever")
|
||||
}
|
||||
}
|
||||
+103
-33
@@ -1,6 +1,7 @@
|
||||
package verifier
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
@@ -325,49 +326,118 @@ func selectorObjectToString(runtime *goja.Runtime, arg goja.Value) string {
|
||||
return strings.Join(parts, " ")
|
||||
}
|
||||
|
||||
// actionField is one property of the lastAction object, in the order the
|
||||
// object is built. Value is a string, an int, or a nested []actionField for
|
||||
// the from/to points.
|
||||
type actionField struct {
|
||||
key string
|
||||
value any
|
||||
}
|
||||
|
||||
// lastActionFields is the ONE description of the lastAction shape. The goja
|
||||
// host turns it into a JS object (lastActionObject); the web host receives the
|
||||
// same fields as JSON (EncodeLastAction) and installs them as state.lastAction
|
||||
// in the page. Both hosts therefore expose identical field names, casing,
|
||||
// presence and order, so a property reading state.lastAction cannot mean one
|
||||
// thing on native and another on web.
|
||||
func lastActionFields(action *Action) []actionField {
|
||||
point := func(x, y int) []actionField {
|
||||
return []actionField{{key: "x", value: x}, {key: "y", value: y}}
|
||||
}
|
||||
fields := []actionField{{key: "kind", value: string(action.Kind)}}
|
||||
if action.On != "" {
|
||||
fields = append(fields, actionField{key: "on", value: action.On})
|
||||
}
|
||||
if action.Text != "" {
|
||||
fields = append(fields, actionField{key: "text", value: action.Text})
|
||||
}
|
||||
switch action.Kind {
|
||||
case ActionKindSwipe:
|
||||
fields = append(fields,
|
||||
actionField{key: "from", value: point(action.FromX, action.FromY)},
|
||||
actionField{key: "to", value: point(action.ToX, action.ToY)})
|
||||
if action.DurationMillis > 0 {
|
||||
fields = append(fields,
|
||||
actionField{key: "durationMillis", value: action.DurationMillis})
|
||||
}
|
||||
case ActionKindScroll:
|
||||
fields = append(fields,
|
||||
actionField{key: "direction", value: action.Direction},
|
||||
actionField{key: "from", value: point(action.FromX, action.FromY)},
|
||||
actionField{key: "to", value: point(action.ToX, action.ToY)})
|
||||
case ActionKindPressKey:
|
||||
fields = append(fields, actionField{key: "key", value: action.Key})
|
||||
case ActionKindWait:
|
||||
fields = append(fields,
|
||||
actionField{key: "durationMillis", value: action.DurationMillis})
|
||||
}
|
||||
return fields
|
||||
}
|
||||
|
||||
func lastActionObject(runtime *goja.Runtime, action *Action) goja.Value {
|
||||
if action == nil {
|
||||
return goja.Null()
|
||||
}
|
||||
return objectFromFields(runtime, lastActionFields(action))
|
||||
}
|
||||
|
||||
func objectFromFields(runtime *goja.Runtime, fields []actionField) *goja.Object {
|
||||
object := runtime.NewObject()
|
||||
_ = object.Set("kind", string(action.Kind))
|
||||
if action.On != "" {
|
||||
_ = object.Set("on", action.On)
|
||||
}
|
||||
if action.Text != "" {
|
||||
_ = object.Set("text", action.Text)
|
||||
}
|
||||
switch action.Kind {
|
||||
case ActionKindSwipe:
|
||||
from := runtime.NewObject()
|
||||
_ = from.Set("x", action.FromX)
|
||||
_ = from.Set("y", action.FromY)
|
||||
to := runtime.NewObject()
|
||||
_ = to.Set("x", action.ToX)
|
||||
_ = to.Set("y", action.ToY)
|
||||
_ = object.Set("from", from)
|
||||
_ = object.Set("to", to)
|
||||
if action.DurationMillis > 0 {
|
||||
_ = object.Set("durationMillis", action.DurationMillis)
|
||||
for _, field := range fields {
|
||||
if nested, ok := field.value.([]actionField); ok {
|
||||
_ = object.Set(field.key, objectFromFields(runtime, nested))
|
||||
continue
|
||||
}
|
||||
case ActionKindScroll:
|
||||
_ = object.Set("direction", action.Direction)
|
||||
from := runtime.NewObject()
|
||||
_ = from.Set("x", action.FromX)
|
||||
_ = from.Set("y", action.FromY)
|
||||
to := runtime.NewObject()
|
||||
_ = to.Set("x", action.ToX)
|
||||
_ = to.Set("y", action.ToY)
|
||||
_ = object.Set("from", from)
|
||||
_ = object.Set("to", to)
|
||||
case ActionKindPressKey:
|
||||
_ = object.Set("key", action.Key)
|
||||
case ActionKindWait:
|
||||
_ = object.Set("durationMillis", action.DurationMillis)
|
||||
_ = object.Set(field.key, field.value)
|
||||
}
|
||||
return object
|
||||
}
|
||||
|
||||
// EncodeLastAction renders the previous step's action for the web host, which
|
||||
// has no Go-side state object to read: the runner pushes this JSON into the
|
||||
// page before each extractor evaluation. A nil action encodes as JSON null,
|
||||
// the same value the goja host reports on the first step of a run and after a
|
||||
// step whose action was never applied.
|
||||
func EncodeLastAction(action *Action) json.RawMessage {
|
||||
if action == nil {
|
||||
return json.RawMessage("null")
|
||||
}
|
||||
return encodeFields(lastActionFields(action))
|
||||
}
|
||||
|
||||
func encodeFields(fields []actionField) json.RawMessage {
|
||||
var buffer bytes.Buffer
|
||||
buffer.WriteByte('{')
|
||||
for index, field := range fields {
|
||||
if index > 0 {
|
||||
buffer.WriteByte(',')
|
||||
}
|
||||
buffer.Write(encodeJSValue(field.key))
|
||||
buffer.WriteByte(':')
|
||||
if nested, ok := field.value.([]actionField); ok {
|
||||
buffer.Write(encodeFields(nested))
|
||||
continue
|
||||
}
|
||||
buffer.Write(encodeJSValue(field.value))
|
||||
}
|
||||
buffer.WriteByte('}')
|
||||
return buffer.Bytes()
|
||||
}
|
||||
|
||||
// encodeJSValue encodes one value the way JS JSON.stringify would, so the JSON
|
||||
// the web host parses is byte-identical to what the goja object stringifies to.
|
||||
// Go escapes <, > and & by default, which JSON.stringify does not, and that
|
||||
// alone would make the two hosts encode the same selector differently.
|
||||
func encodeJSValue(value any) []byte {
|
||||
var buffer bytes.Buffer
|
||||
encoder := json.NewEncoder(&buffer)
|
||||
encoder.SetEscapeHTML(false)
|
||||
if err := encoder.Encode(value); err != nil {
|
||||
return []byte("null")
|
||||
}
|
||||
return bytes.TrimRight(buffer.Bytes(), "\n")
|
||||
}
|
||||
|
||||
func runtimeMillis(stepTime, runStart time.Time) int64 {
|
||||
if stepTime.IsZero() || runStart.IsZero() {
|
||||
return 0
|
||||
|
||||
@@ -147,3 +147,49 @@ func TestLastActionObject_ExposesKindSpecificFields(t *testing.T) {
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestLastAction_WebJSONMatchesTheGojaObject pins the two hosts to ONE shape.
|
||||
// The goja host builds state.lastAction as a JS object; the web host receives
|
||||
// EncodeLastAction's JSON and installs the parsed value as state.lastAction in
|
||||
// the page. A field this side renames, drops or cases differently would leave a
|
||||
// spec reading state.lastAction working on native and silently mismatching on
|
||||
// web, which is the failure this whole path exists to prevent. Comparing
|
||||
// goja's own JSON.stringify against the encoder is the strongest available
|
||||
// statement that the two are the same object.
|
||||
func TestLastAction_WebJSONMatchesTheGojaObject(t *testing.T) {
|
||||
verifier := newVerifier(t)
|
||||
mustLoad(t, verifier, `
|
||||
globalThis.last = __sanderling__.extract(state => JSON.stringify(state.lastAction));
|
||||
`)
|
||||
|
||||
for _, testCase := range []struct {
|
||||
name string
|
||||
action *Action
|
||||
}{
|
||||
{"nil", nil},
|
||||
{"Tap", &Action{Kind: ActionKindTap, On: "id:TxnSubmit", X: 12, Y: 34}},
|
||||
{"TapWithoutSelector", &Action{Kind: ActionKindTap, X: 12, Y: 34}},
|
||||
{"DoubleTap", &Action{Kind: ActionKindDoubleTap, On: `desc:say "hi" <b>`}},
|
||||
{"InputText", &Action{Kind: ActionKindInputText, On: "id:field", Text: "50"}},
|
||||
{"Swipe", &Action{Kind: ActionKindSwipe, FromX: 1, FromY: 2, ToX: 3, ToY: 4, DurationMillis: 250}},
|
||||
{"SwipeNoDuration", &Action{Kind: ActionKindSwipe, FromX: 1, FromY: 2, ToX: 3, ToY: 4}},
|
||||
{"Scroll", &Action{Kind: ActionKindScroll, Direction: "down", FromX: 5, FromY: 6, ToX: 5, ToY: 1}},
|
||||
{"PressKey", &Action{Kind: ActionKindPressKey, Key: "enter"}},
|
||||
{"Wait", &Action{Kind: ActionKindWait, DurationMillis: 500}},
|
||||
} {
|
||||
t.Run(testCase.name, func(t *testing.T) {
|
||||
if err := verifier.PushSnapshot(SnapshotInput{
|
||||
Snapshots: Snapshots{},
|
||||
LastAction: testCase.action,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
handle := verifier.runtime.GlobalObject().Get("last").ToObject(verifier.runtime)
|
||||
goja := handle.Get("current").String()
|
||||
web := string(EncodeLastAction(testCase.action))
|
||||
if goja != web {
|
||||
t.Errorf("the two hosts disagree on state.lastAction\n goja: %s\n web: %s", goja, web)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1382,3 +1382,54 @@ func TestWithPlatform_IOSReachesPicker(t *testing.T) {
|
||||
t.Errorf("key = %q, want back (native press-key pool)", action.Key)
|
||||
}
|
||||
}
|
||||
|
||||
// TestOverrideExtractorValues_EmptyPayloadIsUndefined pins the cross-host
|
||||
// meaning of a page reading with no value. JSON has no undefined, so the web
|
||||
// runtime wraps every reading in a {value} envelope and the chrome driver hands
|
||||
// an absent value through as an empty payload. It has to land here as undefined,
|
||||
// because that is what PushSnapshot records for a getter that returned undefined
|
||||
// on native: decoding it as null instead would make `x.current === undefined`
|
||||
// answer one thing on the native host and another on web, for one spec.
|
||||
func TestOverrideExtractorValues_EmptyPayloadIsUndefined(t *testing.T) {
|
||||
verifier := newVerifier(t)
|
||||
mustLoad(t, verifier, helloSpec)
|
||||
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": json.RawMessage(`42`)}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := verifier.OverrideExtractorValues(map[int]json.RawMessage{
|
||||
0: nil,
|
||||
1: json.RawMessage(`null`),
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
for _, want := range []struct {
|
||||
expression string
|
||||
result bool
|
||||
}{
|
||||
{"screen.current === undefined", true},
|
||||
{"screen.current === null", false},
|
||||
{"balance.current === null", true},
|
||||
{"balance.current === undefined", false},
|
||||
} {
|
||||
value, err := verifier.runtime.RunString(want.expression)
|
||||
if err != nil {
|
||||
t.Fatalf("evaluate %s: %v", want.expression, err)
|
||||
}
|
||||
if value.ToBoolean() != want.result {
|
||||
t.Errorf("a spec reading %s gets %v, want %v", want.expression, value, want.result)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestExtractorCount_ReportsEveryRegisteredExtractor keeps the web path's
|
||||
// completeness check honest: it compares the page's reading count against this
|
||||
// number, so a count that ignored an extractor would let a partial table
|
||||
// through.
|
||||
func TestExtractorCount_ReportsEveryRegisteredExtractor(t *testing.T) {
|
||||
verifier := newVerifier(t)
|
||||
mustLoad(t, verifier, helloSpec)
|
||||
if got := verifier.ExtractorCount(); got != 2 {
|
||||
t.Errorf("ExtractorCount() = %d, want 2 (helloSpec registers screen and balance)", got)
|
||||
}
|
||||
}
|
||||
@@ -398,6 +398,15 @@ func (v *Verifier) ChangedExtractors() map[string]ExtractorChange {
|
||||
return changes
|
||||
}
|
||||
|
||||
// ExtractorCount reports how many extractors the spec registered. The web path
|
||||
// compares it against the number of readings the page sent: a page reporting
|
||||
// fewer leaves the rest holding goja's dump-derived value while the others hold
|
||||
// the page's, and a property comparing previous to current across that split
|
||||
// fires on a healthy app.
|
||||
func (v *Verifier) ExtractorCount() int {
|
||||
return len(v.extractors)
|
||||
}
|
||||
|
||||
// OverrideExtractorValues replaces each extractor's `current` slot with a
|
||||
// caller-supplied value, keyed by registration index. Used by the web tick
|
||||
// path so extractor bodies that ran in V8 (against the real DOM) drive the
|
||||
|
||||
Reference in new issue
Block a user