pj/sanderling
1
0 Fork 0
mirror of https://github.com/priyanshujain/sanderling.git synced 2026-10-02 11:07:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

UX refactor (#52)

Browse Source 
* feat(ltl): bound fields on AlwaysFormula and named thunks

Add StepBound/Duration/Deadline to AlwaysFormula as the dual of bounded
Eventually, give ThunkFormula a Name for stable identity, add ThunkNamed,
and surface both in describe() and MarshalJSON.

* feat(ltl): negation normal form pass

nnf/pushNot rewrite a formula so every Not wraps only a Thunk or Error
leaf, dualizing Always<->Eventually and preserving bounds.

* feat(ltl): NNF in NewEvaluator, bounded-always, Finalize, collapse

Apply nnf on construction, reduce bounded Always symmetric to bounded
Eventually (vacuous holds once the window closes), add Finalize to
resolve undischarged liveness obligations to Violated at run end, and
collapse structurally-identical pending obligations.

* test(ltl): property-based NNF laws

Lock double-negation identity, Always/Eventually duality with bound
preservation, leaf pushdown, and not(always true) reaching Violated.

* test(ltl): Finalize, bounded eventually, latch, collapse

Property tests for monotonic violation latch and eventually-within
violating iff n consecutive false, plus Finalize and collapse cases.

* feat(inspect): within clause on always residual node

A negated bounded eventually serializes as a bounded always; render its
bound instead of dropping it.

* feat(ltl): witness violations and (bool,error) predicate thunks

* test(ltl): migrate thunk call sites to (bool,error)

* feat(ltl): flag thrown-predicate witnesses with IsError

* refactor(verifier): replace predicate err side-channel with violation witness

* test(verifier): witness API for thrown predicates

* feat(trace): witnesses map and skipped-verification marker on Step

* feat(runner): thread violation witnesses, finalize, skip marker into trace

* test(ltl): lock violation witness reason, IsError, and step

* test(verifier): finalize surfaces unmet eventually with witness

* fix(ltl): eliminate implies and bounded-always false-negatives

Rewrite a -> b to (not a) or b in NNF so a pending temporal antecedent
can no longer defer the whole implication and drop a consequent that was
false at the current step. Carry a pending inner past a bounded-Always
window close instead of dropping it to holds, so a deferred obligation is
resolved by a later step or Finalize.

* test(ltl): lock implies and bounded-always false-negative regressions

* fix(web-runtime): seed PRNG for reproducible runs and align weighted pick

* feat(testrun): inject seed into web bundle via SANDERLING_SEED define

* test: cover web-runtime seeded PRNG, weighted pick, and seed define wiring

* test(spec): add Go math/rand/v2 PCG oracle and golden fixture

* feat(spec): bit-exact PCG port of Go math/rand/v2

* test(spec): assert pcg.ts matches the PCG golden fixture

* feat(spec): shared input corpus and press-key pools

* feat(spec): action-tree types and Host interface

* feat(spec): verb support matrix and warn-once helper

* feat(spec): deterministic shared action picker

* test(spec): verb matrix and warn-once semantics

* test(spec): picker draw-order and determinism

* refactor(spec): actions.ts returns pure GeneratorNode data trees

* refactor(spec): wire from() sampling through the picker rng

* feat(spec): shared runtime-entry installs next-action over pick.ts

* feat(spec): export LongPress/Scroll/longPresses/scrolls factories

* test(spec): assert data-tree shapes for action factories

* test(spec): runtime-entry serializeAction wire-contract round-trip

* refactor(spec): bridge data-tree nodes to the legacy goja picker tags

* fix(spec): web runtime walks the spec's globalThis.actions data tree

* test(spec): tolerate legacy bridge fields on builtin nodes

* refactor(spec): installRuntime accepts a lazy root resolver

The web bundle imports the runtime before the spec, so the action root
on globalThis.actions only exists after the spec evaluates. Accept a
function form so the goja and web hosts resolve the root per tick.

* refactor(spec): web-runtime becomes the WEB Host, delegates to shared picker

Delete the duplicate picker (resolveGenerator/pickWeighted/randomTap/
randomInput/randomSwipe/randomPressKey/pickFromArray, the mulberry32 PRNG,
and the snake_case serializeAction) plus the __sanderling__ action factory
binds. web-runtime now implements Host (platform/seedHi/seedLo from the
injected 64-bit seed via BigInt, queryCandidates over the live DOM with a
per-tick cache, reportUnsupported) and calls installRuntime so both engines
run pick.ts over the same Pcg. Swipe/longPress/scroll follow the verbs.ts
matrix instead of silently returning null. Keeps the DOM helpers (selector
translation, queryElement, elementHandle, buildState, sanitize, extractors)
and the global locking. Net -214 lines (741 -> 527).

* test(spec): cover the WEB Host surface and seed precision

Replace the deleted-picker tests with Host coverage: platform()==web,
seedHi() parsing a 64-bit seed without Number precision loss, seedLo()==0,
reportUnsupported warning, the installed next-action/extractor globals, and
queryCandidates verb routing + per-tick caching over a querySelectorAll stub.

* refactor(spec): picker emits native selector + scroll endpoints, setup precedence

* feat(spec): goja runtime entry wires the shared picker over the Go host

* feat(bundler): optional RuntimeFile prepends a runtime-entry import via stdin

* feat(testrun): bundle the goja runtime entry so the verifier runs the shared picker

* refactor(spec): drop the legacy goja bridge fields from action factories

* feat(spec): serialize selector-only string targets for the runner to re-resolve

* refactor(verifier): one DecodeAction reads the unified flat wire contract

* refactor(verifier): goja host + shared picker replace the duplicate Go picker

* refactor(runner): decode V8 actions via the unified DecodeAction; wire goja runtime

* test(verifier): author specs through the shared picker path

* test(runner): bundle authored specs with the goja runtime entry

* feat(verifier): collect unsupported verbs for the run report

* refactor(runner): collapse WebDriver forks behind ActionSource/ExtractorSource

* feat(testrun): surface unsupported verbs in run report

* test(verifier): cross-runtime goja/node parity gate on the shared picker

* test(verifier): unsupported verbs collected deduped in first-seen order

* test(runner): summary reports no unsupported verbs on a clean run

* test(spec): golden-fixture cross-runtime parity gate for the node picker

Replace the env-driven parity harness with a shared scenario module and a
committed golden the node picker asserts independently. The goja side asserts
the same golden, so neither runtime invokes the other at test time.

* test(verifier): assert goja picker against the same cross-runtime golden

Drop the node-subprocess coupling: the goja side now installs a stub
__sanderlingHost__ with the fixed candidate list and asserts the committed
golden, matching pkg/spec/test/parity.test.ts.

* refactor(spec): rename pressKey generator export to pressKeys

* refactor(spec): update barrel re-exports for pressKeys

* test(spec): update pressKeys generator export name

* docs(spec): rename pressKey generator to pressKeys

* refactor(spec): extract samplerRng into shared sampler-rng module

* feat(spec): add fluent seeded value generators (strings/integers/emails/edgeCaseText)

* test(spec): cover fluent value generators determinism and chaining

* refactor(bundler): inject globalThis trailer from spec named exports

* refactor(bundler): reuse registration trailer in web bundler

* test(bundler): cover named-export globalThis registration

* feat(spec): add named() to Extracted handle type

* feat(web-runtime): named() and cross-extractor read guard

* feat(verifier): named() and cross-extractor read guard in goja

* test(verifier): cross-extractor read guard and named()

* test(web-runtime): export runtime and extractors for tests

* test(web-runtime): named() and cross-extractor read guard

* refactor(folio): drop manual globalThis trailer (bundler injects it)

* refactor(folio): seed txn amounts via integers().between(1,500)

* refactor(folio-web): drop manual globalThis trailer (bundler injects it)

* fix(folio-web): seed card/txn-type selection via from().generate() for reproducible runs

* refactor(folio-web): weight valid generators against edgeCaseText for names/amounts

* refactor(folio-web): name extractors so violation witnesses are readable

* fix(web-runtime): propagate extractor getter throws and unpoison locked global

Stop swallowing getter errors in evaluateExtractors so the cross-extractor read guard aborts loudly, matching goja's PushSnapshot. Make the __sanderling__ lock configurable (still non-writable) so a shared test process can reinstall a fake.

* test(spec): install fake runtime via defineProperty to survive locked global

* test(web-runtime): assert uncaught cross-extractor read aborts evaluateExtractors

* feat(runner): add MaxSteps bound to Options

* test(runner): MaxSteps stops after exactly N steps

* test(driverpb): drop proto getter round-trip tautology

* test(sidecar): drop stub-mode placeholder tautology tests

* test(mock): drop default-field-value assertion test

* test(ltl): drop Verdict.String tautology tests

* refactor(runner): extract RenderSummary for snapshot testing

* test(runner): golden snapshots for trace stream and violation summary

* feat(web-runtime): capture uncaught errors into state.exceptions

* test(integration): add throwing and counter web fixtures

* test(integration): add specs for the web fixtures

* test(integration): drive web fixtures through the real pipeline in headless Chrome

* chore(make): add test-browser target for the Chrome-driven suite

* ci: run the Chrome-driven browser suite in a separate job

* refactor(test): relocate browser suite to test/browser

* refactor(permissions): delete dead internal/permissions package

* refactor(test): rename package to browser_test

* refactor(sidecarassets): rename internal/sidecar to internal/sidecarassets

* chore(make): point test-browser at test/browser

* docs(decisions): record internal/permissions deletion

* refactor(doctor): use sidecarassets package

* refactor(testrun): use sidecarassets package

* fix(test): resolve testdata relative to browser_test.go

* refactor(verifier): remove dead __sanderlingIndex compat alias

* refactor(bundler): use encoding/json for JS string literals

* docs(action-space): use vendor-neutral native driver wording

* refactor(hierarchy): scrub backend tool name from comments

* refactor(driver): scrub backend tool name from comments

* refactor(driver): add DoubleTap and DoubleTapSelector to DeviceDriver

* refactor(sidecar): implement DoubleTap with the sub-100ms inter-tap gap

* refactor(chrome): implement DoubleTap as two taps with the gap

* refactor(mock): record DoubleTap and DoubleTapSelector actions

* refactor(runner): delegate double-tap to driver, drop gesture timing

* test(runner): assert double-tap delegates to driver DoubleTap

* docs(cmd): add package docs to CLI and developer tools

* docs(driver): add package docs to driver interface and chrome backend

* docs(driver): add package docs to mock and sidecar backends

* docs(platform): add package docs to android and ios device prep

* docs: add package docs to bundler and inspect

* docs(ltl): add package doc to temporal logic evaluator

* docs: add package docs to runner and testrun pipeline

* docs: add package docs to trace and verifier

* docs(sidecarassets): add package doc for embedded JAR loader

* fix(chrome): add disable-dev-shm-usage so Chrome starts in CI

* test(chrome): gate real-Chrome driver tests behind the browser tag

* chore(make): run chrome driver tests in the browser job

* fix(web-runtime): guard global error listeners for non-browser hosts

The module registered window error/unhandledrejection listeners at top
level, which threw under Node (the spec-api test runner) where
globalThis.addEventListener is absent. Register only when the API exists;
the real browser run is unaffected.

* ci(browser): re-enable unprivileged user namespaces for headless Chrome

ubuntu-latest moved to 24.04, whose AppArmor restriction on unprivileged
user namespaces stops headless Chrome from opening its DevTools socket
even with --no-sandbox, surfacing as the driver's 'websocket url timeout'.
Relax the sysctl for the job and add a direct launch check so a future
breakage shows Chrome's own stderr rather than an opaque driver timeout.

* ci(browser): pin stable Chrome for the driver tests

setup-chrome's default latest pulled a dev Chromium (150) whose remote
debugging socket never came up under chromedp, while plain --dump-dom
worked. Pin the stable channel, which the driver is tested against.

* feat(defaults): add scroll and rebalance action weights

Use relative-integer weights (taps/typing co-primary 100, scrolls 50,
swipes 25, doubleTaps 10); the picker normalizes by their total. Adds
scrolls to defaultActions as a first-class reveal behavior.

* feat(defaults): trim scroll action weight wiring

* fix(build): point sidecar jar ignore and embed paths at sidecarassets

* test(defaults): drop stale longPresses re-export assertion

longPresses is opt-in vocabulary, no longer re-exported from
defaults/actions.ts since e0d3b20; its builtin resolution is already
covered by api.test.ts. Trim the defaults test to scrolls, which is an
actual default export.

* fix(chrome): raise DevTools websocket read timeout to 60s

Chrome cold-start on a loaded CI runner can exceed chromedp's 20s
default for reading the DevTools websocket URL, flaking the browser
tests with "websocket url timeout reached". Give launch more headroom.
This commit is contained in:
pj authored and GitHub committed 2026-06-02 09:52:53 +05:30
1 parent 88db9653e5
commit c5bb176be8
98 files changed
+6898 -2227
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

internal/_oracle/main.go
+122
View File
@@ -0,0 +1,122 @@
//go:build ignore
// Oracle generator for the bit-exact PCG port.
//
// Emits the golden fixture consumed by pkg/spec/test/pcg.test.ts. The fixture
// pins the exact draw sequences produced by Go's math/rand/v2 over a PCG source
// so the TypeScript port (pkg/spec/src/pcg.ts) can be asserted bit-for-bit.
//
// Run from the spec package to regenerate:
//
// go run ./internal/_oracle > pkg/spec/test/fixtures/pcg-golden.json
package main
import (
"encoding/json"
"math/rand/v2"
"os"
)
// seed carries the PCG seed pair. The values are emitted as decimal strings so
// JavaScript parses them into BigInt without the float precision loss it would
// suffer on values above 2^53.
type seed struct {
Hi string `json:"hi"`
Lo string `json:"lo"`
}
type caseEntry struct {
Seed seed `json:"seed"`
Uint64 []string `json:"uint64"`
Float64 []float64 `json:"float64"`
IntN map[string][]int `json:"intN"`
}
const drawCount = 40
var seeds = [][2]uint64{
{1718000000000000000, 0},
{42, 0},
{0, 0},
{1, 2},
{18446744073709551615, 18446744073709551615},
}
var intNValues = []int{2, 3, 7, 15, 1000}
func main() {
cases := make([]caseEntry, 0, len(seeds))
for _, pair := range seeds {
hi, lo := pair[0], pair[1]
entry := caseEntry{
Seed: seed{Hi: formatUint64(hi), Lo: formatUint64(lo)},
IntN: map[string][]int{},
}
r := rand.New(rand.NewPCG(hi, lo))
for i := 0; i < drawCount; i++ {
entry.Uint64 = append(entry.Uint64, formatUint64(r.Uint64()))
}
r = rand.New(rand.NewPCG(hi, lo))
for i := 0; i < drawCount; i++ {
entry.Float64 = append(entry.Float64, r.Float64())
}
for _, n := range intNValues {
r = rand.New(rand.NewPCG(hi, lo))
draws := make([]int, 0, drawCount)
for i := 0; i < drawCount; i++ {
draws = append(draws, r.IntN(n))
}
entry.IntN[itoa(n)] = draws
}
cases = append(cases, entry)
}
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
if err := encoder.Encode(cases); err != nil {
panic(err)
}
}
// formatUint64 renders a uint64 as a decimal string so JavaScript can parse it
// into a BigInt without precision loss.
func formatUint64(v uint64) string {
if v == 0 {
return "0"
}
var buf [20]byte
i := len(buf)
for v > 0 {
i--
buf[i] = byte('0' + v%10)
v /= 10
}
return string(buf[i:])
}
func itoa(n int) string {
if n == 0 {
return "0"
}
neg := n < 0
if neg {
n = -n
}
var buf [20]byte
i := len(buf)
for n > 0 {
i--
buf[i] = byte('0' + n%10)
n /= 10
}
if neg {
i--
buf[i] = '-'
}
return string(buf[i:])
}
internal/android/android.go
+1
View File
@@ -1,3 +1,4 @@
// Package android boots and prepares an Android device or emulator for testing via adb.
package android
import (
internal/bundler/bundler.go
+53 -15
View File
@@ -1,3 +1,4 @@
// Package bundler compiles a TypeScript spec into a single JavaScript bundle via esbuild.
package bundler
import (
@@ -6,6 +7,7 @@ import (
"encoding/json"
"errors"
"fmt"
"path/filepath"
"strings"
esbuild "github.com/evanw/esbuild/pkg/api"
@@ -13,9 +15,14 @@ import (
type Options struct {
EntryFile string
Defines map[string]string
Aliases map[string]string
Sourcemap bool
// RuntimeFile, when set, is imported BEFORE the spec via a stdin entry so
// the bundle installs __sanderlingNextAction__ / __sanderlingExtractors__
// (the goja runtime entry wires the shared picker). Empty bundles the spec
// alone, as the bundle-check tool and unit fixtures do.
RuntimeFile string
Defines map[string]string
Aliases map[string]string
Sourcemap bool
}
type Result struct {
@@ -45,18 +52,36 @@ func Bundle(options Options) (Result, error) {
sourcemap = esbuild.SourceMapInline
}
output := esbuild.Build(esbuild.BuildOptions{
EntryPoints: []string{options.EntryFile},
Bundle: true,
Format: esbuild.FormatIIFE,
Target: esbuild.ES2020,
Platform: esbuild.PlatformNeutral,
Define: defines,
Alias: options.Aliases,
Sourcemap: sourcemap,
Write: false,
LogLevel: esbuild.LogLevelSilent,
})
buildOptions := esbuild.BuildOptions{
Bundle: true,
Format: esbuild.FormatIIFE,
Target: esbuild.ES2020,
Platform: esbuild.PlatformNeutral,
Define: defines,
Alias: options.Aliases,
Sourcemap: sourcemap,
Write: false,
LogLevel: esbuild.LogLevelSilent,
}
if options.RuntimeFile == "" {
buildOptions.EntryPoints = []string{options.EntryFile}
} else {
runtimeAbs, err := filepath.Abs(options.RuntimeFile)
if err != nil {
return Result{}, fmt.Errorf("runtime path: %w", err)
}
specAbs, err := filepath.Abs(options.EntryFile)
if err != nil {
return Result{}, fmt.Errorf("entry path: %w", err)
}
buildOptions.Stdin = &esbuild.StdinOptions{
Contents: fmt.Sprintf("import %q;\n%s", runtimeAbs, registrationEntry(specAbs)),
ResolveDir: filepath.Dir(specAbs),
Loader: esbuild.LoaderTS,
}
}
output := esbuild.Build(buildOptions)
if len(output.Errors) > 0 {
var messages []string
@@ -76,3 +101,16 @@ func Bundle(options Options) (Result, error) {
SHA256: hex.EncodeToString(sum[:]),
}, nil
}
// registrationEntry imports the spec as a namespace and copies its named
// exports onto globalThis so authors write plain `export const properties /
// actionsRoot / setup` without hand-assigning globalThis. The guards let web
// specs omit setup. esbuild keeps the exports because the trailer references
// them; output stays an IIFE.
func registrationEntry(specAbs string) string {
return fmt.Sprintf(`import * as __spec from %q;
if (__spec.actionsRoot !== undefined) globalThis.actions = __spec.actionsRoot;
if (__spec.properties !== undefined) globalThis.properties = __spec.properties;
if (__spec.setup !== undefined) globalThis.setup = __spec.setup;
`, specAbs)
}
internal/bundler/bundler_test.go
+53
View File
@@ -146,6 +146,59 @@ func TestBundle_ImportResolution(t *testing.T) {
}
}
func TestBundle_RegistersNamedExportsOnGlobalThis(t *testing.T) {
directory := t.TempDir()
runtimePath := filepath.Join(directory, "runtime.ts")
specPath := filepath.Join(directory, "spec.ts")
if err := os.WriteFile(runtimePath, []byte(`export {};`), 0o600); err != nil {
t.Fatal(err)
}
spec := `
export const properties = "PROPS_MARKER";
export const actionsRoot = "ACTIONS_MARKER";
export const setup = "SETUP_MARKER";
`
if err := os.WriteFile(specPath, []byte(spec), 0o600); err != nil {
t.Fatal(err)
}
result, err := Bundle(Options{EntryFile: specPath, RuntimeFile: runtimePath})
if err != nil {
t.Fatal(err)
}
body := string(result.JavaScript)
for _, marker := range []string{"PROPS_MARKER", "ACTIONS_MARKER", "SETUP_MARKER"} {
if !strings.Contains(body, marker) {
t.Errorf("named export %q not registered in bundle:\n%s", marker, body)
}
}
if !strings.Contains(body, "globalThis.actions") {
t.Errorf("trailer should assign globalThis.actions:\n%s", body)
}
}
func TestBundle_RegistersWithoutSetupExport(t *testing.T) {
directory := t.TempDir()
runtimePath := filepath.Join(directory, "runtime.ts")
specPath := filepath.Join(directory, "spec.ts")
if err := os.WriteFile(runtimePath, []byte(`export {};`), 0o600); err != nil {
t.Fatal(err)
}
spec := `
export const properties = "PROPS_MARKER";
export const actionsRoot = "ACTIONS_MARKER";
`
if err := os.WriteFile(specPath, []byte(spec), 0o600); err != nil {
t.Fatal(err)
}
result, err := Bundle(Options{EntryFile: specPath, RuntimeFile: runtimePath})
if err != nil {
t.Fatalf("spec without setup export should still bundle: %v", err)
}
if !strings.Contains(string(result.JavaScript), "PROPS_MARKER") {
t.Errorf("properties export not registered")
}
}
func TestBundle_ReportsSyntaxErrors(t *testing.T) {
entry := writeFixture(t, "broken.ts", `const x = ;`)
_, err := Bundle(Options{EntryFile: entry})
internal/bundler/web_bundle.go
+7 -27
View File
@@ -3,6 +3,7 @@ package bundler
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"path/filepath"
@@ -47,12 +48,14 @@ func BundleWeb(options WebOptions) (Result, error) {
defines := map[string]string{}
for key, value := range options.Defines {
defines["process.env."+key] = quoteJSString(value)
encoded, err := json.Marshal(value)
if err != nil {
return Result{}, fmt.Errorf("encode define %q: %w", key, err)
}
defines["process.env."+key] = string(encoded)
}
stdinContents := fmt.Sprintf(`import %q;
import %q;
`, runtimeAbs, specAbs)
stdinContents := fmt.Sprintf("import %q;\n%s", runtimeAbs, registrationEntry(specAbs))
output := esbuild.Build(esbuild.BuildOptions{
Stdin: &esbuild.StdinOptions{
@@ -88,26 +91,3 @@ import %q;
SHA256: hex.EncodeToString(sum[:]),
}, nil
}
func quoteJSString(value string) string {
var builder strings.Builder
builder.WriteByte('"')
for index := 0; index < len(value); index++ {
c := value[index]
switch c {
case '"', '\\':
builder.WriteByte('\\')
builder.WriteByte(c)
case '\n':
builder.WriteString("\\n")
case '\r':
builder.WriteString("\\r")
case '\t':
builder.WriteString("\\t")
default:
builder.WriteByte(c)
}
}
builder.WriteByte('"')
return builder.String()
}
internal/bundler/web_bundle_test.go
+36 -2
View File
@@ -34,8 +34,8 @@ export {};
const fakeSpec = `
const handle = (globalThis as { __sanderling__: { extract: (g: () => unknown) => unknown } }).__sanderling__.extract(() => 42);
(globalThis as { actions?: unknown }).actions = handle;
export {};
export const actionsRoot = handle;
export const properties = "WEB_PROPS_MARKER";
`
func TestBundleWeb_RegistersExpectedGlobals(t *testing.T) {
@@ -61,6 +61,8 @@ func TestBundleWeb_RegistersExpectedGlobals(t *testing.T) {
"__sanderlingExtractors__",
"__sanderlingNextAction__",
"__sanderling__",
"WEB_PROPS_MARKER",
"globalThis.actions",
} {
if !strings.Contains(source, expected) {
t.Errorf("bundle missing %q\nsource head:\n%s", expected, head(source, 500))
@@ -122,6 +124,38 @@ func TestBundleWeb_IsDeterministic(t *testing.T) {
}
}
// TestBundleWeb_InjectsSeedDefine asserts that a SANDERLING_SEED define is
// inlined into the bundle so --seed reaches the web runtime PRNG. esbuild
// replaces process.env.SANDERLING_SEED with the quoted literal at build time.
func TestBundleWeb_InjectsSeedDefine(t *testing.T) {
directory := t.TempDir()
runtimePath := filepath.Join(directory, "web-runtime.ts")
specPath := filepath.Join(directory, "spec.ts")
runtime := fakeRuntime + "\n(globalThis as Record<string, unknown>).__seed = process.env.SANDERLING_SEED;\n"
if err := os.WriteFile(runtimePath, []byte(runtime), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(specPath, []byte(fakeSpec), 0o644); err != nil {
t.Fatal(err)
}
result, err := BundleWeb(WebOptions{
EntryFile: specPath,
WebRuntimeFile: runtimePath,
Defines: map[string]string{"SANDERLING_SEED": "8675309"},
})
if err != nil {
t.Fatalf("BundleWeb: %v", err)
}
source := string(result.JavaScript)
if !strings.Contains(source, "8675309") {
t.Errorf("seed literal not inlined into bundle\nsource head:\n%s", head(source, 500))
}
if strings.Contains(source, "process.env.SANDERLING_SEED") {
t.Errorf("define left unsubstituted in bundle")
}
}
func head(text string, max int) string {
if len(text) <= max {
return text
internal/driver/chrome/driver.go
+35
View File
@@ -1,3 +1,4 @@
// Package chrome implements the device driver for web targets by driving Chrome over the DevTools protocol.
package chrome
import (
@@ -37,6 +38,13 @@ func New() *Driver {
chromedp.Flag("headless", true),
chromedp.Flag("disable-gpu", true),
chromedp.NoSandbox,
// CI runners give Chrome a tiny /dev/shm; without this the browser
// process hangs on startup and never reports its DevTools socket.
chromedp.Flag("disable-dev-shm-usage", true),
// Cold-starting Chrome on a loaded CI runner can take longer than the
// 20s default to print its DevTools websocket URL; give it more room
// so launch does not flake with "websocket url timeout reached".
chromedp.WSURLReadTimeout(60*time.Second),
)...,
)
tabCtx, tabCancel := chromedp.NewContext(allocCtx)
@@ -142,6 +150,33 @@ func (d *Driver) TapSelector(_ context.Context, selector string) error {
return chromedp.Run(d.tabCtx, chromedp.Click(target, chromedp.NodeVisible))
}
// doubleTapGap is the inter-tap delay for DoubleTap: short enough to land both
// events inside a sub-100 ms race window. The browser has no single double-tap
// primitive, so the gesture is two taps with this gap.
const doubleTapGap = 50 * time.Millisecond
func (d *Driver) DoubleTap(ctx context.Context, x, y int) error {
return webDoubleTap(ctx, func() error { return d.Tap(ctx, x, y) })
}
func (d *Driver) DoubleTapSelector(ctx context.Context, selector string) error {
return webDoubleTap(ctx, func() error { return d.TapSelector(ctx, selector) })
}
func webDoubleTap(ctx context.Context, tap func() error) error {
if err := tap(); err != nil {
return err
}
timer := time.NewTimer(doubleTapGap)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
}
return tap()
}
func (d *Driver) InputText(_ context.Context, text string) error {
return chromedp.Run(d.tabCtx,
chromedp.ActionFunc(func(ctx context.Context) error {
internal/driver/chrome/driver_test.go
+2
View File
@@ -1,3 +1,5 @@
//go:build browser
package chrome
import (
internal/driver/chrome/translate.go
+1 -1
View File
@@ -17,7 +17,7 @@ var attrNamePattern = regexp.MustCompile(`^[a-zA-Z][a-zA-Z0-9_-]*$`)
// "descPrefix:bar") into a CSS selector or XPath expression usable from the
// chrome driver's TapSelector fallback path. The boolean return is true when
// the result is XPath rather than CSS. Unknown prefixes pass through to a CSS
// attribute match by the same name so a Maestro-side addition (e.g. a future
// attribute match by the same name so a sidecar-side addition (e.g. a future
// "role:") works without a Sanderling release.
func TranslateStringSelector(selector string) (string, bool, error) {
if selector == "" {
internal/driver/driver.go
+3
View File
@@ -1,3 +1,4 @@
// Package driver defines the platform-agnostic device automation interface and shared types.
package driver
import (
@@ -16,6 +17,8 @@ type DeviceDriver interface {
Tap(ctx context.Context, x, y int) error
TapSelector(ctx context.Context, selector string) error
DoubleTap(ctx context.Context, x, y int) error
DoubleTapSelector(ctx context.Context, selector string) error
InputText(ctx context.Context, text string) error
Swipe(ctx context.Context, fromX, fromY, toX, toY int, duration time.Duration) error
PressKey(ctx context.Context, key string) error
internal/driver/mock/mock.go
+46 -27
View File
@@ -1,3 +1,4 @@
// Package mock provides an in-memory device driver that records actions for tests.
package mock
import (
@@ -11,37 +12,39 @@ import (
type ActionKind string
const (
ActionLaunch ActionKind = "launch"
ActionTerminate ActionKind = "terminate"
ActionTap ActionKind = "tap"
ActionTapSelector ActionKind = "tap_selector"
ActionInputText ActionKind = "input_text"
ActionSwipe ActionKind = "swipe"
ActionPressKey ActionKind = "press_key"
ActionLongPress ActionKind = "long_press"
ActionHierarchy ActionKind = "hierarchy"
ActionScreenshot ActionKind = "screenshot"
ActionSnapshot ActionKind = "snapshot"
ActionRecentLogs ActionKind = "recent_logs"
ActionWaitForIdle ActionKind = "wait_for_idle"
ActionHealth ActionKind = "health"
ActionMetrics ActionKind = "metrics"
ActionLaunch ActionKind = "launch"
ActionTerminate ActionKind = "terminate"
ActionTap ActionKind = "tap"
ActionTapSelector ActionKind = "tap_selector"
ActionDoubleTap ActionKind = "double_tap"
ActionDoubleTapSelector ActionKind = "double_tap_selector"
ActionInputText ActionKind = "input_text"
ActionSwipe ActionKind = "swipe"
ActionPressKey ActionKind = "press_key"
ActionLongPress ActionKind = "long_press"
ActionHierarchy ActionKind = "hierarchy"
ActionScreenshot ActionKind = "screenshot"
ActionSnapshot ActionKind = "snapshot"
ActionRecentLogs ActionKind = "recent_logs"
ActionWaitForIdle ActionKind = "wait_for_idle"
ActionHealth ActionKind = "health"
ActionMetrics ActionKind = "metrics"
)
type Action struct {
Kind ActionKind
BundleID string
ClearState bool
X, Y int
Kind ActionKind
BundleID string
ClearState bool
X, Y int
FromX, FromY int
ToX, ToY int
Duration time.Duration
Selector string
Text string
Key string
LogLevel string
LogSince time.Time
Idle time.Duration
ToX, ToY int
Duration time.Duration
Selector string
Text string
Key string
LogLevel string
LogSince time.Time
Idle time.Duration
}
// Driver is an in-memory Driver implementation for unit tests.
@@ -183,6 +186,22 @@ func (d *Driver) TapSelector(ctx context.Context, selector string) error {
return nil
}
func (d *Driver) DoubleTap(ctx context.Context, x, y int) error {
if err := d.failure(ActionDoubleTap); err != nil {
return err
}
d.record(Action{Kind: ActionDoubleTap, X: x, Y: y})
return nil
}
func (d *Driver) DoubleTapSelector(ctx context.Context, selector string) error {
if err := d.failure(ActionDoubleTapSelector); err != nil {
return err
}
d.record(Action{Kind: ActionDoubleTapSelector, Selector: selector})
return nil
}
func (d *Driver) InputText(ctx context.Context, text string) error {
if err := d.failure(ActionInputText); err != nil {
return err
internal/driver/mock/mock_test.go
-16
View File
@@ -9,22 +9,6 @@ import (
"github.com/priyanshujain/sanderling/internal/driver"
)
func TestNew_Defaults(t *testing.T) {
mock := New()
if !mock.HealthInfo.Ready {
t.Errorf("default HealthInfo should be ready")
}
if mock.HealthInfo.Platform != "android" {
t.Errorf("default platform: got %q", mock.HealthInfo.Platform)
}
if mock.HierarchyJSON == "" {
t.Errorf("default hierarchy should be a non-empty JSON")
}
if len(mock.Actions()) != 0 {
t.Errorf("fresh mock should have zero recorded actions")
}
}
func TestRecordsAllActionsInOrder(t *testing.T) {
mock := New()
ctx := context.Background()
internal/driver/sidecar/client.go
+29
View File
@@ -1,3 +1,4 @@
// Package sidecar implements the device driver by talking to the native sidecar over gRPC.
package sidecar
import (
@@ -103,6 +104,34 @@ func (c *Client) TapSelector(ctx context.Context, selector string) error {
return err
}
// doubleTapGap is the inter-tap delay for DoubleTap: short enough to land both
// events inside a sub-100 ms race window, long enough for the sidecar to
// serialize two MotionEvent streams. The sidecar exposes no native double-tap
// RPC, so the gesture is two Taps with this gap.
const doubleTapGap = 50 * time.Millisecond
func (c *Client) DoubleTap(ctx context.Context, x, y int) error {
return doubleTap(ctx, func() error { return c.Tap(ctx, x, y) })
}
func (c *Client) DoubleTapSelector(ctx context.Context, selector string) error {
return doubleTap(ctx, func() error { return c.TapSelector(ctx, selector) })
}
func doubleTap(ctx context.Context, tap func() error) error {
if err := tap(); err != nil {
return err
}
timer := time.NewTimer(doubleTapGap)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
}
return tap()
}
func (c *Client) InputText(ctx context.Context, text string) error {
_, err := c.stub.InputText(ctx, &driverpb.Text{Value: text})
return err
internal/hierarchy/hierarchy.go
+5 -5
View File
@@ -1,4 +1,4 @@
// Package hierarchy parses the TreeNode JSON produced by the Maestro sidecar
// Package hierarchy parses the TreeNode JSON produced by the native sidecar
// and resolves selectors against it.
//
// Selector grammar (v2.0):
@@ -83,7 +83,7 @@ type Tree struct {
Elements []*Element `json:"elements"`
}
// treeNodeJSON mirrors the Maestro TreeNode JSON structure.
// treeNodeJSON mirrors the sidecar TreeNode JSON structure.
type treeNodeJSON struct {
Attributes map[string]string `json:"attributes"`
Children []treeNodeJSON `json:"children"`
@@ -110,7 +110,7 @@ type AttrFilter struct {
// in the TreeNode attributes map. Both directions are listed so cross-platform
// matching works regardless of which name the caller uses.
var attributeAliases = map[string][]string{
// Android XML legacy name; web driver uses content-desc; Maestro normalises to accessibilityText
// Android XML legacy name; web driver uses content-desc; the sidecar normalises to accessibilityText
"content-desc": {"accessibilityText"},
// iOS AXElement / UIKit names
"label": {"accessibilityText"},
@@ -164,7 +164,7 @@ func matchSelector(element *Element, sel Selector) bool {
return true
}
// Parse parses a Maestro TreeNode JSON hierarchy.
// Parse parses a sidecar TreeNode JSON hierarchy.
func Parse(text string) (*Tree, error) {
text = strings.TrimSpace(text)
if text == "" {
@@ -550,7 +550,7 @@ func match(element *Element, kind, value string) bool {
}
}
// boundsPattern matches "[l,t,r,b]" (4-value Android/Maestro format).
// boundsPattern matches "[l,t,r,b]" (4-value Android/sidecar format).
var boundsPattern = regexp.MustCompile(`^\[(-?\d+),(-?\d+),(-?\d+),(-?\d+)\]$`)
// boundsPatternTwo matches "[x1,y1][x2,y2]" (iOS XCUITest format).
internal/hierarchy/hierarchy_test.go
+1 -1
View File
@@ -2,7 +2,7 @@ package hierarchy
import "testing"
// sampleDump is a Maestro TreeNode JSON equivalent of the old XML fixture.
// sampleDump is a sidecar TreeNode JSON equivalent of the old XML fixture.
const sampleDump = `{
"attributes": {"class": "android.widget.LinearLayout", "package": "app", "bounds": "[0,0,1080,2340]"},
"children": [
internal/inspect/assets.go
+1
View File
@@ -1,3 +1,4 @@
// Package inspect serves the embedded web UI for browsing recorded runs.
package inspect
import (
internal/ios/ios.go
+1
View File
@@ -1,3 +1,4 @@
// Package ios boots and prepares an iOS simulator for testing via simctl.
package ios
import (
internal/ltl/evaluator.go
+228 -28
View File
@@ -1,3 +1,4 @@
// Package ltl evaluates linear temporal logic formulas incrementally over observed steps.
package ltl
import (
@@ -31,13 +32,27 @@ func (v Verdict) String() string {
// violated) or carries them forward as residuals. Once a single obligation
// violates, the overall verdict latches to Violated.
type Evaluator struct {
root Formula
pending []Formula
violated bool
root Formula
pending []Formula
violated bool
steps int
violation *Violation
}
// Violation is the witness for a latched verdict: the failing sub-formula, a
// human-readable reason, and the observation step it fired at. A thrown
// predicate carries the goja error text as its reason and sets IsError; a plain
// false carries "predicate false"; Finalize fills it for liveness obligations
// that never discharged.
type Violation struct {
Formula Formula
Reason string
Step int
IsError bool
}
func NewEvaluator(formula Formula) *Evaluator {
return &Evaluator{root: formula}
return &Evaluator{root: nnf(formula)}
}
// Observe evaluates the formula against the current state and returns the
@@ -52,6 +67,7 @@ func (e *Evaluator) ObserveAt(now time.Time) Verdict {
if e.violated {
return VerdictViolated
}
e.steps++
fresh := rootObligation(e.root)
obligations := append(e.pending, fresh)
@@ -65,18 +81,136 @@ func (e *Evaluator) ObserveAt(now time.Time) Verdict {
case statusViolated:
e.violated = true
e.pending = nil
e.violation = result.witness
if e.violation != nil {
e.violation.Step = e.steps
}
return VerdictViolated
case statusPending:
e.pending = append(e.pending, result.formula)
}
}
e.pending = collapse(e.pending)
if len(e.pending) > 0 {
return VerdictPending
}
return VerdictHolds
}
// collapse removes structurally-identical obligations, keeping the first
// occurrence in order. Distinct predicates never merge because ThunkFormula's
// name participates in its describe() key, so deduping cannot hide a violation.
func collapse(obligations []Formula) []Formula {
if len(obligations) < 2 {
return obligations
}
seen := make(map[string]struct{}, len(obligations))
result := obligations[:0]
for _, obligation := range obligations {
key := obligation.describe()
if _, ok := seen[key]; ok {
continue
}
seen[key] = struct{}{}
result = append(result, obligation)
}
return result
}
// Finalize reports the terminal verdict for the run. Pending obligations that
// can never be discharged by a future step (an unbounded eventually that never
// fired, a strong next with no successor) resolve to Violated; safety
// obligations that were never breached resolve to Holds.
func (e *Evaluator) Finalize() Verdict {
if e.violated {
return VerdictViolated
}
for _, obligation := range e.pending {
if finalize(obligation) == statusViolated {
e.violated = true
e.pending = nil
e.violation = &Violation{
Formula: obligation,
Reason: finalizeReason(obligation),
Step: e.steps,
}
return VerdictViolated
}
}
return VerdictHolds
}
// Violation returns the witness for a latched violation, or nil if the
// evaluator has not violated. The witness is set by ObserveAt at the step a
// reduction first violated, or by Finalize for a liveness obligation that
// never discharged.
func (e *Evaluator) Violation() *Violation {
return e.violation
}
// finalizeReason describes why an undischarged obligation resolves to violated
// at run end.
func finalizeReason(formula Formula) string {
switch formula.(type) {
case EventuallyFormula:
return "eventually never satisfied"
case NextFormula:
return "next obligation unmet at run end"
case ThunkFormula:
return "obligation unmet at run end"
default:
return "liveness obligation unmet at run end"
}
}
// finalize collapses a pending obligation to its terminal status assuming no
// further steps will occur.
func finalize(formula Formula) residualStatus {
switch concrete := formula.(type) {
case PureFormula:
if concrete.Value {
return statusHolds
}
return statusViolated
case ThunkFormula:
return statusViolated
case EventuallyFormula:
return statusViolated
case NextFormula:
return statusViolated
case AlwaysFormula:
return statusHolds
case NowFormula:
return finalize(concrete.Inner)
case NotFormula:
switch finalize(concrete.Inner) {
case statusViolated:
return statusHolds
default:
return statusViolated
}
case AndFormula:
if finalize(concrete.Left) == statusViolated || finalize(concrete.Right) == statusViolated {
return statusViolated
}
return statusHolds
case OrFormula:
if finalize(concrete.Left) == statusHolds || finalize(concrete.Right) == statusHolds {
return statusHolds
}
return statusViolated
case ImpliesFormula:
if finalize(concrete.Antecedent) == statusViolated {
return statusHolds
}
return finalize(concrete.Consequent)
default:
return statusHolds
}
}
// Residual returns a single Formula describing what the evaluator still has
// to prove after the most recent ObserveAt. PureFormula{true} means the
// property holds for the run so far; PureFormula{false} means it has latched
@@ -119,10 +253,46 @@ const (
type reduceResult struct {
status residualStatus
formula Formula
witness *Violation
}
func holds() reduceResult { return reduceResult{status: statusHolds} }
func holds() reduceResult { return reduceResult{status: statusHolds} }
// violated reports a violation without an attached witness. Used where the
// failing sub-formula is recovered from a child result whose own witness is
// carried up by violatedFrom.
func violated() reduceResult { return reduceResult{status: statusViolated} }
// violatedWith reports a violation that originates at the given sub-formula
// with the given reason. The reason distinguishes a thrown predicate from a
// plain false so callers (and the inspect UI) can render the cause.
func violatedWith(formula Formula, reason string) reduceResult {
return reduceResult{
status: statusViolated,
witness: &Violation{Formula: formula, Reason: reason},
}
}
// violatedByError reports a violation caused by a predicate that threw. The
// witness keeps the error text as its reason and flags IsError so callers can
// render it as a thrown-predicate error rather than a plain false.
func violatedByError(formula Formula, reason string) reduceResult {
return reduceResult{
status: statusViolated,
witness: &Violation{Formula: formula, Reason: reason, IsError: true},
}
}
// violatedFrom propagates a child violation, preferring the child's witness so
// the deepest failing leaf survives. When the child carried no witness the
// fallback formula and reason describe this level instead.
func violatedFrom(child reduceResult, fallback Formula, reason string) reduceResult {
if child.witness != nil {
return reduceResult{status: statusViolated, witness: child.witness}
}
return violatedWith(fallback, reason)
}
func pending(f Formula) reduceResult {
return reduceResult{status: statusPending, formula: f}
}
@@ -133,13 +303,17 @@ func reduce(formula Formula, now time.Time) reduceResult {
if concrete.Value {
return holds()
}
return violated()
return violatedWith(concrete, "pure false")
case ThunkFormula:
if concrete.Func() {
result, err := concrete.Func()
if err != nil {
return violatedByError(concrete, err.Error())
}
if result {
return holds()
}
return violated()
return violatedWith(concrete, "predicate false")
case NowFormula:
return reduce(concrete.Inner, now)
@@ -162,10 +336,10 @@ func reduce(formula Formula, now time.Time) reduceResult {
return holds()
}
if concrete.HasStepBound && concrete.StepBound <= 1 {
return violated()
return violatedFrom(innerResult, concrete, "eventually bound exhausted")
}
if concrete.HasDeadline && !now.Before(concrete.Deadline) {
return violated()
return violatedFrom(innerResult, concrete, "eventually deadline reached")
}
next := concrete
if concrete.HasStepBound {
@@ -174,18 +348,14 @@ func reduce(formula Formula, now time.Time) reduceResult {
return pending(next)
case ImpliesFormula:
antecedent := reduce(concrete.Antecedent, now)
switch antecedent.status {
case statusHolds:
return reduce(concrete.Consequent, now)
case statusViolated:
return holds()
case statusPending:
return pending(ImpliesFormula{
Antecedent: antecedent.formula,
Consequent: concrete.Consequent,
})
}
// NewEvaluator runs nnf, which rewrites a -> b to (not a) or b, so this
// case is unreachable from a normal evaluator. A directly-constructed
// formula reduced here is evaluated through the same equivalence so a
// pending antecedent cannot drop the consequent.
return reduce(OrFormula{
Left: pushNot(concrete.Antecedent),
Right: nnf(concrete.Consequent),
}, now)
case OrFormula:
left := reduce(concrete.Left, now)
@@ -194,7 +364,7 @@ func reduce(formula Formula, now time.Time) reduceResult {
return holds()
}
if left.status == statusViolated && right.status == statusViolated {
return violated()
return violatedFrom(left, concrete, "both disjuncts violated")
}
if left.status == statusViolated {
return pending(right.formula)
@@ -207,8 +377,11 @@ func reduce(formula Formula, now time.Time) reduceResult {
case AndFormula:
left := reduce(concrete.Left, now)
right := reduce(concrete.Right, now)
if left.status == statusViolated || right.status == statusViolated {
return violated()
if left.status == statusViolated {
return violatedFrom(left, concrete, "conjunct violated")
}
if right.status == statusViolated {
return violatedFrom(right, concrete, "conjunct violated")
}
if left.status == statusHolds && right.status == statusHolds {
return holds()
@@ -225,7 +398,7 @@ func reduce(formula Formula, now time.Time) reduceResult {
inner := reduce(concrete.Inner, now)
switch inner.status {
case statusHolds:
return violated()
return violatedWith(concrete, "negated formula held")
case statusViolated:
return holds()
case statusPending:
@@ -233,11 +406,38 @@ func reduce(formula Formula, now time.Time) reduceResult {
}
case AlwaysFormula:
// First-reduction deadline resolution mirrors EventuallyFormula so a
// relative duration becomes a stable absolute deadline.
if !concrete.HasDeadline && concrete.Duration > 0 {
concrete.Deadline = now.Add(concrete.Duration)
concrete.HasDeadline = true
}
innerResult := reduce(concrete.Inner, now)
if innerResult.status == statusViolated {
return violated()
return violatedFrom(innerResult, concrete, "always inner violated")
}
// A bounded Always is the dual of a bounded Eventually: once the window
// closes without a breach it is vacuously satisfied. A pending inner at
// the closing step is a deferred obligation (a strong next, or an inner
// liveness that has not discharged); it must be carried so a later step
// or Finalize resolves it, never dropped to holds.
if concrete.HasStepBound && concrete.StepBound <= 1 {
if innerResult.status == statusHolds {
return holds()
}
return pending(innerResult.formula)
}
if concrete.HasDeadline && !now.Before(concrete.Deadline) {
if innerResult.status == statusHolds {
return holds()
}
return pending(innerResult.formula)
}
next := concrete
next.Inner = concrete.Inner
if concrete.HasStepBound {
next.StepBound = concrete.StepBound - 1
}
next := AlwaysFormula{Inner: concrete.Inner}
if innerResult.status == statusHolds {
return pending(next)
}
internal/ltl/evaluator_test.go
+5 -14
View File
@@ -36,10 +36,10 @@ func TestPure_FalseImmediatelyViolates(t *testing.T) {
func TestThunk_TransitionFromHoldToViolate(t *testing.T) {
values := []bool{true, true, false, true, true}
step := 0
evaluator := NewEvaluator(Always(Thunk(func() bool {
evaluator := NewEvaluator(Always(Thunk(func() (bool, error) {
current := values[step]
step++
return current
return current, nil
})))
wantSequence := []Verdict{
@@ -59,7 +59,7 @@ func TestThunk_TransitionFromHoldToViolate(t *testing.T) {
func TestEvaluator_StickinessAfterViolation(t *testing.T) {
state := true
evaluator := NewEvaluator(Always(Thunk(func() bool { return state })))
evaluator := NewEvaluator(Always(Thunk(func() (bool, error) { return state, nil })))
if got := evaluator.Observe(); got != VerdictHolds {
t.Fatalf("step 1: got %v, want holds", got)
@@ -83,7 +83,7 @@ func TestEvaluator_TopLevelPureCountedAtEachStep(t *testing.T) {
func TestEvaluator_TopLevelThunkRespectsObservation(t *testing.T) {
state := true
evaluator := NewEvaluator(Thunk(func() bool { return state }))
evaluator := NewEvaluator(Thunk(func() (bool, error) { return state, nil }))
if got := evaluator.Observe(); got != VerdictHolds {
t.Errorf("expected holds, got %v", got)
}
@@ -93,21 +93,12 @@ func TestEvaluator_TopLevelThunkRespectsObservation(t *testing.T) {
}
}
func TestVerdict_String(t *testing.T) {
if VerdictHolds.String() != "holds" {
t.Errorf("VerdictHolds.String() = %q", VerdictHolds.String())
}
if VerdictViolated.String() != "violated" {
t.Errorf("VerdictViolated.String() = %q", VerdictViolated.String())
}
}
func TestDescribe(t *testing.T) {
formula := Always(Pure(true))
if got := Describe(formula); !strings.Contains(got, "Always") || !strings.Contains(got, "Pure(true)") {
t.Errorf("Describe wrong: %q", got)
}
thunk := Always(Thunk(func() bool { return true }))
thunk := Always(Thunk(func() (bool, error) { return true, nil }))
if got := Describe(thunk); !strings.Contains(got, "Thunk") {
t.Errorf("Describe(thunk) wrong: %q", got)
}
internal/ltl/false_negative_test.go
+92
View File
@@ -0,0 +1,92 @@
package ltl
import (
"testing"
"time"
)
// thunkSeq returns a predicate that yields the given boolean per observation,
// repeating the last value once the sequence is exhausted.
func thunkSeq(values ...bool) func() (bool, error) {
step := 0
return func() (bool, error) {
value := values[len(values)-1]
if step < len(values) {
value = values[step]
}
step++
return value, nil
}
}
func runAndFinalize(formula Formula, steps int) (Verdict, *Evaluator) {
evaluator := NewEvaluator(formula)
var last Verdict
for index := range steps {
last = evaluator.ObserveAt(time.Unix(int64(index), 0))
if last == VerdictViolated {
return last, evaluator
}
}
return evaluator.Finalize(), evaluator
}
// Implies with a temporal antecedent must still evaluate the consequent at the
// current step. Always(p) holds over the observed run, so a false consequent
// Not(q) at the last step makes the implication violated. The pre-fix engine
// deferred the whole implication and reported Holds.
func TestImplies_TemporalAntecedent_ConsequentFalseViolates(t *testing.T) {
p := Thunk(thunkSeq(true, true, true))
q := Thunk(thunkSeq(false, false, true))
formula := Implies(Always(p), Not(q))
if verdict, _ := runAndFinalize(formula, 3); verdict != VerdictViolated {
t.Fatalf("Implies(Always(p),Not(q)) p=TTT q=FFT: got %v, want violated", verdict)
}
}
// Single-step witness of the same class: antecedent holds, consequent false.
func TestImplies_TemporalAntecedent_SingleStepViolates(t *testing.T) {
formula := Implies(Always(Thunk(thunkSeq(true))), Not(Thunk(thunkSeq(true))))
if verdict, _ := runAndFinalize(formula, 1); verdict != VerdictViolated {
t.Fatalf("Implies(Always(true),Not(true)): got %v, want violated", verdict)
}
}
// A consequent inside an Or must not mask the violation either.
func TestImplies_TemporalAntecedent_OrConsequentViolates(t *testing.T) {
formula := Implies(Always(Thunk(thunkSeq(true))), Or(Not(Thunk(thunkSeq(true))), Pure(false)))
if verdict, _ := runAndFinalize(formula, 1); verdict != VerdictViolated {
t.Fatalf("Implies(Always(true),Or(Not(true),false)): got %v, want violated", verdict)
}
}
// Control: a false antecedent makes the implication vacuously hold.
func TestImplies_TemporalAntecedent_FalseAntecedentHolds(t *testing.T) {
p := Thunk(thunkSeq(true, true, false))
q := Thunk(thunkSeq(false, false, true))
formula := Implies(Always(p), Not(q))
if verdict, _ := runAndFinalize(formula, 3); verdict != VerdictHolds {
t.Fatalf("Implies(Always(p),Not(q)) p=TTF q=FFT: got %v, want holds", verdict)
}
}
// A bounded Always whose inner is a still-pending deferred Next obligation must
// carry that obligation past the window, not drop it to holds. The pre-fix
// window-close branch returned holds() unconditionally and lost the violation.
func TestBoundedAlways_PendingInnerCarried(t *testing.T) {
inner := AlwaysFormula{Inner: Next(Thunk(thunkSeq(false))), StepBound: 1, HasStepBound: true}
formula := Always(inner)
if verdict, _ := runAndFinalize(formula, 3); verdict != VerdictViolated {
t.Fatalf("Always(boundedAlways(Next(false),1)): got %v, want violated", verdict)
}
}
// A bounded Always whose inner genuinely holds each step stays satisfied: the
// fix must not turn a satisfied bounded window into a false positive.
func TestBoundedAlways_HoldingInnerStillHolds(t *testing.T) {
inner := AlwaysFormula{Inner: Not(Thunk(thunkSeq(false))), StepBound: 1, HasStepBound: true}
formula := Always(inner)
if verdict, _ := runAndFinalize(formula, 3); verdict != VerdictHolds {
t.Fatalf("Always(boundedAlways(Not(false),1)): got %v, want holds", verdict)
}
}
internal/ltl/finalize_test.go
+161
View File
@@ -0,0 +1,161 @@
package ltl
import (
"testing"
"testing/quick"
"time"
)
func TestFinalize_UnboundedEventuallyUnmetIsViolated(t *testing.T) {
evaluator := NewEvaluator(Eventually(ThunkNamed("p", func() (bool, error) { return false, nil })))
for index := range 3 {
if got := evaluator.ObserveAt(time.Unix(int64(index), 0)); got != VerdictPending {
t.Fatalf("step %d: got %v, want pending", index, got)
}
}
if got := evaluator.Finalize(); got != VerdictViolated {
t.Errorf("Finalize = %v, want violated", got)
}
}
func TestFinalize_FinalStepNextIsViolated(t *testing.T) {
evaluator := NewEvaluator(Next(ThunkNamed("p", func() (bool, error) { return true, nil })))
if got := evaluator.Observe(); got != VerdictPending {
t.Fatalf("step 1: got %v, want pending", got)
}
if got := evaluator.Finalize(); got != VerdictViolated {
t.Errorf("Finalize = %v, want violated", got)
}
}
func TestFinalize_HoldingRunStaysHolds(t *testing.T) {
evaluator := NewEvaluator(Always(Pure(true)))
evaluator.Observe()
if got := evaluator.Finalize(); got != VerdictHolds {
t.Errorf("Finalize = %v, want holds", got)
}
}
func TestFinalize_AlreadyViolatedStaysViolated(t *testing.T) {
evaluator := NewEvaluator(Always(Pure(false)))
if got := evaluator.Observe(); got != VerdictViolated {
t.Fatalf("expected violated, got %v", got)
}
if got := evaluator.Finalize(); got != VerdictViolated {
t.Errorf("Finalize = %v, want violated", got)
}
}
func TestFinalize_BoundedAlwaysVacuouslyHolds(t *testing.T) {
// A bounded Always whose window never closed (still pending) is safe.
evaluator := NewEvaluator(EventuallyWithinSteps(Pure(false), 5))
evaluator.Observe()
// The negated form of this is a bounded Always; build it directly.
bounded := NewEvaluator(Always(Not(EventuallyWithinSteps(ThunkNamed("p", func() (bool, error) { return false, nil }), 5))))
bounded.Observe()
if got := bounded.Finalize(); got == VerdictViolated {
t.Errorf("bounded always should not finalize to violated, got %v", got)
}
}
// TestEventuallyWithin_ViolatesIffNConsecutiveFalse locks the bounded
// eventually contract: with a step bound of n and an inner that is false for
// the first n observations, the verdict violates exactly at step n, and with at
// least one true observation inside the window it holds.
func TestEventuallyWithin_ViolatesIffNConsecutiveFalse(t *testing.T) {
law := func(boundSeed uint8, trueAtSeed uint8) bool {
bound := int(boundSeed%5) + 1
// trueAt < 0 means inner is never true.
trueAt := int(trueAtSeed)%(bound+2) - 1
step := 0
inner := ThunkNamed("p", func() (bool, error) {
current := trueAt >= 0 && step == trueAt
return current, nil
})
evaluator := NewEvaluator(EventuallyWithinSteps(inner, bound))
satisfiedInWindow := trueAt >= 0 && trueAt < bound
var final Verdict = VerdictPending
for index := range bound {
step = index
final = evaluator.ObserveAt(time.Unix(int64(index), 0))
if final == VerdictHolds || final == VerdictViolated {
break
}
}
if satisfiedInWindow {
return final == VerdictHolds
}
return final == VerdictViolated
}
if err := quick.Check(law, nil); err != nil {
t.Error(err)
}
}
// TestViolationLatchIsMonotonic locks: once an evaluator reports Violated, every
// subsequent observation (and Finalize) stays Violated regardless of inputs.
func TestViolationLatchIsMonotonic(t *testing.T) {
law := func(seed uint64) bool {
values := make([]bool, 8)
for index := range values {
values[index] = (seed>>uint(index))&1 == 1
}
step := 0
evaluator := NewEvaluator(Always(ThunkNamed("p", func() (bool, error) {
current := values[step%len(values)]
step++
return current, nil
})))
seenViolated := false
for index := range 16 {
got := evaluator.ObserveAt(time.Unix(int64(index), 0))
if got == VerdictViolated {
seenViolated = true
} else if seenViolated {
return false
}
}
if seenViolated && evaluator.Finalize() != VerdictViolated {
return false
}
return true
}
if err := quick.Check(law, nil); err != nil {
t.Error(err)
}
}
func TestCollapse_IdenticalObligationsMerge(t *testing.T) {
merged := collapse([]Formula{
Next(Pure(true)),
Next(Pure(true)),
Next(Pure(true)),
})
if len(merged) != 1 {
t.Errorf("expected 1 obligation after collapse, got %d", len(merged))
}
}
func TestCollapse_DistinctPredicatesDoNotMerge(t *testing.T) {
merged := collapse([]Formula{
Eventually(ThunkNamed("p3", func() (bool, error) { return false, nil })),
Eventually(ThunkNamed("p4", func() (bool, error) { return false, nil })),
})
if len(merged) != 2 {
t.Errorf("distinct predicates must not merge, got %d", len(merged))
}
}
func TestCollapse_NamedThunkLeakBoundsPendingSet(t *testing.T) {
// Always(Eventually(sameThunk)): each step spawns an identical obligation.
// Without collapse the pending set grows unboundedly.
evaluator := NewEvaluator(Always(Eventually(ThunkNamed("p", func() (bool, error) { return false, nil }))))
for index := range 20 {
evaluator.ObserveAt(time.Unix(int64(index), 0))
}
if len(evaluator.pending) > 2 {
t.Errorf("pending set leaked to %d obligations", len(evaluator.pending))
}
}
internal/ltl/formula.go
+62 -15
View File
@@ -13,9 +13,9 @@ type Formula interface {
describe() string
}
// PredicateLabel lets a ThunkFormula carry a human-readable name for the
// closure it wraps. Verifier wires this in when the spec gives the predicate
// a property name; otherwise it stays empty and serializes without a name.
// PredicateLabel lets a ThunkFormula expose the identity of the closure it
// wraps. ThunkFormula satisfies it through its Name field; an empty name
// serializes without a name.
type PredicateLabel interface {
PredicateName() string
}
@@ -33,18 +33,35 @@ func (e ErrorFormula) describe() string {
return fmt.Sprintf("Error(%q)", e.Message)
}
// AlwaysFormula obliges its inner formula to hold at every step. A bounded
// Always (the dual of a bounded Eventually) holds for the steps inside its
// window and is vacuously satisfied once the window closes. An unbounded
// Always carries no bound fields and is checked at every observed step.
type AlwaysFormula struct {
Inner Formula
Inner Formula
StepBound int
HasStepBound bool
Duration time.Duration
Deadline time.Time
HasDeadline bool
}
type PureFormula struct {
Value bool
}
// ThunkFormula wraps an opaque predicate closure. Func returns the predicate's
// boolean result and a non-nil error when the predicate threw; a thrown
// predicate is a witnessed violation distinct from a plain false. Name carries
// the predicate's identity so two distinct predicates produce distinct
// describe() keys and are never merged during obligation collapse.
type ThunkFormula struct {
Func func() bool
Func func() (bool, error)
Name string
}
func (t ThunkFormula) PredicateName() string { return t.Name }
// NowFormula marks its inner formula for evaluation at the current step only.
// Primarily used so that now(...).implies(...) parses unambiguously.
type NowFormula struct {
@@ -96,7 +113,11 @@ func Always(inner Formula) Formula { return AlwaysFormula{Inner: inner} }
func Pure(value bool) Formula { return PureFormula{Value: value} }
func Thunk(function func() bool) Formula { return ThunkFormula{Func: function} }
func Thunk(function func() (bool, error)) Formula { return ThunkFormula{Func: function} }
func ThunkNamed(name string, function func() (bool, error)) Formula {
return ThunkFormula{Func: function, Name: name}
}
func Now(inner Formula) Formula { return NowFormula{Inner: inner} }
@@ -137,11 +158,27 @@ func (OrFormula) isFormula() {}
func (AndFormula) isFormula() {}
func (NotFormula) isFormula() {}
func (a AlwaysFormula) describe() string { return "Always(" + a.Inner.describe() + ")" }
func (p PureFormula) describe() string { return fmt.Sprintf("Pure(%t)", p.Value) }
func (ThunkFormula) describe() string { return "Thunk(...)" }
func (n NowFormula) describe() string { return "Now(" + n.Inner.describe() + ")" }
func (n NextFormula) describe() string { return "Next(" + n.Inner.describe() + ")" }
func (a AlwaysFormula) describe() string {
parts := []string{a.Inner.describe()}
if a.HasStepBound {
parts = append(parts, fmt.Sprintf("steps=%d", a.StepBound))
}
if a.HasDeadline {
parts = append(parts, "deadline="+a.Deadline.Format(time.RFC3339Nano))
} else if a.Duration > 0 {
parts = append(parts, "within="+a.Duration.String())
}
return "Always(" + strings.Join(parts, ", ") + ")"
}
func (p PureFormula) describe() string { return fmt.Sprintf("Pure(%t)", p.Value) }
func (t ThunkFormula) describe() string {
if t.Name != "" {
return "Thunk(" + t.Name + ")"
}
return "Thunk(...)"
}
func (n NowFormula) describe() string { return "Now(" + n.Inner.describe() + ")" }
func (n NextFormula) describe() string { return "Next(" + n.Inner.describe() + ")" }
func (e EventuallyFormula) describe() string {
parts := []string{e.Inner.describe()}
if e.HasStepBound {
@@ -176,10 +213,20 @@ type withinNode struct {
}
func (a AlwaysFormula) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Op string `json:"op"`
Arg Formula `json:"arg"`
}{"always", a.Inner})
payload := struct {
Op string `json:"op"`
Arg Formula `json:"arg"`
Within *withinNode `json:"within,omitempty"`
}{Op: "always", Arg: a.Inner}
switch {
case a.HasStepBound:
payload.Within = &withinNode{Amount: int64(a.StepBound), Unit: "steps"}
case a.Duration > 0:
payload.Within = &withinNode{Amount: a.Duration.Milliseconds(), Unit: "milliseconds"}
case a.HasDeadline:
payload.Within = &withinNode{Amount: a.Deadline.UnixMilli(), Unit: "deadline"}
}
return json.Marshal(payload)
}
func (n NowFormula) MarshalJSON() ([]byte, error) {
internal/ltl/formula_test.go
+7 -13
View File
@@ -50,7 +50,7 @@ func TestAlways_Now_ViolatesImmediately(t *testing.T) {
func TestAlways_Next_PendingThenViolated(t *testing.T) {
y := true
evaluator := NewEvaluator(Always(Next(Thunk(func() bool { return y }))))
evaluator := NewEvaluator(Always(Next(Thunk(func() (bool, error) { return y, nil }))))
if got := evaluator.Observe(); got != VerdictPending {
t.Errorf("step 1: got %v, want pending", got)
@@ -62,7 +62,7 @@ func TestAlways_Next_PendingThenViolated(t *testing.T) {
}
func TestAlways_Next_StaysPendingWhileInnerHolds(t *testing.T) {
evaluator := NewEvaluator(Always(Next(Thunk(func() bool { return true }))))
evaluator := NewEvaluator(Always(Next(Thunk(func() (bool, error) { return true, nil }))))
for index := range 3 {
if got := evaluator.ObserveAt(time.Unix(int64(index), 0)); got != VerdictPending {
t.Errorf("step %d: got %v, want pending", index+1, got)
@@ -76,8 +76,8 @@ func TestAlways_NowImpliesEventuallyWithin_ViolatesWhenYLate(t *testing.T) {
xValues := []bool{true, false, false, false, false}
yValues := []bool{false, false, false, true, true}
step := 0
predX := Thunk(func() bool { return xValues[step] })
predY := Thunk(func() bool { return yValues[step] })
predX := Thunk(func() (bool, error) { return xValues[step], nil })
predY := Thunk(func() (bool, error) { return yValues[step], nil })
formula := Always(Implies(Now(predX), EventuallyWithinSteps(predY, 3)))
evaluator := NewEvaluator(formula)
@@ -107,8 +107,8 @@ func TestAlways_NowImpliesEventuallyWithin_HoldsWhenYInBound(t *testing.T) {
xValues := []bool{true, false, false}
yValues := []bool{false, false, true}
step := 0
predX := Thunk(func() bool { return xValues[step] })
predY := Thunk(func() bool { return yValues[step] })
predX := Thunk(func() (bool, error) { return xValues[step], nil })
predY := Thunk(func() (bool, error) { return yValues[step], nil })
formula := Always(Implies(Now(predX), EventuallyWithinSteps(predY, 3)))
evaluator := NewEvaluator(formula)
@@ -187,12 +187,6 @@ func TestNot_InvertsPure(t *testing.T) {
}
}
func TestVerdict_StringPending(t *testing.T) {
if got := VerdictPending.String(); got != "pending" {
t.Errorf("VerdictPending.String() = %q", got)
}
}
func TestMarshalJSON_AlwaysImpliesEventually(t *testing.T) {
formula := Always(Implies(Now(Pure(true)), EventuallyWithinSteps(Pure(false), 3)))
body, err := json.Marshal(formula)
@@ -231,7 +225,7 @@ func TestMarshalJSON_NextAndThunkAndError(t *testing.T) {
if string(body) != `{"op":"next","arg":{"op":"true"}}` {
t.Errorf("next marshal wrong: %s", body)
}
body, _ = json.Marshal(Thunk(func() bool { return true }))
body, _ = json.Marshal(Thunk(func() (bool, error) { return true, nil }))
if string(body) != `{"op":"predicate"}` {
t.Errorf("thunk marshal wrong: %s", body)
}
internal/ltl/nnf.go
+87
View File
@@ -0,0 +1,87 @@
package ltl
// nnf rewrites a formula into negation normal form: every NotFormula is pushed
// down until it wraps only an opaque leaf (a ThunkFormula or ErrorFormula).
// Temporal operators are dualized along the way (Always <-> Eventually) so the
// evaluator never has to reduce a negated temporal obligation, which it cannot
// do soundly across steps.
func nnf(formula Formula) Formula {
switch concrete := formula.(type) {
case NotFormula:
return pushNot(concrete.Inner)
case AlwaysFormula:
next := concrete
next.Inner = nnf(concrete.Inner)
return next
case EventuallyFormula:
next := concrete
next.Inner = nnf(concrete.Inner)
return next
case NextFormula:
return NextFormula{Inner: nnf(concrete.Inner)}
case NowFormula:
return NowFormula{Inner: nnf(concrete.Inner)}
case AndFormula:
return AndFormula{Left: nnf(concrete.Left), Right: nnf(concrete.Right)}
case OrFormula:
return OrFormula{Left: nnf(concrete.Left), Right: nnf(concrete.Right)}
case ImpliesFormula:
// a -> b is rewritten to (not a) or b so the consequent is always
// reduced live each step. Keeping it as ImpliesFormula let a pending
// (temporal) antecedent defer the whole implication and silently drop a
// consequent that was false at the current step.
return OrFormula{
Left: pushNot(concrete.Antecedent),
Right: nnf(concrete.Consequent),
}
default:
return formula
}
}
// pushNot returns the negation normal form of NOT f.
func pushNot(formula Formula) Formula {
switch concrete := formula.(type) {
case PureFormula:
return PureFormula{Value: !concrete.Value}
case ThunkFormula:
return NotFormula{Inner: concrete}
case ErrorFormula:
return NotFormula{Inner: concrete}
case NotFormula:
return nnf(concrete.Inner)
case AndFormula:
return OrFormula{Left: pushNot(concrete.Left), Right: pushNot(concrete.Right)}
case OrFormula:
return AndFormula{Left: pushNot(concrete.Left), Right: pushNot(concrete.Right)}
case ImpliesFormula:
return AndFormula{
Left: nnf(concrete.Antecedent),
Right: pushNot(concrete.Consequent),
}
case NowFormula:
return NowFormula{Inner: pushNot(concrete.Inner)}
case NextFormula:
return NextFormula{Inner: pushNot(concrete.Inner)}
case AlwaysFormula:
return EventuallyFormula{
Inner: pushNot(concrete.Inner),
StepBound: concrete.StepBound,
HasStepBound: concrete.HasStepBound,
Duration: concrete.Duration,
Deadline: concrete.Deadline,
HasDeadline: concrete.HasDeadline,
}
case EventuallyFormula:
return AlwaysFormula{
Inner: pushNot(concrete.Inner),
StepBound: concrete.StepBound,
HasStepBound: concrete.HasStepBound,
Duration: concrete.Duration,
Deadline: concrete.Deadline,
HasDeadline: concrete.HasDeadline,
}
default:
return NotFormula{Inner: formula}
}
}
internal/ltl/nnf_test.go
+97
View File
@@ -0,0 +1,97 @@
package ltl
import (
"testing"
"testing/quick"
"time"
)
// leaf builds a small set of representative atomic formulas indexed by a seed.
func leafFor(seed uint8) Formula {
switch seed % 3 {
case 0:
return Pure(true)
case 1:
return Pure(false)
default:
return ThunkNamed("p", func() (bool, error) { return true, nil })
}
}
func TestNNF_DoubleNegationIsIdentity(t *testing.T) {
law := func(seed uint8) bool {
leaf := leafFor(seed)
doubled := nnf(Not(Not(leaf)))
direct := nnf(leaf)
return doubled.describe() == direct.describe()
}
if err := quick.Check(law, nil); err != nil {
t.Error(err)
}
}
func TestNNF_NotAlwaysIsEventuallyNot(t *testing.T) {
law := func(seed uint8) bool {
leaf := leafFor(seed)
negated := nnf(Not(Always(leaf)))
expected := nnf(Eventually(Not(leaf)))
return negated.describe() == expected.describe()
}
if err := quick.Check(law, nil); err != nil {
t.Error(err)
}
}
func TestNNF_NotEventuallyIsAlwaysNot(t *testing.T) {
law := func(seed uint8) bool {
leaf := leafFor(seed)
negated := nnf(Not(Eventually(leaf)))
expected := nnf(Always(Not(leaf)))
return negated.describe() == expected.describe()
}
if err := quick.Check(law, nil); err != nil {
t.Error(err)
}
}
func TestNNF_BoundedEventuallyDualKeepsBound(t *testing.T) {
negated := nnf(Not(EventuallyWithinSteps(Pure(true), 4)))
always, ok := negated.(AlwaysFormula)
if !ok {
t.Fatalf("expected AlwaysFormula, got %T", negated)
}
if !always.HasStepBound || always.StepBound != 4 {
t.Errorf("bound not preserved: %+v", always)
}
}
func TestNNF_PushesNotToThunkLeaf(t *testing.T) {
formula := nnf(Always(Not(Always(ThunkNamed("p", func() (bool, error) { return true, nil })))))
always, ok := formula.(AlwaysFormula)
if !ok {
t.Fatalf("expected AlwaysFormula, got %T", formula)
}
eventually, ok := always.Inner.(EventuallyFormula)
if !ok {
t.Fatalf("expected inner EventuallyFormula, got %T", always.Inner)
}
not, ok := eventually.Inner.(NotFormula)
if !ok {
t.Fatalf("expected NotFormula leaf, got %T", eventually.Inner)
}
if _, ok := not.Inner.(ThunkFormula); !ok {
t.Errorf("expected Not to wrap a Thunk, got %T", not.Inner)
}
}
func TestNNF_NotAlwaysTrueViaEvaluatorReportsViolated(t *testing.T) {
evaluator := NewEvaluator(Always(Not(Always(ThunkNamed("p", func() (bool, error) { return true, nil })))))
for index := range 4 {
if got := evaluator.ObserveAt(time.Unix(int64(index), 0)); got == VerdictViolated {
t.Fatalf("step %d latched violated prematurely", index)
}
}
if got := evaluator.Finalize(); got != VerdictViolated {
t.Errorf("Finalize = %v, want violated", got)
}
}
internal/ltl/witness_test.go
+80
View File
@@ -0,0 +1,80 @@
package ltl
import (
"errors"
"testing"
"time"
)
func TestViolation_PredicateFalseCarriesReasonAndStep(t *testing.T) {
values := []bool{true, false}
step := 0
evaluator := NewEvaluator(Always(ThunkNamed("p", func() (bool, error) {
current := values[step]
step++
return current, nil
})))
if got := evaluator.ObserveAt(time.Unix(0, 0)); got != VerdictHolds {
t.Fatalf("step 1: got %v, want holds", got)
}
if got := evaluator.ObserveAt(time.Unix(1, 0)); got != VerdictViolated {
t.Fatalf("step 2: got %v, want violated", got)
}
witness := evaluator.Violation()
if witness == nil {
t.Fatal("Violation = nil, want non-nil")
}
if witness.Reason != "predicate false" {
t.Errorf("Reason = %q, want %q", witness.Reason, "predicate false")
}
if witness.Step != 2 {
t.Errorf("Step = %d, want 2", witness.Step)
}
if witness.IsError {
t.Errorf("IsError = true, want false for a plain false")
}
}
func TestViolation_ThrownPredicateSetsIsError(t *testing.T) {
evaluator := NewEvaluator(Always(ThunkNamed("p", func() (bool, error) {
return false, errors.New("boom")
})))
if got := evaluator.Observe(); got != VerdictViolated {
t.Fatalf("got %v, want violated", got)
}
witness := evaluator.Violation()
if witness == nil {
t.Fatal("Violation = nil, want non-nil")
}
if !witness.IsError {
t.Errorf("IsError = false, want true for a thrown predicate")
}
if witness.Reason != "boom" {
t.Errorf("Reason = %q, want %q", witness.Reason, "boom")
}
}
func TestViolation_FinalizeFillsWitness(t *testing.T) {
evaluator := NewEvaluator(Eventually(ThunkNamed("p", func() (bool, error) {
return false, nil
})))
evaluator.ObserveAt(time.Unix(0, 0))
if got := evaluator.Finalize(); got != VerdictViolated {
t.Fatalf("Finalize = %v, want violated", got)
}
witness := evaluator.Violation()
if witness == nil {
t.Fatal("Violation = nil after Finalize, want non-nil")
}
if witness.Reason != "eventually never satisfied" {
t.Errorf("Reason = %q, want %q", witness.Reason, "eventually never satisfied")
}
}
func TestViolation_NilBeforeViolation(t *testing.T) {
evaluator := NewEvaluator(Always(Pure(true)))
evaluator.Observe()
if got := evaluator.Violation(); got != nil {
t.Errorf("Violation = %+v, want nil for a holding run", got)
}
}
internal/permissions/permissions.go
-125
View File
@@ -1,125 +0,0 @@
package permissions
import (
"context"
"fmt"
"os/exec"
"regexp"
"strings"
)
// Inspector returns the list of uses-permission entries declared by the APK.
type Inspector func(ctx context.Context, apkPath string) ([]string, error)
// Granter grants a single permission to the given package on the connected
// device. Implementations typically wrap `adb shell pm grant`.
type Granter func(ctx context.Context, packageName, permission string) error
// GrantDangerous filters declared permissions to the dangerous set and grants
// each one via the supplied granter. Errors from individual grants are
// collected as warnings rather than aborting — Android refuses to grant
// non-runtime permissions and we prefer to soldier on.
func GrantDangerous(
ctx context.Context,
apkPath, packageName string,
inspector Inspector,
granter Granter,
) (granted []string, warnings []string, err error) {
declared, err := inspector(ctx, apkPath)
if err != nil {
return nil, nil, fmt.Errorf("inspect permissions: %w", err)
}
for _, permission := range declared {
if !IsDangerous(permission) {
continue
}
if err := granter(ctx, packageName, permission); err != nil {
warnings = append(warnings, fmt.Sprintf("%s: %v", permission, err))
continue
}
granted = append(granted, permission)
}
return granted, warnings, nil
}
var aaptPermissionPattern = regexp.MustCompile(`uses-permission:\s+name='([^']+)'`)
// AaptInspector shells out to `aapt dump permissions` to list permissions.
func AaptInspector(ctx context.Context, apkPath string) ([]string, error) {
output, err := exec.CommandContext(ctx, "aapt", "dump", "permissions", apkPath).Output()
if err != nil {
return nil, fmt.Errorf("aapt dump permissions: %w", err)
}
var permissions []string
for _, match := range aaptPermissionPattern.FindAllStringSubmatch(string(output), -1) {
permissions = append(permissions, match[1])
}
return permissions, nil
}
// AdbGranter returns a Granter that runs `adb shell pm grant` against the
// supplied device serial (empty = default device).
func AdbGranter(adbPath, deviceSerial string) Granter {
return func(ctx context.Context, packageName, permission string) error {
arguments := []string{"shell", "pm", "grant", packageName, permission}
if deviceSerial != "" {
arguments = append([]string{"-s", deviceSerial}, arguments...)
}
command := exec.CommandContext(ctx, adbPath, arguments...)
output, err := command.CombinedOutput()
if err != nil {
return fmt.Errorf("adb pm grant %s: %w (%s)", permission, err, strings.TrimSpace(string(output)))
}
return nil
}
}
// dangerousPermissions captures Android's PROTECTION_DANGEROUS set as of
// API 34. New runtime permissions added in later releases should be appended
// here when needed.
var dangerousPermissions = map[string]bool{
"android.permission.READ_CALENDAR": true,
"android.permission.WRITE_CALENDAR": true,
"android.permission.CAMERA": true,
"android.permission.READ_CONTACTS": true,
"android.permission.WRITE_CONTACTS": true,
"android.permission.GET_ACCOUNTS": true,
"android.permission.ACCESS_FINE_LOCATION": true,
"android.permission.ACCESS_COARSE_LOCATION": true,
"android.permission.ACCESS_BACKGROUND_LOCATION": true,
"android.permission.RECORD_AUDIO": true,
"android.permission.READ_PHONE_STATE": true,
"android.permission.READ_PHONE_NUMBERS": true,
"android.permission.CALL_PHONE": true,
"android.permission.ANSWER_PHONE_CALLS": true,
"android.permission.READ_CALL_LOG": true,
"android.permission.WRITE_CALL_LOG": true,
"android.permission.ADD_VOICEMAIL": true,
"android.permission.USE_SIP": true,
"android.permission.PROCESS_OUTGOING_CALLS": true,
"android.permission.BODY_SENSORS": true,
"android.permission.SEND_SMS": true,
"android.permission.RECEIVE_SMS": true,
"android.permission.READ_SMS": true,
"android.permission.RECEIVE_WAP_PUSH": true,
"android.permission.RECEIVE_MMS": true,
"android.permission.READ_EXTERNAL_STORAGE": true,
"android.permission.WRITE_EXTERNAL_STORAGE": true,
"android.permission.ACCESS_MEDIA_LOCATION": true,
"android.permission.ACTIVITY_RECOGNITION": true,
"android.permission.POST_NOTIFICATIONS": true,
"android.permission.NEARBY_WIFI_DEVICES": true,
"android.permission.READ_MEDIA_IMAGES": true,
"android.permission.READ_MEDIA_VIDEO": true,
"android.permission.READ_MEDIA_AUDIO": true,
"android.permission.READ_MEDIA_VISUAL_USER_SELECTED": true,
"android.permission.BLUETOOTH_CONNECT": true,
"android.permission.BLUETOOTH_ADVERTISE": true,
"android.permission.BLUETOOTH_SCAN": true,
"android.permission.UWB_RANGING": true,
"android.permission.BODY_SENSORS_BACKGROUND": true,
}
func IsDangerous(permission string) bool {
return dangerousPermissions[permission]
}
internal/permissions/permissions_test.go
-127
View File
@@ -1,127 +0,0 @@
package permissions
import (
"context"
"errors"
"slices"
"testing"
)
func TestGrantDangerous_FiltersAndCallsGranter(t *testing.T) {
declared := []string{
"android.permission.INTERNET", // not dangerous
"android.permission.CAMERA", // dangerous
"android.permission.WAKE_LOCK", // not dangerous
"android.permission.ACCESS_FINE_LOCATION", // dangerous
"android.permission.READ_EXTERNAL_STORAGE", // dangerous
"android.permission.SYSTEM_ALERT_WINDOW", // not dangerous
}
inspector := func(_ context.Context, _ string) ([]string, error) { return declared, nil }
var requested []string
granter := func(_ context.Context, packageName, permission string) error {
if packageName != "com.example" {
t.Errorf("granter received wrong package: %q", packageName)
}
requested = append(requested, permission)
return nil
}
granted, warnings, err := GrantDangerous(context.Background(), "/path/to/apk", "com.example", inspector, granter)
if err != nil {
t.Fatal(err)
}
want := []string{
"android.permission.CAMERA",
"android.permission.ACCESS_FINE_LOCATION",
"android.permission.READ_EXTERNAL_STORAGE",
}
if !slices.Equal(granted, want) {
t.Errorf("granted permissions: got %v, want %v", granted, want)
}
if !slices.Equal(requested, want) {
t.Errorf("granter calls: got %v, want %v", requested, want)
}
if len(warnings) != 0 {
t.Errorf("expected no warnings, got %v", warnings)
}
}
func TestGrantDangerous_CollectsGranterFailuresAsWarnings(t *testing.T) {
declared := []string{"android.permission.CAMERA", "android.permission.RECORD_AUDIO"}
inspector := func(_ context.Context, _ string) ([]string, error) { return declared, nil }
granter := func(_ context.Context, _, permission string) error {
if permission == "android.permission.CAMERA" {
return errors.New("device denied")
}
return nil
}
granted, warnings, err := GrantDangerous(context.Background(), "/x", "com.example", inspector, granter)
if err != nil {
t.Fatal(err)
}
if !slices.Equal(granted, []string{"android.permission.RECORD_AUDIO"}) {
t.Errorf("granted: %v", granted)
}
if len(warnings) != 1 || warnings[0] != "android.permission.CAMERA: device denied" {
t.Errorf("warnings: %v", warnings)
}
}
func TestGrantDangerous_InspectorErrorBubbles(t *testing.T) {
inspector := func(_ context.Context, _ string) ([]string, error) {
return nil, errors.New("aapt missing")
}
_, _, err := GrantDangerous(context.Background(), "/x", "com.example", inspector, nil)
if err == nil || err.Error() == "" {
t.Errorf("expected wrapped inspector error, got %v", err)
}
}
func TestIsDangerous_KnownPermissions(t *testing.T) {
dangerous := []string{
"android.permission.CAMERA",
"android.permission.RECORD_AUDIO",
"android.permission.POST_NOTIFICATIONS",
}
for _, permission := range dangerous {
if !IsDangerous(permission) {
t.Errorf("expected %q to be dangerous", permission)
}
}
normal := []string{
"android.permission.INTERNET",
"android.permission.WAKE_LOCK",
"android.permission.SYSTEM_ALERT_WINDOW",
}
for _, permission := range normal {
if IsDangerous(permission) {
t.Errorf("expected %q not to be dangerous", permission)
}
}
}
func TestAaptInspector_ParsesFixtureOutput(t *testing.T) {
// Direct test of the regex; avoids spawning aapt.
output := `
package: com.example
sdkVersion:'24'
uses-permission: name='android.permission.INTERNET'
uses-permission: name='android.permission.CAMERA'
uses-permission: name='android.permission.WAKE_LOCK'
`
matches := aaptPermissionPattern.FindAllStringSubmatch(output, -1)
got := make([]string, 0, len(matches))
for _, match := range matches {
got = append(got, match[1])
}
want := []string{
"android.permission.INTERNET",
"android.permission.CAMERA",
"android.permission.WAKE_LOCK",
}
if !slices.Equal(got, want) {
t.Errorf("got %v, want %v", got, want)
}
}
internal/runner/golden_test.go
+143
View File
@@ -0,0 +1,143 @@
package runner
import (
"bufio"
"bytes"
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/priyanshujain/sanderling/internal/verifier"
)
// mustSeededVerifier builds a verifier with a fixed seed so the JS picker's PRNG
// is reproducible across runs, making the trace byte-stable.
func mustSeededVerifier(t *testing.T, spec string, seed uint64) *verifier.Verifier {
t.Helper()
instance, err := verifier.New(verifier.WithSeed(seed))
if err != nil {
t.Fatal(err)
}
if err := instance.Load(bundleSpec(t, spec)); err != nil {
t.Fatal(err)
}
return instance
}
// goldenSeed and goldenSteps fix the run so the mock-driven trace and summary
// are byte-reproducible. Set UPDATE_GOLDEN=1 to rewrite the committed goldens;
// the checked-in files are the asserted source of truth.
const (
goldenSeed = 0x5eed
goldenSteps = 4
)
// canonicalizeTrace rewrites the sole non-deterministic field, "timestamp"
// (wall-clock at step start), to a fixed zero value so equality compares only
// the run's observable shape. Every other field is a pure function of the seed,
// the spec, and the programmed mock hierarchy.
func canonicalizeTrace(t *testing.T, raw []byte) []byte {
t.Helper()
var out bytes.Buffer
scanner := bufio.NewScanner(bytes.NewReader(raw))
scanner.Buffer(make([]byte, 0, 64*1024), 8*1024*1024)
for scanner.Scan() {
var line map[string]json.RawMessage
if err := json.Unmarshal(scanner.Bytes(), &line); err != nil {
t.Fatalf("decode trace line: %v", err)
}
line["timestamp"] = json.RawMessage(`"0001-01-01T00:00:00Z"`)
encoded, err := json.Marshal(line)
if err != nil {
t.Fatalf("encode trace line: %v", err)
}
out.Write(encoded)
out.WriteByte('\n')
}
if err := scanner.Err(); err != nil {
t.Fatalf("scan trace: %v", err)
}
return out.Bytes()
}
func assertGolden(t *testing.T, name string, got []byte) {
t.Helper()
path := filepath.Join("testdata", name)
if os.Getenv("UPDATE_GOLDEN") == "1" {
if err := os.WriteFile(path, got, 0o644); err != nil {
t.Fatalf("write golden %s: %v", name, err)
}
return
}
want, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read golden %s (run with UPDATE_GOLDEN=1 to create): %v", name, err)
}
if !bytes.Equal(want, got) {
t.Errorf("golden %s mismatch\n--- want ---\n%s\n--- got ---\n%s", name, want, got)
}
}
// TestGolden_TraceStreamIsReproducible drives a fixed-seed, fixed-step run
// against the mock driver and snapshots the canonicalized trace.jsonl stream.
// A drift here means the run is no longer deterministic or the trace shape
// changed; both must be reviewed, not papered over.
func TestGolden_TraceStreamIsReproducible(t *testing.T) {
state := newHarnessWithSpec(t, fixtureSpec)
state.verifier = mustSeededVerifier(t, fixtureSpec, goldenSeed)
state.mock.HierarchyJSON = `{"attributes":{"resource-id":"HomeScreen"},"children":[{"attributes":{"resource-id":"next","bounds":"[40,80,240,160]"},"children":[],"clickable":true,"enabled":true}]}`
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
_, err := Run(ctx, Options{
Duration: time.Hour,
IdleTimeout: 10 * time.Millisecond,
MaxSteps: goldenSteps,
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
raw, err := os.ReadFile(filepath.Join(state.writer.Directory(), "trace.jsonl"))
if err != nil {
t.Fatal(err)
}
assertGolden(t, "trace.jsonl", canonicalizeTrace(t, raw))
}
// TestGolden_ViolationSummaryIsReproducible drives a known violation and
// snapshots the rendered summary string produced by RenderSummary (the same
// function the CLI prints). It also asserts the violated property is named, so
// a regression that drops the property from the report cannot pass silently.
func TestGolden_ViolationSummaryIsReproducible(t *testing.T) {
state := newHarnessWithSpec(t, violationSpec)
state.verifier = mustSeededVerifier(t, violationSpec, goldenSeed)
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
summary, err := Run(ctx, Options{
Duration: time.Hour,
IdleTimeout: 10 * time.Millisecond,
MaxSteps: goldenSteps,
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
var rendered bytes.Buffer
RenderSummary(&rendered, summary, "android")
if !strings.Contains(rendered.String(), "balanceNonNegative") {
t.Errorf("summary must name the violated property balanceNonNegative, got:\n%s", rendered.String())
}
assertGolden(t, "violation-summary.txt", rendered.Bytes())
}
internal/runner/runner.go
+112 -105
View File
@@ -1,3 +1,4 @@
// Package runner drives the observe-decide-act loop that steps a spec against a device.
package runner
import (
@@ -5,6 +6,7 @@ import (
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"strings"
"time"
@@ -20,15 +22,15 @@ import (
"github.com/priyanshujain/sanderling/internal/verifier"
)
// doubleTapGap is the inter-tap delay for ActionKindDoubleTap: short enough to
// land both events inside a sub-100 ms race window, long enough for adb
// `input tap` to serialize two MotionEvent streams.
const doubleTapGap = 50 * time.Millisecond
type Options struct {
Duration time.Duration
IdleTimeout time.Duration
// MaxSteps caps the run at a fixed number of steps for reproducible
// bounded runs. 0 means unbounded (the duration deadline governs); a
// positive value stops the loop once that many steps have run.
MaxSteps int
BundleID string
Driver driver.DeviceDriver
Verifier *verifier.Verifier
@@ -41,6 +43,10 @@ type Summary struct {
EndTime time.Time
Steps int
Violations []ViolationRecord
// UnsupportedVerbs lists verbs the picker requested that the platform
// could not dispatch, deduped, so the report can flag a spec exercising
// gestures this target does not support.
UnsupportedVerbs []string
}
type ViolationRecord struct {
@@ -65,6 +71,11 @@ func Run(ctx context.Context, options Options) (Summary, error) {
// before the deadline is set so the settle time does not eat the run.
waitForForeground(ctx, options, logger)
// Pick the action and extractor sources once from the driver's
// capabilities so the step loop runs one uniform path with no per-step
// driver type assertion.
actionSource, extractorSource := pickSources(options)
summary := Summary{StartTime: time.Now()}
deadline := summary.StartTime.Add(options.Duration)
stepIndex := 0
@@ -74,6 +85,9 @@ func Run(ctx context.Context, options Options) (Summary, error) {
if err := ctx.Err(); err != nil {
break
}
if options.MaxSteps > 0 && stepIndex >= options.MaxSteps {
break
}
stepIndex++
stepStart := time.Now()
@@ -116,17 +130,15 @@ func Run(ctx context.Context, options Options) (Summary, error) {
return nil
})
var v8Overrides map[int]json.RawMessage
if web, ok := options.Driver.(driver.WebDriver); ok {
g.Go(func() error {
overrides, err := web.EvaluateExtractors(gctx)
if err != nil {
logger.Warn("v8 extractor evaluation failed", "step", si, "err", err)
return nil
}
v8Overrides = overrides
g.Go(func() error {
overrides, err := extractorSource.ExtractorOverrides(gctx)
if err != nil {
logger.Warn("v8 extractor evaluation failed", "step", si, "err", err)
return nil
})
}
}
v8Overrides = overrides
return nil
})
// All goroutines write to local variables and return nil, so the Wait
// error is always nil; ignored intentionally.
_ = g.Wait()
@@ -166,6 +178,8 @@ func Run(ctx context.Context, options Options) (Summary, error) {
// progressing.
var violations []string
var extractorChanges map[string]trace.ExtractorChange
var witnesses map[string]trace.Witness
skippedVerification := false
if !transitional {
if err := options.Verifier.PushSnapshot(verifier.SnapshotInput{
Tree: tree,
@@ -186,25 +200,16 @@ func Run(ctx context.Context, options Options) (Summary, error) {
}
options.Verifier.EvaluateProperties()
violations = options.Verifier.NewlyViolatedProperties()
for _, name := range violations {
if predicateErr := options.Verifier.PredicateError(name); predicateErr != nil {
logger.Warn("predicate error", "step", stepIndex, "property", name, "err", predicateErr)
}
}
witnesses = collectWitnesses(options.Verifier, violations, logger, stepIndex)
extractorChanges = encodeExtractorChanges(options.Verifier.ChangedExtractors())
} else {
skippedVerification = true
logger.Warn("transitional tree after retry budget; skipping verifier",
"step", stepIndex, "screen", screen, "nodes", treeSize)
}
logger.Info("step", "index", stepIndex, "screen", screen, "nodes", treeSize)
var nextAction verifier.Action
var nextErr error
if web, ok := options.Driver.(driver.WebDriver); ok {
nextAction, nextErr = nextActionFromV8(ctx, web)
} else {
nextAction, nextErr = options.Verifier.NextAction()
}
nextAction, nextErr := actionSource.NextAction(ctx)
var traceAction *trace.Action
if nextErr == nil {
traceAction = traceActionFor(nextAction, tree)
@@ -240,16 +245,18 @@ func Run(ctx context.Context, options Options) (Summary, error) {
}
step := trace.Step{
Index: stepIndex,
Timestamp: stepStart,
Screen: screen,
NextAction: traceAction,
Violations: violations,
Hierarchy: tree,
Residuals: residuals,
Metrics: metrics,
ExtractorChanges: extractorChanges,
Transitional: transitional,
Index: stepIndex,
Timestamp: stepStart,
Screen: screen,
NextAction: traceAction,
Violations: violations,
Hierarchy: tree,
Residuals: residuals,
Metrics: metrics,
ExtractorChanges: extractorChanges,
Transitional: transitional,
SkippedVerification: skippedVerification,
Witnesses: witnesses,
}
if err := options.TraceWriter.WriteStep(step); err != nil {
return summary, fmt.Errorf("step %d trace: %w", stepIndex, err)
@@ -276,10 +283,51 @@ func Run(ctx context.Context, options Options) (Summary, error) {
}
}
// Finalize each evaluator once the loop ends so liveness obligations that
// never discharged (an unbounded eventually that never fired, a strong
// next with no successor) are reported as violations rather than silently
// left pending. Properties already violated mid-run are not re-reported.
if ended := options.Verifier.Finalize(); len(ended) > 0 {
witnesses := collectWitnesses(options.Verifier, ended, logger, stepIndex)
summary.Violations = append(summary.Violations, ViolationRecord{
StepIndex: stepIndex,
Properties: ended,
})
finalStep := trace.Step{
Index: stepIndex,
Timestamp: time.Now(),
Violations: ended,
Witnesses: witnesses,
}
if err := options.TraceWriter.WriteStep(finalStep); err != nil {
return summary, fmt.Errorf("finalize trace: %w", err)
}
}
summary.UnsupportedVerbs = options.Verifier.UnsupportedVerbs()
summary.EndTime = time.Now()
return summary, nil
}
// RenderSummary writes the human-facing run summary: step count, each violation
// record, and any unsupported verbs. The wall-clock duration is excluded so the
// output is deterministic and snapshot-testable; the CLI prints it separately.
func RenderSummary(w io.Writer, summary Summary, platform string) {
fmt.Fprintf(w, "\nrun complete: %d steps\n", summary.Steps)
if len(summary.Violations) == 0 {
fmt.Fprintln(w, "no violations.")
} else {
fmt.Fprintf(w, "%d violation record(s):\n", len(summary.Violations))
for _, violation := range summary.Violations {
fmt.Fprintf(w, " step %d: %v\n", violation.StepIndex, violation.Properties)
}
}
if len(summary.UnsupportedVerbs) > 0 {
fmt.Fprintf(w, "unsupported on %s: %s\n",
platform, strings.Join(summary.UnsupportedVerbs, ", "))
}
}
func validate(options Options) error {
if options.Driver == nil {
return errors.New("runner: Driver is required")
@@ -417,26 +465,13 @@ func applyAction(ctx context.Context, drv driver.DeviceDriver, action verifier.A
return drv.Tap(ctx, x, y)
case verifier.ActionKindDoubleTap:
x, y, ok := resolveCoordinates(action, tree)
tap := func() error {
if !ok {
if action.On == "" {
return nil
}
return drv.TapSelector(ctx, action.On)
if !ok {
if action.On == "" {
return nil
}
return drv.Tap(ctx, x, y)
return drv.DoubleTapSelector(ctx, action.On)
}
if err := tap(); err != nil {
return err
}
timer := time.NewTimer(doubleTapGap)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
}
return tap()
return drv.DoubleTap(ctx, x, y)
case verifier.ActionKindLongPress:
x, y, ok := resolveCoordinates(action, tree)
if !ok {
@@ -748,59 +783,31 @@ func captureMetrics(ctx context.Context, options Options, logger *slog.Logger, s
}
}
// nextActionFromV8 invokes the V8-side action generator and decodes the
// resulting JSON into a verifier.Action. ErrNoAction is returned when the
// generator declined to act this tick.
func nextActionFromV8(ctx context.Context, web driver.WebDriver) (verifier.Action, error) {
raw, err := web.NextActionFromV8(ctx)
if err != nil {
return verifier.Action{}, fmt.Errorf("v8 next action: %w", err)
// collectWitnesses gathers the violation witness for each newly-violated
// property, logs its cause, and returns them keyed by property name for the
// trace. Properties without a captured witness are skipped.
func collectWitnesses(verifierInstance *verifier.Verifier, properties []string, logger *slog.Logger, stepIndex int) map[string]trace.Witness {
if len(properties) == 0 {
return nil
}
if len(raw) == 0 || string(raw) == "null" {
return verifier.Action{}, verifier.ErrNoAction
witnesses := map[string]trace.Witness{}
for _, name := range properties {
witness := verifierInstance.Witness(name)
if witness == nil {
continue
}
logger.Warn("property violated",
"step", stepIndex, "property", name, "reason", witness.Reason, "error", witness.IsError)
witnesses[name] = trace.Witness{
Reason: witness.Reason,
IsError: witness.IsError,
Extractors: witness.Extractors,
}
}
var decoded struct {
Kind string `json:"kind"`
X int `json:"x"`
Y int `json:"y"`
FromX int `json:"from_x"`
FromY int `json:"from_y"`
ToX int `json:"to_x"`
ToY int `json:"to_y"`
Key string `json:"key"`
Text string `json:"text"`
DurationMillis int `json:"duration_millis"`
}
if err := json.Unmarshal(raw, &decoded); err != nil {
return verifier.Action{}, fmt.Errorf("decode v8 action: %w", err)
}
switch decoded.Kind {
case "Tap":
return verifier.Action{Kind: verifier.ActionKindTap, X: decoded.X, Y: decoded.Y}, nil
case "DoubleTap":
return verifier.Action{Kind: verifier.ActionKindDoubleTap, X: decoded.X, Y: decoded.Y}, nil
case "InputText":
return verifier.Action{
Kind: verifier.ActionKindInputText,
X: decoded.X, Y: decoded.Y,
Text: decoded.Text,
}, nil
case "Swipe":
return verifier.Action{
Kind: verifier.ActionKindSwipe,
FromX: decoded.FromX,
FromY: decoded.FromY,
ToX: decoded.ToX,
ToY: decoded.ToY,
DurationMillis: decoded.DurationMillis,
}, nil
case "PressKey":
return verifier.Action{Kind: verifier.ActionKindPressKey, Key: decoded.Key}, nil
case "Wait":
return verifier.Action{Kind: verifier.ActionKindWait, DurationMillis: decoded.DurationMillis}, nil
default:
return verifier.Action{}, verifier.ErrNoAction
if len(witnesses) == 0 {
return nil
}
return witnesses
}
func encodeExtractorChanges(changes map[string]verifier.ExtractorChange) map[string]trace.ExtractorChange {
internal/runner/runner_test.go
+78 -22
View File
@@ -18,6 +18,7 @@ import (
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"github.com/priyanshujain/sanderling/internal/bundler"
"github.com/priyanshujain/sanderling/internal/driver"
mockdriver "github.com/priyanshujain/sanderling/internal/driver/mock"
"github.com/priyanshujain/sanderling/internal/hierarchy"
@@ -26,18 +27,20 @@ import (
)
const fixtureSpec = `
const balance = __sanderling__.extract(state => state.snapshots.balance ?? 0);
import { actions, always, extract, Tap } from "@sanderling/spec";
const balance = extract(state => state.snapshots.balance ?? 0);
globalThis.properties = {
balanceNonNegative: __sanderling__.always(() => balance.current >= 0),
balanceNonNegative: always(() => balance.current >= 0),
};
globalThis.actions = __sanderling__.actions(() => [__sanderling__.tap({ on: "id:next" })]);
globalThis.actions = actions(() => [Tap({ on: "id:next" })]);
`
const violationSpec = `
import { actions, always } from "@sanderling/spec";
globalThis.properties = {
balanceNonNegative: __sanderling__.always(() => false),
balanceNonNegative: always(() => false),
};
globalThis.actions = __sanderling__.actions(() => []);
globalThis.actions = actions(() => []);
`
type harness struct {
@@ -50,6 +53,34 @@ func newHarness(t *testing.T) *harness {
return newHarnessWithSpec(t, fixtureSpec)
}
// bundleSpec compiles an authored TS spec with the goja runtime entry so the
// loaded bundle installs __sanderlingNextAction__ (the shared picker).
func bundleSpec(t *testing.T, specSource string) string {
t.Helper()
dir := t.TempDir()
specPath := filepath.Join(dir, "spec.ts")
if err := os.WriteFile(specPath, []byte(specSource), 0o600); err != nil {
t.Fatal(err)
}
apiPath, err := filepath.Abs("../../pkg/spec/src/index.ts")
if err != nil {
t.Fatal(err)
}
runtimePath, err := filepath.Abs("../../pkg/spec/src/goja-runtime.ts")
if err != nil {
t.Fatal(err)
}
bundle, err := bundler.Bundle(bundler.Options{
EntryFile: specPath,
RuntimeFile: runtimePath,
Aliases: map[string]string{"@sanderling/spec": apiPath},
})
if err != nil {
t.Fatal(err)
}
return string(bundle.JavaScript)
}
func newHarnessWithSpec(t *testing.T, spec string) *harness {
t.Helper()
directory := t.TempDir()
@@ -61,7 +92,7 @@ func newHarnessWithSpec(t *testing.T, spec string) *harness {
if err != nil {
t.Fatal(err)
}
if err := verifierInstance.Load(spec); err != nil {
if err := verifierInstance.Load(bundleSpec(t, spec)); err != nil {
t.Fatal(err)
}
state := &harness{
@@ -94,6 +125,11 @@ func TestRunner_HappyPathStepsAndTraces(t *testing.T) {
if len(summary.Violations) != 0 {
t.Errorf("no violations expected, got %v", summary.Violations)
}
// Every builtin verb is supported on every platform, so a clean run must
// report no unsupported verbs (the runner still wires the field through).
if len(summary.UnsupportedVerbs) != 0 {
t.Errorf("expected no unsupported verbs, got %v", summary.UnsupportedVerbs)
}
actions := state.mock.Actions()
if !containsAction(actions, mockdriver.ActionTapSelector, "id:next") {
@@ -101,6 +137,29 @@ func TestRunner_HappyPathStepsAndTraces(t *testing.T) {
}
}
func TestRunner_MaxStepsStopsAfterExactlyNSteps(t *testing.T) {
state := newHarness(t)
const maxSteps = 3
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
// A long duration ensures MaxSteps, not the deadline, ends the run.
summary, err := Run(ctx, Options{
Duration: time.Hour,
IdleTimeout: 10 * time.Millisecond,
MaxSteps: maxSteps,
Driver: state.mock,
Verifier: state.verifier,
TraceWriter: state.writer,
})
if err != nil {
t.Fatalf("Run: %v", err)
}
if summary.Steps != maxSteps {
t.Errorf("expected exactly %d steps, got %d", maxSteps, summary.Steps)
}
}
func TestRunner_ViolationSurfacesInSummary(t *testing.T) {
state := newHarnessWithSpec(t, violationSpec)
@@ -196,10 +255,11 @@ func TestRunner_ViolationSurfacesOnlyOnOnsetStep(t *testing.T) {
func TestRunner_ThrowingPredicateIsLoggedNotPanic(t *testing.T) {
const throwingSpec = `
import { actions, always, Tap } from "@sanderling/spec";
globalThis.properties = {
broken: __sanderling__.always(() => { throw new Error("bad predicate"); }),
broken: always(() => { throw new Error("bad predicate"); }),
};
globalThis.actions = __sanderling__.actions(() => [__sanderling__.tap({ on: "id:next" })]);
globalThis.actions = actions(() => [Tap({ on: "id:next" })]);
`
state := newHarnessWithSpec(t, throwingSpec)
@@ -370,30 +430,25 @@ func TestApplyAction_V8InputTextAtOriginStillTaps(t *testing.T) {
}
}
func TestApplyAction_DoubleTapDispatchesTwoTapsAtCoordinates(t *testing.T) {
func TestApplyAction_DoubleTapDispatchesDoubleTapAtCoordinates(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindDoubleTap, X: 100, Y: 200}
start := time.Now()
if err := applyAction(context.Background(), driverMock, action, nil); err != nil {
t.Fatalf("apply action: %v", err)
}
elapsed := time.Since(start)
if elapsed < 40*time.Millisecond {
t.Errorf("expected >= 40ms gap between taps, elapsed %v", elapsed)
}
taps := 0
for _, a := range driverMock.Actions() {
if a.Kind == mockdriver.ActionTap && a.X == 100 && a.Y == 200 {
if a.Kind == mockdriver.ActionDoubleTap && a.X == 100 && a.Y == 200 {
taps++
}
}
if taps != 2 {
t.Errorf("expected 2 Tap calls at (100,200), got %d in %v", taps, driverMock.Actions())
if taps != 1 {
t.Errorf("expected 1 DoubleTap call at (100,200), got %d in %v", taps, driverMock.Actions())
}
}
func TestApplyAction_DoubleTapDispatchesTwoSelectorTaps(t *testing.T) {
func TestApplyAction_DoubleTapDispatchesDoubleTapSelector(t *testing.T) {
driverMock := mockdriver.New()
action := verifier.Action{Kind: verifier.ActionKindDoubleTap, On: "id:save"}
@@ -402,12 +457,12 @@ func TestApplyAction_DoubleTapDispatchesTwoSelectorTaps(t *testing.T) {
}
taps := 0
for _, a := range driverMock.Actions() {
if a.Kind == mockdriver.ActionTapSelector && a.Selector == "id:save" {
if a.Kind == mockdriver.ActionDoubleTapSelector && a.Selector == "id:save" {
taps++
}
}
if taps != 2 {
t.Errorf("expected 2 TapSelector calls with id:save, got %d in %v", taps, driverMock.Actions())
if taps != 1 {
t.Errorf("expected 1 DoubleTapSelector call with id:save, got %d in %v", taps, driverMock.Actions())
}
}
@@ -962,7 +1017,8 @@ func TestIsTransientApplyError_Classification(t *testing.T) {
// after a Wait action - the action already provides settling time.
func TestRunner_WaitActionSkipsIdle(t *testing.T) {
const waitSpec = `
globalThis.actions = __sanderling__.actions(() => [__sanderling__.wait({ durationMillis: 5 })]);
import { actions, Wait } from "@sanderling/spec";
globalThis.actions = actions(() => [Wait({ durationMillis: 5 })]);
`
state := newHarnessWithSpec(t, waitSpec)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
internal/runner/source.go
+72
View File
@@ -0,0 +1,72 @@
package runner
import (
"context"
"encoding/json"
"fmt"
"github.com/priyanshujain/sanderling/internal/driver"
"github.com/priyanshujain/sanderling/internal/verifier"
)
// ActionSource resolves the next action for a step. Both runtimes (the goja
// picker and the web/V8 picker) implement it so the runner loop has one path
// and no per-step driver type assertion. NextAction returns verifier.ErrNoAction
// when the picker declined to act this tick.
type ActionSource interface {
NextAction(ctx context.Context) (verifier.Action, error)
}
// ExtractorSource yields per-step extractor overrides the runner applies after
// PushSnapshot. The mobile path has none (returns nil); the web path returns the
// values its extractors computed in V8 against the real DOM.
type ExtractorSource interface {
ExtractorOverrides(ctx context.Context) (map[int]json.RawMessage, error)
}
// gojaSource drives both action selection and (trivially) extractor overrides
// for the mobile path, where the goja-bundled picker runs in-process and no V8
// extractor values exist.
type gojaSource struct {
verifier *verifier.Verifier
}
func (s gojaSource) NextAction(context.Context) (verifier.Action, error) {
return s.verifier.NextAction()
}
func (gojaSource) ExtractorOverrides(context.Context) (map[int]json.RawMessage, error) {
return nil, nil
}
// webSource adapts the chrome driver's V8 picker: it evaluates the bundled
// __sanderlingNextAction__ / __sanderlingExtractors__ globals and decodes their
// JSON into the unified action and override shapes.
type webSource struct {
web driver.WebDriver
}
func (s webSource) NextAction(ctx context.Context) (verifier.Action, error) {
raw, err := s.web.NextActionFromV8(ctx)
if err != nil {
return verifier.Action{}, fmt.Errorf("v8 next action: %w", err)
}
// Both engines emit the unified flat camelCase wire contract; one decoder
// reads it. A null payload means the generator declined to act this tick.
return verifier.DecodeAction(raw)
}
func (s webSource) ExtractorOverrides(ctx context.Context) (map[int]json.RawMessage, error) {
return s.web.EvaluateExtractors(ctx)
}
// pickSources selects the runtime's action and extractor sources ONCE at setup
// from the driver's capabilities, so the step loop never type-asserts.
func pickSources(options Options) (ActionSource, ExtractorSource) {
if web, ok := options.Driver.(driver.WebDriver); ok {
source := webSource{web: web}
return source, source
}
source := gojaSource{verifier: options.Verifier}
return source, source
}
internal/runner/testdata/trace.jsonl
Vendored
+4
View File
@@ -0,0 +1,4 @@
{"extractor_changes":{"extractor_0":{"prev":null,"curr":0}},"hierarchy":{"elements":[{"resourceId":"HomeScreen","bounds":{"left":0,"top":0,"right":0,"bottom":0},"attrs":{"editable":"false","resource-id":"HomeScreen"}},{"resourceId":"next","clickable":true,"enabled":true,"bounds":{"left":40,"top":80,"right":240,"bottom":160},"attrs":{"bounds":"[40,80,240,160]","clickable":"true","editable":"false","enabled":"true","resource-id":"next"}}]},"next_action":{"kind":"Tap","selector":"id:next","resolved_bounds":{"x":40,"y":80,"width":200,"height":80},"tap_point":{"x":140,"y":120}},"residuals":{"balanceNonNegative":{"op":"true"}},"step":1,"timestamp":"0001-01-01T00:00:00Z"}
{"hierarchy":{"elements":[{"resourceId":"HomeScreen","bounds":{"left":0,"top":0,"right":0,"bottom":0},"attrs":{"editable":"false","resource-id":"HomeScreen"}},{"resourceId":"next","clickable":true,"enabled":true,"bounds":{"left":40,"top":80,"right":240,"bottom":160},"attrs":{"bounds":"[40,80,240,160]","clickable":"true","editable":"false","enabled":"true","resource-id":"next"}}]},"next_action":{"kind":"Tap","selector":"id:next","resolved_bounds":{"x":40,"y":80,"width":200,"height":80},"tap_point":{"x":140,"y":120}},"residuals":{"balanceNonNegative":{"op":"true"}},"step":2,"timestamp":"0001-01-01T00:00:00Z"}
{"hierarchy":{"elements":[{"resourceId":"HomeScreen","bounds":{"left":0,"top":0,"right":0,"bottom":0},"attrs":{"editable":"false","resource-id":"HomeScreen"}},{"resourceId":"next","clickable":true,"enabled":true,"bounds":{"left":40,"top":80,"right":240,"bottom":160},"attrs":{"bounds":"[40,80,240,160]","clickable":"true","editable":"false","enabled":"true","resource-id":"next"}}]},"next_action":{"kind":"Tap","selector":"id:next","resolved_bounds":{"x":40,"y":80,"width":200,"height":80},"tap_point":{"x":140,"y":120}},"residuals":{"balanceNonNegative":{"op":"true"}},"step":3,"timestamp":"0001-01-01T00:00:00Z"}
{"hierarchy":{"elements":[{"resourceId":"HomeScreen","bounds":{"left":0,"top":0,"right":0,"bottom":0},"attrs":{"editable":"false","resource-id":"HomeScreen"}},{"resourceId":"next","clickable":true,"enabled":true,"bounds":{"left":40,"top":80,"right":240,"bottom":160},"attrs":{"bounds":"[40,80,240,160]","clickable":"true","editable":"false","enabled":"true","resource-id":"next"}}]},"next_action":{"kind":"Tap","selector":"id:next","resolved_bounds":{"x":40,"y":80,"width":200,"height":80},"tap_point":{"x":140,"y":120}},"residuals":{"balanceNonNegative":{"op":"true"}},"step":4,"timestamp":"0001-01-01T00:00:00Z"}
internal/runner/testdata/violation-summary.txt
Vendored
+4
View File
@@ -0,0 +1,4 @@
run complete: 4 steps
1 violation record(s):
step 1: [balanceNonNegative]
internal/sidecar/embed_test.go
-20
View File
@@ -1,20 +0,0 @@
//go:build !withsidecar
package sidecar
import "testing"
func TestStub_IsPlaceholder(t *testing.T) {
if !IsPlaceholder() {
t.Errorf("default build should be a placeholder")
}
if EmbeddedSize() != 0 {
t.Errorf("stub should have empty embedded JAR, got size %d", EmbeddedSize())
}
}
func TestStub_ExtractFails(t *testing.T) {
if _, err := Extract(t.TempDir()); err == nil {
t.Fatal("expected Extract to fail in stub mode")
}
}
internal/sidecar/embed.go → internal/sidecarassets/embed.go
+2 -1
View File
@@ -1,4 +1,5 @@
package sidecar
// Package sidecarassets embeds the native sidecar JAR and extracts it to disk at runtime.
package sidecarassets
import (
"crypto/sha256"
internal/sidecar/embed_real.go → internal/sidecarassets/embed_real.go
+1 -1
View File
@@ -1,6 +1,6 @@
//go:build withsidecar
package sidecar
package sidecarassets
import _ "embed"
internal/sidecar/embed_stub.go → internal/sidecarassets/embed_stub.go
+1 -1
View File
@@ -1,6 +1,6 @@
//go:build !withsidecar
package sidecar
package sidecarassets
var embeddedJAR []byte
internal/sidecar/embed_withsidecar_test.go → internal/sidecarassets/embed_withsidecar_test.go
+1 -1
View File
@@ -1,6 +1,6 @@
//go:build withsidecar
package sidecar
package sidecarassets
import (
"crypto/sha256"
internal/testrun/driver.go
+3 -3
View File
@@ -14,7 +14,7 @@ import (
"github.com/priyanshujain/sanderling/internal/driver/chrome"
driverSidecar "github.com/priyanshujain/sanderling/internal/driver/sidecar"
"github.com/priyanshujain/sanderling/internal/ios"
"github.com/priyanshujain/sanderling/internal/sidecar"
"github.com/priyanshujain/sanderling/internal/sidecarassets"
)
// buildDriver creates the appropriate DeviceDriver for the platform and returns
@@ -30,11 +30,11 @@ func buildDriver(ctx context.Context, options Options, stdout io.Writer) (driver
}
sidecarDirectory := os.TempDir() + "/sanderling-sidecar"
jarPath, err := sidecar.Extract(sidecarDirectory)
jarPath, err := sidecarassets.Extract(sidecarDirectory)
if err != nil {
return nil, nil, fmt.Errorf("extract sidecar: %w", err)
}
fmt.Fprintf(stdout, "sidecar JAR: %s (size=%d)\n", jarPath, sidecar.EmbeddedSize())
fmt.Fprintf(stdout, "sidecar JAR: %s (size=%d)\n", jarPath, sidecarassets.EmbeddedSize())
sidecarPort, err := pickFreePort()
if err != nil {
internal/testrun/testrun.go
+41 -23
View File
@@ -1,12 +1,13 @@
// Package testrun wires together the device, bundler, runner, and verifier into a single test pipeline.
package testrun
import (
"context"
"fmt"
"io"
"math/rand/v2"
"os"
"path/filepath"
"strconv"
"time"
"github.com/priyanshujain/sanderling/internal/android"
@@ -53,14 +54,21 @@ func Execute(ctx context.Context, options Options, stdout io.Writer) error {
aliases["@sanderling/spec/defaults"] = filepath.Join(base, "defaults/index.ts")
aliases["@sanderling/spec/defaults/properties"] = filepath.Join(base, "defaults/properties.ts")
}
seed := resolveSeed(options.Seed)
defines := map[string]string{
"SANDERLING_TEST_PHONE": os.Getenv("SANDERLING_TEST_PHONE"),
"SANDERLING_TEST_OTP": os.Getenv("SANDERLING_TEST_OTP"),
"SANDERLING_SEED": strconv.FormatInt(seed, 10),
}
gojaRuntimePath := resolveGojaRuntimePath(specAPIPath, options.Spec)
if gojaRuntimePath == "" {
return fmt.Errorf("goja-runtime.ts not found near %s; checkout pkg/spec or set @sanderling/spec alias", options.Spec)
}
bundle, err := bundler.Bundle(bundler.Options{
EntryFile: options.Spec,
Defines: defines,
Aliases: aliases,
EntryFile: options.Spec,
RuntimeFile: gojaRuntimePath,
Defines: defines,
Aliases: aliases,
})
if err != nil {
return fmt.Errorf("bundle spec: %w", err)
@@ -101,12 +109,9 @@ func Execute(ctx context.Context, options Options, stdout io.Writer) error {
}
}
seed := options.Seed
if seed == 0 {
seed = time.Now().UnixNano()
}
verifierInstance, err := verifier.New(
verifier.WithRand(rand.New(rand.NewPCG(uint64(seed), 0))),
verifier.WithSeed(uint64(seed)),
verifier.WithPlatform(options.Platform),
verifier.WithAppPackage(options.BundleID),
)
if err != nil {
@@ -161,25 +166,38 @@ func Execute(ctx context.Context, options Options, stdout io.Writer) error {
return fmt.Errorf("runner: %w", err)
}
fmt.Fprintf(stdout, "\nrun complete: %d steps in %s\n", summary.Steps, summary.EndTime.Sub(summary.StartTime).Round(time.Millisecond))
if len(summary.Violations) == 0 {
fmt.Fprintln(stdout, "no violations.")
} else {
fmt.Fprintf(stdout, "%d violation record(s):\n", len(summary.Violations))
for _, violation := range summary.Violations {
fmt.Fprintf(stdout, " step %d: %v\n", violation.StepIndex, violation.Properties)
}
}
fmt.Fprintf(stdout, "\nelapsed: %s\n", summary.EndTime.Sub(summary.StartTime).Round(time.Millisecond))
runner.RenderSummary(stdout, summary, options.Platform)
return nil
}
// resolveSeed returns the configured seed, or a time-derived one when unset.
// The same value seeds both the goja PRNG and the web bundle's SANDERLING_SEED
// define, so a single run is reproducible across both runtimes.
func resolveSeed(configured int64) int64 {
if configured != 0 {
return configured
}
return time.Now().UnixNano()
}
// resolveWebRuntimePath returns the path to pkg/spec/src/web-runtime.ts.
// Tries the spec-API checkout first (so monorepo development works without
// publishing the package), then falls back to a sibling of the resolved
// @sanderling/spec entry, and finally to a node_modules path.
func resolveWebRuntimePath(specAPIPath, userSpecPath string) string {
return resolveRuntimeSibling(specAPIPath, userSpecPath, "web-runtime.ts")
}
// resolveGojaRuntimePath returns the path to pkg/spec/src/goja-runtime.ts, the
// native verifier's runtime entry that installs __sanderlingNextAction__.
func resolveGojaRuntimePath(specAPIPath, userSpecPath string) string {
return resolveRuntimeSibling(specAPIPath, userSpecPath, "goja-runtime.ts")
}
// resolveRuntimeSibling finds a runtime-entry file that sits beside the spec-API
// index.ts. Tries the spec-API checkout first (so monorepo development works
// without publishing the package), then falls back to a node_modules path.
func resolveRuntimeSibling(specAPIPath, userSpecPath, filename string) string {
if specAPIPath != "" {
candidate := filepath.Join(filepath.Dir(specAPIPath), "web-runtime.ts")
candidate := filepath.Join(filepath.Dir(specAPIPath), filename)
if _, err := os.Stat(candidate); err == nil {
return candidate
}
@@ -187,7 +205,7 @@ func resolveWebRuntimePath(specAPIPath, userSpecPath string) string {
if absoluteSpec, err := filepath.Abs(userSpecPath); err == nil {
directory := filepath.Dir(absoluteSpec)
for {
candidate := filepath.Join(directory, "node_modules", "@sanderling", "spec", "src", "web-runtime.ts")
candidate := filepath.Join(directory, "node_modules", "@sanderling", "spec", "src", filename)
if _, err := os.Stat(candidate); err == nil {
return candidate
}
internal/testrun/testrun_test.go
+22
View File
@@ -3,9 +3,31 @@ package testrun
import (
"os"
"path/filepath"
"strconv"
"testing"
)
func TestResolveSeed_UsesConfiguredWhenNonZero(t *testing.T) {
if got := resolveSeed(42); got != 42 {
t.Fatalf("got %d, want 42", got)
}
}
func TestResolveSeed_DerivesWhenZero(t *testing.T) {
if got := resolveSeed(0); got == 0 {
t.Fatal("expected a non-zero time-derived seed")
}
}
// TestSeedDefineFormatsAsDecimal guards the contract that Execute uses to put
// the seed into the web bundle: strconv.FormatInt base 10, which web-runtime
// folds to a 32-bit PRNG seed.
func TestSeedDefineFormatsAsDecimal(t *testing.T) {
if got := strconv.FormatInt(resolveSeed(8675309), 10); got != "8675309" {
t.Fatalf("got %q, want 8675309", got)
}
}
func TestResolveSpecAPIPath_FindsUpwardSibling(t *testing.T) {
root := t.TempDir()
apiPath := filepath.Join(root, "pkg", "spec", "src", "index.ts")
internal/trace/writer.go
+35 -19
View File
@@ -1,3 +1,4 @@
// Package trace records each run's steps, snapshots, and violations to disk for later inspection.
package trace
import (
@@ -13,23 +14,38 @@ import (
)
type Step struct {
Index int `json:"step"`
Timestamp time.Time `json:"timestamp"`
Screen string `json:"screen,omitempty"`
Snapshots map[string]json.RawMessage `json:"snapshots,omitempty"`
Index int `json:"step"`
Timestamp time.Time `json:"timestamp"`
Screen string `json:"screen,omitempty"`
Snapshots map[string]json.RawMessage `json:"snapshots,omitempty"`
// NextAction is the action chosen for the next iteration based on observing this step.
NextAction *Action `json:"next_action,omitempty"`
Exceptions []Exception `json:"exceptions,omitempty"`
Violations []string `json:"violations,omitempty"`
Hierarchy *hierarchy.Tree `json:"hierarchy,omitempty"`
Residuals map[string]json.RawMessage `json:"residuals,omitempty"`
Metrics *Metrics `json:"metrics,omitempty"`
ExtractorChanges map[string]ExtractorChange `json:"extractor_changes,omitempty"`
NextAction *Action `json:"next_action,omitempty"`
Exceptions []Exception `json:"exceptions,omitempty"`
Violations []string `json:"violations,omitempty"`
Hierarchy *hierarchy.Tree `json:"hierarchy,omitempty"`
Residuals map[string]json.RawMessage `json:"residuals,omitempty"`
Metrics *Metrics `json:"metrics,omitempty"`
ExtractorChanges map[string]ExtractorChange `json:"extractor_changes,omitempty"`
// Transitional marks a step whose hierarchy still showed a NavHost
// cross-fade (multiple route-level *Screen ids) after the runner's
// retry budget. The verifier is skipped for these steps so transient
// state does not poison the previous/current extractor advance.
Transitional bool `json:"transitional,omitempty"`
// SkippedVerification is set true exactly when the verifier was skipped
// for this step, so downstream tooling can tell a deliberately-skipped
// step from one that was verified and came back clean.
SkippedVerification bool `json:"skipped_verification,omitempty"`
// Witnesses records the violation witness for each property that newly
// violated at this step: the cause and the extractor values at onset.
Witnesses map[string]Witness `json:"witnesses,omitempty"`
}
// Witness is the trace-side record of a property violation: why it fired and a
// snapshot of every extractor's value at the violating step.
type Witness struct {
Reason string `json:"reason,omitempty"`
IsError bool `json:"is_error,omitempty"`
Extractors map[string]json.RawMessage `json:"extractors,omitempty"`
}
// ExtractorChange records the prev/curr JSON values of an extractor whose
@@ -83,14 +99,14 @@ type Exception struct {
}
type Meta struct {
Seed int64 `json:"seed"`
SpecPath string `json:"spec_path"`
BundleSHA256 string `json:"bundle_sha256"`
Platform string `json:"platform"`
BundleID string `json:"bundle_id"`
StartedAt time.Time `json:"started_at"`
EndedAt *time.Time `json:"ended_at,omitempty"`
SanderlingVersion string `json:"sanderling_version"`
Seed int64 `json:"seed"`
SpecPath string `json:"spec_path"`
BundleSHA256 string `json:"bundle_sha256"`
Platform string `json:"platform"`
BundleID string `json:"bundle_id"`
StartedAt time.Time `json:"started_at"`
EndedAt *time.Time `json:"ended_at,omitempty"`
SanderlingVersion string `json:"sanderling_version"`
}
type Writer struct {
internal/verifier/ax_integration_test.go
+1 -1
View File
@@ -61,7 +61,7 @@ func TestStateAxObjectSelectorTestTagAlias(t *testing.T) {
// TestStateAxFindWorks verifies that a Parse+PushSnapshot+extract round trip
// actually lets the spec resolve selectors through state.ax.find.
// Reads /tmp/live-dump.json (Maestro TreeNode JSON format); skipped if absent.
// Reads /tmp/live-dump.json (sidecar TreeNode JSON format); skipped if absent.
func TestStateAxFindWorks(t *testing.T) {
jsonText, err := os.ReadFile("/tmp/live-dump.json")
if err != nil {
internal/verifier/bindings.go
+105 -245
View File
@@ -2,6 +2,7 @@ package verifier
import (
"fmt"
"math/big"
"time"
"github.com/dop251/goja"
@@ -9,8 +10,10 @@ import (
type extractorState struct {
getter goja.Callable
handle *goja.Object
name string
// currentValue/previousValue back the handle's current/previous accessors.
currentValue goja.Value
previousValue goja.Value
// prev/curr cache the JSON-encoded extractor values from the prior and
// current PushSnapshot, used by ChangedExtractors to surface per-step
// diffs in the trace.
@@ -20,11 +23,6 @@ type extractorState struct {
type formulaState struct {
predicate goja.Callable
// err holds the goja error from this thunk's most recent invocation, or
// nil if the latest call succeeded. The thunk returns false on error so
// the LTL evaluator marks the property violated; PredicateError surfaces
// the underlying cause for the current step.
err error
}
type specKind int
@@ -62,20 +60,7 @@ type formulaSpec struct {
const (
tagFormula = "__sanderlingFormula"
tagFormulaSpecIndex = "__sanderlingFormulaSpec"
tagActionGenerator = "__sanderlingActionGenerator"
tagInternalKind = "__sanderlingKind"
tagSelector = "__sanderlingSelector"
internalKindActions = "actions"
internalKindWeighted = "weighted"
internalKindBuiltinTaps = "taps"
internalKindBuiltinDoubleTaps = "doubleTaps"
internalKindBuiltinTyping = "typing"
internalKindBuiltinSwipes = "swipes"
internalKindBuiltinWaitOnce = "waitOnce"
internalKindBuiltinPressKey = "pressKey"
internalKindBuiltinLongPresses = "longPresses"
internalKindBuiltinScrolls = "scrolls"
)
// installRuntimeBindings exposes globalThis.__sanderling__ to the loaded spec.
@@ -97,65 +82,75 @@ func (v *Verifier) installRuntimeBindings() error {
if err := sanderling.Set("eventually", v.bindEventually); err != nil {
return err
}
if err := sanderling.Set("actions", v.bindActions); err != nil {
return err
}
if err := sanderling.Set("weighted", v.bindWeighted); err != nil {
return err
}
if err := sanderling.Set("from", v.bindFrom); err != nil {
return err
}
if err := sanderling.Set("tap", v.bindTap); err != nil {
return err
}
if err := sanderling.Set("doubleTap", v.bindDoubleTap); err != nil {
return err
}
if err := sanderling.Set("longPress", v.bindLongPress); err != nil {
return err
}
if err := sanderling.Set("scroll", v.bindScroll); err != nil {
return err
}
if err := sanderling.Set("inputText", v.bindInputText); err != nil {
return err
}
if err := sanderling.Set("swipe", v.bindSwipe); err != nil {
return err
}
if err := sanderling.Set("pressKey", v.bindPressKey); err != nil {
return err
}
if err := sanderling.Set("wait", v.bindWait); err != nil {
return err
}
if err := sanderling.Set("taps", v.builtinGenerator(internalKindBuiltinTaps)); err != nil {
return err
}
if err := sanderling.Set("doubleTaps", v.builtinGenerator(internalKindBuiltinDoubleTaps)); err != nil {
return err
}
if err := sanderling.Set("typing", v.builtinGenerator(internalKindBuiltinTyping)); err != nil {
return err
}
if err := sanderling.Set("swipes", v.builtinGenerator(internalKindBuiltinSwipes)); err != nil {
return err
}
if err := sanderling.Set("waitOnce", v.builtinGenerator(internalKindBuiltinWaitOnce)); err != nil {
return err
}
if err := sanderling.Set("pressKeys", v.builtinGenerator(internalKindBuiltinPressKey)); err != nil {
return err
}
if err := sanderling.Set("longPresses", v.builtinGenerator(internalKindBuiltinLongPresses)); err != nil {
return err
}
if err := sanderling.Set("scrolls", v.builtinGenerator(internalKindBuiltinScrolls)); err != nil {
return err
}
return v.runtime.GlobalObject().Set("__sanderling__", sanderling)
if err := v.runtime.GlobalObject().Set("__sanderling__", sanderling); err != nil {
return err
}
return v.installHost()
}
// installHost exposes globalThis.__sanderlingHost__ for the goja runtime entry.
// The shared picker (pick.ts) draws against it: platform() drives the verb
// matrix and press-key pool; seedHi/seedLo construct its Pcg; queryCandidates
// enumerates targets over the hierarchy tree; reportUnsupported records the
// verb for the run report.
func (v *Verifier) installHost() error {
host := v.runtime.NewObject()
if err := host.Set("platform", func(goja.FunctionCall) goja.Value {
return v.runtime.ToValue(v.platform)
}); err != nil {
return err
}
if err := host.Set("seedHi", func(goja.FunctionCall) goja.Value {
return v.runtime.ToValue(new(big.Int).SetUint64(v.seed))
}); err != nil {
return err
}
if err := host.Set("seedLo", func(goja.FunctionCall) goja.Value {
return v.runtime.ToValue(big.NewInt(0))
}); err != nil {
return err
}
if err := host.Set("queryCandidates", v.bindQueryCandidates); err != nil {
return err
}
if err := host.Set("reportUnsupported", func(call goja.FunctionCall) goja.Value {
v.recordUnsupported(call.Argument(0).String())
return goja.Undefined()
}); err != nil {
return err
}
return v.runtime.GlobalObject().Set("__sanderlingHost__", host)
}
// recordUnsupported notes a verb the picker requested that this platform
// cannot dispatch, deduped and in first-seen order.
func (v *Verifier) recordUnsupported(verb string) {
if verb == "" || v.unsupportedSeen[verb] {
return
}
v.unsupportedSeen[verb] = true
v.unsupported = append(v.unsupported, verb)
}
// bindQueryCandidates returns the host-enumerated targets for a verb as an
// array of {x, y, selector, width, height}, in tree order.
func (v *Verifier) bindQueryCandidates(call goja.FunctionCall) goja.Value {
verb := call.Argument(0).String()
candidates := v.candidatesForVerb(verb)
array := v.runtime.NewArray()
for index, candidate := range candidates {
item := v.runtime.NewObject()
_ = item.Set("x", candidate.x)
_ = item.Set("y", candidate.y)
_ = item.Set("width", candidate.width)
_ = item.Set("height", candidate.height)
if candidate.selector != "" {
_ = item.Set("selector", candidate.selector)
}
_ = array.Set(fmt.Sprintf("%d", index), item)
}
return array
}
func (v *Verifier) bindExtract(call goja.FunctionCall) goja.Value {
@@ -177,14 +172,43 @@ func (v *Verifier) bindExtract(call goja.FunctionCall) goja.Value {
name = fmt.Sprintf("extractor_%d", len(v.extractors))
}
handle := v.runtime.NewObject()
_ = handle.Set("current", goja.Undefined())
_ = handle.Set("previous", goja.Undefined())
state := &extractorState{
getter: getter,
name: name,
currentValue: goja.Undefined(),
previousValue: goja.Undefined(),
}
v.extractors = append(v.extractors, &extractorState{getter: getter, handle: handle, name: name})
handle := v.runtime.NewObject()
_ = handle.DefineAccessorProperty("current", v.runtime.ToValue(func(goja.FunctionCall) goja.Value {
v.checkNotExtracting("current")
return state.currentValue
}), nil, goja.FLAG_FALSE, goja.FLAG_TRUE)
_ = handle.DefineAccessorProperty("previous", v.runtime.ToValue(func(goja.FunctionCall) goja.Value {
v.checkNotExtracting("previous")
return state.previousValue
}), nil, goja.FLAG_FALSE, goja.FLAG_TRUE)
_ = handle.Set("named", func(call goja.FunctionCall) goja.Value {
state.name = call.Argument(0).String()
return handle
})
v.extractors = append(v.extractors, state)
return handle
}
// checkNotExtracting panics with a JS error when an extractor getter tries to
// read another extractor handle's current/previous. The message is identical to
// the web runtime's so authors see one diagnostic across engines.
func (v *Verifier) checkNotExtracting(slot string) {
if v.extracting {
panic(v.runtime.NewGoError(fmt.Errorf(
"reading .%s of an extractor inside another extractor is not allowed; extractor getters may read only from the state argument",
slot,
)))
}
}
// bindAlways accepts either a predicate function (legacy shape) or a formula
// handle (new shape). Both produce a formula handle tagged with
// __sanderlingFormulaSpec.
@@ -273,9 +297,6 @@ func (v *Verifier) formulaHandle(kind specKind, index int) *goja.Object {
handle := v.runtime.NewObject()
_ = handle.Set(tagFormula, true)
_ = handle.Set(tagFormulaSpecIndex, index)
// Keep __sanderlingIndex as an alias so older property shapes that read it keep
// working during backward-compat transitions.
_ = handle.Set("__sanderlingIndex", index)
_ = handle.Set("implies", v.binaryChain(index, specKindImplies))
_ = handle.Set("or", v.binaryChain(index, specKindOr))
@@ -356,164 +377,3 @@ func (v *Verifier) extractSpecIndex(value goja.Value) (int, bool) {
}
return int(indexValue.ToInteger()), true
}
func (v *Verifier) bindActions(call goja.FunctionCall) goja.Value {
if len(call.Arguments) != 1 {
panic(v.runtime.NewTypeError("actions requires a single generator argument"))
}
if _, ok := goja.AssertFunction(call.Arguments[0]); !ok {
panic(v.runtime.NewTypeError("actions argument must be a function"))
}
handle := v.runtime.NewObject()
_ = handle.Set(tagActionGenerator, true)
_ = handle.Set(tagInternalKind, internalKindActions)
_ = handle.Set("generate", call.Arguments[0])
return handle
}
func (v *Verifier) bindWeighted(call goja.FunctionCall) goja.Value {
entries := v.runtime.NewArray()
for index, argument := range call.Arguments {
object := argument.ToObject(v.runtime)
if object == nil {
panic(v.runtime.NewTypeError(fmt.Sprintf("weighted entry %d must be a [number, generator] tuple", index)))
}
if err := entries.Set(fmt.Sprintf("%d", index), object); err != nil {
panic(v.runtime.NewGoError(err))
}
}
handle := v.runtime.NewObject()
_ = handle.Set(tagActionGenerator, true)
_ = handle.Set(tagInternalKind, internalKindWeighted)
_ = handle.Set("entries", entries)
return handle
}
// bindFrom returns a `{ generate }` that picks uniformly at random from the
// provided items using the verifier's seeded rng.
func (v *Verifier) bindFrom(call goja.FunctionCall) goja.Value {
if len(call.Arguments) != 1 {
panic(v.runtime.NewTypeError("from requires an array argument"))
}
itemsValue := call.Arguments[0]
itemsObject := itemsValue.ToObject(v.runtime)
if itemsObject == nil {
panic(v.runtime.NewTypeError("from argument must be an array"))
}
lengthValue := itemsObject.Get("length")
if lengthValue == nil {
panic(v.runtime.NewTypeError("from argument must be array-like"))
}
length := int(lengthValue.ToInteger())
handle := v.runtime.NewObject()
_ = handle.Set("generate", func(goja.FunctionCall) goja.Value {
if length == 0 {
return goja.Undefined()
}
index := v.rng.IntN(length)
return itemsObject.Get(fmt.Sprintf("%d", index))
})
return handle
}
func (v *Verifier) bindTap(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("Tap requires {on}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "Tap")
_ = handle.Set("on", parameters.Get("on"))
return handle
}
func (v *Verifier) bindDoubleTap(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("DoubleTap requires {on}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "DoubleTap")
_ = handle.Set("on", parameters.Get("on"))
return handle
}
func (v *Verifier) bindLongPress(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("LongPress requires {on}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "LongPress")
_ = handle.Set("on", parameters.Get("on"))
return handle
}
func (v *Verifier) bindScroll(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("Scroll requires {direction}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "Scroll")
_ = handle.Set("direction", parameters.Get("direction"))
_ = handle.Set("in", parameters.Get("in"))
return handle
}
func (v *Verifier) bindInputText(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("InputText requires {into, text}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "InputText")
_ = handle.Set("into", parameters.Get("into"))
_ = handle.Set("text", parameters.Get("text"))
return handle
}
func (v *Verifier) bindSwipe(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("Swipe requires {from, to}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "Swipe")
_ = handle.Set("from", parameters.Get("from"))
_ = handle.Set("to", parameters.Get("to"))
if duration := parameters.Get("durationMillis"); duration != nil && !goja.IsUndefined(duration) {
_ = handle.Set("durationMillis", duration)
}
return handle
}
func (v *Verifier) bindPressKey(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("PressKey requires {key}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "PressKey")
_ = handle.Set("key", parameters.Get("key"))
return handle
}
func (v *Verifier) bindWait(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("Wait requires {durationMillis}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "Wait")
_ = handle.Set("durationMillis", parameters.Get("durationMillis"))
return handle
}
func (v *Verifier) builtinGenerator(kind string) *goja.Object {
handle := v.runtime.NewObject()
_ = handle.Set(tagActionGenerator, true)
_ = handle.Set(tagInternalKind, kind)
return handle
}
internal/verifier/extractor_guard_test.go
+34
View File
@@ -0,0 +1,34 @@
package verifier
import (
"strings"
"testing"
)
func TestPushSnapshot_CrossExtractorReadIsGuarded(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
const a = __sanderling__.extract(state => 1);
const b = __sanderling__.extract(() => a.current);
globalThis.a = a;
globalThis.b = b;
`)
err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
if err == nil {
t.Fatal("expected guard error, got nil")
}
if !strings.Contains(err.Error(), "inside another extractor is not allowed") {
t.Errorf("guard error wrong: %v", err)
}
}
func TestExtract_NamedSetsExtractorName(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
const route = __sanderling__.extract(state => "home").named("route");
globalThis.route = route;
`)
if got := verifier.extractors[0].name; got != "route" {
t.Errorf("named() did not set name: got %q, want %q", got, "route")
}
}
internal/verifier/marshal.go
+52 -116
View File
@@ -266,10 +266,10 @@ func selectorPathFromJS(runtime *goja.Runtime, arg goja.Value) ([]hierarchy.Sele
}
// selectorStringFromJS returns a string representation of the selector argument
// for tagging returned element objects (used by selectorOf to reconstruct the
// selector when the element is passed back as an action target). Output
// follows the canonical hierarchy selector grammar: "k:v" pairs space-joined
// per object, chains joined by " > ".
// for tagging returned ax element objects (state.ax.find/findAll), so a spec
// reading an element's selector sees the canonical form. Output follows the
// hierarchy selector grammar: "k:v" pairs space-joined per object, chains
// joined by " > ".
func selectorStringFromJS(runtime *goja.Runtime, arg goja.Value) string {
if arg == nil || goja.IsUndefined(arg) || goja.IsNull(arg) {
return ""
@@ -414,132 +414,68 @@ func jsonToJSValue(runtime *goja.Runtime, raw json.RawMessage) (goja.Value, erro
return runtime.ToValue(generic), nil
}
// jsValueToAction converts a JS-side action object into a Go Action.
func jsValueToAction(runtime *goja.Runtime, value goja.Value) (Action, error) {
if value == nil || goja.IsNull(value) || goja.IsUndefined(value) {
return Action{}, fmt.Errorf("nil action")
// wireAction is the unified flat wire contract both runtime entries emit
// (runtime-entry.ts serializeAction). ONE decoder reads it on both the goja
// (native) and the runner (web) sides, so a field rename cannot silently turn
// an action into a no-op on one path only.
type wireAction struct {
Kind string `json:"kind"`
X int `json:"x"`
Y int `json:"y"`
Selector string `json:"selector"`
Text string `json:"text"`
Key string `json:"key"`
Direction string `json:"direction"`
FromX int `json:"fromX"`
FromY int `json:"fromY"`
ToX int `json:"toX"`
ToY int `json:"toY"`
DurationMillis int `json:"durationMillis"`
}
// DecodeAction turns one serialized action (the flat camelCase wire contract)
// into a Go Action. An empty payload or JSON "null" reports ErrNoAction.
func DecodeAction(raw json.RawMessage) (Action, error) {
if len(raw) == 0 || string(raw) == "null" {
return Action{}, ErrNoAction
}
object := value.ToObject(runtime)
kindValue := object.Get("kind")
if kindValue == nil {
return Action{}, fmt.Errorf("action missing kind")
var wire wireAction
if err := json.Unmarshal(raw, &wire); err != nil {
return Action{}, fmt.Errorf("decode action: %w", err)
}
kind := kindValue.String()
switch kind {
switch wire.Kind {
case "Tap":
on := object.Get("on")
x, y := coordinatesOf(runtime, on)
return Action{Kind: ActionKindTap, On: selectorOf(runtime, on), X: x, Y: y}, nil
return Action{Kind: ActionKindTap, On: wire.Selector, X: wire.X, Y: wire.Y}, nil
case "DoubleTap":
on := object.Get("on")
x, y := coordinatesOf(runtime, on)
return Action{Kind: ActionKindDoubleTap, On: selectorOf(runtime, on), X: x, Y: y}, nil
return Action{Kind: ActionKindDoubleTap, On: wire.Selector, X: wire.X, Y: wire.Y}, nil
case "LongPress":
return Action{Kind: ActionKindLongPress, On: wire.Selector, X: wire.X, Y: wire.Y}, nil
case "InputText":
into := object.Get("into")
text := object.Get("text")
x, y := coordinatesOf(runtime, into)
return Action{Kind: ActionKindInputText, On: selectorOf(runtime, into), Text: stringOf(text), X: x, Y: y}, nil
return Action{Kind: ActionKindInputText, On: wire.Selector, Text: wire.Text, X: wire.X, Y: wire.Y}, nil
case "Swipe":
from := object.Get("from")
to := object.Get("to")
fromX, fromY := coordinatesOf(runtime, from)
toX, toY := coordinatesOf(runtime, to)
if fromX == 0 && fromY == 0 {
fromX, fromY = pointCoordinates(runtime, from)
}
if toX == 0 && toY == 0 {
toX, toY = pointCoordinates(runtime, to)
}
return Action{
Kind: ActionKindSwipe,
FromX: fromX,
FromY: fromY,
ToX: toX,
ToY: toY,
DurationMillis: intField(object, "durationMillis"),
FromX: wire.FromX,
FromY: wire.FromY,
ToX: wire.ToX,
ToY: wire.ToY,
DurationMillis: wire.DurationMillis,
}, nil
case "LongPress":
on := object.Get("on")
x, y := coordinatesOf(runtime, on)
return Action{Kind: ActionKindLongPress, On: selectorOf(runtime, on), X: x, Y: y}, nil
case "Scroll":
in := object.Get("in")
x, y := coordinatesOf(runtime, in)
return Action{
Kind: ActionKindScroll,
Direction: stringOf(object.Get("direction")),
On: selectorOf(runtime, in),
X: x,
Y: y,
Kind: ActionKindScroll,
Direction: wire.Direction,
FromX: wire.FromX,
FromY: wire.FromY,
ToX: wire.ToX,
ToY: wire.ToY,
DurationMillis: wire.DurationMillis,
}, nil
case "PressKey":
return Action{Kind: ActionKindPressKey, Key: stringOf(object.Get("key"))}, nil
return Action{Kind: ActionKindPressKey, Key: wire.Key}, nil
case "Wait":
return Action{Kind: ActionKindWait, DurationMillis: intField(object, "durationMillis")}, nil
return Action{Kind: ActionKindWait, DurationMillis: wire.DurationMillis}, nil
default:
return Action{}, fmt.Errorf("unknown action kind %q", kind)
return Action{}, fmt.Errorf("unknown action kind %q", wire.Kind)
}
}
// pointCoordinates reads a plain {x, y} literal (not an AX element), which is
// how Swipe endpoints are commonly expressed in specs.
func pointCoordinates(runtime *goja.Runtime, value goja.Value) (int, int) {
if value == nil || goja.IsNull(value) || goja.IsUndefined(value) {
return 0, 0
}
object := value.ToObject(runtime)
if object == nil {
return 0, 0
}
x := object.Get("x")
y := object.Get("y")
if x == nil || y == nil {
return 0, 0
}
return int(x.ToInteger()), int(y.ToInteger())
}
func intField(object *goja.Object, name string) int {
value := object.Get(name)
if value == nil || goja.IsUndefined(value) || goja.IsNull(value) {
return 0
}
return int(value.ToInteger())
}
func selectorOf(runtime *goja.Runtime, value goja.Value) string {
if value == nil || goja.IsNull(value) || goja.IsUndefined(value) {
return ""
}
object := value.ToObject(runtime)
if object == nil {
return value.String()
}
if tag := object.Get(tagSelector); tag != nil && !goja.IsUndefined(tag) {
return tag.String()
}
return value.String()
}
func coordinatesOf(runtime *goja.Runtime, value goja.Value) (int, int) {
if value == nil || goja.IsNull(value) || goja.IsUndefined(value) {
return 0, 0
}
object := value.ToObject(runtime)
if object == nil {
return 0, 0
}
xValue := object.Get("x")
yValue := object.Get("y")
if xValue == nil || yValue == nil || goja.IsUndefined(xValue) || goja.IsUndefined(yValue) {
return 0, 0
}
return int(xValue.ToInteger()), int(yValue.ToInteger())
}
func stringOf(value goja.Value) string {
if value == nil || goja.IsNull(value) || goja.IsUndefined(value) {
return ""
}
return value.String()
}
internal/verifier/parity_test.go
+103
View File
@@ -0,0 +1,103 @@
package verifier
import (
"encoding/json"
"os"
"path/filepath"
"testing"
)
// TestCrossRuntimeParity is the W2 acceptance gate: for a FIXED seed and a FIXED
// candidate state, the goja-bundled picker must emit the SAME action stream as
// the node/web pick.ts. Both engines run the SHARED pick.ts over the SHARED Pcg,
// so each asserts the SAME committed golden (pkg/spec/test/fixtures/
// parity-golden.json); the node side does so in pkg/spec/test/parity.test.ts.
// Matching one golden on both sides proves they match each other without either
// invoking the other.
//
// The candidate ORDER and the per-tick PCG draw order are the parity contract.
// A stub __sanderlingHost__ feeds the SAME three candidates (in the SAME order)
// for every verb, so the hierarchy filter is out of the picture and the only
// variable left is the JS engine. The weighted root mixes a tap branch with a
// typing branch, exercising weighted selection, a builtin, and the input corpus
// within the 20-tick window; reordering candidates or adding/dropping a draw on
// either side shifts the stream and fails the golden.
func TestCrossRuntimeParity(t *testing.T) {
const seed uint64 = 0x9e3779b97f4a7c15
golden := loadParityGolden(t)
verifier := newVerifier(t, WithSeed(seed))
installStubHost(t, verifier)
loadActionSpec(t, verifier, `
import { taps, typing, weighted } from "@sanderling/spec";
globalThis.actions = weighted([3, taps], [1, typing]);
`)
if len(golden) == 0 {
t.Fatal("golden stream is empty")
}
for step, want := range golden {
got, err := verifier.NextAction()
if err != nil {
t.Fatalf("goja next action at step %d: %v", step, err)
}
if got != want {
t.Fatalf("step %d diverged from golden:\n goja=%+v\n want=%+v", step, got, want)
}
}
}
// installStubHost replaces the verifier's hierarchy-backed __sanderlingHost__
// with one returning a FIXED candidate list for every verb, keeping the seed the
// verifier was constructed with. It must run BEFORE Load, because the bundled
// goja runtime entry captures the host when the spec evaluates.
func installStubHost(t *testing.T, verifier *Verifier) {
t.Helper()
const stub = `
const candidates = [
{ x: 50, y: 60, selector: "id:alpha", width: 100, height: 40 },
{ x: 150, y: 160, selector: "id:beta", width: 120, height: 48 },
{ x: 250, y: 260, selector: "id:gamma", width: 80, height: 32 },
];
const seedHi = globalThis.__sanderlingHost__.seedHi;
const seedLo = globalThis.__sanderlingHost__.seedLo;
globalThis.__sanderlingHost__ = {
platform: () => "android",
queryCandidates: () => candidates,
reportUnsupported: () => {},
seedHi,
seedLo,
};
`
if _, err := verifier.runtime.RunString(stub); err != nil {
t.Fatalf("install stub host: %v", err)
}
}
// loadParityGolden decodes the shared golden stream with the SAME DecodeAction
// the runner uses, so the goja comparison is apples-to-apples with the wire the
// node picker emits.
func loadParityGolden(t *testing.T) []Action {
t.Helper()
path, err := filepath.Abs("../../pkg/spec/test/fixtures/parity-golden.json")
if err != nil {
t.Fatal(err)
}
body, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read golden: %v", err)
}
var raw []json.RawMessage
if err := json.Unmarshal(body, &raw); err != nil {
t.Fatalf("decode golden: %v", err)
}
stream := make([]Action, len(raw))
for i, message := range raw {
action, err := DecodeAction(message)
if err != nil {
t.Fatalf("decode golden action %d: %v", i, err)
}
stream[i] = action
}
return stream
}
internal/verifier/scope_test.go
+38 -11
View File
@@ -33,7 +33,10 @@ func pushTree(t *testing.T, v *Verifier, treeJSON string) {
// the result lands on its center regardless of seed.
func TestTaps_ExcludeOffAppPackage(t *testing.T) {
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.taps;`)
loadActionSpec(t, verifier, `
import { taps } from "@sanderling/spec";
globalThis.actions = taps;
`)
pushTree(t, verifier, scopedTreeJSON)
action, err := verifier.NextAction()
@@ -56,7 +59,10 @@ func TestTyping_ExcludeOffAppPackage(t *testing.T) {
]
}`
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.typing;`)
loadActionSpec(t, verifier, `
import { typing } from "@sanderling/spec";
globalThis.actions = typing;
`)
pushTree(t, verifier, treeJSON)
action, err := verifier.NextAction()
@@ -72,7 +78,10 @@ func TestTyping_ExcludeOffAppPackage(t *testing.T) {
// exploration never scrolls the keyboard's emoji list instead of the app.
func TestSwipes_ExcludeOffAppPackage(t *testing.T) {
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.swipes;`)
loadActionSpec(t, verifier, `
import { swipes } from "@sanderling/spec";
globalThis.actions = swipes;
`)
pushTree(t, verifier, scopedTreeJSON)
// Both the root and SubmitButton (com.folio) are valid anchors; only the
@@ -103,7 +112,10 @@ func TestTaps_AllOffAppYieldsErrNoAction(t *testing.T) {
]
}`
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.taps;`)
loadActionSpec(t, verifier, `
import { taps } from "@sanderling/spec";
globalThis.actions = taps;
`)
pushTree(t, verifier, treeJSON)
if _, err := verifier.NextAction(); !errors.Is(err, ErrNoAction) {
@@ -121,7 +133,10 @@ func TestTaps_UnsetAppPackageKeepsAllNodes(t *testing.T) {
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `globalThis.actions = __sanderling__.taps;`)
loadActionSpec(t, verifier, `
import { taps } from "@sanderling/spec";
globalThis.actions = taps;
`)
pushTree(t, verifier, treeJSON)
action, err := verifier.NextAction()
@@ -137,7 +152,10 @@ func TestTaps_UnsetAppPackageKeepsAllNodes(t *testing.T) {
// mirrors taps: it yields a LongPress on the only clickable in-app node.
func TestLongPresses_TargetsClickableElement(t *testing.T) {
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.longPresses;`)
loadActionSpec(t, verifier, `
import { longPresses } from "@sanderling/spec";
globalThis.actions = longPresses;
`)
pushTree(t, verifier, scopedTreeJSON)
action, err := verifier.NextAction()
@@ -163,7 +181,10 @@ func TestScrolls_TargetsScrollableContainer(t *testing.T) {
]
}`
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.scrolls;`)
loadActionSpec(t, verifier, `
import { scrolls } from "@sanderling/spec";
globalThis.actions = scrolls;
`)
pushTree(t, verifier, treeJSON)
for range 50 {
@@ -190,8 +211,8 @@ func TestScrolls_TargetsScrollableContainer(t *testing.T) {
default:
t.Fatalf("unexpected direction %q", action.Direction)
}
if action.DurationMillis != 300 {
t.Fatalf("durationMillis = %d, want 300", action.DurationMillis)
if action.DurationMillis != 250 {
t.Fatalf("durationMillis = %d, want 250", action.DurationMillis)
}
}
}
@@ -200,7 +221,10 @@ func TestScrolls_TargetsScrollableContainer(t *testing.T) {
// no scrollable container is present.
func TestScrolls_NoScrollableYieldsErrNoAction(t *testing.T) {
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.scrolls;`)
loadActionSpec(t, verifier, `
import { scrolls } from "@sanderling/spec";
globalThis.actions = scrolls;
`)
pushTree(t, verifier, scopedTreeJSON)
if _, err := verifier.NextAction(); !errors.Is(err, ErrNoAction) {
@@ -219,7 +243,10 @@ func TestTaps_EmptyPackageNodeStaysInScope(t *testing.T) {
]
}`
verifier := newVerifier(t, WithAppPackage("com.folio"))
mustLoad(t, verifier, `globalThis.actions = __sanderling__.taps;`)
loadActionSpec(t, verifier, `
import { taps } from "@sanderling/spec";
globalThis.actions = taps;
`)
pushTree(t, verifier, treeJSON)
action, err := verifier.NextAction()
internal/verifier/spec_integration_test.go
+6 -1
View File
@@ -23,12 +23,17 @@ func bundleIntegrationSpec(t *testing.T) string {
if err != nil {
t.Fatal(err)
}
runtimePath, err := filepath.Abs("../../pkg/spec/src/goja-runtime.ts")
if err != nil {
t.Fatal(err)
}
defaultsPath, err := filepath.Abs("../../pkg/spec/src/defaults/properties.ts")
if err != nil {
t.Fatal(err)
}
bundle, err := bundler.Bundle(bundler.Options{
EntryFile: specPath,
EntryFile: specPath,
RuntimeFile: runtimePath,
Aliases: map[string]string{
"@sanderling/spec": apiPath,
"@sanderling/spec/defaults/properties": defaultsPath,
internal/verifier/verifier_test.go
+318 -81
View File
@@ -3,13 +3,15 @@ package verifier
import (
"encoding/json"
"errors"
"math/rand/v2"
"os"
"path/filepath"
"slices"
"strings"
"testing"
"github.com/dop251/goja"
"github.com/priyanshujain/sanderling/internal/bundler"
"github.com/priyanshujain/sanderling/internal/hierarchy"
"github.com/priyanshujain/sanderling/internal/ltl"
)
@@ -30,6 +32,43 @@ func mustLoad(t *testing.T, verifier *Verifier, source string) {
}
}
// bundleActionSpec bundles an inline TS spec authored against @sanderling/spec
// together with the goja runtime entry, so loading it installs
// __sanderlingNextAction__ (the shared picker). Action targets must be resolved
// ax elements (carrying x/y) or builtins; raw selector strings no longer
// resolve to coordinates in the unified contract.
func bundleActionSpec(t *testing.T, specSource string) string {
t.Helper()
dir := t.TempDir()
specPath := filepath.Join(dir, "spec.ts")
if err := os.WriteFile(specPath, []byte(specSource), 0o600); err != nil {
t.Fatal(err)
}
apiPath, err := filepath.Abs("../../pkg/spec/src/index.ts")
if err != nil {
t.Fatal(err)
}
runtimePath, err := filepath.Abs("../../pkg/spec/src/goja-runtime.ts")
if err != nil {
t.Fatal(err)
}
bundle, err := bundler.Bundle(bundler.Options{
EntryFile: specPath,
RuntimeFile: runtimePath,
Aliases: map[string]string{"@sanderling/spec": apiPath},
})
if err != nil {
t.Fatal(err)
}
return string(bundle.JavaScript)
}
// loadActionSpec bundles and loads an inline authored spec into the verifier.
func loadActionSpec(t *testing.T, verifier *Verifier, specSource string) {
t.Helper()
mustLoad(t, verifier, bundleActionSpec(t, specSource))
}
const helloSpec = `
const screen = __sanderling__.extract(state => state.snapshots.screen ?? "");
const balance = __sanderling__.extract(state => state.snapshots["ledger.balance"] ?? 0);
@@ -40,10 +79,6 @@ globalThis.balance = balance;
globalThis.properties = {
balanceNonNegative: __sanderling__.always(() => balance.current >= 0),
};
globalThis.actions = __sanderling__.actions(() => [
__sanderling__.tap({ on: "id:home_button" }),
]);
`
func TestLoad_ExposesRuntimeBindings(t *testing.T) {
@@ -231,9 +266,25 @@ globalThis.properties = {
}
func TestNextAction_FromActionsGenerator(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"resource-id": "home_button", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
loadActionSpec(t, verifier, `
import { actions, Tap } from "@sanderling/spec";
globalThis.actions = actions(() => {
const home = state.ax.find("id:home_button");
return home ? [Tap({ on: home })] : [];
});
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
@@ -248,16 +299,31 @@ func TestNextAction_FromActionsGenerator(t *testing.T) {
}
func TestNextAction_WeightedSelectsByWeight(t *testing.T) {
verifier := newVerifier(t, WithRand(rand.New(rand.NewPCG(42, 0))))
mustLoad(t, verifier, `
const tapHome = __sanderling__.actions(() => [__sanderling__.tap({ on: "id:home" })]);
const tapAway = __sanderling__.actions(() => [__sanderling__.tap({ on: "id:away" })]);
globalThis.actions = __sanderling__.weighted(
[1, tapHome],
[99, tapAway],
);
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"resource-id": "home", "bounds": "[0,0,100,40]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "away", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t, WithSeed(42))
loadActionSpec(t, verifier, `
import { actions, weighted, Tap } from "@sanderling/spec";
const tapHome = actions(() => {
const home = state.ax.find("id:home");
return home ? [Tap({ on: home })] : [];
});
const tapAway = actions(() => {
const away = state.ax.find("id:away");
return away ? [Tap({ on: away })] : [];
});
globalThis.actions = weighted([1, tapHome], [99, tapAway]);
`)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
awayCount := 0
homeCount := 0
@@ -280,8 +346,9 @@ func TestNextAction_WeightedSelectsByWeight(t *testing.T) {
func TestNextAction_EmptyGeneratorReturnsErrNoAction(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.actions = __sanderling__.actions(() => []);
loadActionSpec(t, verifier, `
import { actions } from "@sanderling/spec";
globalThis.actions = actions(() => []);
`)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
@@ -291,13 +358,34 @@ func TestNextAction_EmptyGeneratorReturnsErrNoAction(t *testing.T) {
}
}
// setupTree carries the three targets the setup-precedence tests resolve.
const setupTree = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,120]"},
"children": [
{"attributes": {"resource-id": "setup", "bounds": "[0,0,100,40]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "main", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "login", "bounds": "[0,80,100,120]"}, "clickable": true, "enabled": true, "children": []}
]
}`
func TestNextAction_SetupTakesPrecedenceWhenYielding(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.setup = __sanderling__.actions(() => [__sanderling__.tap({ on: "id:setup" })]);
globalThis.actions = __sanderling__.actions(() => [__sanderling__.tap({ on: "id:main" })]);
loadActionSpec(t, verifier, `
import { actions, Tap } from "@sanderling/spec";
globalThis.setup = actions(() => {
const target = state.ax.find("id:setup");
return target ? [Tap({ on: target })] : [];
});
globalThis.actions = actions(() => {
const target = state.ax.find("id:main");
return target ? [Tap({ on: target })] : [];
});
`)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
tree, err := hierarchy.Parse(setupTree)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
@@ -310,11 +398,19 @@ func TestNextAction_SetupTakesPrecedenceWhenYielding(t *testing.T) {
func TestNextAction_FallsThroughToActionsWhenSetupEmpty(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.setup = __sanderling__.actions(() => []);
globalThis.actions = __sanderling__.actions(() => [__sanderling__.tap({ on: "id:main" })]);
loadActionSpec(t, verifier, `
import { actions, Tap } from "@sanderling/spec";
globalThis.setup = actions(() => []);
globalThis.actions = actions(() => {
const target = state.ax.find("id:main");
return target ? [Tap({ on: target })] : [];
});
`)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
tree, err := hierarchy.Parse(setupTree)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
@@ -327,21 +423,30 @@ func TestNextAction_FallsThroughToActionsWhenSetupEmpty(t *testing.T) {
func TestNextAction_SetupReengagesAfterRegression(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.loggedIn = __sanderling__.extract(state => state.snapshots["loggedIn"] === true);
globalThis.setup = __sanderling__.actions(() => {
loadActionSpec(t, verifier, `
import { actions, extract, Tap } from "@sanderling/spec";
const loggedIn = extract(state => state.snapshots["loggedIn"] === true);
globalThis.setup = actions(() => {
if (loggedIn.current) return [];
return [__sanderling__.tap({ on: "id:login" })];
const target = state.ax.find("id:login");
return target ? [Tap({ on: target })] : [];
});
globalThis.actions = actions(() => {
const target = state.ax.find("id:main");
return target ? [Tap({ on: target })] : [];
});
globalThis.actions = __sanderling__.actions(() => [__sanderling__.tap({ on: "id:main" })]);
`)
tree, err := hierarchy.Parse(setupTree)
if err != nil {
t.Fatal(err)
}
push := func(loggedIn bool) {
raw := json.RawMessage(`false`)
if loggedIn {
raw = json.RawMessage(`true`)
}
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"loggedIn": raw}}); err != nil {
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"loggedIn": raw}, Tree: tree}); err != nil {
t.Fatal(err)
}
}
@@ -376,14 +481,19 @@ func TestNextAction_SetupReengagesAfterRegression(t *testing.T) {
func TestNextAction_NoSetupRegistered(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.actions = __sanderling__.actions(() => [__sanderling__.tap({ on: "id:main" })]);
loadActionSpec(t, verifier, `
import { actions, Tap } from "@sanderling/spec";
globalThis.actions = actions(() => {
const target = state.ax.find("id:main");
return target ? [Tap({ on: target })] : [];
});
`)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
if verifier.setupGenerator != nil {
t.Errorf("setupGenerator should be nil when spec does not export setup")
tree, err := hierarchy.Parse(setupTree)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
t.Fatal(err)
@@ -403,7 +513,10 @@ func TestDoubleTapsBuiltin_TargetsClickable(t *testing.T) {
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `globalThis.actions = __sanderling__.doubleTaps;`)
loadActionSpec(t, verifier, `
import { doubleTaps } from "@sanderling/spec";
globalThis.actions = doubleTaps;
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
@@ -442,13 +555,25 @@ func TestDoubleTapsBuiltin_TargetsClickable(t *testing.T) {
}
func TestDoubleTap_RoundTrip(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"resource-id": "save", "bounds": "[0,0,100,40]"}, "clickable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.actions = __sanderling__.actions(() => [
__sanderling__.doubleTap({ on: "id:save" }),
]);
loadActionSpec(t, verifier, `
import { actions, DoubleTap } from "@sanderling/spec";
globalThis.actions = actions(() => {
const save = state.ax.find("id:save");
return save ? [DoubleTap({ on: save })] : [];
});
`)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
@@ -463,13 +588,25 @@ func TestDoubleTap_RoundTrip(t *testing.T) {
}
func TestInputText_RoundTrip(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,100]"},
"children": [
{"attributes": {"resource-id": "phone", "bounds": "[0,0,100,40]"}, "editable": true, "enabled": true, "children": []}
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `
globalThis.actions = __sanderling__.actions(() => [
__sanderling__.inputText({ into: "id:phone", text: "+919876543210" }),
]);
loadActionSpec(t, verifier, `
import { actions, InputText } from "@sanderling/spec";
globalThis.actions = actions(() => {
const phone = state.ax.find("id:phone");
return phone ? [InputText({ into: phone, text: "+919876543210" })] : [];
});
`)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
action, err := verifier.NextAction()
if err != nil {
@@ -495,7 +632,10 @@ func TestTypingBuiltin_TargetsEditableField(t *testing.T) {
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `globalThis.actions = __sanderling__.typing;`)
loadActionSpec(t, verifier, `
import { typing } from "@sanderling/spec";
globalThis.actions = typing;
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
@@ -513,11 +653,21 @@ func TestTypingBuiltin_TargetsEditableField(t *testing.T) {
if action.X != 50 || action.Y != 20 {
t.Errorf("coords = (%d,%d), want (50,20) at EmailField center", action.X, action.Y)
}
if !slices.Contains(inputCorpus, action.Text) {
t.Errorf("text %q not drawn from inputCorpus", action.Text)
// The typing builtin draws from corpus.ts INPUT_CORPUS; the single editable
// field means any draw lands here, so the text must be a corpus member.
if !slices.Contains(testInputCorpus, action.Text) {
t.Errorf("text %q not drawn from the input corpus", action.Text)
}
}
// testInputCorpus mirrors pkg/spec/src/corpus.ts INPUT_CORPUS so the typing
// builtin's emitted text can be asserted to come from the shared pool.
var testInputCorpus = []string{
"", "a", strings.Repeat("a", 4096), "🙂🔥💸", " ", "\t\n", "-1",
"999999999999999999999", "0.0000001", "1e10", "'; DROP TABLE--",
"<script>alert(1)</script>", "../../etc/passwd", "%s%n", "NaN",
}
// TestTypingBuiltin_NoEditableYieldsErrNoAction verifies the typing generator
// declines (ErrNoAction) when no editable element is present, so a weighted
// layer falls through to another generator.
@@ -529,7 +679,10 @@ func TestTypingBuiltin_NoEditableYieldsErrNoAction(t *testing.T) {
]
}`
verifier := newVerifier(t)
mustLoad(t, verifier, `globalThis.actions = __sanderling__.typing;`)
loadActionSpec(t, verifier, `
import { typing } from "@sanderling/spec";
globalThis.actions = typing;
`)
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
@@ -582,12 +735,50 @@ globalThis.properties = {
t.Errorf("verdict: got %v, want %v", got, ltl.VerdictViolated)
}
predicateErr := verifier.PredicateError("broken")
if predicateErr == nil {
t.Fatal("PredicateError: got nil, want non-nil")
witness := verifier.Witness("broken")
if witness == nil {
t.Fatal("Witness: got nil, want non-nil")
}
if !strings.Contains(predicateErr.Error(), "bad predicate") {
t.Errorf("PredicateError message: got %q, want to contain %q", predicateErr.Error(), "bad predicate")
if !witness.IsError {
t.Errorf("Witness.IsError = false, want true for a thrown predicate")
}
if !strings.Contains(witness.Reason, "bad predicate") {
t.Errorf("Witness.Reason: got %q, want to contain %q", witness.Reason, "bad predicate")
}
}
// An unbounded eventually that never fires stays pending during the run and is
// only reported by Finalize at run end. The witness records why it failed.
func TestFinalize_UnmetEventuallyReportedWithWitness(t *testing.T) {
const spec = `
globalThis.flag = __sanderling__.extract(state => state.snapshots["flag"] ?? false, "flag");
globalThis.properties = {
flagEventuallyTrue: __sanderling__.always(__sanderling__.eventually(() => flag.current === true)),
};
`
verifier := newVerifier(t)
mustLoad(t, verifier, spec)
for step := 0; step < 3; step++ {
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"flag": json.RawMessage(`false`)}}); err != nil {
t.Fatal(err)
}
verdicts := verifier.EvaluateProperties()
if got := verdicts["flagEventuallyTrue"]; got != ltl.VerdictPending {
t.Fatalf("step %d: verdict = %v, want pending", step, got)
}
}
ended := verifier.Finalize()
if !slices.Contains(ended, "flagEventuallyTrue") {
t.Fatalf("Finalize ended = %v, want to contain flagEventuallyTrue", ended)
}
witness := verifier.Witness("flagEventuallyTrue")
if witness == nil {
t.Fatal("Witness = nil after Finalize, want non-nil")
}
if !strings.Contains(witness.Reason, "eventually") {
t.Errorf("Witness.Reason = %q, want to mention eventually", witness.Reason)
}
}
@@ -702,15 +893,30 @@ func TestSelector_BooleanValue(t *testing.T) {
// dynamic array picks the same element under the same seed across runs. The
// folio spec relies on this to replace Math.random() in account-card taps.
func TestFrom_SeededReplayIsDeterministic(t *testing.T) {
const treeJSON = `{
"attributes": {"resource-id": "root", "bounds": "[0,0,100,200]"},
"children": [
{"attributes": {"resource-id": "card_a", "bounds": "[0,0,100,40]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "card_b", "bounds": "[0,40,100,80]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "card_c", "bounds": "[0,80,100,120]"}, "clickable": true, "enabled": true, "children": []},
{"attributes": {"resource-id": "card_d", "bounds": "[0,120,100,160]"}, "clickable": true, "enabled": true, "children": []}
]
}`
tree, err := hierarchy.Parse(treeJSON)
if err != nil {
t.Fatal(err)
}
pickedSequence := func(seed uint64) []string {
verifier := newVerifier(t, WithRand(rand.New(rand.NewPCG(seed, 0))))
mustLoad(t, verifier, `
globalThis.actions = __sanderling__.actions(() => {
const cards = ["card_a", "card_b", "card_c", "card_d"];
return [__sanderling__.tap({ on: __sanderling__.from(cards).generate() })];
verifier := newVerifier(t, WithSeed(seed))
loadActionSpec(t, verifier, `
import { actions, from, Tap } from "@sanderling/spec";
const cards = ["id:card_a", "id:card_b", "id:card_c", "id:card_d"];
globalThis.actions = actions(() => {
const target = state.ax.find(from(cards).generate());
return target ? [Tap({ on: target })] : [];
});
`)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree})
var picks []string
for range 20 {
action, err := verifier.NextAction()
@@ -741,13 +947,14 @@ func TestFrom_SeededReplayIsDeterministic(t *testing.T) {
}
}
// PredicateError must reflect the most recent step's predicate result, not a
// latched first-step error. The runner logs PredicateError once per step; if it
// stays pinned to step 1 forever, downstream debugging looks frozen even though
// the underlying state is changing.
func TestPredicateError_ReflectsCurrentStepNotFirstStep(t *testing.T) {
// A thrown predicate is a witnessed violation at the step it first throws, and
// the property latches violated thereafter (sticky always semantics). The
// witness captures the onset step's error text (count=1) and the extractor
// snapshot at that step; it does not keep updating to later counts because the
// verdict has latched.
func TestThrowingPredicate_WitnessCapturesOnsetStep(t *testing.T) {
const spec = `
globalThis.counter = __sanderling__.extract(state => state.snapshots["count"]);
globalThis.counter = __sanderling__.extract(state => state.snapshots["count"], "counter");
globalThis.properties = {
reportsCounter: __sanderling__.always(() => { throw new Error("count=" + counter.current); }),
};
@@ -760,16 +967,24 @@ globalThis.properties = {
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"count": raw}}); err != nil {
t.Fatal(err)
}
_ = verifier.EvaluateProperties()
verdicts := verifier.EvaluateProperties()
if got := verdicts["reportsCounter"]; got != ltl.VerdictViolated {
t.Fatalf("step %d: verdict = %v, want violated", step, got)
}
}
got := verifier.PredicateError("reportsCounter")
if got == nil {
t.Fatalf("step %d: PredicateError = nil, want non-nil", step)
}
want := "count=" + string(rune('0'+step))
if !strings.Contains(got.Error(), want) {
t.Errorf("step %d: PredicateError = %q, want to contain %q", step, got.Error(), want)
}
witness := verifier.Witness("reportsCounter")
if witness == nil {
t.Fatal("Witness = nil, want non-nil")
}
if witness.Step != 1 {
t.Errorf("Witness.Step = %d, want 1 (onset)", witness.Step)
}
if !strings.Contains(witness.Reason, "count=1") {
t.Errorf("Witness.Reason = %q, want to contain %q", witness.Reason, "count=1")
}
if got := string(witness.Extractors["counter"]); got != `"1"` {
t.Errorf("Witness.Extractors[counter] = %s, want %q", got, `"1"`)
}
}
@@ -845,8 +1060,6 @@ globalThis.card = card;
globalThis.properties = {
hasTestTag: __sanderling__.always(() => typeof card.current.attrs.testTag === "string"),
};
globalThis.actions = __sanderling__.actions(() => []);
`
// TestSelectorStringFromJS_CanonicalGrammar guarantees the selector tag stamped
@@ -1019,3 +1232,27 @@ func TestOverrideExtractorValues_PropagatesNestedObjectFields(t *testing.T) {
t.Errorf("scalar field missing: card.current.balance = %d, want 12345", got)
}
}
// TestUnsupportedVerbs_CollectedDedupedInOrder drives the real host binding the
// shared picker invokes (__sanderlingHost__.reportUnsupported) and asserts the
// verifier collects each verb once, in first-seen order, for the run report.
func TestUnsupportedVerbs_CollectedDedupedInOrder(t *testing.T) {
verifier := newVerifier(t)
report, ok := goja.AssertFunction(
verifier.runtime.GlobalObject().Get("__sanderlingHost__").
ToObject(verifier.runtime).Get("reportUnsupported"),
)
if !ok {
t.Fatal("reportUnsupported host binding missing")
}
for _, verb := range []string{"scrolls", "swipes", "scrolls", "longPresses"} {
if _, err := report(goja.Undefined(), verifier.runtime.ToValue(verb)); err != nil {
t.Fatalf("reportUnsupported(%q): %v", verb, err)
}
}
got := verifier.UnsupportedVerbs()
want := []string{"scrolls", "swipes", "longPresses"}
if !slices.Equal(got, want) {
t.Errorf("UnsupportedVerbs = %v, want %v", got, want)
}
}
internal/verifier/worker.go
+222 -429
View File
@@ -1,3 +1,4 @@
// Package verifier runs the spec's extractors and property formulas against each observed step.
package verifier
import (
@@ -6,9 +7,7 @@ import (
"errors"
"fmt"
"maps"
"math/rand/v2"
"sort"
"strings"
"time"
"github.com/dop251/goja"
@@ -23,14 +22,17 @@ type Verifier struct {
formulas []*formulaState
formulaSpecs []formulaSpec
properties map[string]int // property name -> formula-spec index
actionGenerator goja.Value
setupGenerator goja.Value
properties map[string]int // property name -> formula-spec index
// nextActionFn is the bundle-installed __sanderlingNextAction__, which runs
// the shared picker (pick.ts) over the shared Pcg.
nextActionFn goja.Callable
evaluators map[string]*ltl.Evaluator
priorVerdicts map[string]ltl.Verdict
newlyViolated []string
witnesses map[string]Witness
lastTree *hierarchy.Tree
lastAction *Action
@@ -40,14 +42,41 @@ type Verifier struct {
runStart time.Time
appPackage string
platform string
seed uint64
rng *rand.Rand
// unsupported collects verbs the picker requested but the platform cannot
// dispatch (reportUnsupported host callback), deduped and in first-seen
// order, so the runner can surface them in the run report.
unsupported []string
unsupportedSeen map[string]bool
// extracting is true only while an extractor getter is running. The handle's
// current/previous accessors consult it so a getter that reaches into
// another extractor's handle throws instead of reading a stale value.
extracting bool
}
// UnsupportedVerbs returns the verbs the picker requested that this platform
// cannot dispatch, deduped and in first-seen order.
func (v *Verifier) UnsupportedVerbs() []string {
return v.unsupported
}
type Option func(*Verifier)
func WithRand(rng *rand.Rand) Option {
return func(v *Verifier) { v.rng = rng }
// WithSeed sets the 64-bit seed the JS picker constructs its Pcg from
// (new Pcg(seed, 0), matching the web bundle's SANDERLING_SEED). The verifier
// exposes it to the bundle via the __sanderlingHost__.seedHi/seedLo binds.
func WithSeed(seed uint64) Option {
return func(v *Verifier) { v.seed = seed }
}
// WithPlatform names the platform the host reports to the picker
// ("android"/"ios"/"web"); it drives the verb-support matrix and the press-key
// pool. Empty defaults to "android".
func WithPlatform(platform string) Option {
return func(v *Verifier) { v.platform = platform }
}
// WithAppPackage scopes random-action target selection to the app under test.
@@ -60,15 +89,20 @@ func WithAppPackage(appPackage string) Option {
func New(options ...Option) (*Verifier, error) {
verifier := &Verifier{
runtime: goja.New(),
properties: map[string]int{},
evaluators: map[string]*ltl.Evaluator{},
priorVerdicts: map[string]ltl.Verdict{},
rng: rand.New(rand.NewPCG(0, 0)),
runtime: goja.New(),
properties: map[string]int{},
evaluators: map[string]*ltl.Evaluator{},
priorVerdicts: map[string]ltl.Verdict{},
witnesses: map[string]Witness{},
platform: "android",
unsupportedSeen: map[string]bool{},
}
for _, option := range options {
option(verifier)
}
if verifier.platform == "" {
verifier.platform = "android"
}
if err := verifier.installRuntimeBindings(); err != nil {
return nil, fmt.Errorf("install bindings: %w", err)
}
@@ -105,12 +139,14 @@ func (v *Verifier) Load(source string) error {
}
}
if actionsValue := v.runtime.GlobalObject().Get("actions"); actionsValue != nil && !goja.IsUndefined(actionsValue) && !goja.IsNull(actionsValue) {
v.actionGenerator = actionsValue
}
if setupValue := v.runtime.GlobalObject().Get("setup"); setupValue != nil && !goja.IsUndefined(setupValue) && !goja.IsNull(setupValue) {
v.setupGenerator = setupValue
// The bundle's goja runtime entry installs __sanderlingNextAction__ once the
// spec assigned globalThis.actions. Capture it; a spec bundled without the
// runtime entry (raw-JS unit fixtures) leaves it nil and NextAction reports
// ErrNoAction.
if fn := v.runtime.GlobalObject().Get("__sanderlingNextAction__"); fn != nil {
if callable, ok := goja.AssertFunction(fn); ok {
v.nextActionFn = callable
}
}
return nil
@@ -139,7 +175,8 @@ func (v *Verifier) buildFormulaNode(index int) (ltl.Formula, error) {
case specKindPure:
return ltl.Pure(spec.pureValue), nil
case specKindThunk:
return ltl.Thunk(v.formulaThunk(spec.predicateIndex)), nil
name := fmt.Sprintf("p%d", spec.predicateIndex)
return ltl.ThunkNamed(name, v.formulaThunk(spec.predicateIndex)), nil
case specKindNow:
child, err := v.buildFormulaNode(spec.childA)
if err != nil {
@@ -244,21 +281,28 @@ func (v *Verifier) PushSnapshot(input SnapshotInput) error {
// Extractor previous/current advance exactly once per PushSnapshot.
// Predicate thunks read these slots but never trigger advancement, so
// invoking a thunk multiple times between snapshots is value-stable.
// refreshPredicateErrors relies on this to safely re-call predicates.
for index, extractor := range v.extractors {
previous := extractor.handle.Get("current")
_ = extractor.handle.Set("previous", previous)
newValue, err := extractor.getter(goja.Undefined(), state)
extractor.previousValue = extractor.currentValue
newValue, err := v.runExtractor(extractor, state)
if err != nil {
return fmt.Errorf("extractor %d: %w", index, err)
}
_ = extractor.handle.Set("current", newValue)
extractor.currentValue = newValue
extractor.prev = extractor.curr
extractor.curr = encodeExtractorValue(newValue)
}
return nil
}
// runExtractor invokes an extractor's getter with the extracting flag set, so a
// getter that reads another extractor's current/previous throws. The flag is
// cleared even if the getter panics.
func (v *Verifier) runExtractor(extractor *extractorState, state goja.Value) (goja.Value, error) {
v.extracting = true
defer func() { v.extracting = false }()
return extractor.getter(goja.Undefined(), state)
}
// encodeExtractorValue produces a stable JSON encoding of an extractor's
// current value for diff comparison. goja values that don't survive Export
// (e.g. wrapped host functions) yield nil; callers treat nil as "unknown" and
@@ -328,7 +372,7 @@ func (v *Verifier) OverrideExtractorValues(overrides map[int]json.RawMessage) (s
if conversionErr != nil {
return skipped, fmt.Errorf("extractor override %d: %w", index, conversionErr)
}
_ = v.extractors[index].handle.Set("current", value)
v.extractors[index].currentValue = value
}
return skipped, nil
}
@@ -362,12 +406,12 @@ func (v *Verifier) EvaluateProperties() map[string]ltl.Verdict {
for name, evaluator := range v.evaluators {
verdicts[name] = evaluator.ObserveAt(stepTime)
}
v.refreshPredicateErrors()
var onset []string
for name, verdict := range verdicts {
if verdict == ltl.VerdictViolated && v.priorVerdicts[name] != ltl.VerdictViolated {
onset = append(onset, name)
v.captureWitness(name)
}
}
sort.Strings(onset)
@@ -380,6 +424,89 @@ func (v *Verifier) EvaluateProperties() map[string]ltl.Verdict {
return verdicts
}
// Witness is the verifier-level record of a property violation: the LTL reason
// (a predicate's thrown-error text, "predicate false", or a liveness failure),
// the step it fired at, and a snapshot of every extractor's current value at
// that step. The snapshot lets a reader see the state that produced the
// violation without replaying the run.
type Witness struct {
Property string
Reason string
Step int
IsError bool
Extractors map[string]json.RawMessage
}
// captureWitness records the witness for a property that just transitioned to
// violated, snapshotting the current extractor values so the cause is visible
// after the run.
func (v *Verifier) captureWitness(name string) {
evaluator, ok := v.evaluators[name]
if !ok {
return
}
violation := evaluator.Violation()
if violation == nil {
return
}
v.witnesses[name] = Witness{
Property: name,
Reason: violation.Reason,
Step: violation.Step,
IsError: violation.IsError,
Extractors: v.extractorSnapshot(),
}
}
// extractorSnapshot encodes every named extractor's current value as JSON. A
// nil value (extractor never advanced or its value did not survive Export)
// is recorded as JSON null.
func (v *Verifier) extractorSnapshot() map[string]json.RawMessage {
if len(v.extractors) == 0 {
return nil
}
snapshot := make(map[string]json.RawMessage, len(v.extractors))
for _, extractor := range v.extractors {
value := extractor.curr
if value == nil {
value = []byte("null")
}
snapshot[extractor.name] = append(json.RawMessage(nil), value...)
}
return snapshot
}
// Witness returns the captured violation witness for a property, or nil if the
// property has not violated. Callers consult this after EvaluateProperties (or
// Finalize) reports a violation to surface the cause and the state at onset.
func (v *Verifier) Witness(name string) *Witness {
witness, ok := v.witnesses[name]
if !ok {
return nil
}
return &witness
}
// Finalize drives each evaluator to its terminal verdict and returns the names
// of properties that violate only at run end (a liveness obligation that never
// discharged), capturing a witness for each. Properties already violated
// mid-run are not re-reported here.
func (v *Verifier) Finalize() []string {
var ended []string
for name, evaluator := range v.evaluators {
if v.priorVerdicts[name] == ltl.VerdictViolated {
continue
}
if evaluator.Finalize() == ltl.VerdictViolated {
ended = append(ended, name)
v.captureWitness(name)
v.priorVerdicts[name] = ltl.VerdictViolated
}
}
sort.Strings(ended)
return ended
}
// NewlyViolatedProperties returns the names of properties whose verdict
// transitioned from non-Violated to Violated on the most recent
// EvaluateProperties call, sorted lexicographically. Returns nil if no
@@ -396,14 +523,14 @@ func (v *Verifier) NewlyViolatedProperties() []string {
}
// Residuals returns the residual formula for each registered property after
// the most recent EvaluateProperties call. Properties that errored during
// predicate evaluation surface as ErrorFormula so the inspect UI can render
// "predicate threw" inline.
// the most recent EvaluateProperties call. Properties whose violation was
// caused by a thrown predicate surface as ErrorFormula, sourced from the
// captured witness, so the inspect UI can render "predicate threw" inline.
func (v *Verifier) Residuals() map[string]ltl.Formula {
residuals := map[string]ltl.Formula{}
for name, evaluator := range v.evaluators {
if predicateErr := v.PredicateError(name); predicateErr != nil {
residuals[name] = ltl.ErrorFormula{Message: predicateErr.Error()}
if witness, ok := v.witnesses[name]; ok && witness.IsError {
residuals[name] = ltl.ErrorFormula{Message: witness.Reason}
continue
}
residuals[name] = evaluator.Residual()
@@ -411,174 +538,42 @@ func (v *Verifier) Residuals() map[string]ltl.Formula {
return residuals
}
// NextAction resolves an action for the current step. The setup generator,
// when registered, runs first; if it yields an action, that wins. When setup
// returns ErrNoAction (all branches empty) the call falls through to the
// root action generator with the existing retry semantics. Setup is consulted
// every step, so state regression (e.g. a logout under fuzz) automatically
// re-engages the precondition.
// NextAction resolves an action for the current step by invoking the bundled
// __sanderlingNextAction__(), which runs the SHARED picker (pick.ts) over the
// shared Pcg. Setup-generator precedence and the 16-attempt retry both live in
// runtime-entry.ts now, so this is a thin call-and-decode. A null result (the
// generator declined to act) reports ErrNoAction.
func (v *Verifier) NextAction() (Action, error) {
if v.setupGenerator != nil {
action, err := v.resolveGenerator(v.setupGenerator)
if err == nil {
return action, nil
}
if !errors.Is(err, ErrNoAction) {
return Action{}, err
}
}
if v.actionGenerator == nil {
if v.nextActionFn == nil {
return Action{}, ErrNoAction
}
const maxRetries = 16
for range maxRetries {
action, err := v.resolveGenerator(v.actionGenerator)
if err == nil {
return action, nil
}
if !errors.Is(err, ErrNoAction) {
return Action{}, err
}
value, err := v.nextActionFn(goja.Undefined())
if err != nil {
return Action{}, fmt.Errorf("next action: %w", err)
}
return Action{}, ErrNoAction
if value == nil || goja.IsNull(value) || goja.IsUndefined(value) {
return Action{}, ErrNoAction
}
raw, err := json.Marshal(value.Export())
if err != nil {
return Action{}, fmt.Errorf("marshal action: %w", err)
}
return DecodeAction(raw)
}
var ErrNoAction = errors.New("verifier: no action available")
func (v *Verifier) formulaThunk(index int) func() bool {
return func() bool {
func (v *Verifier) formulaThunk(index int) func() (bool, error) {
return func() (bool, error) {
formula := v.formulas[index]
result, err := formula.predicate(goja.Undefined())
if err != nil {
formula.err = err
return false
return false, err
}
formula.err = nil
return result.ToBoolean()
return result.ToBoolean(), nil
}
}
// refreshPredicateErrors re-invokes every registered predicate so that
// formula.err reflects the current step rather than a latched first-step
// throw. EvaluateProperties short-circuits once a property has latched to
// violated, so without this refresh the runner's per-step "predicate error"
// log freezes on whatever the predicate threw at step 1. The refreshed errors
// have no effect on verdicts.
//
// Invariant: predicates may be re-invoked here outside the LTL gate that
// would normally skip them (e.g. an `implies` consequent whose antecedent is
// false). They must therefore be side-effect-free reads of extractor state;
// any spec that asserts internal preconditions inside a predicate could
// surface a spurious error in the inspect UI without affecting verdicts.
func (v *Verifier) refreshPredicateErrors() {
for _, formula := range v.formulas {
result, err := formula.predicate(goja.Undefined())
if err != nil {
formula.err = err
continue
}
_ = result
formula.err = nil
}
}
// PredicateError returns the first goja error raised by any thunk in the
// named property's formula tree, or nil if none fired. Callers typically
// consult this after EvaluateProperties reports a violation to distinguish
// a genuine predicate-false verdict from a malformed spec.
func (v *Verifier) PredicateError(name string) error {
rootIndex, ok := v.properties[name]
if !ok {
return nil
}
return v.firstThunkError(rootIndex)
}
func (v *Verifier) firstThunkError(index int) error {
if index < 0 || index >= len(v.formulaSpecs) {
return nil
}
spec := v.formulaSpecs[index]
switch spec.kind {
case specKindThunk:
return v.formulas[spec.predicateIndex].err
case specKindImplies, specKindOr, specKindAnd:
if err := v.firstThunkError(spec.childA); err != nil {
return err
}
return v.firstThunkError(spec.childB)
case specKindNow, specKindNext, specKindEventually, specKindNot, specKindAlways:
return v.firstThunkError(spec.childA)
}
return nil
}
func (v *Verifier) resolveGenerator(generator goja.Value) (Action, error) {
object := generator.ToObject(v.runtime)
if object == nil {
return Action{}, fmt.Errorf("generator is not an object")
}
kindValue := object.Get(tagInternalKind)
if kindValue == nil {
return Action{}, fmt.Errorf("generator missing internal kind tag")
}
switch kindValue.String() {
case internalKindActions:
generateValue := object.Get("generate")
generate, ok := goja.AssertFunction(generateValue)
if !ok {
return Action{}, fmt.Errorf("actions handle missing generate function")
}
result, err := generate(goja.Undefined())
if err != nil {
return Action{}, fmt.Errorf("generate: %w", err)
}
return v.pickFromResult(result)
case internalKindWeighted:
entries := object.Get("entries").ToObject(v.runtime)
if entries == nil {
return Action{}, fmt.Errorf("weighted handle missing entries")
}
picked, err := v.pickWeighted(entries)
if err != nil {
return Action{}, err
}
return v.resolveGenerator(picked)
case internalKindBuiltinTaps:
return v.generateRandomTap()
case internalKindBuiltinDoubleTaps:
return v.generateRandomDoubleTap()
case internalKindBuiltinTyping:
return v.generateRandomInput()
case internalKindBuiltinSwipes:
return v.generateRandomSwipe()
case internalKindBuiltinWaitOnce:
return Action{Kind: ActionKindWait, DurationMillis: 500}, nil
case internalKindBuiltinPressKey:
return v.generateRandomPressKey()
case internalKindBuiltinLongPresses:
return v.generateRandomLongPress()
case internalKindBuiltinScrolls:
return v.generateRandomScroll()
default:
return Action{}, fmt.Errorf("unknown generator kind %q", kindValue.String())
}
}
// generateRandomTap picks a visible, tappable element from the last
// hierarchy snapshot and returns a Tap action targeting its center.
func (v *Verifier) generateRandomTap() (Action, error) {
return v.generateRandomTapKind(ActionKindTap)
}
// generateRandomDoubleTap is the DoubleTap counterpart of generateRandomTap.
// Real user gestures include double-tap (image zoom, like-to-favorite,
// play/pause); a fuzzer that never emits one cannot exercise either those
// features or the sub-100ms race windows that single-step Tap cadence misses.
func (v *Verifier) generateRandomDoubleTap() (Action, error) {
return v.generateRandomTapKind(ActionKindDoubleTap)
}
// inScope reports whether an element belongs to the app under test. Nodes from
// another package (the soft keyboard, system UI, permission dialogs) are out of
// scope. An unset app package or an element with no package falls through to in
@@ -590,31 +585,6 @@ func (v *Verifier) inScope(element *hierarchy.Element) bool {
return element.Package == v.appPackage
}
func (v *Verifier) generateRandomTapKind(kind ActionKind) (Action, error) {
if v.lastTree == nil {
return Action{}, ErrNoAction
}
candidates := make([]*hierarchy.Element, 0, len(v.lastTree.Elements))
for _, element := range v.lastTree.Elements {
if !element.Clickable || !element.Enabled {
continue
}
if !v.inScope(element) {
continue
}
if element.Bounds.Right-element.Bounds.Left <= 0 || element.Bounds.Bottom-element.Bounds.Top <= 0 {
continue
}
candidates = append(candidates, element)
}
if len(candidates) == 0 {
return Action{}, ErrNoAction
}
picked := candidates[v.rng.IntN(len(candidates))]
x, y := picked.Bounds.Center()
return Action{Kind: kind, On: selectorForElement(v.lastTree, picked), X: x, Y: y}, nil
}
// selectorForElement builds a canonical "key:value" selector that resolves
// back to the given element via hierarchy.Tree.Find. Prefers resource-id (the
// testTag carrier on Android / accessibilityIdentifier on iOS), falling back
@@ -656,237 +626,60 @@ func selectorForElement(tree *hierarchy.Tree, element *hierarchy.Element) string
return ""
}
// inputCorpus is the edge-case string pool the typing builtin draws from to
// stress field parsing: empty, whitespace, overflow length, unicode, numeric
// boundaries, and common injection payloads.
var inputCorpus = []string{
"",
"a",
strings.Repeat("a", 4096),
"🙂🔥💸",
" ",
"\t\n",
"-1",
"999999999999999999999",
"0.0000001",
"1e10",
"'; DROP TABLE--",
"<script>alert(1)</script>",
"../../etc/passwd",
"%s%n",
"NaN",
}
// generateRandomInput picks a visible, editable, enabled element from the last
// hierarchy snapshot and types a random edge-case value into it. The runner
// taps the target coordinates to focus before typing, so this works on both
// native and web with no driver-side dispatch change.
func (v *Verifier) generateRandomInput() (Action, error) {
// candidatesForVerb enumerates the host-side targets a builtin verb may draw
// from, in v.lastTree.Elements ORDER (the order is part of the picker's parity
// contract). The filters are LIFTED from the old Go picker:
//
// taps/doubleTaps/longPresses: clickable + enabled + positive bounds
// typing: editable + enabled + positive bounds
// scrolls: scrollable attribute + positive bounds
// swipes: any in-scope element
//
// Every candidate carries the resolving selector so the runner can re-route by
// id/text. Out-of-scope nodes (the soft keyboard, system UI) are always dropped.
func (v *Verifier) candidatesForVerb(verb string) []candidate {
if v.lastTree == nil {
return Action{}, ErrNoAction
return nil
}
candidates := make([]*hierarchy.Element, 0, len(v.lastTree.Elements))
var result []candidate
for _, element := range v.lastTree.Elements {
if !element.Editable || !element.Enabled {
continue
}
if !v.inScope(element) {
continue
}
if element.Bounds.Right-element.Bounds.Left <= 0 || element.Bounds.Bottom-element.Bounds.Top <= 0 {
if !verbAccepts(verb, element) {
continue
}
candidates = append(candidates, element)
x, y := element.Bounds.Center()
result = append(result, candidate{
x: x,
y: y,
width: element.Bounds.Width(),
height: element.Bounds.Height(),
selector: selectorForElement(v.lastTree, element),
})
}
if len(candidates) == 0 {
return Action{}, ErrNoAction
}
picked := candidates[v.rng.IntN(len(candidates))]
x, y := picked.Bounds.Center()
value := inputCorpus[v.rng.IntN(len(inputCorpus))]
return Action{Kind: ActionKindInputText, X: x, Y: y, Text: value}, nil
return result
}
// generateRandomSwipe emits a swipe over a random enabled element or the
// whole screen, in a random direction. Returns ErrNoAction only when we have
// no tree to size a gesture off of.
func (v *Verifier) generateRandomSwipe() (Action, error) {
if v.lastTree == nil || len(v.lastTree.Elements) == 0 {
return Action{}, ErrNoAction
}
candidates := make([]*hierarchy.Element, 0, len(v.lastTree.Elements))
for _, element := range v.lastTree.Elements {
if v.inScope(element) {
candidates = append(candidates, element)
}
}
if len(candidates) == 0 {
return Action{}, ErrNoAction
}
element := candidates[v.rng.IntN(len(candidates))]
cx, cy := element.Bounds.Center()
if cx <= 0 || cy <= 0 {
return Action{}, ErrNoAction
}
// Pick a direction: 0=up 1=down 2=left 3=right; magnitude 200-600 px.
magnitude := 200 + v.rng.IntN(401)
toX, toY := cx, cy
switch v.rng.IntN(4) {
case 0:
toY = cy - magnitude
case 1:
toY = cy + magnitude
case 2:
toX = cx - magnitude
case 3:
toX = cx + magnitude
}
if toX < 0 {
toX = 0
}
if toY < 0 {
toY = 0
}
return Action{
Kind: ActionKindSwipe,
FromX: cx,
FromY: cy,
ToX: toX,
ToY: toY,
DurationMillis: 250,
}, nil
type candidate struct {
x, y int
width, height int
selector string
}
// generateRandomLongPress mirrors generateRandomTap: it picks a visible,
// clickable, enabled, in-scope element and targets its center. Real users
// long-press (context menus, reorder handles, multi-select), so a fuzzer that
// never emits one cannot reach those affordances.
func (v *Verifier) generateRandomLongPress() (Action, error) {
return v.generateRandomTapKind(ActionKindLongPress)
}
// generateRandomScroll picks a scrollable, in-scope container and emits a swipe
// across it in a random direction.
func (v *Verifier) generateRandomScroll() (Action, error) {
if v.lastTree == nil {
return Action{}, ErrNoAction
// verbAccepts applies the per-verb element filter.
func verbAccepts(verb string, element *hierarchy.Element) bool {
positiveBounds := element.Bounds.Width() > 0 && element.Bounds.Height() > 0
switch verb {
case "taps", "doubleTaps", "longPresses":
return element.Clickable && element.Enabled && positiveBounds
case "typing":
return element.Editable && element.Enabled && positiveBounds
case "scrolls":
return element.Attributes["scrollable"] == "true" && positiveBounds
case "swipes":
return true
default:
return false
}
candidates := make([]*hierarchy.Element, 0, len(v.lastTree.Elements))
for _, element := range v.lastTree.Elements {
if element.Attributes["scrollable"] != "true" {
continue
}
if !v.inScope(element) {
continue
}
if element.Bounds.Width() <= 0 || element.Bounds.Height() <= 0 {
continue
}
candidates = append(candidates, element)
}
if len(candidates) == 0 {
return Action{}, ErrNoAction
}
picked := candidates[v.rng.IntN(len(candidates))]
cx, cy := picked.Bounds.Center()
width := picked.Bounds.Width()
height := picked.Bounds.Height()
directions := []string{"up", "down", "left", "right"}
dir := directions[v.rng.IntN(len(directions))]
toX, toY := cx, cy
// Scroll direction names the content motion; the gesture swipes the
// opposite way. Revealing content below ("down") means dragging the
// finger up, so toY decreases, and likewise for the other directions.
switch dir {
case "down":
toY = cy - (4*height)/10
case "up":
toY = cy + (4*height)/10
case "left":
toX = cx + (4*width)/10
case "right":
toX = cx - (4*width)/10
}
if toX < 0 {
toX = 0
}
if toY < 0 {
toY = 0
}
return Action{
Kind: ActionKindScroll,
Direction: dir,
FromX: cx,
FromY: cy,
ToX: toX,
ToY: toY,
DurationMillis: 300,
}, nil
}
func (v *Verifier) generateRandomPressKey() (Action, error) {
// Keep exploration gentle: only "back" for now. Home/menu would navigate
// away from the app under test.
return Action{Kind: ActionKindPressKey, Key: "back"}, nil
}
func (v *Verifier) pickFromResult(result goja.Value) (Action, error) {
if result == nil || goja.IsUndefined(result) || goja.IsNull(result) {
return Action{}, ErrNoAction
}
object := result.ToObject(v.runtime)
if object == nil {
return Action{}, ErrNoAction
}
lengthValue := object.Get("length")
if lengthValue == nil {
return jsValueToAction(v.runtime, result)
}
length := int(lengthValue.ToInteger())
if length == 0 {
return Action{}, ErrNoAction
}
pick := v.rng.IntN(length)
return jsValueToAction(v.runtime, object.Get(fmt.Sprintf("%d", pick)))
}
func (v *Verifier) pickWeighted(entries *goja.Object) (goja.Value, error) {
lengthValue := entries.Get("length")
if lengthValue == nil {
return nil, fmt.Errorf("weighted entries missing length")
}
length := int(lengthValue.ToInteger())
if length == 0 {
return nil, ErrNoAction
}
weights := make([]float64, length)
generators := make([]goja.Value, length)
totalWeight := 0.0
for index := range length {
entry := entries.Get(fmt.Sprintf("%d", index)).ToObject(v.runtime)
if entry == nil {
return nil, fmt.Errorf("weighted entry %d not an array", index)
}
weight := entry.Get("0").ToFloat()
generator := entry.Get("1")
if weight < 0 {
weight = 0
}
weights[index] = weight
generators[index] = generator
totalWeight += weight
}
if totalWeight == 0 {
return nil, ErrNoAction
}
pick := v.rng.Float64() * totalWeight
cumulative := 0.0
for index := range length {
cumulative += weights[index]
if pick < cumulative {
return generators[index], nil
}
}
return generators[length-1], nil
}