diff --git a/README.md b/README.md index 443bcf1..7431583 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,15 @@ # sanderling -Autonomous property-based testing for mobile apps. Specs in TypeScript. Core in Go. Drives the app under test through Maestro. +Autonomous property-based testing for mobile and web apps. Specs in TypeScript. Core in Go. -> Alpha. Android emulator only. Full scope in the [v0.1.0 roadmap](https://github.com/priyanshujain/sanderling/issues/4). +> Alpha. Android, iOS, and web (Chrome). Full scope in the [v0.1.0 roadmap](https://github.com/priyanshujain/sanderling/issues/4). ## Docs - [Getting started](https://priyanshujain.github.io/sanderling/manual/getting-started.html) - [Writing specs](https://priyanshujain.github.io/sanderling/manual/writing-specs.html) - [`sanderling inspect` UI](https://priyanshujain.github.io/sanderling/manual/inspect.html) -- [Example](https://github.com/priyanshujain/sanderling/tree/master/examples/folio) +- Examples: [folio](https://github.com/priyanshujain/sanderling/tree/master/examples/folio) (KMP, Android/iOS/web), [folio-web](https://github.com/priyanshujain/sanderling/tree/master/examples/folio-web) (React + Vite) - [Architecture](https://priyanshujain.github.io/sanderling/development/architecture.html) After a `sanderling test` run, browse traces locally with `sanderling inspect`. It opens a web UI for stepping through actions, screenshots, snapshots, residual formulas, and exceptions. diff --git a/docs/development/architecture.md b/docs/development/architecture.md index ad335a8..ec2b5e3 100644 --- a/docs/development/architecture.md +++ b/docs/development/architecture.md @@ -14,7 +14,7 @@ flowchart TB R --> T["Trace writer\nJSONL + PNG"] end - SC["Maestro sidecar (JVM)"] + SC["Native sidecar (JVM)"] DC["Device / Emulator"] CH["Chrome (CDP)"] RD[("runs/")] @@ -32,7 +32,7 @@ flowchart TB **sanderling (Go).** The top-level binary. Bundles the spec with esbuild, evaluates it in goja, runs the main loop, dispatches actions through the `DeviceDriver` interface, writes the trace. -**Maestro sidecar (JVM).** A Kotlin process that wraps `maestro-client` and exposes a gRPC surface matching the `DeviceDriver` interface. Handles UI input, screenshots, the system accessibility tree, and OS-level alerts. Native platforms only. +**Native sidecar (JVM).** A Kotlin process that exposes a gRPC surface matching the `DeviceDriver` interface. Handles UI input, screenshots, the system accessibility tree, and OS-level alerts. Native platforms only. **Chrome (CDP).** For web targets, the Go binary drives Chrome directly over the Chrome DevTools Protocol. No sidecar is involved. @@ -40,7 +40,7 @@ flowchart TB | Channel | Platform | Transport | Purpose | |---|---|---|---| -| Go to Maestro sidecar | Native | gRPC (localhost TCP) | UI input, screenshots, system alerts | +| Go to native sidecar | Native | gRPC (localhost TCP) | UI input, screenshots, system alerts | | Go to Chrome | Web | Chrome DevTools Protocol | UI input, screenshots, DOM hierarchy, console logs | On native, the transport split exists because only real UI events need to cross process and OS-API boundaries. Introspection is cheap, frequent, and lives on a fast local socket directly to the app. On web, CDP handles both. @@ -63,7 +63,7 @@ fetch state ─► evaluate properties ─► pick action ─► dispatch 2. The runner fetches the UI hierarchy and logs from the sidecar. 3. The runner feeds state into goja. Extractors re-read; properties re-evaluate; the action generator returns a weighted tree. 4. The runner writes the trace entry for this step. -5. The runner picks an action by weight and dispatches it through the driver (gRPC to sidecar -> Maestro -> UIAutomator or XCTest). +5. The runner picks an action by weight and dispatches it through the driver (gRPC to sidecar -> UIAutomator or XCTest). 6. Loop. **Web (Chrome):** diff --git a/docs/development/decisions.md b/docs/development/decisions.md index 2a83171..454b9de 100644 --- a/docs/development/decisions.md +++ b/docs/development/decisions.md @@ -20,7 +20,7 @@ Go's `internal/` directory restriction prevents any code outside this module fro ### `internal/driver/` is an interface + subdirectory implementations -The `driver.go` file defines the `DeviceDriver` interface. Concrete implementations live in subdirectories: `sidecar/` (Maestro gRPC), `chrome/` (CDP), `mock/` (tests). This pattern keeps the runner and verifier decoupled from any specific platform. +The `driver.go` file defines the `DeviceDriver` interface. Concrete implementations live in subdirectories: `sidecar/` (gRPC to the native sidecar), `chrome/` (CDP), `mock/` (tests). This pattern keeps the runner and verifier decoupled from any specific platform. ### `internal/verifier/marshal.go` moves to `internal/inspect/` diff --git a/docs/development/design-principles.md b/docs/development/design-principles.md index f401f0b..9ca4209 100644 --- a/docs/development/design-principles.md +++ b/docs/development/design-principles.md @@ -6,14 +6,14 @@ title: Design principles ## 1. The app owns introspection; the driver owns input -On native platforms, the in-app SDK knows the state: view hierarchy, coverage, logs, exceptions, custom extractors. Maestro causes the state to change through taps, swipes, typed text, and deep links. The Go runner decides what to do. +On native platforms, the in-app SDK knows the state: view hierarchy, coverage, logs, exceptions, custom extractors. The driver causes the state to change through taps, swipes, typed text, and deep links. The Go runner decides what to do. -Splitting these responsibilities is what makes the system work across iOS and Android with one spec surface. Neither Maestro nor the SDK alone is sufficient. +Splitting these responsibilities is what makes the system work across iOS and Android with one spec surface. Neither the driver nor the SDK alone is sufficient. -- Maestro can read a coarse accessibility tree, but not the real `UIView` or `View` hierarchy, not coverage, not in-process logs. -- The SDK can see everything inside the app, but cannot dispatch UI events the way the OS would. Touch injection through Maestro goes through XCTest or UIAutomator, which the OS treats as real input. +- The driver can read a coarse accessibility tree, but not the real `UIView` or `View` hierarchy, not coverage, not in-process logs. +- The SDK can see everything inside the app, but cannot dispatch UI events the way the OS would. Touch injection goes through the OS UI-test pipeline (XCTest on iOS, UIAutomator on Android), which the OS treats as real input. -On web, Chrome DevTools Protocol handles both input and introspection. No in-app SDK is needed. +On web, the driver speaks the Chrome DevTools Protocol and handles both input and introspection. No in-app SDK is needed. ## 2. One TypeScript surface across platforms @@ -23,13 +23,13 @@ Corollary: if a concept only exists on one platform, it does not belong in the s ## 3. The driver is an interface -`DeviceDriver` has two production implementations: `sidecar` (Maestro gRPC, for native) and `chrome` (CDP, for web), plus a `mock` for tests. The runner never knows which is wired in. Adding a new platform means adding a new implementation; nothing else changes. +`DeviceDriver` has two production implementations: `sidecar` (gRPC to the native sidecar) and `chrome` (CDP, for web), plus a `mock` for tests. The runner never knows which is wired in. Adding a new platform means adding a new implementation; nothing else changes. -## 4. Hot loops bypass Maestro (native) +## 4. Hot loops bypass the sidecar (native) -On native, per-step introspection (hierarchy dump, coverage read, pause and resume) goes over a local Unix socket directly to the SDK. Only physical UI events go through Maestro's gRPC. +On native, per-step introspection (hierarchy dump, coverage read, pause and resume) goes over a local Unix socket directly to the SDK. Only physical UI events go through the sidecar's gRPC surface. -A 30-minute run is about 10,000 steps. Every step has at least one hierarchy dump and one coverage read. If those went through Maestro, the JVM sidecar would be the bottleneck. Instead the hot path is a 2 ms round-trip to an in-process Swift or Kotlin SDK. +A 30-minute run is about 10,000 steps. Every step has at least one hierarchy dump and one coverage read. If those went through the JVM sidecar, the JVM hop would be the bottleneck. Instead the hot path is a 2 ms round-trip to an in-process Swift or Kotlin SDK. On web, CDP is fast enough that a separate introspection channel is not needed. diff --git a/docs/manual/cli.md b/docs/manual/cli.md index 9d375f6..e44c55b 100644 --- a/docs/manual/cli.md +++ b/docs/manual/cli.md @@ -19,6 +19,7 @@ Run a spec against an app for a fixed duration. | `--launcher-activity` | resolved | Optional `/` to launch. Overrides default resolution. | | `--platform` | `android` | Target platform: `android`, `ios`, or `web`. | | `--avd` | optional (android) | Android AVD name to boot if no device is connected. Required only when no device is connected and multiple AVDs exist. | +| `--ios-device` | optional (ios) | iOS simulator name or UDID to boot if none is running. | | `--duration` | `5m` | Total test duration (`30s`, `5m`, `2h`, `1d`). | | `--seed` | `0` | PRNG seed. `0` uses a random seed and records it in `meta.json`. | | `--output` | `./runs` | Output directory for traces. | @@ -38,7 +39,19 @@ See [the inspect UI page](./inspect/) for the panel reference and keyboard short ## `sanderling doctor` -Check the host environment for a working sanderling setup: Go toolchain, JDK, Maestro availability, emulator reachability, SDK linkage hints. +Check the host environment for a working sanderling setup. + +``` +sanderling doctor [--platform web|android|ios|all] +``` + +`--platform` defaults to `all`, which runs every platform's checks (deduped). Pass a specific platform to scope the output. + +| Platform | Checks | +|---|---| +| `web` | headless Chromium can launch (the bundled CDP surface boots a real browser). | +| `android` | `adb` on PATH; `emulator` on PATH or under `ANDROID_HOME`; Java 17+; embedded native sidecar JAR is real. | +| `ios` | `xcrun` on PATH; `simctl` on PATH; Java 17+; embedded native sidecar JAR is real. | ## `sanderling version` diff --git a/docs/manual/getting-started.md b/docs/manual/getting-started.md index 9762120..d872fc2 100644 --- a/docs/manual/getting-started.md +++ b/docs/manual/getting-started.md @@ -35,9 +35,11 @@ npm install --save-dev @sanderling/spec ## Your first run +The repo ships two sample apps. `examples/folio` is a Kotlin Multiplatform personal-ledger app that covers Android, iOS, and web (wasmJs) from one shared codebase. `examples/folio-web` is a smaller React + Vite app that covers only the web path. Both carry a TypeScript spec under `sanderling/spec.ts`. Install `just`, then pick a target below. + ### Android -The repo ships a working sample at `examples/folio`, a Kotlin Multiplatform app with a TypeScript spec under `sanderling/spec.ts`. Install `just`, then from `examples/folio`: +From `examples/folio`: ```sh just install # build and install the folio APK on a booted emulator or device @@ -52,15 +54,32 @@ AVD=Pixel_7 just test Persistent settings can live in a `.env` alongside the justfile (`AVD=Pixel_7`, `DURATION=5m`, and so on). -### Web +### iOS -The repo also ships a web sample at `examples/folio-web`, a React/Vite app with the same domain logic. From `examples/folio-web`: +From `examples/folio` (requires Xcode 16+ and `xcodegen`): ```sh -just test # starts Chrome, runs the spec +just test-ios # default simulator: iPhone 17 Pro +IOS_DEVICE="iPhone 15" just test-ios # pick a different simulator ``` -No emulator or SDK setup needed. +`just test-ios` boots the simulator if needed, builds and installs the app, then runs `sanderling test --platform ios`. + +### Web + +From either example. For the KMP wasmJs build, use `examples/folio`: + +```sh +just web # serve the wasmJs app on a webpack dev server +``` + +For the React + Vite build, use `examples/folio-web`: + +```sh +just test # starts the Vite dev server, then sanderling drives Chrome via CDP +``` + +No emulator or SDK setup needed for either web path. ### Trace output diff --git a/docs/manual/inspect.md b/docs/manual/inspect.md index a41152f..4767126 100644 --- a/docs/manual/inspect.md +++ b/docs/manual/inspect.md @@ -14,6 +14,19 @@ The positional argument can be a runs directory or a single run directory (auto- ![sanderling inspect](../../_assets/inspect-ui.png) +## Panels + +| Panel | What it shows | +|---|---| +| Screenshot | Device screenshot for the focused step, with the dispatched action's target spotlit. | +| ActionList | Ordered list of steps with the verb and target for each action. | +| Timeline | Per-property lane chart over the run; cells coloured by violated, pending, or holds. | +| ViolationsPanel | Property statuses at the focused step with residual formulas for any that failed. | +| HierarchyPanel | Filterable table of every UI element captured at the focused step; mirrors the selectors documented in [Spec language reference](./spec-language/). | +| SnapshotTable | Flattened `snapshots` map for the focused step, with diffs against the previous step. | +| MetricsChart | Heap and other host-side metrics sampled per step. | +| ExceptionsPanel | Uncaught exceptions captured during the run, with a jump-to-first control. | + ## Keyboard shortcuts | Key | Action | diff --git a/docs/manual/runs.md b/docs/manual/runs.md index 9261fb9..384f30c 100644 --- a/docs/manual/runs.md +++ b/docs/manual/runs.md @@ -13,7 +13,7 @@ A run is not analogous to a unit test. A closer framing is: boot a fuzzer for an ``` sanderling test --spec spec.ts --bundle-id com.example.app --duration 30m │ - ├── uninstall and reinstall the app (clean slate, every run) + ├── launch the app under test (pass --clear-data to wipe app data first) ├── boot the sidecar, connect the agent socket ├── bundle the spec, load it into goja │ @@ -26,6 +26,10 @@ sanderling test --spec spec.ts --bundle-id com.example.app --duration 30m └── meta.json ``` +## App state across runs + +By default the installed app is left in place between runs. Whatever state the previous run left behind (account, cached responses, onboarding completion) carries over. Pass `--clear-data` to wipe app data before launch and start cold every run. See [CLI reference](./cli/#sanderling-test) for the flag. + ## Why runs are long and linear sanderling does not restart the app every N steps. Each restart throws away two things. diff --git a/docs/manual/spec-language.md b/docs/manual/spec-language.md index 143fdf2..fd0b202 100644 --- a/docs/manual/spec-language.md +++ b/docs/manual/spec-language.md @@ -125,7 +125,7 @@ Fields available on every element returned by `find` / `findAll`: ### iOS - `id` maps to the `accessibilityIdentifier` set via `.accessibilityIdentifier` in SwiftUI/UIKit. -- `desc` maps to `accessibilityText`, which Maestro builds by merging `accessibilityLabel` and the element's value (e.g., `"Close, icon description"`). The `desc:` selector handles this by also matching when the description starts with `, `. +- `desc` maps to `accessibilityText`, which the iOS sidecar builds by merging `accessibilityLabel` and the element's value (e.g., `"Close, icon description"`). The `desc:` selector handles this by also matching when the description starts with `, `. - `class` is the XCUITest element type (e.g., `XCUIElementTypeButton`). - `attrs` contains raw XCUITest attributes: `title`, `placeholderValue`, `hasFocus`, etc. diff --git a/docs/manual/writing-specs.md b/docs/manual/writing-specs.md index 4b46753..9603a8a 100644 --- a/docs/manual/writing-specs.md +++ b/docs/manual/writing-specs.md @@ -183,10 +183,12 @@ import { noUncaughtExceptions, noLogcatErrors } from "@sanderling/spec/defaults/ export const properties = { noUncaughtExceptions, // fails if the app throws an uncaught exception - noLogcatErrors, // fails if logcat emits any error-level lines + noLogcatErrors, // android-only; reads logcat, no-ops on ios/web }; ``` +`noLogcatErrors` reads from logcat and only applies on Android. Including it in a spec that targets iOS or web is harmless; it silently holds. + ## Pattern: preconditions sanderling has no setup phase. Preconditions are action generators with high weight that self-disable once the condition is satisfied. @@ -221,6 +223,25 @@ export const actionsRoot = weighted( Once `onLoginScreen.current` is false, `doLogin` returns `[]` and drops out of the eligible set automatically. +## Pattern: setup export + +Preconditions that drive the app from a fresh state into the surface you actually want to fuzz (login, onboarding, permission grants, seed data) can be exported as `setup` instead of mixing into `actionsRoot`. The runner tries `setup` first; if it yields no action, it falls through to `actionsRoot`. State regressing back across the precondition (logout under fuzz) automatically re-engages setup. + +```ts +const login = actions(() => { + if (loggedIn.current) return []; + const email = loginEmailField.current; + const submit = loginSubmit.current; + if (!email || !submit) return []; + return [InputText({ into: email, text: "demo@app.test" }), Tap({ on: submit })]; +}); + +export const setup = login; +export const actionsRoot = weighted([60, browse], [40, edit]); +``` + +`setup` is just an `ActionGenerator`; compose with `actions`, `weighted`, or `whenRoute` exactly like the main pool. Works identically across Android, iOS, and web. + ## Pattern: conditional properties Gate a property so it only applies when a precondition holds: diff --git a/examples/folio-web/README.md b/examples/folio-web/README.md new file mode 100644 index 0000000..f8cae4f --- /dev/null +++ b/examples/folio-web/README.md @@ -0,0 +1,59 @@ +# Folio (web) + +A React + Vite port of the folio personal-ledger app: login with demo +credentials, create accounts, add credits and debits. Exists as the +minimal web example sanderling runs its property-based specs against. + +## Stack + +- React 19 + Vite + TypeScript +- Stable HTML `id` attributes on every interactive element so the spec + targets DOM nodes by id, not by aria-label parsing +- `data-*` attributes (`data-cents`, `data-account-id`, `data-txn-count`) + carry structured numeric state for property assertions + +## Prerequisites + +- `just` +- [`bun`](https://bun.sh) (installs the JS toolchain; the project uses + `bun.lock`, not `package-lock.json` or `pnpm-lock.yaml`) +- Chrome installed (sanderling drives it via CDP) + +```sh +bun install +``` + +## Demo credentials + +``` +email: demo@ledger.app +password: ledger123 +``` + +## Run a sanderling test + +```sh +just test +``` + +The justfile boots the Vite dev server on port 5173, waits for the port +to respond, then runs `sanderling test --platform web --bundle-id +http://localhost:5173` against it. Override the port with `PORT=4000 just +test`; `DURATION`, `SEED`, and `OUTPUT` work the same way as in the KMP +folio example. + +Traces land in `./sanderling/runs//`. + +## How it connects to sanderling + +- The spec at `sanderling/spec.ts` finds elements by HTML `id` + (`#email`, `#add-account`, `#ledger`, `#txn-amount`, ...). Anything + the spec touches has a stable id in the markup. +- Numeric state the spec asserts on is exposed via `data-cents` and + similar attributes. The spec reads `el.attrs["data-cents"]` rather + than parsing currency strings out of `aria-label`. +- Account cards expose `data-account-id` + `data-balance` so the + spec can correlate cards across steps without depending on DOM order. +- `just test` invokes `sanderling test --platform web`, which connects + to Chrome via the Chrome DevTools Protocol; no extension or app SDK + is involved. diff --git a/examples/folio/README.md b/examples/folio/README.md index c82204b..503c8a0 100644 --- a/examples/folio/README.md +++ b/examples/folio/README.md @@ -2,8 +2,8 @@ A minimal Kotlin Multiplatform personal-ledger app: login with demo credentials, create accounts, add credits and debits. Shared UI across -Android and iOS via Compose Multiplatform. Doubles as the example -sanderling runs its property-based specs against. +Android, iOS, and web (wasmJs via Compose for Web). Doubles as the +example sanderling runs its property-based specs against. ## Stack @@ -39,6 +39,16 @@ IOS_DEVICE="iPhone 15" just ios # pick a different simulator builds the KMP framework (`Shared.framework` from `:app:shared`), links it into the SwiftUI host, installs, and launches. +## Web + +```sh +just web # webpack dev server with COOP/COEP headers +just web-build # produce a webpack distributable bundle +``` + +`just web` runs `:app:webApp:wasmJsBrowserDevelopmentRun --continuous`, so +edits to shared code reload in the browser. + ## Demo credentials ``` @@ -68,6 +78,17 @@ DURATION=5m Traces land in `./sanderling/runs//`. +## Run a sanderling test (iOS) + +```sh +just test-ios # default simulator: iPhone 17 Pro +IOS_DEVICE="iPhone 15" just test-ios # pick a different simulator +``` + +`just test-ios` boots the simulator if needed, runs `just ios` to install +and launch the app, then invokes `sanderling test --platform ios`. Same +`DURATION`, `SEED`, and `OUTPUT` env vars as the Android target. + ## How it connects to sanderling - Each screen sets a stable Compose `testTag` (`HomeScreen`, `AccountCard`,