mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
docs: pandoc-based site and v0.1.0 groundwork (#5)
* chore(prose): remove em-dashes from config files * chore(prose): remove em-dashes from android sdk config * docs(spec-api): remove em-dash from README * fix(doctor): reword sidecar-jar error without em-dash * test(sidecar): reword assertion message without em-dash * docs: add CLAUDE.md with project conventions * build: add docs target for pandoc site * docs(site): add pandoc template and stylesheet * docs(site): add pandoc build script * docs(site): add landing pages * docs(manual): add getting-started * docs(manual): add writing-specs * docs(manual): add runs * docs(manual): add cli reference * docs(dev): add design principles * docs(dev): add architecture * ci: deploy docs site to github pages * docs: rewrite README as entry point to docs site
This commit is contained in:
25 files changed
+920
-56
No files matched your search
@@ -0,0 +1,41 @@
|
||||
---
|
||||
title: CLI reference
|
||||
---
|
||||
|
||||
# CLI reference
|
||||
|
||||
```
|
||||
uatu <command> [flags]
|
||||
```
|
||||
|
||||
## `uatu test`
|
||||
|
||||
Run a spec against an app for a fixed duration.
|
||||
|
||||
| Flag | Default | Description |
|
||||
|---|---|---|
|
||||
| `--spec` | required | Path to the TypeScript spec. |
|
||||
| `--bundle-id` | required | Target app bundle ID (Android: applicationId). |
|
||||
| `--launcher-activity` | resolved | Optional `<pkg>/<activity>` to launch. Overrides default resolution. |
|
||||
| `--platform` | `android` | Target platform. Only `android` in the current alpha. |
|
||||
| `--avd` | required (android) | Android AVD name. |
|
||||
| `--duration` | `5m` | Total test duration (`30s`, `5m`, `2h`, `1d`). |
|
||||
| `--seed` | `0` | PRNG seed. `0` uses a random seed and records it in `meta.json`. |
|
||||
| `--output` | `./runs` | Output directory for traces. |
|
||||
|
||||
## `uatu doctor`
|
||||
|
||||
Check the host environment for a working uatu setup: Go toolchain, JDK, Maestro availability, emulator reachability, SDK linkage hints.
|
||||
|
||||
## `uatu version`
|
||||
|
||||
Print the CLI version.
|
||||
|
||||
## Flags coming in v0.1.0
|
||||
|
||||
- `--permissions` to pre-set OS-level permissions (for example `--permissions location=allow,notifications=deny`).
|
||||
- `--max-steps` hard cap on step count.
|
||||
- `--exit-on-violation` stop the run on the first property violation.
|
||||
- `uatu inspect` command for browsing traces in the built-in UI.
|
||||
|
||||
Tracked in [issue #4](https://github.com/priyanshujain/uatu/issues/4).
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: Getting started
|
||||
---
|
||||
|
||||
# Getting started
|
||||
|
||||
Install the CLI, link the SDK into your debug build, run a spec.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- An Android emulator with API level 30 or newer.
|
||||
- The app under test built as a debug variant with the uatu Android SDK linked in.
|
||||
- `adb` on your PATH.
|
||||
|
||||
Run `uatu doctor` to check the host environment.
|
||||
|
||||
## Install
|
||||
|
||||
### CLI
|
||||
|
||||
macOS arm64:
|
||||
|
||||
```sh
|
||||
curl -L https://github.com/priyanshujain/uatu/releases/latest/download/uatu_<version>_darwin_arm64.tar.gz | tar xz
|
||||
./uatu version
|
||||
```
|
||||
|
||||
Linux amd64:
|
||||
|
||||
```sh
|
||||
curl -L https://github.com/priyanshujain/uatu/releases/latest/download/uatu_<version>_linux_amd64.tar.gz | tar xz
|
||||
./uatu version
|
||||
```
|
||||
|
||||
Pre-built for `darwin/arm64`, `darwin/amd64`, `linux/amd64`, `linux/arm64`.
|
||||
|
||||
### Spec package (npm)
|
||||
|
||||
```sh
|
||||
npm install --save-dev @uatu/spec
|
||||
```
|
||||
|
||||
### Android SDK (Maven Central)
|
||||
|
||||
```kotlin
|
||||
dependencies {
|
||||
implementation("io.github.priyanshujain:sdk-android:<version>")
|
||||
}
|
||||
```
|
||||
|
||||
## Your first run
|
||||
|
||||
The repo ships a working sample at `examples/sample-app`. From that directory:
|
||||
|
||||
```sh
|
||||
./gradlew :sample-app:installDebug
|
||||
uatu test \
|
||||
--spec spec.ts \
|
||||
--bundle-id dev.uatu.sample \
|
||||
--platform android \
|
||||
--avd Pixel_7 \
|
||||
--duration 2m
|
||||
```
|
||||
|
||||
When the run ends, the trace lands in `runs/<timestamp>/`:
|
||||
|
||||
```
|
||||
runs/2026-04-18T12-34-56/
|
||||
├── trace.jsonl
|
||||
├── screenshots/
|
||||
└── meta.json
|
||||
```
|
||||
|
||||
Open the screenshots directory to scrub visually, or read `trace.jsonl` step by step.
|
||||
|
||||
Next: [writing specs](./writing-specs.html).
|
||||
@@ -0,0 +1,10 @@
|
||||
---
|
||||
title: Manual
|
||||
---
|
||||
|
||||
# Manual
|
||||
|
||||
- [Getting started](./getting-started.html)
|
||||
- [Writing specs](./writing-specs.html)
|
||||
- [Runs](./runs.html)
|
||||
- [CLI reference](./cli.html)
|
||||
@@ -0,0 +1,65 @@
|
||||
---
|
||||
title: Runs
|
||||
---
|
||||
|
||||
# What is a run?
|
||||
|
||||
One `uatu test` invocation. Fresh install, spec-driven exploration, then the trace lands in `runs/<timestamp>/`. Typically minutes to hours, not seconds.
|
||||
|
||||
A run is not analogous to a unit test. A closer framing is: boot a fuzzer for an hour and see what breaks. Violations are recorded in the trace and exploration continues, so one run can surface many bugs.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
```
|
||||
uatu test --spec spec.ts --bundle-id com.example.app --avd Pixel_7 --duration 30m
|
||||
│
|
||||
├── uninstall and reinstall the app (clean slate, every run)
|
||||
├── boot the sidecar, connect the agent socket
|
||||
├── bundle the spec, load it into goja
|
||||
│
|
||||
├── step 0..N: pause, capture state, evaluate properties, pick action, resume, dispatch
|
||||
│
|
||||
└── terminate when --duration elapses (or SIGINT)
|
||||
└── trace written to ./runs/<timestamp>/
|
||||
├── trace.jsonl
|
||||
├── screenshots/
|
||||
└── meta.json
|
||||
```
|
||||
|
||||
## Why runs are long and linear
|
||||
|
||||
uatu does not restart the app every N steps. Each restart throws away two things.
|
||||
|
||||
**Novelty and coverage signal.** The exploration strategy weights actions by whether they reach previously unseen state. Restarting resets that history.
|
||||
|
||||
**Deep app states.** Many screens take many actions to reach: nested settings, a loaded cart, post-checkout flows. A 50-step prefix to reach "cart with 3 items" does not happen if every run starts cold.
|
||||
|
||||
Long-linear trajectories find bugs that restart-based testing structurally cannot.
|
||||
|
||||
## Setup cost amortizes
|
||||
|
||||
Preconditions (login, onboarding, consent dialogs) are written as weighted action generators gated on extractors. See [writing specs](./writing-specs.html#pattern-preconditions-login-onboarding). They fire only when applicable, so login happens once per run, not per step.
|
||||
|
||||
| Run length | Login cost | % of run |
|
||||
|---|---|---|
|
||||
| 5 min | ~15s | 5% |
|
||||
| 30 min | ~15s | 0.8% |
|
||||
| 1 hour | ~15s | 0.4% |
|
||||
| CI: 3 seeds × 10 min | ~45s total | 2.5% |
|
||||
|
||||
At any non-trivial run length, preconditions are a rounding error.
|
||||
|
||||
## Session state
|
||||
|
||||
Session tokens, keychain, shared prefs, cookies, and other app-managed persistence survive the full run. If the app logs the user out mid-run, the `doLogin` generator re-fires automatically because its gating extractor (`onLoginScreen`) becomes true again. No retry logic. No special-casing.
|
||||
|
||||
## Termination
|
||||
|
||||
A run ends when either of these happens.
|
||||
|
||||
- `--duration` elapses.
|
||||
- The process is interrupted (SIGINT).
|
||||
|
||||
The trace is written incrementally, so an interrupted run is still fully inspectable.
|
||||
|
||||
Additional termination conditions (`--max-steps`, `--exit-on-violation`, hard crash handling) land in [v0.1.0](https://github.com/priyanshujain/uatu/issues/4).
|
||||
@@ -0,0 +1,193 @@
|
||||
---
|
||||
title: Writing specs
|
||||
---
|
||||
|
||||
# Writing specs
|
||||
|
||||
A spec has three parts: extractors, properties, and actions.
|
||||
|
||||
```ts
|
||||
import { extract, always, actions, weighted, Tap, taps, swipes } from "@uatu/spec";
|
||||
|
||||
// 1. Extractors pull values from each observed state.
|
||||
const loggedIn = extract((s) => !!s.ax.find("id:home-tab-bar"));
|
||||
const cartCount = extract<number>((s) => (s.snapshots.cart_count as number) ?? 0);
|
||||
|
||||
// 2. Properties are LTL formulas evaluated every step.
|
||||
export const properties = {
|
||||
cartNeverNegative: always(() => cartCount.current >= 0),
|
||||
};
|
||||
|
||||
// 3. Actions are a weighted tree of what uatu is allowed to do.
|
||||
export const actions = weighted(
|
||||
[10, taps],
|
||||
[2, swipes],
|
||||
);
|
||||
```
|
||||
|
||||
The Go runner calls into the JS runtime each step. Extractors re-read the current state. Properties re-evaluate with their residual formulas. The action generator returns a tree, and one leaf is sampled by weight and dispatched.
|
||||
|
||||
## The `State` object
|
||||
|
||||
What extractors see:
|
||||
|
||||
```ts
|
||||
interface State {
|
||||
ax: AccessibilityTree; // view hierarchy
|
||||
snapshots: Record<string, unknown>; // values registered by the in-app SDK
|
||||
screen: { id: string; hash: string };
|
||||
lastAction: Action | null;
|
||||
logs: LogEntry[]; // since previous state
|
||||
exceptions: Exception[];
|
||||
time: number; // ms since run start
|
||||
}
|
||||
```
|
||||
|
||||
`ax.find("text:Click me")`, `ax.find("id:login-form")`, `ax.findAll("role:todo-row")` are the common accessors. Prefer stable testID-style identifiers over positional selectors, for the same reason you would in Espresso or XCUITest.
|
||||
|
||||
`snapshots` is populated by the in-app SDK via `Uatu.extract("name") { value }`. Use it when the UI does not expose a value you need, such as business-logic state or hidden fields.
|
||||
|
||||
## Pattern: preconditions (login, onboarding)
|
||||
|
||||
uatu has no setup phase and no fixtures. Preconditions are action generators with two properties:
|
||||
|
||||
1. High weight, so they fire whenever applicable.
|
||||
2. Gated on a state extractor, so they return an empty tree when not applicable and self-disable once the precondition is met.
|
||||
|
||||
```ts
|
||||
const onLoginScreen = extract((s) => !!s.ax.find("id:login-form"));
|
||||
|
||||
const doLogin = actions(() => {
|
||||
if (!onLoginScreen.current) return [];
|
||||
const emailField = state.ax.find("id:email-field");
|
||||
const signInButton = state.ax.find("id:sign-in-button");
|
||||
if (!emailField || !signInButton) return [];
|
||||
return [
|
||||
InputText({ into: emailField, text: "[email protected]" }),
|
||||
Tap({ on: signInButton }),
|
||||
];
|
||||
});
|
||||
```
|
||||
|
||||
Stack these for onboarding, consent dialogs, cold-start flows:
|
||||
|
||||
```ts
|
||||
const dismissOnboarding = actions(() => {
|
||||
const skip = state.ax.find("text:Skip");
|
||||
return skip ? [Tap({ on: skip })] : [];
|
||||
});
|
||||
|
||||
export const actions = weighted(
|
||||
[100, dismissOnboarding], // clear the path first
|
||||
[50, doLogin], // log in when the login screen appears
|
||||
[10, taps], // exploration
|
||||
[2, swipes],
|
||||
);
|
||||
```
|
||||
|
||||
Lifecycle of a run:
|
||||
|
||||
```
|
||||
Step 1: fresh install, onboarding visible
|
||||
eligible: dismissOnboarding (weight 100)
|
||||
picks: Tap "Skip"
|
||||
|
||||
Step 2-3: login screen visible
|
||||
eligible: doLogin (weight 50)
|
||||
picks: InputText / Tap to sign in
|
||||
|
||||
Step 4+: home screen, onboarding and login generators return []
|
||||
eligible: taps, swipes
|
||||
picks: autonomous exploration
|
||||
```
|
||||
|
||||
Session state (tokens, keychain, prefs) persists through the rest of the run. If the app logs the user out mid-run, `doLogin` re-fires automatically. No retry logic, no special-casing.
|
||||
|
||||
## Pattern: conditional properties
|
||||
|
||||
Use gating extractors the same way inside properties. Express "only check X when Y holds":
|
||||
|
||||
```ts
|
||||
const loggedIn = extract((s) => !!s.ax.find("id:home-tab-bar"));
|
||||
|
||||
export const properties = {
|
||||
cartPersistsWhenLoggedIn: always(() => {
|
||||
if (!loggedIn.current) return true;
|
||||
return state.snapshots.cart_count !== undefined;
|
||||
}),
|
||||
};
|
||||
```
|
||||
|
||||
When `implies` ships in v0.1.0, this becomes:
|
||||
|
||||
```ts
|
||||
cartPersistsWhenLoggedIn: always(() =>
|
||||
implies(loggedIn.current, () => cartCount.current !== undefined)
|
||||
),
|
||||
```
|
||||
|
||||
## Pattern: eventually (once the operator lands)
|
||||
|
||||
`always` asserts something holds at every step. `eventually` asserts it holds at some step, usually with a time bound:
|
||||
|
||||
```ts
|
||||
// v0.1.0+
|
||||
loginSucceedsWithin30s: eventually(
|
||||
() => loggedIn.current
|
||||
).within(30, "seconds"),
|
||||
```
|
||||
|
||||
Useful for liveness checks: the loading spinner eventually goes away, the deep link eventually lands on `/home`.
|
||||
|
||||
## Pattern: snapshot-backed properties
|
||||
|
||||
When the UI does not expose a value but the app knows it, use the SDK's extractor registry:
|
||||
|
||||
```kotlin
|
||||
// in the app (Android)
|
||||
Uatu.extract("cart_count") { store.cart.size }
|
||||
```
|
||||
|
||||
```ts
|
||||
// in the spec
|
||||
const cartCount = extract<number>((s) => (s.snapshots.cart_count as number) ?? 0);
|
||||
|
||||
export const properties = {
|
||||
cartMonotonicAfterAdd: always(() => {
|
||||
const previous = cartCount.previous;
|
||||
return previous === undefined || cartCount.current >= previous;
|
||||
}),
|
||||
};
|
||||
```
|
||||
|
||||
This pattern lets you write properties against business logic that no UI element exposes.
|
||||
|
||||
## Pattern: weighted exploration sub-trees
|
||||
|
||||
Nest `weighted` to group related actions and tune their collective rate:
|
||||
|
||||
```ts
|
||||
export const actions = weighted(
|
||||
[100, dismissOnboarding],
|
||||
[50, doLogin],
|
||||
[10, taps],
|
||||
[2, swipes],
|
||||
[1, weighted(
|
||||
[3, openLink("todos://home")],
|
||||
[1, openLink("todos://settings")],
|
||||
[1, openLink("todos://item/42/edit")],
|
||||
)],
|
||||
);
|
||||
```
|
||||
|
||||
Weights are relative within a tree, so nested trees get their own local budget. This is how you keep low-frequency but high-value actions (deep links, background/foreground, rotate) from drowning out normal tapping.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
**Positional taps.** `Tap({ on: { x: 100, y: 200 } })` works for a demo but breaks on any layout change. Always prefer an `ax.find("id:...")` reference.
|
||||
|
||||
**Sleep or wait-for-time.** `Wait(3000)` inside an action generator is a smell. If you need to wait for a condition, use an extractor and gate the next action on it.
|
||||
|
||||
**Retry logic inside generators.** Generators should be pure: given the same state they produce the same actions. Retry is the runner's responsibility.
|
||||
|
||||
**Unbounded `eventually`.** Without a `.within(...)`, `eventually` never fails within a finite run. It just stays residual. Almost always you want a bound.
|
||||
Reference in new issue
Block a user