mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
13bb2feb82946524cc4f3ce194974f82359d1bd7
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 pj
|
13bb2feb82 |
feat: uatu inspect UI (web trace explorer) (#23)
* feat(trace): extend Step/Action/Meta schema for inspect UI
Add Step.Hierarchy, Step.Residuals, Action.Selector/ResolvedBounds/TapPoint,
Meta.EndedAt and JSON tags on hierarchy.Element/Bounds/Tree so trace.jsonl
can drive the upcoming uatu inspect web UI.
* test(trace): cover EndedAt + new step fields round-trip
* feat(ltl): MarshalJSON for Formula AST + Evaluator.Residual()
Each Formula concrete type now serializes to a closed-set residual node
(true/false/not/and/or/implies/always/now/next/eventually/predicate/error)
that mirrors the TS spec API surface. Evaluator.Residual() folds pending
obligations into a single Formula so the runner can stamp one residual
per property per step into trace.jsonl.
* feat(runner): stamp residuals, hierarchy, selector targets, ended_at
Each Step now carries the captured hierarchy, per-property residual ASTs,
and (for Tap/InputText) the selector + resolved bounds + tap point. The
test_run command writes meta.ended_at on graceful shutdown so the inspect
UI can distinguish completed runs from in-progress ones.
* feat(inspect): scaffold embed dist for SPA assets
Stage 2 stub for the inspect server. Real web bundle gets wired in
Stage 4 (Makefile copies web/dist into internal/inspect/dist).
* chore(web): ignore web/ build output in root .gitignore
* chore(web): add bun + vite + vitest scaffold config
* feat(web): monochrome design tokens, typography, app shell CSS
* chore(web): placeholder for self-hosted JetBrains Mono fonts
* feat(web): index.html entry with style links and root mount
* feat(web): typescript types mirroring run/step trace schema
* feat(web): typed fetchers for runs/steps/screenshots
* feat(web): App shell with router and run/step routes
* feat(inspect): runs scan, lazy step parse, mtime-aware cache
* feat(web): RunList route with table, loading, and error states
* feat(web): RunDetail route shell with three placeholder panels
* feat(inspect): fsnotify-backed runs watcher with debounce
* fix(web): use jest-dom/vitest entry so matchers register
* test(web): cover listRuns happy path and error response
* test(web): render RunList with mocked fetch and assert row
* chore(web): commit bun lockfile
* feat(inspect): http handlers for runs/steps/screenshots/SSE
* test(inspect): cover handlers, screenshot whitelist, SSE, dev proxy
* feat(cmd): add 'uatu inspect' subcommand
* fix(web): align TS types with snake_case wire format
Go inspect server serializes RunSummary, StepSummary, Step, Meta with
snake_case JSON tags (matching the on-disk trace.jsonl/meta.json). Update
the TS types and consumers to match so API responses parse without
runtime undefined fields. Action keeps resolvedBounds/tapPoint as camelCase
because those keys were defined that way in the trace schema.
* feat(web): add ActionList panel for run-detail step navigation
* feat(web): add SnapshotTable panel with diff highlighting
Renders snapshots dictionary as a flat sorted dotted-path tree.
Changed leaves get data-changed plus a hover title with the previous value.
* test(web): cover SnapshotTable rendering and diff behavior
Eight cases: empty state, sort order, dotted-path expansion,
changed/unchanged/missing-previous flagging, and inline-vs-expanded arrays.
* feat(web): add Screenshot panel with bounds and tap overlays
Center column of run-detail page. Renders the device screenshot
scaled to fit, with an SVG overlay drawing resolvedBounds as a
violation-colored rect, tapPoint as a contrast ring, and swipes
as an arrow. Falls back to a placeholder when src is missing or
the image fails to load.
* fix(web): guard scrollIntoView call for jsdom compatibility
* test(web): cover Screenshot panel rendering and overlays
* test(web): cover ActionList rendering, selection, keyboard, and markers
* feat(web): add ExceptionsPanel component
* test(web): add ExceptionsPanel tests
* feat(web): add Timeline panel with property swimlanes
Renders SVG swimlanes per property with violated/pending/holds cells,
action-marker dots, click-to-seek, and a selected-step highlight bar.
* test(web): cover Timeline empty state, cells, status, click, highlight
* feat(web): add ResidualNode recursive AST renderer
* test(web): cover ResidualNode operators, predicate, and error chip
* feat(web): add ViolationsPanel with status badges and jump button
* test(web): cover ViolationsPanel rows, status grouping, and jump button
* test(web): register testing-library cleanup globally
All six panel test files added local afterEach(cleanup); centralize it in
the shared setup so future tests inherit DOM isolation by default.
* feat(web): hooks for url/keyboard/theme/sse
* feat(web): wire all panels into run-detail with phone-dominant grid
ActionList left, Screenshot center, Snapshots/Properties/Exceptions
stacked right, Timeline bottom. URL-synced step index (useStep), keyboard
shortcuts (j/k/arrows/g/G/.), light+dark theme toggle stored in
localStorage, SSE auto-refresh on the run index.
* test(web): add three reference run fixtures (clean, violation, exception)
* build: web targets in Makefile + bun in CI; docs(inspect)
- Makefile: web-build/web-dev/inspect-dev/test-web targets; uatu and
install now depend on web-build so the binary embeds the latest SPA.
- ci.yml: setup-bun + cache; existing make test now runs web typecheck +
vitest as part of the full suite.
- docs/manual/inspect.md: panel reference, keyboard shortcuts, URLs.
- docs/manual/cli.md: document uatu inspect.
- README: link to inspect docs.
* feat(runner): capture a screenshot per step
The driver already exposes Screenshot(ctx), but the runner never called
it. Each step now writes <run>/screenshots/step-NNNNN.png right after
the trace line, using the same failure-is-a-warning posture as other
best-effort observability hooks. Makes the inspect UI's center panel
actually useful.
* feat(inspect): include action_label in StepSummary
Tap/InputText/Swipe/PressKey/Wait each get a short human-readable
label (selector, quoted text, swipe direction, key name, duration) so
the action list panel can render readable rows instead of just 'Tap'
with no target.
* test(inspect): accept either #app or #root in SPA shell fallback
* feat(web): render action_label and screen in ActionList rows
Step rows now show 'Tap id:save', 'InputText "alice"', 'Swipe up',
'PressKey back', etc. Steps with no action fall back to
'observe @ <screen>' so the list reads as a flow instead of a wall
of '--' placeholders.
* feat(sidecar): implement screencap for android driver backend
Was stubbed to return an empty byte array, which made the runner's
per-step screenshot capture a no-op. Shell out to 'adb exec-out
screencap -p' and stream the PNG bytes back. Width/height stay zero
because the PNG header carries them; the Go side can parse if needed.
* feat(proto): add Metrics RPC for per-step CPU and memory capture
* feat(driver): Metrics(bundleID) returns cpu_percent + heap/total bytes
* feat(sidecar): implement Metrics RPC via adb top + /proc/<pid>/status
* feat(runner): capture metrics + before/after screenshots per step
Each step now writes step-NNNNN.png (before applyAction) and
step-NNNNN-after.png (after the action + wait-for-idle). The runner
samples Driver.Metrics(bundleID) before writing the trace line and
stamps Step.Metrics with cpu_percent, heap_bytes, total_memory_bytes
so the inspect UI can chart CPU and heap over the run.
* fix(runner,sidecar): measure CPU across step via /proc stat delta
'top -d 0.3 -n 2' measures CPU in a 300ms window that coincides with
the SDK-paused app, always reporting 0%. Switch to reading
/proc/<pid>/stat utime+stime and computing the delta between successive
calls; the natural step cadence gives a 2-5s measurement window that
captures the action response and render cycle. Also moved the sample
to before snapshotStep so the delta starts before the SDK pause.
* feat(web): add Metrics type for per-step cpu and memory
* refactor(web): replace --accent-change with --accent-positive token
* refactor(web): recolor chip-progress as neutral outlined chip
* refactor(web): use neutral border for changed snapshot rows
* feat(web): add MetricsChart panel with HEAP and CPU lanes
SVG-based time-series chart rendering heap bytes and CPU percent per
step across two stacked lanes, with a shared step axis below. Lines are
monochrome; a vertical highlight marks the selected step; per-step hit
rects make any click seek to that step.
* feat(web): revamp ActionList with tag targets, elapsed time, and expandable rows
Render selector-based Tap actions as <tag/> markup, show zero-padded MM:SS.mmm
elapsed time per row, and expand the active row with Position/Content sub-rows
when a full Step is available. Adds formatActionRow/formatElapsed helpers and
covers both with unit tests.
* fix(runner): stop copying Tap selector into action.text
The 'Content' inspect row should show the user-supplied text for
InputText actions and stay empty for Taps. Previously the runner copied
action.On into traceAction.Text for both, so the inspect UI showed the
selector as the tap's 'Content'.
* fix(web): use text-muted for swipe arrow after accent-change removal
* fix(web): snapshot values truncate with ellipsis + title tooltip
Long JSON values were breaking one character per line due to
overflow-wrap:anywhere in a narrow column. Switch to single-line ellipsis
with the full value exposed via the title attribute on hover.
* feat(web): state-before/after columns + metrics chart at bottom
RunDetail now renders a four-column grid:
actions | state-before | state-after | side (exceptions + timeline)
with MetricsChart spanning the bottom row. Each state column shows its
own screenshot (step-NNNNN.png vs step-NNNNN-after.png), snapshot table,
and violations panel. ActionList now receives runStartMillis and the
selected Step so the active row can expand Position/Content sub-rows.
* fix(web): skip zero-value ticks + add exception markers to metrics
HEAP '0B' and CPU '100%' labels overlapped at the lane boundary. Drop
the bottom-of-range tick on both lanes (baseline is implied) and widen
LANE_GAP so the remaining labels have breathing room. Accept an
exceptionStepIndices prop and draw a dashed red vertical line at each
to surface exception spikes directly on the CPU/heap chart.
* fix(web): let action body column shrink below its content
Required minmax(0, 1fr) so the row grid honours the column's min-size of
0 instead of the implicit 'auto', preventing the action-list from
overflowing its parent when the target string is long.
* feat(web): bigger state screenshots + single properties row
Collapse snapshots into a summary chip ('SNAPSHOTS · N violations') so
the screenshot fills its state card. Deduplicate ViolationsPanel —
show it once in a new full-width 'properties' row between the state
cards and the timeline. Drop the right sidebar; exceptions now surface
as dashed markers on the metrics chart with the ExceptionsPanel only
rendering when there are actual exceptions to report.
* feat(web): add minimal Tabs component
Monochrome tab strip with underline-on-active. Used by state-before
and state-after cards to swap between Screenshot, Snapshots, Properties.
Pane scrolls internally so the outer grid stays fixed-height.
* feat(web): fold timeline into MetricsChart as STEPS lane
Adds a thin per-step status row above HEAP showing violated (red),
pending (dim gray) or holds (green-tinted). Extends highlight +
exception markers to span the status lane. Frees a whole row in the
detail grid so the page can fit in 100vh.
* refactor(web): tabbed state cards, drop standalone Timeline panel
State-before/after now use Tabs (Screenshot / Snapshots / Properties,
default Screenshot). Removes the dedicated timeline row; status lane
lives on the metrics chart. Banner is gone from the shell.
* feat(web): lock app shell to 100vh with no page scroll
html/body/#root fill the viewport, body gets overflow:hidden, and the
detail grid uses minmax(0, 1fr) rows so inner panels own their scroll.
Tightens toolbar + panel padding for a denser feel.
* feat(web): arrow-key nav + badges on Tabs (WAI-ARIA tablist)
Roving tabindex, ArrowLeft/Right/Up/Down/Home/End navigation, explicit
aria-selected/aria-controls/id wiring, and support for an optional
badge inside each tab (used for violation counts).
* feat(web): ViolationsPanel supports violationsOnly filter
* feat(web): ActionList arrow-key nav + listbox semantics + smaller font
Promote the list to role=listbox with role=option rows; roving tabindex
lets ArrowUp/Down (and Home/End) seek between steps with focus. Font
size dropped to 11px and padding tightened so long selector-tag labels
fit in the 340px actions column.
* fix(web): useKeyboardNav yields arrow keys to tablist/listbox targets
Previously pressing ArrowRight on a focused tab switched tabs AND
advanced the step. Skip arrow handling when the event target is inside
an element with an arrow-owning ARIA role.
* feat(web): fourth 'Violations' tab + wider actions + shorter metrics
Adds a Violations tab to each state card showing only violated properties
(with count badge on the tab label when > 0). Actions column widened
from 280px to 340px, bottom metrics strip trimmed from 220px to 140px
with tighter lane heights, so the whole page still fits in 100vh with
no scrollbar.
* feat(web): compact RunDetail layout using 1px borders instead of panel padding
* refactor(inspect): simplify MetricsChart to HEAP+CPU with time axis
Drop the STEPS status lane and per-sample circle markers, switch the
x-axis from step indices to mm:ss clock time, trim y-axis ticks to
min/max with compact units, rotate lane labels into the left gutter,
and replace the thin playhead line with a wider dotted red band.
Traces stay grayscale; red appears only on the playhead pattern.
* fix(web): RunList rows no longer stretch to fill viewport height
Tables inherited flex: 1 1 auto from .app-main > * and distributed extra
vertical space across rows. Override with flex: 0 0 auto + align-self.
* misc changes
* fix(web): hoist useState above early return in MetricsChart
Calling useState after an unconditional early return violates React's
Rules of Hooks: the empty-samples branch renders 0 hooks while the
populated branch calls 1. On the initial null->loaded transition of
history the hook count changes and React throws.
* fix(web): subscribe to named SSE event instead of 'message'
Server emits 'event: runs.changed' frames; the WHATWG EventSource spec
dispatches those as events of type 'runs.changed', not 'message'. The
listener registered on 'message' was never fired, so RunList never
auto-refreshed on run create/finish/delete.
* fix(inspect): unsubscribe SSE clients on disconnect
Watcher.Subscribe appended to a slice with no matching removal path,
so every closed EventSource connection leaked its channel. Over a
long-running server the slice grew unbounded and every fs event paid
O(N) iterating dead channels. Add Unsubscribe + defer it in
handleEvents.
Unsubscribe does not close the channel: broadcast snapshots the
slice without holding the mutex, so a concurrent close would race
with its non-blocking send.
* fix(trace): rename resolvedBounds/tapPoint to snake_case
Every other json tag in the trace schema (from_x, duration_millis,
bundle_sha256, etc.) uses snake_case. The two new Action fields
introduced with the inspect UI broke that pattern. Rename them
before the format ships to external consumers.
* chore(web): drop vitest and remove UI tests from CI
No UI tests wanted in web. Removes vitest, jsdom, testing-library
devDeps and the vitest.setup.ts + vite.config.ts test block.
Makefile test-web becomes web-typecheck (typecheck only).
Fixes CI failure where `vitest run` exits 1 with no test files.
* chore(make): dedupe sidecar embed and drop recursive make
Make $(SIDECAR_JAR) the real recipe and $(SIDECAR_EMBED) a file
target, so uatu/install/inspect-dev share one copy step and
sidecar/release-cli just depend on the jar instead of re-invoking make.
|
||
|
pj
|
323878c34a |
fix(verifier): don't crash on throwing JS predicates (#21)
* fix(verifier): don't crash on throwing JS predicates formulaThunk used to panic whenever goja returned an error from a predicate callable, and nothing on the LTL -> runner path recovered, so a malformed spec (e.g. a property whose body throws or touches an undefined field) would kill the verifier process. Latch the first error on formulaState, return false so LTL marks the property violated, and expose PredicateError(name) that walks the property's formula-spec tree and surfaces the latched cause. * fix(runner): log predicate errors alongside violations For each violated property, surface the verifier's latched predicate error via logger.Warn so operators can distinguish a genuine false verdict from a malformed spec. Add a runner-level test asserting that a throwing predicate no longer crashes the run and that the error message appears in the log. |
||
|
pj
|
69002b07b1 |
fix(runner,sample-app): surface silent errors and demo a failing property (#15)
* fix(runner): surface non-deadline WaitForIdle errors Previously the WaitForIdle return value was discarded entirely, hiding real driver failures (gRPC transport errors, sidecar crashes) behind the expected deadline-exceeded case. Log non-deadline errors so they are visible without changing control flow. * chore(sample-app): drop unused uptime_millis extractor Registered in SampleApplication but never consumed by spec.ts. * fix(sample-app): drop trivial appIsRunning property app_state was hardcoded to 'running' so the property was a tautology that could never fail. Removing both the extractor and the property is the simplest fix; demo-grade properties that can fail land next. * feat(sample-app): add Reset button that zeroes clickCount Pairs with the next commit's tap-reset action so the fuzzer can violate clickCountNeverDecreases and demonstrate uatu actually finding a property violation. * feat(sample-app): add tap-reset action to exercise Reset button Weighted at 10/122, fuzzer reaches it within a short run. Pairs with the Reset button to demonstrate uatu detecting the clickCountNeverDecreases violation. * fix(runner): filter WaitForIdle errors via context state, not errors.Is errors.Is(err, context.DeadlineExceeded) misses gRPC's wrapped status.DeadlineExceeded, so every step under the maestro driver logged a spurious warning. Check idleCtx.Err() instead — captures both deadline-fired and parent-canceled cases regardless of how the driver wraps them. * chore(sample-app): tune action weights so demo violates in ~30s Prior weights left tap-reset rare enough that short demo runs missed the violation by chance. Bumped to 30/107, with typeUsername reduced since username noise doesn't help exercise clickCount. * refactor(runner): route warnings through slog Adds Options.Logger (defaults to slog.Default()) and converts the three warning sites that were using fmt.Printf. Progress line stays on Printf since it's user-facing UI, not a log. Makes the warnings testable via a capturing handler. * test(runner): assert WaitForIdle driver errors are logged Captures slog output via TextHandler into a buffer and asserts the warning message + injected error text appear when the mock driver returns a non-context error from WaitForIdle. Guards against a regression of the silent-error swallow. |
||
|
pj
|
d0578dbaaa |
fix(runner): warn on malformed screen snapshot (#13)
* fix(runner): warn on malformed screen snapshot screenFromSnapshot swallowed json.Unmarshal errors, so a non-string screen value silently became "" in the step log and trace while the verifier still saw the raw JSON. Return the error and warn at the call site, matching the hierarchy warning pattern. * docs: clarify --avd is optional for uatu test The CLI accepts --avd as an empty-string default (cmd/uatu/main.go:49) and only requires it when no device is connected and multiple AVDs exist (cmd/uatu/android_env.go:63). Docs and examples that showed it as required or always-passed were misleading. |
||
|
pj
|
16e55086d8 |
fix(runner): surface focus-tap errors in InputText (#12)
* fix(runner): surface focus-tap errors in InputText action A failed Tap/TapSelector before InputText was swallowed, so text typed into the wrong field (or no field) still reported success. Return the error so the step fails explicitly. * feat(sample-app): add username EditText and snapshot Gives the spec a real EditText target (content-desc: username_field) so the InputText action path can be exercised end-to-end. The typed value is mirrored into MainActivity.username and surfaced as the "username" snapshot for spec assertions. * feat(sample-app): exercise InputText action against username field Adds typeUsername action and usernameNeverShrinks property to the sample spec, and extends the integration test to assert the bundled spec emits an InputText(desc:username_field, "alice") action and that the property correctly violates when a snapshot reports a shorter string. |
||
|
pj
|
e7b3e2ba9c |
refactor(runner): caller manages app launch/terminate
Removes Launch + Terminate from runner.Run so the CLI can launch the app first, wait for the SDK to connect, then start the loop. The previous shape forced runner to launch internally which fought with the SDK-must-be-connected-first ordering. BundleID/ClearState fields go away too since runner no longer launches; the CLI keeps them on its testOptions struct. |
||
|
pj
|
d4a6e33aa6 |
feat(runner): pause-snapshot-evaluate-resume loop
Wires agent.Conn + driver.Driver + verifier.Verifier + trace.Writer into the v0.1 step cycle: snapshot the SDK, push to verifier, evaluate properties, write the trace step (with violations), release the SDK pause, apply the next action via the driver, wait for idle. Driver.Launch happens once before the loop and Terminate runs in defer so even an early error tears down the app cleanly. Summary returns step count and per-step violation records for the caller to print or persist. |