llm action-selection backend (#68)

* feat(spec): add llm() action-backend marker

* feat(spec): make llm marker inert on the JS picker

* feat(spec): expose __sanderlingSampleInput__ corpus draw

* feat(openrouter): minimal chat-completions client

* test(openrouter): cover request shape, parse, and errors

* feat(verifier): thread screenshot + capture corpus sampler

* feat(verifier): LLM accessors — candidates, config, sampler

* test(verifier): cover AllCandidates, LLMConfig, SampleInput

* feat(trace): record action Source and LLMReasoning

* feat(runner): thread step screenshot into PushSnapshot

* feat(runner): llmSource selects actions via OpenRouter

* feat(runner): wire llmSource selection and trace stamping

* test(runner): cover llmSource selection, mapping, downscale

* docs(folio): add llm action-backend example spec

* docs(folio): document the LLM action backend run

* feat(llmclient): support OPENAI_API_KEY, openrouter wins

* refactor(runner): rename openrouter package to llmclient

* docs: both api keys, example model gpt-5.4-nano

* docs: add pr style rules to claude.md

* fix(runner): explain action kinds in llm prompt to stop swipe loops

* feat(trace): record llm ranked list and chosen rank

* feat(runner): stamp llm ranked list and chosen rank on trace

* fix(runner): tap by selector to survive layout shift after observe

* revert(runner): drop selector-first tap; broke path/testTag selectors

* feat(spec): llm() accepts optional instructions

* feat(verifier): read llm instructions off config

* feat(runner): append spec instructions to llm system prompt

* docs(folio): describe app in llm spec instructions

* feat(bundler): map generator export to globalThis.generator

* feat(verifier): read llm config off globalThis.generator

* feat(runner): gate llm source on --generator flag

* feat(cmd): add --generator llm|seeded flag

* test: cover --generator flag parsing and pickSources gating

* feat(verifier): enumerate llm candidates by walking actionsRoot

collect-walk the weighted action tree: recurse weighted branches
accumulating selection probability, call authored leaves once for
concrete actions, enumerate builtins per element. label controls by
visible text (borrowing descendant text), fold gestures into directional
scrolls over scrollable containers, drop disabled, dedup descriptions.

* test(verifier): cover candidate enumeration walk

* feat(verifier): add SetupAction to walk setup without the seeded root

* test(verifier): cover SetupAction setup-only precedence

* refactor(llmclient): make JSONSchema.Schema raw json for pinned field order

* feat(trace): record llm choice number and chosen_action echo

* feat(runner): llm picks one number from weighted candidates

drop the seeded-root call for a setup-only precedence path, render a
numbered weighted candidate list, pin a reasoning-first choice schema,
strict-skip when chosen_action does not echo the numbered entry, and let
the model supply typed values (corpus fallback when empty).

* test(runner): cover choice schema, strict-skip, and setup precedence

* refactor(verifier): drop the superseded AllCandidates enumeration

* feat(folio): drive spec.ts under --generator llm; drop spec-llm.ts

* fix(verifier): label editable fields by hint, not the typed value

an editable field's own text is its transient content; prefer the hint
so the field is named by purpose and the label stays stable.

* test(runner): cover weight-suffixed echo and stripWeightSuffix

* fix(runner): accept chosen_action echo that carries the weight suffix

real runs showed the model copies the whole numbered line including the
trailing (w34) weight annotation, so strict-skip rejected ~91% of picks
and the llm was paralyzed. strip the weight suffix before comparing. also
nudge the prompt to stress-test repeated submissions (idempotency).

* fix(verifier): skip llm enumeration on cross-fade frames

a navhost mid-transition carries >1 route *Screen in a collapsed
coordinate space; acting on it taps garbage (soft keyboard). real runs
showed the llm acting on 44% of steps being such frames. skip them so the
llm re-observes a settled frame next step.

* feat(folio): show current balance on the add-transaction screen

renders the account's balance (testTag TxnCurrentBalance) below the
account name, above the credit/debit toggle, so before/after screenshots
carry comparison data.

* fix(replay): derive device space from screen extent, not first node

the first positive-bounds element is often a short status-bar node
(320x24 on android); using it gave a 320/24 aspect ratio that squashed
the screenshot overlay into a grey horizontal band. use the max extent
across elements (like the runner's screenBounds) instead.

* fix(folio): show balance as a compact one-line label

per review: one line, account-name-sized, e.g. "Balance: $0.00"
instead of a large balance card.

* fix(folio): move balance into the header, one compact line under the account name

* fix(replay): attribute deferred violations to the causing step, not detection

* fix(replay): show a step's own violations in both panels, no next-step bleed

* refactor(hierarchy): one Tree.Transitional, drop the duplicated cross-fade check

* chore: ignore .playwright-mcp scratch output

* docs: document the llm generator and --generator flag

* docs(spec): correct the llm() comment; config reads off globalThis.generator

* docs: add pr description rules

replay-ui/src/__tests__/device-space.test.ts

@@ -44,4 +44,18 @@ describe("deviceSpaceOf", () => {
     };
     expect(deviceSpaceOf(hierarchy)).toEqual({ width: 402, height: 874 });
   });
+
+  it("uses the screen extent, not a short status-bar node listed first", () => {
+    // Regression: a 320x24 status bar precedes the 320x640 screen. Picking the
+    // first positive-bounds element gave a 320/24 aspect ratio, squashing the
+    // screenshot overlay into a grey horizontal band.
+    const hierarchy: Hierarchy = {
+      elements: [
+        { bounds: { left: 0, top: 0, right: 320, bottom: 24 } },
+        { bounds: { left: 0, top: 0, right: 320, bottom: 640 } },
+        { bounds: { left: 20, top: 271, right: 286, bottom: 319 } },
+      ],
+    };
+    expect(deviceSpaceOf(hierarchy)).toEqual({ width: 320, height: 640 });
+  });
 });

replay-ui/src/__tests__/run-history.test.ts

@@ -2,6 +2,7 @@ import { describe, it, expect } from "bun:test";
 import {
   buildRunHistory,
   collectPropertyNames,
+  relocateViolationsToCause,
   sortLanes,
   statusForProperty,
 } from "../lib/run-history";
@@ -52,7 +53,67 @@ describe("sortLanes", () => {
   });
 });
 
+// Bug class: a next/eventually violation records on the DETECTION step but is
+// caused earlier; leaving it on the detection step lights up an unrelated
+// action in the Violations tab while the timeline dot sits on the cause step.
+describe("relocateViolationsToCause", () => {
+  it("moves a deferred violation to the step its witness blames", () => {
+    const steps = [
+      step({ step: 290, residuals: { p: { op: "predicate", name: "p3" } } }),
+      step({
+        step: 291,
+        violations: ["p"],
+        witnesses: { p: { step: 290, reason: "predicate false" } },
+        residuals: { p: { op: "false" } },
+      }),
+      step({ step: 292 }),
+    ];
+    const moved = relocateViolationsToCause(steps);
+
+    expect(moved[0]?.violations).toEqual(["p"]);
+    expect(moved[0]?.witnesses?.p?.reason).toBe("predicate false");
+    expect(moved[1]?.violations).toEqual([]);
+    expect(moved[1]?.witnesses?.p).toBeUndefined();
+    // originals are cloned, never mutated
+    expect(steps[1]?.violations).toEqual(["p"]);
+    expect(steps[1]?.witnesses?.p?.step).toBe(290);
+  });
+
+  it("leaves a violation without a witness on its detection step", () => {
+    const steps = [step({ step: 5, violations: ["p"], residuals: { p: { op: "false" } } })];
+    expect(relocateViolationsToCause(steps)).toBe(steps);
+  });
+
+  it("keeps the violation put when the blamed step is absent from the trace", () => {
+    const steps = [step({ step: 9, violations: ["p"], witnesses: { p: { step: 3 } } })];
+    expect(relocateViolationsToCause(steps)[0]?.violations).toEqual(["p"]);
+  });
+});
+
 describe("buildRunHistory", () => {
+  it("anchors a deferred violation's lane cell to the cause step, not detection", () => {
+    const run = {
+      id: "run-2",
+      steps: [summary({ index: 290, has_violations: true }), summary({ index: 291 })],
+    } as unknown as Run;
+    const responses = [
+      step({ step: 290, residuals: { p: { op: "predicate" } } }),
+      step({
+        step: 291,
+        violations: ["p"],
+        witnesses: { p: { step: 290 } },
+        residuals: { p: { op: "false" } },
+      }),
+    ];
+
+    const history = buildRunHistory(run, responses);
+
+    expect(history.lanes[0].statuses).toEqual(["violated", "pending"]);
+    expect(history.steps[0]?.violations).toEqual(["p"]);
+    expect(history.steps[1]?.violations).toEqual([]);
+    expect(history.firstViolationStep).toBe(290);
+  });
+
   it("aligns lane statuses, metrics samples, and first-violation index by position", () => {
     const run = {
       id: "run-1",

replay-ui/src/lib/device-space.ts

@@ -3,18 +3,23 @@ import type { Hierarchy } from "../types";
 // Tap points and resolved bounds in the trace share the hierarchy root's
 // coordinate space (iOS points, Android pixels, web CSS px). Screenshots may
 // be scaled (iOS 3x, web DPR>1), so the overlay viewBox must come from the
-// root bounds, not the image's natural pixel size. Elements are in pre-order;
-// the first one with positive extent is the root window (iOS prepends a
-// synthetic zero-bounds node, so plain elements[0] is not enough).
+// device bounds, not the image's natural pixel size.
+//
+// Use the maximum extent across all elements (mirroring the runner's
+// screenBounds), NOT the first positive-bounds element: the first element is
+// often a status-bar/decor node a few px tall (e.g. 320x24 on Android), which
+// would give a wildly wrong aspect ratio and squash the overlay into a band.
 export function deviceSpaceOf(
   hierarchy?: Hierarchy,
 ): { width: number; height: number } | undefined {
+  let width = 0;
+  let height = 0;
   for (const element of hierarchy?.elements ?? []) {
-    const width = element.bounds.right;
-    const height = element.bounds.bottom;
-    if (width > 0 && height > 0) {
-      return { width, height };
-    }
+    if (element.bounds.right > width) width = element.bounds.right;
+    if (element.bounds.bottom > height) height = element.bounds.bottom;
+  }
+  if (width > 0 && height > 0) {
+    return { width, height };
   }
   return undefined;
 }

replay-ui/src/lib/run-history.ts

@@ -14,6 +14,55 @@ export interface RunHistory {
   steps: (Step | null)[];
 }
 
+// A next/eventually obligation is evaluated one or more steps AFTER the action
+// that armed it, so the checker records the violation on the DETECTION step
+// while its witness names the CAUSE step. The timeline dot (the backend's
+// markViolations) already sits on the cause step; mirror that here so the
+// Violations tab and property lanes light up on the same step — the guilty
+// action — not the unrelated action that happened to be running when the
+// obligation resolved. Steps are cloned, never mutated in place.
+export function relocateViolationsToCause(
+  steps: (Step | null)[],
+): (Step | null)[] {
+  const byIndex = new Map<number, Step>();
+  for (const s of steps) if (s) byIndex.set(s.step, s);
+
+  const clones = new Map<number, Step>();
+  const clone = (s: Step): Step => {
+    let c = clones.get(s.step);
+    if (!c) {
+      c = {
+        ...s,
+        violations: [...(s.violations ?? [])],
+        witnesses: { ...(s.witnesses ?? {}) },
+      };
+      clones.set(s.step, c);
+    }
+    return c;
+  };
+
+  for (const s of steps) {
+    for (const name of s?.violations ?? []) {
+      const cause = s?.witnesses?.[name]?.step;
+      if (cause === undefined || cause === s?.step) continue;
+      const target = byIndex.get(cause);
+      if (!target) continue;
+      const from = clone(s as Step);
+      const to = clone(target);
+      from.violations = (from.violations ?? []).filter((n) => n !== name);
+      const witness = s?.witnesses?.[name];
+      if (witness) {
+        delete from.witnesses?.[name];
+        (to.witnesses ??= {})[name] = witness;
+      }
+      if (!(to.violations ?? []).includes(name)) (to.violations ??= []).push(name);
+    }
+  }
+
+  if (clones.size === 0) return steps;
+  return steps.map((s) => (s ? clones.get(s.step) ?? s : s));
+}
+
 export function collectPropertyNames(steps: (Step | null)[]): string[] {
   const names = new Set<string>();
   for (const step of steps) {
@@ -47,10 +96,11 @@ export function buildRunHistory(
   run: Run,
   responses: (Step | null)[],
 ): RunHistory {
-  const propertyNames = collectPropertyNames(responses);
+  const steps = relocateViolationsToCause(responses);
+  const propertyNames = collectPropertyNames(steps);
   const lanes: PropertyLane[] = propertyNames.map((name) => ({
     name,
-    statuses: responses.map((step) => statusForProperty(name, step)),
+    statuses: steps.map((step) => statusForProperty(name, step)),
   }));
   const firstViolationStep = run.steps.find((entry) => entry.has_violations)?.index;
   const firstExceptionStep = run.steps.find((entry) => entry.has_exceptions)?.index;
@@ -63,7 +113,7 @@ export function buildRunHistory(
   const metricsSamples: MetricsSample[] = run.steps.map((entry, position) => ({
     stepIndex: entry.index,
     timestamp: entry.timestamp,
-    metrics: responses[position]?.metrics,
+    metrics: steps[position]?.metrics,
   }));
   return {
     names: propertyNames,
@@ -73,6 +123,6 @@ export function buildRunHistory(
     exceptionStepIndices,
     violationStepIndices,
     metricsSamples,
-    steps: responses,
+    steps,
   };
 }

replay-ui/src/routes/RunDetail.tsx

@@ -113,12 +113,13 @@ export default function RunDetail() {
     return <div className="status-block">loading run...</div>;
   }
 
-  const violationsBefore = currentStep?.violations ?? [];
-  const violationsAfter = nextStep?.violations ?? violationsBefore;
+  // A violation belongs to the action that caused it (relocateViolationsToCause
+  // already anchored it to that step), so show this step's own violations in
+  // both panels rather than bleeding the next step's record into "after".
+  const stepViolations = currentStep?.violations ?? [];
+  const stepWitnesses = currentStep?.witnesses;
   const residualsBefore = currentStep?.residuals;
   const residualsAfter = nextStep?.residuals ?? residualsBefore;
-  const witnessesBefore = currentStep?.witnesses;
-  const witnessesAfter = nextStep?.witnesses ?? witnessesBefore;
   const exceptionsForStep = currentStep?.exceptions;
   const beforeSpace = deviceSpaceOf(currentStep?.hierarchy);
   const afterSpace = deviceSpaceOf(nextStep?.hierarchy ?? currentStep?.hierarchy);
@@ -157,9 +158,9 @@ export default function RunDetail() {
       content: (
         <ViolationsPanel
           propertyNames={history?.names ?? []}
-          violations={violationsBefore}
+          violations={stepViolations}
           residuals={residualsBefore}
-          witnesses={witnessesBefore}
+          witnesses={stepWitnesses}
           onJumpToFirstViolation={jumpToFirstViolation}
           hasFirstViolation={history?.firstViolationStep !== undefined}
           onJumpToStep={goTo}
@@ -170,17 +171,17 @@ export default function RunDetail() {
       id: "violations",
       label: "Violations",
       badge:
-        violationsBefore.length > 0 ? (
+        stepViolations.length > 0 ? (
           <span className="tabs-badge" data-kind="violation">
-            {violationsBefore.length}
+            {stepViolations.length}
           </span>
         ) : undefined,
       content: (
         <ViolationsPanel
           propertyNames={history?.names ?? []}
-          violations={violationsBefore}
+          violations={stepViolations}
           residuals={residualsBefore}
-          witnesses={witnessesBefore}
+          witnesses={stepWitnesses}
           onJumpToFirstViolation={jumpToFirstViolation}
           hasFirstViolation={history?.firstViolationStep !== undefined}
           onJumpToStep={goTo}
@@ -224,9 +225,9 @@ export default function RunDetail() {
       content: (
         <ViolationsPanel
           propertyNames={history?.names ?? []}
-          violations={violationsAfter}
+          violations={stepViolations}
           residuals={residualsAfter}
-          witnesses={witnessesAfter}
+          witnesses={stepWitnesses}
           onJumpToFirstViolation={jumpToFirstViolation}
           hasFirstViolation={history?.firstViolationStep !== undefined}
           onJumpToStep={goTo}
@@ -237,17 +238,17 @@ export default function RunDetail() {
       id: "violations",
       label: "Violations",
       badge:
-        violationsAfter.length > 0 ? (
+        stepViolations.length > 0 ? (
           <span className="tabs-badge" data-kind="violation">
-            {violationsAfter.length}
+            {stepViolations.length}
           </span>
         ) : undefined,
       content: (
         <ViolationsPanel
           propertyNames={history?.names ?? []}
-          violations={violationsAfter}
+          violations={stepViolations}
           residuals={residualsAfter}
-          witnesses={witnessesAfter}
+          witnesses={stepWitnesses}
           onJumpToFirstViolation={jumpToFirstViolation}
           hasFirstViolation={history?.firstViolationStep !== undefined}
           onJumpToStep={goTo}