llm action-selection backend (#68)

* feat(spec): add llm() action-backend marker

* feat(spec): make llm marker inert on the JS picker

* feat(spec): expose __sanderlingSampleInput__ corpus draw

* feat(openrouter): minimal chat-completions client

* test(openrouter): cover request shape, parse, and errors

* feat(verifier): thread screenshot + capture corpus sampler

* feat(verifier): LLM accessors — candidates, config, sampler

* test(verifier): cover AllCandidates, LLMConfig, SampleInput

* feat(trace): record action Source and LLMReasoning

* feat(runner): thread step screenshot into PushSnapshot

* feat(runner): llmSource selects actions via OpenRouter

* feat(runner): wire llmSource selection and trace stamping

* test(runner): cover llmSource selection, mapping, downscale

* docs(folio): add llm action-backend example spec

* docs(folio): document the LLM action backend run

* feat(llmclient): support OPENAI_API_KEY, openrouter wins

* refactor(runner): rename openrouter package to llmclient

* docs: both api keys, example model gpt-5.4-nano

* docs: add pr style rules to claude.md

* fix(runner): explain action kinds in llm prompt to stop swipe loops

* feat(trace): record llm ranked list and chosen rank

* feat(runner): stamp llm ranked list and chosen rank on trace

* fix(runner): tap by selector to survive layout shift after observe

* revert(runner): drop selector-first tap; broke path/testTag selectors

* feat(spec): llm() accepts optional instructions

* feat(verifier): read llm instructions off config

* feat(runner): append spec instructions to llm system prompt

* docs(folio): describe app in llm spec instructions

* feat(bundler): map generator export to globalThis.generator

* feat(verifier): read llm config off globalThis.generator

* feat(runner): gate llm source on --generator flag

* feat(cmd): add --generator llm|seeded flag

* test: cover --generator flag parsing and pickSources gating

* feat(verifier): enumerate llm candidates by walking actionsRoot

collect-walk the weighted action tree: recurse weighted branches
accumulating selection probability, call authored leaves once for
concrete actions, enumerate builtins per element. label controls by
visible text (borrowing descendant text), fold gestures into directional
scrolls over scrollable containers, drop disabled, dedup descriptions.

* test(verifier): cover candidate enumeration walk

* feat(verifier): add SetupAction to walk setup without the seeded root

* test(verifier): cover SetupAction setup-only precedence

* refactor(llmclient): make JSONSchema.Schema raw json for pinned field order

* feat(trace): record llm choice number and chosen_action echo

* feat(runner): llm picks one number from weighted candidates

drop the seeded-root call for a setup-only precedence path, render a
numbered weighted candidate list, pin a reasoning-first choice schema,
strict-skip when chosen_action does not echo the numbered entry, and let
the model supply typed values (corpus fallback when empty).

* test(runner): cover choice schema, strict-skip, and setup precedence

* refactor(verifier): drop the superseded AllCandidates enumeration

* feat(folio): drive spec.ts under --generator llm; drop spec-llm.ts

* fix(verifier): label editable fields by hint, not the typed value

an editable field's own text is its transient content; prefer the hint
so the field is named by purpose and the label stays stable.

* test(runner): cover weight-suffixed echo and stripWeightSuffix

* fix(runner): accept chosen_action echo that carries the weight suffix

real runs showed the model copies the whole numbered line including the
trailing (w34) weight annotation, so strict-skip rejected ~91% of picks
and the llm was paralyzed. strip the weight suffix before comparing. also
nudge the prompt to stress-test repeated submissions (idempotency).

* fix(verifier): skip llm enumeration on cross-fade frames

a navhost mid-transition carries >1 route *Screen in a collapsed
coordinate space; acting on it taps garbage (soft keyboard). real runs
showed the llm acting on 44% of steps being such frames. skip them so the
llm re-observes a settled frame next step.

* feat(folio): show current balance on the add-transaction screen

renders the account's balance (testTag TxnCurrentBalance) below the
account name, above the credit/debit toggle, so before/after screenshots
carry comparison data.

* fix(replay): derive device space from screen extent, not first node

the first positive-bounds element is often a short status-bar node
(320x24 on android); using it gave a 320/24 aspect ratio that squashed
the screenshot overlay into a grey horizontal band. use the max extent
across elements (like the runner's screenBounds) instead.

* fix(folio): show balance as a compact one-line label

per review: one line, account-name-sized, e.g. "Balance: $0.00"
instead of a large balance card.

* fix(folio): move balance into the header, one compact line under the account name

* fix(replay): attribute deferred violations to the causing step, not detection

* fix(replay): show a step's own violations in both panels, no next-step bleed

* refactor(hierarchy): one Tree.Transitional, drop the duplicated cross-fade check

* chore: ignore .playwright-mcp scratch output

* docs: document the llm generator and --generator flag

* docs(spec): correct the llm() comment; config reads off globalThis.generator

* docs: add pr description rules

pkg/spec/src/action-tree.ts

@@ -57,10 +57,13 @@ export type ActionDescriptor =
 //   weighted: probabilistic choice over child nodes, scanned ascending.
 //   actions:  author callback returning a list to pick uniformly from.
 //   builtin:  host-backed leaf identified by a verb.
+//   llm:      marker selecting the LLM action backend; inert on the JS picker
+//             (walk returns null), Go reads config.model off globalThis.actions.
 export type GeneratorNode =
   | { kind: "weighted"; branches: ReadonlyArray<readonly [number, GeneratorNode]> }
   | { kind: "actions"; generate: () => ActionDescriptor[] }
-  | { kind: "builtin"; verb: BuiltinVerb };
+  | { kind: "builtin"; verb: BuiltinVerb }
+  | { kind: "llm"; config: { model: string; instructions?: string } };
 
 // Candidate is one host-enumerated target for a builtin verb. The host
 // resolves geometry (and a native selector) so no element handle crosses into

pkg/spec/src/actions.ts

@@ -34,6 +34,18 @@ export function actions(generator: () => Action[]): GeneratorNode {
   return { kind: "actions", generate: generator as () => ActionDescriptor[] };
 }
 
+// llm selects the LLM action backend: instead of the seeded picker drawing a
+// random candidate, Go drives an OpenAI-compatible model that picks one
+// candidate from the screenshot + the numbered candidate list. Assign it to the
+// `generator` export; Go reads the config off globalThis.generator and the
+// marker is inert on the JS picker (pick.ts walks it to null). API key comes
+// from OPENROUTER_API_KEY (OpenRouter) or OPENAI_API_KEY (OpenAI); OpenRouter
+// wins when both are set. Optional `instructions` are appended to the model's
+// prompt to describe the app under test.
+export function llm(config: { model: string; instructions?: string }): GeneratorNode {
+  return { kind: "llm", config };
+}
+
 export function whenRoute(
   routeExtractor: { readonly current: string | null },
   routes: string | readonly string[],

pkg/spec/src/index.ts

@@ -42,6 +42,7 @@ export {
   actions,
   doubleTaps,
   from,
+  llm,
   longPresses,
   pressKeys,
   scrolls,

pkg/spec/src/pick.ts

@@ -86,6 +86,12 @@ export function walk(
     }
     case "builtin":
       return walkBuiltin(node.verb, rng, host);
+    case "llm":
+      // The LLM backend is driven by Go (it reads config.model off
+      // globalThis.actions and selects via OpenRouter). On the JS picker the
+      // marker is inert, so the goja NextAction reports no action and the Go
+      // llmSource takes over.
+      return null;
   }
 }
 

pkg/spec/src/runtime-entry.ts

@@ -9,6 +9,7 @@
 
 import { Pcg } from "./pcg.ts";
 import { nextAction, walk } from "./pick.ts";
+import { INPUT_CORPUS } from "./corpus.ts";
 import type { ActionDescriptor, GeneratorNode, Host } from "./action-tree.ts";
 import type { Point } from "./types.ts";
 
@@ -130,7 +131,24 @@ export function installRuntime(
   const resolveRoot = typeof root === "function" ? root : () => root;
   const resolveSetup = () =>
     (globalThis as { setup?: GeneratorNode }).setup ?? null;
+  // The LLM action backend (Go) types InputText values by drawing from the same
+  // edge-case corpus the seeded `typing` builtin uses. Expose that draw here so
+  // Go reuses the exact sampler rather than reimplementing the corpus.
+  defineLockedGlobal(
+    "__sanderlingSampleInput__",
+    () => INPUT_CORPUS[rng.intN(INPUT_CORPUS.length)] ?? "",
+  );
   defineLockedGlobal("__sanderlingExtractors__", () => evaluateExtractors());
+  // __sanderlingSetupAction__ walks ONLY the setup generator once, for the LLM
+  // action generator (Go), which drives selection itself and must not run the
+  // seeded action root — but still wants setup's precondition steps (e.g. login)
+  // to run first. Returns null when setup is unset or yields nothing.
+  defineLockedGlobal("__sanderlingSetupAction__", () => {
+    resolveRoot();
+    const setup = resolveSetup();
+    if (!setup) return null;
+    return serializeAction(walk(setup, rng, host));
+  });
   defineLockedGlobal("__sanderlingNextAction__", () => {
     // resolveRoot runs first: on web it also resets the per-tick candidate
     // cache, which setup's walk below must see fresh.