feat: LTL operators, sampling, and default generators (#17)

* feat(ltl): add Now/Next/Eventually/Implies/Or/And/Not formulas

Replace the fold-with-latch evaluator with a residual-formula reducer.
Each Observe() instantiates a fresh obligation from the root (stripping
an outer Always), reduces each pending obligation against current state,
latches Violated on first failure, and surfaces Pending verdicts for
deferred obligations. Existing Always/Pure/Thunk tests continue to pass.

* feat(ltl): support relative duration for eventually().within()

* feat(proto): add Swipe, PressKey, RecentLogs RPCs

* feat(verifier,runner): formula handles, new action kinds, rich state

- verifier: add formula-spec registry; bindNow/bindNext/bindEventually with
  chainable .implies/.or/.and/.not and .within(n,unit) on eventually; bindFrom
  for uniform sampling. bindAlways keeps accepting plain predicates.
- verifier: store lastTree, lastAction, step time, logs, exceptions on the
  Verifier; SnapshotInput replaces the (snapshots, tree) pair. stateObject now
  produces state.lastAction/time/logs/exceptions matching the TS State type.
- verifier: make taps/swipes/waitOnce/pressKey built-in generators actually
  fire; taps picks a clickable, enabled element from the last hierarchy.
- agent: add exceptions field to Message wire format.
- driver: add Swipe/PressKey/RecentLogs to Driver interface; wire maestro
  client and mock driver. LogEntry exposed for runner consumption.
- runner: apply Swipe/PressKey/Wait actions; collect logcat and exceptions;
  pass lastAction and step time into PushSnapshot.

* feat(spec-api): LTL operators, new actions, richer State

- ltl.ts exports now/next/eventually; always overload accepts a Formula
- types.ts: Formula gains implies/or/and/not; EventuallyFormula adds .within;
  State gains lastAction/time/logs/exceptions; Swipe/PressKey/Wait action types
- actions.ts: Swipe/PressKey/Wait/from constructors; waitOnce + pressKey
  default generators
- tests exercise the chaining, sampling, and new actions through a recorded
  fake runtime

* feat(sidecar): add swipe, pressKey, recentLogs RPC handlers

* feat(sdk-android): capture uncaught exceptions

Install a default uncaught handler on Uatu.start, chained with any
existing handler so Android's crash reporter still runs. Expose
Uatu.reportError for callers to forward caught throwables. A bounded
circular buffer (default 50) drains into each STATE message's new
exceptions field. Protocol.kt serializes/deserializes the field,
matching the Go wire format added to internal/agent/protocol.go.

* feat(spec-api): add @uatu/spec/defaults/properties bundle

* feat(sample-app): exercise new LTL operators + defaults

spec.ts now imports eventually/next/now/from from @uatu/spec and
noUncaughtExceptions from @uatu/spec/defaults/properties. It declares
three properties that exercise the new surface:

- accountCountNonNegative: plain always() safety
- addAccountAdvances: always(now(x).implies(next(y)))
- eventuallyLoggedIn: eventually(p).within(30, "seconds")
- noUncaughtExceptions: imported default

The weighted actions root uses from() for random phone/name sampling
and entries for taps/swipes/waitOnce/pressKey built-ins.

SampleApplication gains a debug hook gated on the system property
uatu.inject_error so the e2e run can synthesize an Uatu.reportError and
verify noUncaughtExceptions violates.

cmd/uatu/test_run.go adds a subpath alias so specs importing
"@uatu/spec/defaults/properties" resolve against the in-tree source
when running from the uatu checkout. The spec-integration tests swap
the old click-counter fixtures for the new login hierarchy.

* feat(trace): record swipe/key/wait details + exceptions

trace.Step gains an Exceptions array so the trace captures the
class/message/stackTrace for each SDK-reported throwable in a step.
trace.Action gains FromX/FromY/ToX/ToY/Key/DurationMillis so the full
payload of Swipe/PressKey/Wait actions is visible in trace.jsonl.

sample-app's debug error hook now gates on ApplicationInfo.DEBUGGABLE
instead of a system property (adb setprop fails on non-rooted
emulators).
This commit is contained in:
pj authored and GitHub committed 2026-04-20 02:19:39 +07:00
1 parent 5b5594ee05
commit 7493945251
40 files changed
+2980 -262

No files matched your search

+1 -1
View File
@@ -31,7 +31,7 @@ func TestStateAxFindWorks(t *testing.T) {
globalThis.count = __uatu__.extract(state => state.ax.findAll("id:select_language").length);
`)
if err := verifier.PushSnapshot(Snapshots{}, tree); err != nil {
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}, Tree: tree}); err != nil {
t.Fatal(err)
}
+305 -13
View File
@@ -2,6 +2,7 @@ package verifier
import (
"fmt"
"time"
"github.com/dop251/goja"
)
@@ -11,6 +12,9 @@ type ActionKind string
const (
ActionKindTap ActionKind = "Tap"
ActionKindInputText ActionKind = "InputText"
ActionKindSwipe ActionKind = "Swipe"
ActionKindPressKey ActionKind = "PressKey"
ActionKindWait ActionKind = "Wait"
)
type Action struct {
@@ -21,6 +25,13 @@ type Action struct {
// Tap/InputText. Zero means the runner must resolve On against the
// current hierarchy.
X, Y int
// Swipe coordinates (raw px). Used only for ActionKindSwipe.
FromX, FromY int
ToX, ToY int
// DurationMillis is the Swipe gesture duration or the Wait duration.
DurationMillis int
// Key is the logical key name for ActionKindPressKey.
Key string
}
type extractorState struct {
@@ -32,15 +43,51 @@ type formulaState struct {
predicate goja.Callable
}
type specKind int
const (
tagFormula = "__uatuFormula"
tagActionGenerator = "__uatuActionGenerator"
tagInternalKind = "__uatuKind"
tagSelector = "__uatuSelector"
internalKindActions = "actions"
internalKindWeighted = "weighted"
internalKindBuiltinTaps = "taps"
internalKindBuiltinSwipes = "swipes"
specKindPure specKind = iota
specKindThunk
specKindNow
specKindNext
specKindEventually
specKindImplies
specKindOr
specKindAnd
specKindNot
specKindAlways
)
// formulaSpec is the Go-side registry entry that mirrors a chainable JS
// formula handle. Handles reference specs by index; chaining creates new
// specs that reference their operands' indices.
type formulaSpec struct {
kind specKind
pureValue bool
predicateIndex int
childA int
childB int
stepBound int
hasStepBound bool
duration time.Duration
}
const (
tagFormula = "__uatuFormula"
tagFormulaSpecIndex = "__uatuFormulaSpec"
tagActionGenerator = "__uatuActionGenerator"
tagInternalKind = "__uatuKind"
tagSelector = "__uatuSelector"
internalKindActions = "actions"
internalKindWeighted = "weighted"
internalKindBuiltinTaps = "taps"
internalKindBuiltinSwipes = "swipes"
internalKindBuiltinWaitOnce = "waitOnce"
internalKindBuiltinPressKey = "pressKey"
)
// installRuntimeBindings exposes globalThis.__uatu__ to the loaded spec.
@@ -53,24 +100,51 @@ func (v *Verifier) installRuntimeBindings() error {
if err := uatu.Set("always", v.bindAlways); err != nil {
return err
}
if err := uatu.Set("now", v.bindNow); err != nil {
return err
}
if err := uatu.Set("next", v.bindNext); err != nil {
return err
}
if err := uatu.Set("eventually", v.bindEventually); err != nil {
return err
}
if err := uatu.Set("actions", v.bindActions); err != nil {
return err
}
if err := uatu.Set("weighted", v.bindWeighted); err != nil {
return err
}
if err := uatu.Set("from", v.bindFrom); err != nil {
return err
}
if err := uatu.Set("tap", v.bindTap); err != nil {
return err
}
if err := uatu.Set("inputText", v.bindInputText); err != nil {
return err
}
if err := uatu.Set("swipe", v.bindSwipe); err != nil {
return err
}
if err := uatu.Set("pressKey", v.bindPressKey); err != nil {
return err
}
if err := uatu.Set("wait", v.bindWait); err != nil {
return err
}
if err := uatu.Set("taps", v.builtinGenerator(internalKindBuiltinTaps)); err != nil {
return err
}
if err := uatu.Set("swipes", v.builtinGenerator(internalKindBuiltinSwipes)); err != nil {
return err
}
if err := uatu.Set("waitOnce", v.builtinGenerator(internalKindBuiltinWaitOnce)); err != nil {
return err
}
if err := uatu.Set("pressKeys", v.builtinGenerator(internalKindBuiltinPressKey)); err != nil {
return err
}
return v.runtime.GlobalObject().Set("__uatu__", uatu)
}
@@ -92,25 +166,178 @@ func (v *Verifier) bindExtract(call goja.FunctionCall) goja.Value {
return handle
}
// bindAlways accepts either a predicate function (legacy shape) or a formula
// handle (new shape). Both produce a formula handle tagged with
// __uatuFormulaSpec.
func (v *Verifier) bindAlways(call goja.FunctionCall) goja.Value {
if len(call.Arguments) != 1 {
panic(v.runtime.NewTypeError("always requires exactly one argument"))
}
arg := call.Arguments[0]
if predicate, ok := goja.AssertFunction(arg); ok {
thunkIndex := v.registerThunk(predicate)
return v.makeFormulaHandle(specKindAlways, formulaSpec{
kind: specKindAlways,
childA: thunkIndex,
})
}
childIndex, ok := v.extractSpecIndex(arg)
if !ok {
panic(v.runtime.NewTypeError("always argument must be a predicate or formula"))
}
return v.makeFormulaHandle(specKindAlways, formulaSpec{
kind: specKindAlways,
childA: childIndex,
})
}
func (v *Verifier) bindNow(call goja.FunctionCall) goja.Value {
thunkIndex := v.requirePredicate(call, "now")
return v.makeFormulaHandle(specKindNow, formulaSpec{
kind: specKindNow,
childA: thunkIndex,
})
}
func (v *Verifier) bindNext(call goja.FunctionCall) goja.Value {
thunkIndex := v.requirePredicate(call, "next")
return v.makeFormulaHandle(specKindNext, formulaSpec{
kind: specKindNext,
childA: thunkIndex,
})
}
func (v *Verifier) bindEventually(call goja.FunctionCall) goja.Value {
thunkIndex := v.requirePredicate(call, "eventually")
return v.makeFormulaHandle(specKindEventually, formulaSpec{
kind: specKindEventually,
childA: thunkIndex,
})
}
func (v *Verifier) requirePredicate(call goja.FunctionCall, name string) int {
if len(call.Arguments) != 1 {
panic(v.runtime.NewTypeError(name + " requires exactly one argument"))
}
predicate, ok := goja.AssertFunction(call.Arguments[0])
if !ok {
panic(v.runtime.NewTypeError("always argument must be a function"))
panic(v.runtime.NewTypeError(name + " argument must be a function"))
}
return v.registerThunk(predicate)
}
formula := &formulaState{predicate: predicate}
v.formulas = append(v.formulas, formula)
formulaIndex := len(v.formulas) - 1
// registerThunk stores a predicate in v.formulas and returns its index, which
// reduce can later invoke via formulaThunk.
func (v *Verifier) registerThunk(predicate goja.Callable) int {
spec := formulaSpec{kind: specKindThunk}
// predicateIndex points into v.formulas, which is a parallel slice.
spec.predicateIndex = len(v.formulas)
v.formulas = append(v.formulas, &formulaState{predicate: predicate})
v.formulaSpecs = append(v.formulaSpecs, spec)
return len(v.formulaSpecs) - 1
}
// registerSpec appends a spec and returns its index.
func (v *Verifier) registerSpec(spec formulaSpec) int {
v.formulaSpecs = append(v.formulaSpecs, spec)
return len(v.formulaSpecs) - 1
}
// makeFormulaHandle registers the spec and returns a JS handle exposing
// chainable combinators. Eventually handles additionally expose .within.
func (v *Verifier) makeFormulaHandle(kind specKind, spec formulaSpec) *goja.Object {
index := v.registerSpec(spec)
return v.formulaHandle(kind, index)
}
func (v *Verifier) formulaHandle(kind specKind, index int) *goja.Object {
handle := v.runtime.NewObject()
_ = handle.Set(tagFormula, true)
_ = handle.Set("__uatuIndex", formulaIndex)
_ = handle.Set(tagFormulaSpecIndex, index)
// Keep __uatuIndex as an alias so older property shapes that read it keep
// working during backward-compat transitions.
_ = handle.Set("__uatuIndex", index)
_ = handle.Set("implies", v.binaryChain(index, specKindImplies))
_ = handle.Set("or", v.binaryChain(index, specKindOr))
_ = handle.Set("and", v.binaryChain(index, specKindAnd))
_ = handle.Set("not", v.unaryChain(index, specKindNot))
if kind == specKindEventually {
_ = handle.Set("within", v.eventuallyWithin(index))
}
return handle
}
func (v *Verifier) binaryChain(selfIndex int, kind specKind) func(call goja.FunctionCall) goja.Value {
return func(call goja.FunctionCall) goja.Value {
if len(call.Arguments) != 1 {
panic(v.runtime.NewTypeError("operator requires exactly one argument"))
}
otherIndex, ok := v.extractSpecIndex(call.Arguments[0])
if !ok {
panic(v.runtime.NewTypeError("operator argument must be a formula"))
}
return v.makeFormulaHandle(kind, formulaSpec{
kind: kind,
childA: selfIndex,
childB: otherIndex,
})
}
}
func (v *Verifier) unaryChain(selfIndex int, kind specKind) func(call goja.FunctionCall) goja.Value {
return func(call goja.FunctionCall) goja.Value {
return v.makeFormulaHandle(kind, formulaSpec{
kind: kind,
childA: selfIndex,
})
}
}
func (v *Verifier) eventuallyWithin(selfIndex int) func(call goja.FunctionCall) goja.Value {
return func(call goja.FunctionCall) goja.Value {
if len(call.Arguments) != 2 {
panic(v.runtime.NewTypeError("within requires (amount, unit)"))
}
amount := call.Argument(0).ToInteger()
unit := call.Argument(1).String()
base := v.formulaSpecs[selfIndex]
if base.kind != specKindEventually {
panic(v.runtime.NewTypeError("within only applies to eventually"))
}
switch unit {
case "steps":
base.stepBound = int(amount)
base.hasStepBound = true
case "milliseconds":
base.duration = time.Duration(amount) * time.Millisecond
case "seconds":
base.duration = time.Duration(amount) * time.Second
default:
panic(v.runtime.NewTypeError("within unit must be 'milliseconds', 'seconds', or 'steps'"))
}
return v.makeFormulaHandle(specKindEventually, base)
}
}
// extractSpecIndex reads __uatuFormulaSpec from a JS formula handle.
func (v *Verifier) extractSpecIndex(value goja.Value) (int, bool) {
if value == nil || goja.IsNull(value) || goja.IsUndefined(value) {
return 0, false
}
object := value.ToObject(v.runtime)
if object == nil {
return 0, false
}
indexValue := object.Get(tagFormulaSpecIndex)
if indexValue == nil || goja.IsUndefined(indexValue) {
return 0, false
}
return int(indexValue.ToInteger()), true
}
func (v *Verifier) bindActions(call goja.FunctionCall) goja.Value {
if len(call.Arguments) != 1 {
panic(v.runtime.NewTypeError("actions requires a single generator argument"))
@@ -143,6 +370,34 @@ func (v *Verifier) bindWeighted(call goja.FunctionCall) goja.Value {
return handle
}
// bindFrom returns a `{ generate }` that picks uniformly at random from the
// provided items using the verifier's seeded rng.
func (v *Verifier) bindFrom(call goja.FunctionCall) goja.Value {
if len(call.Arguments) != 1 {
panic(v.runtime.NewTypeError("from requires an array argument"))
}
itemsValue := call.Arguments[0]
itemsObject := itemsValue.ToObject(v.runtime)
if itemsObject == nil {
panic(v.runtime.NewTypeError("from argument must be an array"))
}
lengthValue := itemsObject.Get("length")
if lengthValue == nil {
panic(v.runtime.NewTypeError("from argument must be array-like"))
}
length := int(lengthValue.ToInteger())
handle := v.runtime.NewObject()
_ = handle.Set("generate", func(goja.FunctionCall) goja.Value {
if length == 0 {
return goja.Undefined()
}
index := v.rng.IntN(length)
return itemsObject.Get(fmt.Sprintf("%d", index))
})
return handle
}
func (v *Verifier) bindTap(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
@@ -166,6 +421,43 @@ func (v *Verifier) bindInputText(call goja.FunctionCall) goja.Value {
return handle
}
func (v *Verifier) bindSwipe(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("Swipe requires {from, to}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "Swipe")
_ = handle.Set("from", parameters.Get("from"))
_ = handle.Set("to", parameters.Get("to"))
if duration := parameters.Get("durationMillis"); duration != nil && !goja.IsUndefined(duration) {
_ = handle.Set("durationMillis", duration)
}
return handle
}
func (v *Verifier) bindPressKey(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("PressKey requires {key}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "PressKey")
_ = handle.Set("key", parameters.Get("key"))
return handle
}
func (v *Verifier) bindWait(call goja.FunctionCall) goja.Value {
parameters := call.Argument(0).ToObject(v.runtime)
if parameters == nil {
panic(v.runtime.NewTypeError("Wait requires {durationMillis}"))
}
handle := v.runtime.NewObject()
_ = handle.Set("kind", "Wait")
_ = handle.Set("durationMillis", parameters.Get("durationMillis"))
return handle
}
func (v *Verifier) builtinGenerator(kind string) *goja.Object {
handle := v.runtime.NewObject()
_ = handle.Set(tagActionGenerator, true)
+147 -7
View File
@@ -3,6 +3,7 @@ package verifier
import (
"encoding/json"
"fmt"
"time"
"github.com/dop251/goja"
@@ -12,13 +13,23 @@ import (
// Snapshots is the per-step extractor output forwarded by the SDK.
type Snapshots map[string]json.RawMessage
// stateObject builds a JS-side `{ snapshots, ax }` matching the State type
// from pkg/spec-api. ax is backed by the parsed uiautomator hierarchy when
// one is provided.
func stateObject(runtime *goja.Runtime, snapshots Snapshots, tree *hierarchy.Tree) (*goja.Object, error) {
type stateInput struct {
snapshots Snapshots
tree *hierarchy.Tree
lastAction *Action
stepTime time.Time
runStart time.Time
logs []LogEntry
exceptions []Exception
}
// stateObject builds the JS-side `state` object matching the State type from
// pkg/spec-api. Fields beyond snapshots/ax are included when the caller
// populated them on stateInput.
func stateObject(runtime *goja.Runtime, input stateInput) (*goja.Object, error) {
state := runtime.NewObject()
snapshotsObject := runtime.NewObject()
for key, raw := range snapshots {
for key, raw := range input.snapshots {
value, err := jsonToJSValue(runtime, raw)
if err != nil {
return nil, fmt.Errorf("snapshot %q: %w", key, err)
@@ -30,7 +41,19 @@ func stateObject(runtime *goja.Runtime, snapshots Snapshots, tree *hierarchy.Tre
if err := state.Set("snapshots", snapshotsObject); err != nil {
return nil, err
}
if err := state.Set("ax", accessibilityObject(runtime, tree)); err != nil {
if err := state.Set("ax", accessibilityObject(runtime, input.tree)); err != nil {
return nil, err
}
if err := state.Set("lastAction", lastActionObject(runtime, input.lastAction)); err != nil {
return nil, err
}
if err := state.Set("time", runtimeMillis(input.stepTime, input.runStart)); err != nil {
return nil, err
}
if err := state.Set("logs", logsArray(runtime, input.logs)); err != nil {
return nil, err
}
if err := state.Set("exceptions", exceptionsArray(runtime, input.exceptions)); err != nil {
return nil, err
}
return state, nil
@@ -88,6 +111,74 @@ func elementObject(runtime *goja.Runtime, element *hierarchy.Element, selector s
return object
}
func lastActionObject(runtime *goja.Runtime, action *Action) goja.Value {
if action == nil {
return goja.Null()
}
object := runtime.NewObject()
_ = object.Set("kind", string(action.Kind))
if action.On != "" {
_ = object.Set("on", action.On)
}
if action.Text != "" {
_ = object.Set("text", action.Text)
}
switch action.Kind {
case ActionKindSwipe:
from := runtime.NewObject()
_ = from.Set("x", action.FromX)
_ = from.Set("y", action.FromY)
to := runtime.NewObject()
_ = to.Set("x", action.ToX)
_ = to.Set("y", action.ToY)
_ = object.Set("from", from)
_ = object.Set("to", to)
if action.DurationMillis > 0 {
_ = object.Set("durationMillis", action.DurationMillis)
}
case ActionKindPressKey:
_ = object.Set("key", action.Key)
case ActionKindWait:
_ = object.Set("durationMillis", action.DurationMillis)
}
return object
}
func runtimeMillis(stepTime, runStart time.Time) int64 {
if stepTime.IsZero() || runStart.IsZero() {
return 0
}
return stepTime.Sub(runStart).Milliseconds()
}
func logsArray(runtime *goja.Runtime, logs []LogEntry) *goja.Object {
array := runtime.NewArray()
for index, entry := range logs {
item := runtime.NewObject()
_ = item.Set("unixMillis", entry.UnixMillis)
_ = item.Set("level", entry.Level)
_ = item.Set("tag", entry.Tag)
_ = item.Set("message", entry.Message)
_ = array.Set(fmt.Sprintf("%d", index), item)
}
return array
}
func exceptionsArray(runtime *goja.Runtime, exceptions []Exception) *goja.Object {
array := runtime.NewArray()
for index, exception := range exceptions {
item := runtime.NewObject()
_ = item.Set("class", exception.Class)
_ = item.Set("message", exception.Message)
_ = item.Set("stackTrace", exception.StackTrace)
if exception.UnixMillis > 0 {
_ = item.Set("unixMillis", exception.UnixMillis)
}
_ = array.Set(fmt.Sprintf("%d", index), item)
}
return array
}
func jsonToJSValue(runtime *goja.Runtime, raw json.RawMessage) (goja.Value, error) {
if len(raw) == 0 {
return goja.Undefined(), nil
@@ -99,7 +190,7 @@ func jsonToJSValue(runtime *goja.Runtime, raw json.RawMessage) (goja.Value, erro
return runtime.ToValue(generic), nil
}
// jsValueToAction converts a JS-side {kind, on?, into?, text?} into a Go Action.
// jsValueToAction converts a JS-side action object into a Go Action.
func jsValueToAction(runtime *goja.Runtime, value goja.Value) (Action, error) {
if value == nil || goja.IsNull(value) || goja.IsUndefined(value) {
return Action{}, fmt.Errorf("nil action")
@@ -120,11 +211,60 @@ func jsValueToAction(runtime *goja.Runtime, value goja.Value) (Action, error) {
text := object.Get("text")
x, y := coordinatesOf(runtime, into)
return Action{Kind: ActionKindInputText, On: selectorOf(runtime, into), Text: stringOf(text), X: x, Y: y}, nil
case "Swipe":
from := object.Get("from")
to := object.Get("to")
fromX, fromY := coordinatesOf(runtime, from)
toX, toY := coordinatesOf(runtime, to)
if fromX == 0 && fromY == 0 {
fromX, fromY = pointCoordinates(runtime, from)
}
if toX == 0 && toY == 0 {
toX, toY = pointCoordinates(runtime, to)
}
return Action{
Kind: ActionKindSwipe,
FromX: fromX,
FromY: fromY,
ToX: toX,
ToY: toY,
DurationMillis: intField(object, "durationMillis"),
}, nil
case "PressKey":
return Action{Kind: ActionKindPressKey, Key: stringOf(object.Get("key"))}, nil
case "Wait":
return Action{Kind: ActionKindWait, DurationMillis: intField(object, "durationMillis")}, nil
default:
return Action{}, fmt.Errorf("unknown action kind %q", kind)
}
}
// pointCoordinates reads a plain {x, y} literal (not an AX element), which is
// how Swipe endpoints are commonly expressed in specs.
func pointCoordinates(runtime *goja.Runtime, value goja.Value) (int, int) {
if value == nil || goja.IsNull(value) || goja.IsUndefined(value) {
return 0, 0
}
object := value.ToObject(runtime)
if object == nil {
return 0, 0
}
x := object.Get("x")
y := object.Get("y")
if x == nil || y == nil {
return 0, 0
}
return int(x.ToInteger()), int(y.ToInteger())
}
func intField(object *goja.Object, name string) int {
value := object.Get(name)
if value == nil || goja.IsUndefined(value) || goja.IsNull(value) {
return 0
}
return int(value.ToInteger())
}
func selectorOf(runtime *goja.Runtime, value goja.Value) string {
if value == nil || goja.IsNull(value) || goja.IsUndefined(value) {
return ""
+55 -66
View File
@@ -14,10 +14,9 @@ const sampleAppHierarchyXML = `<?xml version="1.0" encoding="UTF-8"?>
<hierarchy rotation="0">
<node index="0" class="android.widget.FrameLayout" package="dev.uatu.sample" bounds="[0,0][1080,2400]">
<node index="0" class="android.widget.LinearLayout" bounds="[64,96][1016,2336]">
<node index="0" class="android.widget.TextView" text="Clicks: 0" bounds="[100,200][900,300]" />
<node index="1" class="android.widget.Button" text="Click me" clickable="true" enabled="true" bounds="[400,800][680,920]" />
<node index="2" class="android.widget.TextView" text="Username: " bounds="[100,1000][900,1080]" />
<node index="3" class="android.widget.EditText" content-desc="username_field" clickable="true" enabled="true" bounds="[100,1200][900,1320]" />
<node index="0" class="android.widget.TextView" text="Sign in" bounds="[100,200][900,300]" />
<node index="1" class="android.widget.EditText" content-desc="phone_field" clickable="true" enabled="true" bounds="[100,400][900,520]" />
<node index="2" class="android.widget.Button" text="Continue" clickable="true" enabled="true" bounds="[400,800][680,920]" />
</node>
</node>
</hierarchy>`
@@ -34,9 +33,16 @@ func bundleSampleAppSpec(t *testing.T) string {
if err != nil {
t.Fatal(err)
}
defaultsPath, err := filepath.Abs("../../pkg/spec-api/src/defaults/properties.ts")
if err != nil {
t.Fatal(err)
}
bundle, err := bundler.Bundle(bundler.Options{
EntryFile: specPath,
Aliases: map[string]string{"@uatu/spec": apiPath},
Aliases: map[string]string{
"@uatu/spec": apiPath,
"@uatu/spec/defaults/properties": defaultsPath,
},
})
if err != nil {
t.Fatal(err)
@@ -44,9 +50,10 @@ func bundleSampleAppSpec(t *testing.T) string {
return string(bundle.JavaScript)
}
// TestSampleAppSpecTapsClickMe verifies the bundled sample-app spec emits a
// Tap on the "Click me" button when that button is present in the hierarchy.
func TestSampleAppSpecTapsClickMe(t *testing.T) {
// TestSampleAppSpecFiresLoginActions verifies the bundled sample-app spec
// emits Tap/InputText actions targeting the login screen elements when they
// are present in the hierarchy.
func TestSampleAppSpecFiresLoginActions(t *testing.T) {
v := newVerifier(t)
if err := v.Load(bundleSampleAppSpec(t)); err != nil {
t.Fatal(err)
@@ -57,85 +64,67 @@ func TestSampleAppSpecTapsClickMe(t *testing.T) {
t.Fatal(err)
}
snapshots := Snapshots{
"app_state": json.RawMessage(`"running"`),
"click_count": json.RawMessage(`0`),
"route": json.RawMessage(`"login"`),
"logged_in": json.RawMessage(`false`),
"account_count": json.RawMessage(`0`),
}
if err := v.PushSnapshot(snapshots, tree); err != nil {
if err := v.PushSnapshot(SnapshotInput{Snapshots: snapshots, Tree: tree}); err != nil {
t.Fatal(err)
}
tapHits := 0
inputHits := 0
tapContinueHits := 0
typePhoneHits := 0
for range 400 {
action, err := v.NextAction()
if err != nil {
continue
}
switch {
case action.Kind == ActionKindTap && action.On == "text:Click me":
tapHits++
case action.Kind == ActionKindInputText && action.On == "desc:username_field" && action.Text == "alice":
inputHits++
case action.Kind == ActionKindTap && action.On == "text:Continue":
tapContinueHits++
case action.Kind == ActionKindInputText && action.On == "desc:phone_field":
typePhoneHits++
}
}
if tapHits == 0 {
t.Fatal("tapClickMe never fired on sample-app hierarchy")
if tapContinueHits == 0 {
t.Fatal("tapContinue never fired on sample-app hierarchy")
}
if inputHits == 0 {
t.Fatal("typeUsername never fired on sample-app hierarchy")
if typePhoneHits == 0 {
t.Fatal("typePhone never fired on sample-app hierarchy")
}
}
// TestSampleAppSpecPropertiesHold checks the three properties declared in the
// sample-app spec evaluate correctly across a realistic snapshot sequence.
func TestSampleAppSpecPropertiesHold(t *testing.T) {
// TestSampleAppSpecPropertiesEvaluate checks the properties declared in the
// sample-app spec evaluate sensibly across a small snapshot sequence. The
// spec mixes safety and liveness properties; Pending verdicts are expected
// for liveness properties that haven't had time to resolve yet.
func TestSampleAppSpecPropertiesEvaluate(t *testing.T) {
v := newVerifier(t)
if err := v.Load(bundleSampleAppSpec(t)); err != nil {
t.Fatal(err)
}
steps := []struct {
appState string
clickCount int
username string
want map[string]ltl.Verdict
}{
{"running", 0, "", map[string]ltl.Verdict{
"appIsRunning": ltl.VerdictHolds,
"clickCountNonNegative": ltl.VerdictHolds,
"clickCountNeverDecreases": ltl.VerdictHolds,
"usernameNeverShrinks": ltl.VerdictHolds,
}},
{"running", 5, "alice", map[string]ltl.Verdict{
"appIsRunning": ltl.VerdictHolds,
"clickCountNonNegative": ltl.VerdictHolds,
"clickCountNeverDecreases": ltl.VerdictHolds,
"usernameNeverShrinks": ltl.VerdictHolds,
}},
{"running", 3, "al", map[string]ltl.Verdict{
"appIsRunning": ltl.VerdictHolds,
"clickCountNonNegative": ltl.VerdictHolds,
"clickCountNeverDecreases": ltl.VerdictViolated,
"usernameNeverShrinks": ltl.VerdictViolated,
}},
tree, err := hierarchy.Parse(sampleAppHierarchyXML)
if err != nil {
t.Fatal(err)
}
for index, step := range steps {
stateRaw, _ := json.Marshal(step.appState)
countRaw, _ := json.Marshal(step.clickCount)
usernameRaw, _ := json.Marshal(step.username)
if err := v.PushSnapshot(Snapshots{
"app_state": stateRaw,
"click_count": countRaw,
"username": usernameRaw,
}, nil); err != nil {
t.Fatalf("step %d: %v", index, err)
}
got := v.EvaluateProperties()
for property, want := range step.want {
if got[property] != want {
t.Errorf("step %d %q: got %v, want %v", index, property, got[property], want)
}
}
snapshots := Snapshots{
"route": json.RawMessage(`"login"`),
"logged_in": json.RawMessage(`false`),
"account_count": json.RawMessage(`0`),
}
if err := v.PushSnapshot(SnapshotInput{Snapshots: snapshots, Tree: tree}); err != nil {
t.Fatal(err)
}
verdicts := v.EvaluateProperties()
if verdicts["accountCountNonNegative"] != ltl.VerdictHolds {
t.Errorf("accountCountNonNegative: got %v, want holds", verdicts["accountCountNonNegative"])
}
if verdicts["noUncaughtExceptions"] != ltl.VerdictHolds {
t.Errorf("noUncaughtExceptions: got %v, want holds", verdicts["noUncaughtExceptions"])
}
// Liveness: eventuallyLoggedIn hasn't resolved yet.
if verdicts["eventuallyLoggedIn"] != ltl.VerdictPending {
t.Errorf("eventuallyLoggedIn: got %v, want pending", verdicts["eventuallyLoggedIn"])
}
}
+9 -9
View File
@@ -60,10 +60,10 @@ func TestPushSnapshot_UpdatesExtractorCurrentAndPrevious(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
if err := verifier.PushSnapshot(Snapshots{
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{
"screen": json.RawMessage(`"customer_ledger"`),
"ledger.balance": json.RawMessage(`1500`),
}, nil); err != nil {
}}); err != nil {
t.Fatal(err)
}
@@ -78,7 +78,7 @@ func TestPushSnapshot_UpdatesExtractorCurrentAndPrevious(t *testing.T) {
}
// Push again: previous should mirror the prior current.
if err := verifier.PushSnapshot(Snapshots{"ledger.balance": json.RawMessage(`2000`)}, nil); err != nil {
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": json.RawMessage(`2000`)}}); err != nil {
t.Fatal(err)
}
balanceValue = verifier.runtime.GlobalObject().Get("balance").ToObject(verifier.runtime)
@@ -105,7 +105,7 @@ func TestEvaluateProperties_HoldsThenViolates(t *testing.T) {
}
for index, testCase := range cases {
raw, _ := json.Marshal(testCase.balance)
if err := verifier.PushSnapshot(Snapshots{"ledger.balance": raw}, nil); err != nil {
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"ledger.balance": raw}}); err != nil {
t.Fatal(err)
}
verdicts := verifier.EvaluateProperties()
@@ -118,7 +118,7 @@ func TestEvaluateProperties_HoldsThenViolates(t *testing.T) {
func TestNextAction_FromActionsGenerator(t *testing.T) {
verifier := newVerifier(t)
mustLoad(t, verifier, helloSpec)
_ = verifier.PushSnapshot(Snapshots{}, nil)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
action, err := verifier.NextAction()
if err != nil {
@@ -142,7 +142,7 @@ func TestNextAction_WeightedSelectsByWeight(t *testing.T) {
[99, tapAway],
);
`)
_ = verifier.PushSnapshot(Snapshots{}, nil)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
awayCount := 0
homeCount := 0
@@ -168,7 +168,7 @@ func TestNextAction_EmptyGeneratorReturnsErrNoAction(t *testing.T) {
mustLoad(t, verifier, `
globalThis.actions = __uatu__.actions(() => []);
`)
_ = verifier.PushSnapshot(Snapshots{}, nil)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
_, err := verifier.NextAction()
if !errors.Is(err, ErrNoAction) {
@@ -183,7 +183,7 @@ func TestInputText_RoundTrip(t *testing.T) {
__uatu__.inputText({ into: "id:phone", text: "+919876543210" }),
]);
`)
_ = verifier.PushSnapshot(Snapshots{}, nil)
_ = verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{}})
action, err := verifier.NextAction()
if err != nil {
@@ -202,7 +202,7 @@ func TestPushSnapshot_FeedsSnapshotsToExtractorState(t *testing.T) {
mustLoad(t, verifier, `
globalThis.captured = __uatu__.extract(state => state.snapshots["k"]);
`)
if err := verifier.PushSnapshot(Snapshots{"k": json.RawMessage(`"hello"`)}, nil); err != nil {
if err := verifier.PushSnapshot(SnapshotInput{Snapshots: Snapshots{"k": json.RawMessage(`"hello"`)}}); err != nil {
t.Fatal(err)
}
value := verifier.runtime.GlobalObject().Get("captured").ToObject(verifier.runtime).Get("current")
+253 -15
View File
@@ -4,6 +4,7 @@ import (
"errors"
"fmt"
"math/rand/v2"
"time"
"github.com/dop251/goja"
@@ -12,15 +13,23 @@ import (
)
type Verifier struct {
runtime *goja.Runtime
extractors []*extractorState
formulas []*formulaState
runtime *goja.Runtime
extractors []*extractorState
formulas []*formulaState
formulaSpecs []formulaSpec
properties map[string]int // property name -> formula index
properties map[string]int // property name -> formula-spec index
actionGenerator goja.Value
evaluators map[string]*ltl.Evaluator
lastTree *hierarchy.Tree
lastAction *Action
lastLogs []LogEntry
lastExceptions []Exception
stepTime time.Time
runStart time.Time
rng *rand.Rand
}
@@ -62,13 +71,16 @@ func (v *Verifier) Load(source string) error {
if handle == nil {
return fmt.Errorf("property %q is not an object", name)
}
indexValue := handle.Get("__uatuIndex")
if indexValue == nil {
specIndex, ok := v.extractSpecIndex(handle)
if !ok {
return fmt.Errorf("property %q was not produced by always()", name)
}
index := int(indexValue.ToInteger())
v.properties[name] = index
v.evaluators[name] = ltl.NewEvaluator(ltl.Always(ltl.Thunk(v.formulaThunk(index))))
formula, err := v.buildFormula(specIndex)
if err != nil {
return fmt.Errorf("property %q: %w", name, err)
}
v.properties[name] = specIndex
v.evaluators[name] = ltl.NewEvaluator(formula)
}
}
@@ -79,11 +91,125 @@ func (v *Verifier) Load(source string) error {
return nil
}
// buildFormula walks the formula-spec registry and produces a Go ltl.Formula
// tree rooted at the given spec index. Specs built at the top level are
// always wrapped in Always unless the top-level spec is already an Always.
func (v *Verifier) buildFormula(rootIndex int) (ltl.Formula, error) {
inner, err := v.buildFormulaNode(rootIndex)
if err != nil {
return nil, err
}
if _, ok := inner.(ltl.AlwaysFormula); ok {
return inner, nil
}
return ltl.Always(inner), nil
}
func (v *Verifier) buildFormulaNode(index int) (ltl.Formula, error) {
if index < 0 || index >= len(v.formulaSpecs) {
return nil, fmt.Errorf("formula spec index %d out of range", index)
}
spec := v.formulaSpecs[index]
switch spec.kind {
case specKindPure:
return ltl.Pure(spec.pureValue), nil
case specKindThunk:
return ltl.Thunk(v.formulaThunk(spec.predicateIndex)), nil
case specKindNow:
child, err := v.buildFormulaNode(spec.childA)
if err != nil {
return nil, err
}
return ltl.Now(child), nil
case specKindNext:
child, err := v.buildFormulaNode(spec.childA)
if err != nil {
return nil, err
}
return ltl.Next(child), nil
case specKindEventually:
child, err := v.buildFormulaNode(spec.childA)
if err != nil {
return nil, err
}
formula := ltl.EventuallyFormula{Inner: child}
if spec.hasStepBound {
formula.StepBound = spec.stepBound
formula.HasStepBound = true
}
if spec.duration > 0 {
formula.Duration = spec.duration
}
return formula, nil
case specKindImplies:
left, err := v.buildFormulaNode(spec.childA)
if err != nil {
return nil, err
}
right, err := v.buildFormulaNode(spec.childB)
if err != nil {
return nil, err
}
return ltl.Implies(left, right), nil
case specKindOr:
left, err := v.buildFormulaNode(spec.childA)
if err != nil {
return nil, err
}
right, err := v.buildFormulaNode(spec.childB)
if err != nil {
return nil, err
}
return ltl.Or(left, right), nil
case specKindAnd:
left, err := v.buildFormulaNode(spec.childA)
if err != nil {
return nil, err
}
right, err := v.buildFormulaNode(spec.childB)
if err != nil {
return nil, err
}
return ltl.And(left, right), nil
case specKindNot:
child, err := v.buildFormulaNode(spec.childA)
if err != nil {
return nil, err
}
return ltl.Not(child), nil
case specKindAlways:
child, err := v.buildFormulaNode(spec.childA)
if err != nil {
return nil, err
}
return ltl.Always(child), nil
default:
return nil, fmt.Errorf("unknown formula spec kind %d", spec.kind)
}
}
// PushSnapshot updates the JS-side state and refreshes every extractor's
// current/previous values in registration order. Passing a nil tree is
// allowed and yields an empty ax scope.
func (v *Verifier) PushSnapshot(snapshots Snapshots, tree *hierarchy.Tree) error {
state, err := stateObject(v.runtime, snapshots, tree)
func (v *Verifier) PushSnapshot(input SnapshotInput) error {
v.lastTree = input.Tree
v.lastAction = input.LastAction
v.lastLogs = input.Logs
v.lastExceptions = input.Exceptions
v.stepTime = input.StepTime
if v.runStart.IsZero() {
v.runStart = input.RunStart
}
state, err := stateObject(v.runtime, stateInput{
snapshots: input.Snapshots,
tree: input.Tree,
lastAction: input.LastAction,
stepTime: input.StepTime,
runStart: v.runStart,
logs: input.Logs,
exceptions: input.Exceptions,
})
if err != nil {
return fmt.Errorf("build state: %w", err)
}
@@ -102,12 +228,47 @@ func (v *Verifier) PushSnapshot(snapshots Snapshots, tree *hierarchy.Tree) error
return nil
}
// SnapshotInput bundles everything a step feeds into the verifier. Fields
// other than Snapshots are optional; callers that only have snapshots can
// populate Snapshots alone and leave the rest zero.
type SnapshotInput struct {
Snapshots Snapshots
Tree *hierarchy.Tree
LastAction *Action
StepTime time.Time
RunStart time.Time
Logs []LogEntry
Exceptions []Exception
}
// LogEntry mirrors a logcat line captured between steps.
type LogEntry struct {
UnixMillis int64
Level string
Tag string
Message string
}
// Exception mirrors an SDK-captured uncaught throwable.
type Exception struct {
Class string
Message string
StackTrace string
UnixMillis int64
}
// EvaluateProperties returns each registered property's running verdict
// after the most recent PushSnapshot.
// after the most recent PushSnapshot. The step time passed in PushSnapshot is
// forwarded to each evaluator so deadline-bound operators see the snapshot's
// wall clock rather than time.Now().
func (v *Verifier) EvaluateProperties() map[string]ltl.Verdict {
verdicts := map[string]ltl.Verdict{}
stepTime := v.stepTime
if stepTime.IsZero() {
stepTime = time.Now()
}
for name, evaluator := range v.evaluators {
verdicts[name] = evaluator.Observe()
verdicts[name] = evaluator.ObserveAt(stepTime)
}
return verdicts
}
@@ -178,13 +339,90 @@ func (v *Verifier) resolveGenerator(generator goja.Value) (Action, error) {
return Action{}, err
}
return v.resolveGenerator(picked)
case internalKindBuiltinTaps, internalKindBuiltinSwipes:
return Action{}, ErrNoAction
case internalKindBuiltinTaps:
return v.generateRandomTap()
case internalKindBuiltinSwipes:
return v.generateRandomSwipe()
case internalKindBuiltinWaitOnce:
return Action{Kind: ActionKindWait, DurationMillis: 500}, nil
case internalKindBuiltinPressKey:
return v.generateRandomPressKey()
default:
return Action{}, fmt.Errorf("unknown generator kind %q", kindValue.String())
}
}
// generateRandomTap picks a visible, tappable element from the last
// hierarchy snapshot and returns a Tap action targeting its center.
func (v *Verifier) generateRandomTap() (Action, error) {
if v.lastTree == nil {
return Action{}, ErrNoAction
}
candidates := make([]*hierarchy.Element, 0, len(v.lastTree.Elements))
for _, element := range v.lastTree.Elements {
if !element.Clickable || !element.Enabled {
continue
}
if element.Bounds.Right-element.Bounds.Left <= 0 || element.Bounds.Bottom-element.Bounds.Top <= 0 {
continue
}
candidates = append(candidates, element)
}
if len(candidates) == 0 {
return Action{}, ErrNoAction
}
picked := candidates[v.rng.IntN(len(candidates))]
x, y := picked.Bounds.Center()
return Action{Kind: ActionKindTap, X: x, Y: y}, nil
}
// generateRandomSwipe emits a swipe over a random enabled element or the
// whole screen, in a random direction. Returns ErrNoAction only when we have
// no tree to size a gesture off of.
func (v *Verifier) generateRandomSwipe() (Action, error) {
if v.lastTree == nil || len(v.lastTree.Elements) == 0 {
return Action{}, ErrNoAction
}
element := v.lastTree.Elements[v.rng.IntN(len(v.lastTree.Elements))]
cx, cy := element.Bounds.Center()
if cx <= 0 || cy <= 0 {
return Action{}, ErrNoAction
}
// Pick a direction: 0=up 1=down 2=left 3=right; magnitude 200-600 px.
magnitude := 200 + v.rng.IntN(401)
toX, toY := cx, cy
switch v.rng.IntN(4) {
case 0:
toY = cy - magnitude
case 1:
toY = cy + magnitude
case 2:
toX = cx - magnitude
case 3:
toX = cx + magnitude
}
if toX < 0 {
toX = 0
}
if toY < 0 {
toY = 0
}
return Action{
Kind: ActionKindSwipe,
FromX: cx,
FromY: cy,
ToX: toX,
ToY: toY,
DurationMillis: 250,
}, nil
}
func (v *Verifier) generateRandomPressKey() (Action, error) {
// Keep exploration gentle: only "back" for now. Home/menu would navigate
// away from the app under test.
return Action{Kind: ActionKindPressKey, Key: "back"}, nil
}
func (v *Verifier) pickFromResult(result goja.Value) (Action, error) {
if result == nil || goja.IsUndefined(result) || goja.IsNull(result) {
return Action{}, ErrNoAction