mirror of
https://github.com/priyanshujain/sanderling.git
synced 2026-10-02 19:17:10 +00:00
iOS simulator driver: Go-native companion-backed backend (#62)
* perf(ios): use prebuilt XCTest runner to cut startup * chore(ioscompanion): add companion asset prepare script * feat(ioscompanion): embed and extract simulator companion bundle * test(ioscompanion): cover companion stub and embedded extraction * docs: add third party notices for vendored companion * chore: ignore vendored companion bundle artifact * build(proto): pin simulator companion proto v1.1.8 * build(proto): add dedicated buf module and gen template for pinned proto * build(proto): exclude pinned companion proto from root buf workspace * feat(ioscompanion): commit generated companion gRPC stubs * feat(ioscompanion): map flat companion describe dump to TreeNode JSON * test(ioscompanion): add hierarchy-map golden and unit tests * feat(ioscompanion): port screen-settle stability polling to Go * test(ioscompanion): cover settle transitional, hash, streak, and cap rules * feat(ioscompanion): add USB HID keymap module * test(ioscompanion): cover keymap branches and paste-chord constants * build: embed companion assets via withcompanion tag * feat(ioscompanion): add transport companion interface * feat(ioscompanion): add HID event wrapper and builders * feat(ioscompanion): wire gRPC companion client and Dial * test(ioscompanion): cover HID builders and unit conversions * test(ioscompanion): cover Dial, process-state mapping, and install archive * test(ioscompanion): add gated simulator integration smoke test * feat(ioscompanion): text input and gesture HID composition with pasteboard fallback * test(ioscompanion): cover input composers, paste dialog loop, and pure helpers * feat(ioscompanion): add Describe to companion transport * feat(ioscompanion): implement DeviceDriver with companion supervision * test(ioscompanion): unit tests with fake companion transport * test(ioscompanion): gated companion smoke test * feat(ios): add ResolveTarget for simulator vs physical-device routing * feat(testrun): route iOS simulators through the native companion driver * refactor(testrun): defer the java preflight check to the physical-device path * feat(cli): add --ios-app-path flag * feat(doctor): split iOS checks into simulator and physical-device paths * test(folio): add gate-analyzer fixtures for G1-G5 * feat(folio): add iOS conformance gate script * chore(folio): wire gates recipe, app path, and ignore gate output * style: gofmt struct alignment drift * fix(doctor): probe simctl via xcrun instead of PATH lookup * fix(ioscompanion): spawn companion under driver-lifetime context * test(ioscompanion): prove companion child outlives startup context * fix(ioscompanion): chunk install payload under companion message cap * test(ioscompanion): cover install payload chunking * fix(ioscompanion): reinstall via simctl and sanitize companion env * fix(ioscompanion): wait out unresolved accessibility values after launch * perf(ioscompanion): paste long text for atomic landing * test(ioscompanion): cover paste threshold, retry flow, and sentinel detection * fix(ioscompanion): treat unresolved bridge values as transitional, never as content * fix(ioscompanion): accept masked secure-field values as paste landing * test(ioscompanion): cover sentinel mapping and masked-field landing * fix(ioscompanion): atomic erase and single-send paste to prevent doubling * test(ioscompanion): cover atomic erase, single chord, unverifiable field * fix(ioscompanion): verify paste on a time budget that outlasts the bridge blackout * test(ioscompanion): cover bridge-blackout paste verification * fix(ioscompanion): drop unresolved-value settle gate that never let empty-field screens settle * refactor(ioscompanion): name the empty-editable-field sentinel for what it is * perf(ioscompanion): tighten settle streak for the fast companion transport * feat(ioscompanion): pre-grant pasteboard access so unicode input skips the OS prompt * refactor(ioscompanion): drop paste warm-up now that the grant suppresses the prompt * test(ioscompanion): cover pasteboard grant on launch, drop warm-up tests * fix(ioscompanion): retry describe past transient collapsed accessibility dumps * test(ioscompanion): cover collapsed-dump detection * perf(ioscompanion): split raw and retrying describe so settle does not double-wait collapses * perf(ioscompanion): tighten settle now that collapses are handled separately * fix(ioscompanion): replace field content on input so blackout-skipped erase cannot accumulate text * test(ioscompanion): cover replace-on-input and TextReplacer capability * refactor(ioscompanion): neutralize HID events behind the transport seam * feat(companion): add simulator runner project skeleton * feat(companion): serve accessibility snapshots over the wire protocol * feat(companion): synthesize timestamped touch gestures * feat(companion): type text with replace semantics * feat(companion): serve the wire protocol from a parked runner * feat(ioscompanion): add TextEditor capability and unavailable sentinel to the transport seam * feat(ioscompanion): route text input through a text-editing companion when available * fix(companion): bind listener by port and source screen size from snapshot * feat(ioscompanion): add runner companion JSON transport * test(ioscompanion): cover runner transport protocol mapping * fix(companion): synthesize gestures synchronously to avoid the async completion crash * fix(companion): type on the main thread and recover from focus assertions * fix(companion): keep serving after an automation failure * refactor(companion): tidy snapshot serialization * fix(companion): honor sequential tap gaps and survive synthesis exceptions * feat(ioscompanion): expose native typing with an explicit replace flag * chore(companion): add runner asset prepare script * feat(ioscompanion): embed and extract the runner test bundle * test(ioscompanion): cover runner asset extraction * build(ioscompanion): commit runner asset archive * feat(ioscompanion): pair the legacy companion with the in-simulator runner * test(ioscompanion): cover hybrid routing, paste-grant skip, and port binding * fix(ioscompanion): reconnect after interrupted runner calls instead of restarting * fix(ioscompanion): route hybrid lifecycle through the runner and harden restarts * feat(companion): launch and terminate apps through the automation session * build(ioscompanion): refresh runner asset with session lifecycle * fix(ioscompanion): classify connection deadline expiry as caller budget * fix(companion): capture snapshots on the main thread inside the catch bridge * build(ioscompanion): refresh runner asset with main-thread snapshots * perf(ioscompanion): count read spans toward settle and capture snapshots concurrently * feat(ioscompanion): make the hybrid simulator companion the default * test(folio): cover runner-session orphans in the gate harness * test(ioscompanion): pin the child-lifetime test to the legacy path * fix(ioscompanion): keep mappable text on one HID stream and verify unicode clears * fix(ioscompanion): pause the clear chord so selection applies before the delete * fix(companion): prune the keyboard subtree from snapshots * build(ioscompanion): refresh runner asset without keyboard elements * fix(ioscompanion): capture the screenshot transport before a recovery can reassign it * fix(companion): pin the runner listener to loopback * fix(companion): size the replace delete prefix to cover any focused field * build(ioscompanion): refresh runner asset with loopback bind and replace fix * fix(cli): cancel the run context on SIGINT so spawned children are reaped * fix(testrun): point the device java preflight hint at the ios-device doctor * fix(folio): word-bound the G2 ERROR scan and drop the dead objc allowlist glob * test(ioscompanion): cover stopProcess, restart, and failed bring-up supervision * chore: add test-companion target for the withcompanion-tagged suite * chore(ioscompanion): stop tracking the runner archive build artifact * build: produce the runner archive from source like the companion bundle * refactor(conformance): move the gate harness out of examples/folio * chore(folio): drop the gate harness wiring from the example app
This commit is contained in:
97 files changed
+22104
-83
No files matched your search
@@ -0,0 +1 @@
|
||||
runs/
|
||||
Executable
+405
@@ -0,0 +1,405 @@
|
||||
#!/usr/bin/env bash
|
||||
# Scripted conformance gate for the iOS simulator driver. Runs five serial,
|
||||
# non-overlapping 3-minute fuzz runs against the folio example app
|
||||
# (examples/folio) and scores five gates (G1..G5) over the captured traces and
|
||||
# output. Exits non-zero if any gate fails.
|
||||
#
|
||||
# Backends:
|
||||
# BACKEND=simulator (default) drive the booted iOS simulator
|
||||
# BACKEND=device drive an attached physical iPhone via the
|
||||
# native sidecar; select it with
|
||||
# IOS_DEVICE="<name>" (passed as --ios-device)
|
||||
#
|
||||
# Usage:
|
||||
# ./gates.sh run the simulator gates
|
||||
# BACKEND=device IOS_DEVICE="iPhone" ./gates.sh
|
||||
# ./gates.sh --self-test run the offline analyzer tests only
|
||||
#
|
||||
# Tunables (environment):
|
||||
# RUNS=5 number of serial runs
|
||||
# DURATION=3m per-run fuzz duration
|
||||
# SEED=0 fuzz seed
|
||||
# P95_LIMIT_MS=2500 G5 p95 step-latency ceiling in milliseconds
|
||||
# SANDERLING=sanderling binary to invoke
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
script_directory="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
folio_directory="$(cd "${script_directory}/../examples/folio" && pwd)"
|
||||
|
||||
BACKEND="${BACKEND:-simulator}"
|
||||
RUNS="${RUNS:-5}"
|
||||
DURATION="${DURATION:-3m}"
|
||||
SEED="${SEED:-0}"
|
||||
P95_LIMIT_MS="${P95_LIMIT_MS:-2500}"
|
||||
SANDERLING="${SANDERLING:-sanderling}"
|
||||
IOS_DEVICE="${IOS_DEVICE:-iPhone 17 Pro}"
|
||||
|
||||
bundle_id="app.folio"
|
||||
spec_path="${folio_directory}/sanderling/spec.ts"
|
||||
ios_app="${folio_directory}/app/iosApp/build/Build/Products/Debug-iphonesimulator/iosApp.app"
|
||||
|
||||
# The companion binary, embedded for simulator runs. Referenced by file name
|
||||
# only for the orphan-process check; prose elsewhere says "the companion".
|
||||
companion_process_name="idb_companion"
|
||||
|
||||
# Known-benign stderr lines the companion always prints. These are matched as
|
||||
# fixed substrings and excluded from the G2 ERROR scan. Keep this list tight:
|
||||
# only lines that are provably harmless and emitted on every healthy run.
|
||||
benign_stderr_substrings=(
|
||||
# The dynamic linker reports the same Objective-C class registered by two
|
||||
# loaded images. The companion runs fine; this is cosmetic. The real line
|
||||
# reads "objc[<pid>]: Class ...", so the fixed substring is "]: Class".
|
||||
"]: Class"
|
||||
"is implemented in both"
|
||||
"One of the two will be used. Which one is undefined."
|
||||
# gRPC and absl emit informational banner lines on startup.
|
||||
"WARNING: All log messages before absl::InitializeLog()"
|
||||
)
|
||||
|
||||
# ---- gate analyzers (pure, operate on a single run directory) --------------
|
||||
|
||||
# G2 helper: strip benign companion noise, then report sanderling ERROR lines.
|
||||
# sanderling's progress logger renders error-level records as lines beginning
|
||||
# "error:" (see internal/testrun/progress.go). We also catch the upper-case
|
||||
# ERROR token (word-bounded, so ERRORS/ERRORLESS and path fragments do not
|
||||
# trip the gate) for safety against future handlers.
|
||||
error_lines_in() {
|
||||
local output_file="$1"
|
||||
local filtered
|
||||
filtered="$(cat "$output_file")"
|
||||
local pattern
|
||||
for pattern in "${benign_stderr_substrings[@]}"; do
|
||||
filtered="$(printf '%s\n' "$filtered" | grep -vF "$pattern" || true)"
|
||||
done
|
||||
printf '%s\n' "$filtered" | grep -E '(^error:|\bERROR\b)' || true
|
||||
}
|
||||
|
||||
# G1: process exit status recorded by the runner loop.
|
||||
gate_exit_zero() {
|
||||
local run_directory="$1"
|
||||
[[ "$(cat "${run_directory}/exit_status")" == "0" ]]
|
||||
}
|
||||
|
||||
# G2: no sanderling ERROR lines after filtering benign companion noise.
|
||||
gate_no_error_lines() {
|
||||
local run_directory="$1"
|
||||
local found
|
||||
found="$(error_lines_in "${run_directory}/output.log")"
|
||||
[[ -z "$found" ]]
|
||||
}
|
||||
|
||||
# G3: the first hierarchy snapshot shows the login screen with empty email and
|
||||
# password fields (clear-state proof). Reads the first trace line carrying a
|
||||
# hierarchy and asserts LoginEmail/LoginPassword carry no text.
|
||||
gate_clear_state() {
|
||||
local run_directory="$1"
|
||||
local trace_file="${run_directory}/trace.jsonl"
|
||||
[[ -f "$trace_file" ]] || return 1
|
||||
local verdict
|
||||
verdict="$(jq -s -r '
|
||||
[ .[] | select(.hierarchy != null) ] as $withHierarchy
|
||||
| if ($withHierarchy | length) == 0 then "fail:no-hierarchy"
|
||||
else ($withHierarchy[0].hierarchy.elements // []) as $elements
|
||||
| ($elements | map(select(.resourceId == "LoginEmail")) | first) as $email
|
||||
| ($elements | map(select(.resourceId == "LoginPassword")) | first) as $password
|
||||
| if $email == null or $password == null then "fail:no-login-fields"
|
||||
elif (($email.attrs.text // $email.text // "") != "") then "fail:email-not-empty"
|
||||
elif (($password.attrs.text // $password.text // "") != "") then "fail:password-not-empty"
|
||||
else "pass" end
|
||||
end
|
||||
' "$trace_file")"
|
||||
[[ "$verdict" == "pass" ]]
|
||||
}
|
||||
|
||||
# G4: no doubled text after InputText. For each InputText action targeting a
|
||||
# field, the field's value in the NEXT hierarchy must not contain the input
|
||||
# concatenated with itself (catches append-vs-replace and double-paste bugs).
|
||||
# The action chosen at step N is applied before step N+1 is observed, so the
|
||||
# effect lands in the following snapshot.
|
||||
gate_no_doubled_text() {
|
||||
local run_directory="$1"
|
||||
local trace_file="${run_directory}/trace.jsonl"
|
||||
[[ -f "$trace_file" ]] || return 1
|
||||
local verdict
|
||||
verdict="$(jq -s -r '
|
||||
# Map a selector like "testTag:LoginScreen > testTag:LoginEmail" to its
|
||||
# target field id: the token after the final ":" of the last segment.
|
||||
def target_field(selector):
|
||||
(selector | split(">") | last | gsub("^\\s+|\\s+$";"")) as $last
|
||||
| ($last | split(":") | last);
|
||||
|
||||
[ .[] | select(.hierarchy != null) ] as $steps
|
||||
| reduce range(0; ($steps | length)) as $i ([];
|
||||
($steps[$i]) as $current
|
||||
| (if $i + 1 < ($steps | length) then $steps[$i + 1] else null end) as $next
|
||||
| if ($current.next_action.kind == "InputText") and ($next != null)
|
||||
then
|
||||
(target_field($current.next_action.selector // "")) as $field
|
||||
| ($current.next_action.text // "") as $typed
|
||||
| (($next.hierarchy.elements // [])
|
||||
| map(select(.resourceId == $field)) | first) as $element
|
||||
| if $element != null and $typed != ""
|
||||
then
|
||||
(($element.attrs.text // $element.text // "")) as $value
|
||||
| if ($value | contains($typed + $typed))
|
||||
then . + [{field: $field, typed: $typed, value: $value}]
|
||||
else . end
|
||||
else . end
|
||||
else . end)
|
||||
| if length == 0 then "pass"
|
||||
else "fail:" + (.[0].field) + ":" + (.[0].value) end
|
||||
' "$trace_file")"
|
||||
[[ "$verdict" == "pass" ]]
|
||||
}
|
||||
|
||||
# Emit one step-latency sample per consecutive trace-step pair, in
|
||||
# milliseconds, on stdout. The trace records a wall-clock timestamp per step;
|
||||
# the latency of a step is the gap to the next step's observation. Used by G5,
|
||||
# which aggregates samples across all runs before computing the p95. Timestamps
|
||||
# are RFC3339 with fractional seconds and a numeric offset, so they are parsed
|
||||
# with python3 rather than jq's UTC-only fromdateiso8601.
|
||||
emit_step_latencies() {
|
||||
local trace_file="$1"
|
||||
[[ -f "$trace_file" ]] || return 0
|
||||
jq -r 'select(.timestamp != null) | .timestamp' "$trace_file" \
|
||||
| python3 -c '
|
||||
import sys
|
||||
from datetime import datetime
|
||||
stamps = [datetime.fromisoformat(line.strip()) for line in sys.stdin if line.strip()]
|
||||
for earlier, later in zip(stamps, stamps[1:]):
|
||||
print(int((later - earlier).total_seconds() * 1000))
|
||||
'
|
||||
}
|
||||
|
||||
# Compute the p95 (nearest-rank) of the latency samples on stdin, in ms.
|
||||
p95_of() {
|
||||
python3 -c '
|
||||
import sys, math
|
||||
samples = sorted(int(float(line)) for line in sys.stdin if line.strip())
|
||||
if not samples:
|
||||
print(0)
|
||||
sys.exit(0)
|
||||
rank = max(1, math.ceil(0.95 * len(samples)))
|
||||
print(samples[rank - 1])
|
||||
'
|
||||
}
|
||||
|
||||
# G5 orphan check: report any lingering companion, runner session (the hybrid
|
||||
# simulator driver hosts an in-simulator runner), and, on the device backend,
|
||||
# the XCTest runner java sidecar. Empty output means clean.
|
||||
orphan_processes() {
|
||||
local found=""
|
||||
if pgrep -f "$companion_process_name" >/dev/null 2>&1; then
|
||||
found+="companion "
|
||||
fi
|
||||
if pgrep -f "sanderling-runner.*xctestrun" >/dev/null 2>&1; then
|
||||
found+="runner-session "
|
||||
fi
|
||||
if pgrep -f "CompanionRunnerUITests-Runner" >/dev/null 2>&1; then
|
||||
found+="runner-app "
|
||||
fi
|
||||
if [[ "$BACKEND" == "device" ]]; then
|
||||
# The native sidecar that drives the XCTest runner for physical devices.
|
||||
if pgrep -f "sanderling.*sidecar.jar" >/dev/null 2>&1; then
|
||||
found+="sidecar "
|
||||
fi
|
||||
fi
|
||||
printf '%s' "$found"
|
||||
}
|
||||
|
||||
# ---- run orchestration -----------------------------------------------------
|
||||
|
||||
invoke_sanderling() {
|
||||
local output_directory="$1"
|
||||
local output_log="$2"
|
||||
local exit_status_file="$3"
|
||||
|
||||
local target_flags=()
|
||||
if [[ "$BACKEND" == "device" ]]; then
|
||||
target_flags=(--ios-device "$IOS_DEVICE")
|
||||
else
|
||||
# Simulator: build + install the current app so each run starts from the
|
||||
# current build, matching the test-ios recipe. clear-state reinstall uses
|
||||
# --ios-app-path. just ios boots IOS_DEVICE if nothing is booted.
|
||||
target_flags=(--ios-device "$IOS_DEVICE" --ios-app-path "$ios_app")
|
||||
fi
|
||||
|
||||
local status=0
|
||||
"$SANDERLING" test \
|
||||
--platform ios \
|
||||
--spec "$spec_path" \
|
||||
--bundle-id "$bundle_id" \
|
||||
"${target_flags[@]}" \
|
||||
--duration "$DURATION" \
|
||||
--seed "$SEED" \
|
||||
--output "$output_directory" \
|
||||
>"$output_log" 2>&1 || status=$?
|
||||
printf '%s' "$status" >"$exit_status_file"
|
||||
}
|
||||
|
||||
# Locate the run directory sanderling created under output_directory (it nests
|
||||
# a timestamped subdirectory) and normalise its artifacts up one level.
|
||||
collect_run_artifacts() {
|
||||
local output_directory="$1"
|
||||
[[ -f "${output_directory}/trace.jsonl" ]] && return 0
|
||||
local produced
|
||||
produced="$(find "$output_directory" -mindepth 2 -name 'trace.jsonl' 2>/dev/null | head -1 || true)"
|
||||
if [[ -n "$produced" ]]; then
|
||||
cp "$produced" "${output_directory}/trace.jsonl" 2>/dev/null || true
|
||||
fi
|
||||
}
|
||||
|
||||
run_gates() {
|
||||
if [[ "$BACKEND" == "simulator" ]]; then
|
||||
echo "preparing folio build for the simulator backend"
|
||||
( cd "$folio_directory" && just ios >/dev/null )
|
||||
fi
|
||||
|
||||
local timestamp
|
||||
timestamp="$(date +%Y%m%d-%H%M%S)"
|
||||
local gate_root="${script_directory}/runs/${timestamp}"
|
||||
mkdir -p "$gate_root"
|
||||
echo "gate root: ${gate_root}"
|
||||
echo "backend=${BACKEND} runs=${RUNS} duration=${DURATION} seed=${SEED} p95_limit_ms=${P95_LIMIT_MS}"
|
||||
|
||||
local all_latencies="${gate_root}/all-latencies.txt"
|
||||
: >"$all_latencies"
|
||||
|
||||
local -a g1 g2 g3 g4 g5
|
||||
local run_index
|
||||
for ((run_index = 1; run_index <= RUNS; run_index++)); do
|
||||
local run_directory="${gate_root}/run-${run_index}"
|
||||
mkdir -p "$run_directory"
|
||||
echo "run ${run_index}/${RUNS} -> ${run_directory}"
|
||||
|
||||
invoke_sanderling "$run_directory" "${run_directory}/output.log" "${run_directory}/exit_status"
|
||||
collect_run_artifacts "$run_directory"
|
||||
|
||||
g1[run_index]=$(gate_exit_zero "$run_directory" && echo PASS || echo FAIL)
|
||||
g2[run_index]=$(gate_no_error_lines "$run_directory" && echo PASS || echo FAIL)
|
||||
g3[run_index]=$(gate_clear_state "$run_directory" && echo PASS || echo FAIL)
|
||||
g4[run_index]=$(gate_no_doubled_text "$run_directory" && echo PASS || echo FAIL)
|
||||
|
||||
emit_step_latencies "${run_directory}/trace.jsonl" >>"$all_latencies"
|
||||
|
||||
local orphans
|
||||
orphans="$(orphan_processes)"
|
||||
if [[ -n "$orphans" ]]; then
|
||||
g5[run_index]="FAIL"
|
||||
echo " orphaned processes after run ${run_index}: ${orphans}"
|
||||
else
|
||||
g5[run_index]="PENDING"
|
||||
fi
|
||||
done
|
||||
|
||||
local p95
|
||||
p95="$(p95_of <"$all_latencies")"
|
||||
local final_orphans
|
||||
final_orphans="$(orphan_processes)"
|
||||
|
||||
# G5 is global: p95 is computed over every run's samples and an orphan after
|
||||
# any single run fails the whole gate. Decide once, then stamp every row.
|
||||
local g5_global="PASS"
|
||||
if [[ "$p95" -ge "$P95_LIMIT_MS" ]]; then
|
||||
g5_global="FAIL"
|
||||
fi
|
||||
if [[ -n "$final_orphans" ]]; then
|
||||
g5_global="FAIL"
|
||||
echo "orphaned processes at end: ${final_orphans}"
|
||||
fi
|
||||
for ((run_index = 1; run_index <= RUNS; run_index++)); do
|
||||
if [[ "${g5[run_index]}" == "FAIL" ]]; then
|
||||
g5_global="FAIL"
|
||||
fi
|
||||
done
|
||||
for ((run_index = 1; run_index <= RUNS; run_index++)); do
|
||||
g5[run_index]="$g5_global"
|
||||
done
|
||||
|
||||
echo
|
||||
printf 'run G1 G2 G3 G4 G5\n'
|
||||
local verdict="PASS"
|
||||
for ((run_index = 1; run_index <= RUNS; run_index++)); do
|
||||
printf '%-4s %-5s %-5s %-5s %-5s %-5s\n' \
|
||||
"$run_index" "${g1[run_index]}" "${g2[run_index]}" "${g3[run_index]}" "${g4[run_index]}" "${g5[run_index]}"
|
||||
for cell in "${g1[run_index]}" "${g2[run_index]}" "${g3[run_index]}" "${g4[run_index]}" "${g5[run_index]}"; do
|
||||
[[ "$cell" == "PASS" ]] || verdict="FAIL"
|
||||
done
|
||||
done
|
||||
echo
|
||||
echo "p95 step latency: ${p95}ms (limit ${P95_LIMIT_MS}ms)"
|
||||
if [[ "$verdict" == "PASS" ]]; then
|
||||
echo "GATES PASS"
|
||||
return 0
|
||||
fi
|
||||
echo "GATES FAIL"
|
||||
return 1
|
||||
}
|
||||
|
||||
# ---- offline self-test -----------------------------------------------------
|
||||
# Exercises every analyzer against canned passing/failing run directories under
|
||||
# testdata/ so the parsing logic can be checked without a device.
|
||||
|
||||
self_test() {
|
||||
local testdata="${script_directory}/testdata"
|
||||
local failures=0
|
||||
|
||||
assert() {
|
||||
local label="$1" expected="$2" actual="$3"
|
||||
if [[ "$expected" == "$actual" ]]; then
|
||||
printf 'ok %s\n' "$label"
|
||||
else
|
||||
printf 'FAIL %s (expected %s, got %s)\n' "$label" "$expected" "$actual"
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
}
|
||||
|
||||
assert "G1 pass run exits zero" PASS \
|
||||
"$(gate_exit_zero "${testdata}/pass" && echo PASS || echo FAIL)"
|
||||
assert "G1 fail run nonzero exit" FAIL \
|
||||
"$(gate_exit_zero "${testdata}/g1-nonzero-exit" && echo PASS || echo FAIL)"
|
||||
|
||||
assert "G2 pass run no errors" PASS \
|
||||
"$(gate_no_error_lines "${testdata}/pass" && echo PASS || echo FAIL)"
|
||||
assert "G2 benign noise tolerated" PASS \
|
||||
"$(gate_no_error_lines "${testdata}/g2-benign-only" && echo PASS || echo FAIL)"
|
||||
assert "G2 real error caught" FAIL \
|
||||
"$(gate_no_error_lines "${testdata}/g2-real-error" && echo PASS || echo FAIL)"
|
||||
|
||||
assert "G3 pass clear state" PASS \
|
||||
"$(gate_clear_state "${testdata}/pass" && echo PASS || echo FAIL)"
|
||||
assert "G3 dirty fields caught" FAIL \
|
||||
"$(gate_clear_state "${testdata}/g3-dirty-field" && echo PASS || echo FAIL)"
|
||||
|
||||
assert "G4 pass no doubling" PASS \
|
||||
"$(gate_no_doubled_text "${testdata}/pass" && echo PASS || echo FAIL)"
|
||||
assert "G4 doubled text caught" FAIL \
|
||||
"$(gate_no_doubled_text "${testdata}/g4-doubled-text" && echo PASS || echo FAIL)"
|
||||
|
||||
local pass_p95 slow_p95
|
||||
pass_p95="$(emit_step_latencies "${testdata}/pass/trace.jsonl" | p95_of)"
|
||||
slow_p95="$(emit_step_latencies "${testdata}/g5-slow-p95/trace.jsonl" | p95_of)"
|
||||
assert "G5 pass p95 under limit" PASS \
|
||||
"$([[ "$pass_p95" -lt "$P95_LIMIT_MS" ]] && echo PASS || echo FAIL)"
|
||||
assert "G5 slow p95 over limit" FAIL \
|
||||
"$([[ "$slow_p95" -lt "$P95_LIMIT_MS" ]] && echo PASS || echo FAIL)"
|
||||
|
||||
echo
|
||||
if [[ "$failures" -eq 0 ]]; then
|
||||
echo "SELF-TEST PASS"
|
||||
return 0
|
||||
fi
|
||||
echo "SELF-TEST FAIL (${failures} failed)"
|
||||
return 1
|
||||
}
|
||||
|
||||
main() {
|
||||
case "${1:-}" in
|
||||
--self-test) self_test ;;
|
||||
"") run_gates ;;
|
||||
*) echo "usage: $0 [--self-test]" >&2; exit 2 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -0,0 +1 @@
|
||||
1
|
||||
@@ -0,0 +1 @@
|
||||
run complete
|
||||
@@ -0,0 +1,4 @@
|
||||
{"step": 1, "timestamp": "2026-06-06T12:00:00.600000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "[email protected]", "selector": "testTag:LoginScreen > testTag:LoginEmail"}}
|
||||
{"step": 2, "timestamp": "2026-06-06T12:00:01.200000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "ledger123", "selector": "testTag:LoginScreen > testTag:LoginPassword"}}
|
||||
{"step": 3, "timestamp": "2026-06-06T12:00:01.800000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "ledger123"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
{"step": 4, "timestamp": "2026-06-06T12:00:02.400000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "ledger123"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1,5 @@
|
||||
companion pid=4242 listening on 127.0.0.1:51000
|
||||
objc[4242]: Class FBSDKError is implemented in both /a (0x1) and /b (0x2). One of the two will be used. Which one is undefined.
|
||||
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
|
||||
archived previous runs under runs/ERRORS_archive
|
||||
run complete
|
||||
@@ -0,0 +1,4 @@
|
||||
{"step": 1, "timestamp": "2026-06-06T12:00:00.600000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "[email protected]", "selector": "testTag:LoginScreen > testTag:LoginEmail"}}
|
||||
{"step": 2, "timestamp": "2026-06-06T12:00:01.200000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "ledger123", "selector": "testTag:LoginScreen > testTag:LoginPassword"}}
|
||||
{"step": 3, "timestamp": "2026-06-06T12:00:01.800000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "ledger123"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
{"step": 4, "timestamp": "2026-06-06T12:00:02.400000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "ledger123"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1,5 @@
|
||||
companion pid=4242 listening on 127.0.0.1:51000
|
||||
objc[4242]: Class FBSDKError is implemented in both /a (0x1) and /b (0x2). One of the two will be used. Which one is undefined.
|
||||
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
|
||||
archived previous runs under runs/ERRORS_archive
|
||||
error: hierarchy fetch failed: companion connection lost
|
||||
@@ -0,0 +1,4 @@
|
||||
{"step": 1, "timestamp": "2026-06-06T12:00:00.600000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "[email protected]", "selector": "testTag:LoginScreen > testTag:LoginEmail"}}
|
||||
{"step": 2, "timestamp": "2026-06-06T12:00:01.200000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "ledger123", "selector": "testTag:LoginScreen > testTag:LoginPassword"}}
|
||||
{"step": 3, "timestamp": "2026-06-06T12:00:01.800000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "ledger123"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
{"step": 4, "timestamp": "2026-06-06T12:00:02.400000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "ledger123"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1 @@
|
||||
run complete
|
||||
@@ -0,0 +1,2 @@
|
||||
{"step": 1, "timestamp": "2026-06-06T12:00:00.600000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "leftover"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "[email protected]", "selector": "testTag:LoginScreen > testTag:LoginEmail"}}
|
||||
{"step": 2, "timestamp": "2026-06-06T12:00:01.200000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "leftover"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
@@ -0,0 +1 @@
|
||||
run complete
|
||||
@@ -0,0 +1,2 @@
|
||||
{"step": 1, "timestamp": "2026-06-06T12:00:00.600000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "[email protected]", "selector": "testTag:LoginScreen > testTag:LoginEmail"}}
|
||||
{"step": 2, "timestamp": "2026-06-06T12:00:01.200000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]@folio.app"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
@@ -0,0 +1 @@
|
||||
0
|
||||
+1
@@ -0,0 +1 @@
|
||||
run complete
|
||||
@@ -0,0 +1,4 @@
|
||||
{"step": 1, "timestamp": "2026-06-06T12:00:04+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "[email protected]", "selector": "testTag:LoginScreen > testTag:LoginEmail"}}
|
||||
{"step": 2, "timestamp": "2026-06-06T12:00:08+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "ledger123", "selector": "testTag:LoginScreen > testTag:LoginPassword"}}
|
||||
{"step": 3, "timestamp": "2026-06-06T12:00:12+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "ledger123"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
{"step": 4, "timestamp": "2026-06-06T12:00:16+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "ledger123"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
Vendored
+121
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env python3
|
||||
# Regenerates the canned gate-analyzer fixtures. Each fixture is a minimal but
|
||||
# format-accurate run directory (trace.jsonl + output.log + exit_status) that
|
||||
# isolates one gate's pass or fail condition. Run from this directory:
|
||||
# python3 generate.py
|
||||
import json
|
||||
import os
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
BASE = datetime(2026, 6, 6, 12, 0, 0, tzinfo=timezone(timedelta(hours=5, minutes=30)))
|
||||
|
||||
|
||||
def element(resource_id, text="", element_class="android.widget.EditText"):
|
||||
return {
|
||||
"resourceId": resource_id,
|
||||
"class": element_class,
|
||||
"editable": True,
|
||||
"bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135},
|
||||
"attrs": {"resource-id": resource_id, "class": element_class, "text": text},
|
||||
}
|
||||
|
||||
|
||||
def login_screen(email_text="", password_text=""):
|
||||
return {
|
||||
"elements": [
|
||||
element("LoginScreen", element_class="android.view.View"),
|
||||
element("LoginEmail", email_text),
|
||||
element("LoginPassword", password_text),
|
||||
element("LoginSubmit", element_class="android.view.View"),
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
def input_action(field, text):
|
||||
return {
|
||||
"kind": "InputText",
|
||||
"text": text,
|
||||
"selector": f"testTag:LoginScreen > testTag:{field}",
|
||||
}
|
||||
|
||||
|
||||
def step(index, gap_ms, hierarchy, next_action=None):
|
||||
timestamp = (BASE + timedelta(milliseconds=index * gap_ms)).isoformat()
|
||||
record = {"step": index, "timestamp": timestamp, "hierarchy": hierarchy}
|
||||
if next_action is not None:
|
||||
record["next_action"] = next_action
|
||||
return record
|
||||
|
||||
|
||||
def write_run(name, steps, exit_status="0", output_lines=None):
|
||||
directory = os.path.join(HERE, name)
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
with open(os.path.join(directory, "trace.jsonl"), "w") as handle:
|
||||
for record in steps:
|
||||
handle.write(json.dumps(record) + "\n")
|
||||
with open(os.path.join(directory, "exit_status"), "w") as handle:
|
||||
handle.write(exit_status)
|
||||
lines = output_lines if output_lines is not None else ["run complete\n"]
|
||||
with open(os.path.join(directory, "output.log"), "w") as handle:
|
||||
handle.writelines(lines)
|
||||
|
||||
|
||||
# A clean run: empty login fields first, single (not doubled) typed values in
|
||||
# the following snapshots, sub-second step gaps.
|
||||
def healthy_steps(gap_ms=600):
|
||||
return [
|
||||
step(1, gap_ms, login_screen("", ""), input_action("LoginEmail", "[email protected]")),
|
||||
step(2, gap_ms, login_screen("[email protected]", ""), input_action("LoginPassword", "ledger123")),
|
||||
step(3, gap_ms, login_screen("[email protected]", "ledger123")),
|
||||
step(4, gap_ms, login_screen("[email protected]", "ledger123")),
|
||||
]
|
||||
|
||||
|
||||
BENIGN_NOISE = [
|
||||
"companion pid=4242 listening on 127.0.0.1:51000\n",
|
||||
"objc[4242]: Class FBSDKError is implemented in both /a (0x1) and /b (0x2). "
|
||||
"One of the two will be used. Which one is undefined.\n",
|
||||
"WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n",
|
||||
# An incidental ERROR substring inside a longer word: the G2 scan is
|
||||
# word-bounded, so this must not trip the gate.
|
||||
"archived previous runs under runs/ERRORS_archive\n",
|
||||
]
|
||||
|
||||
write_run("pass", healthy_steps())
|
||||
|
||||
write_run("g1-nonzero-exit", healthy_steps(), exit_status="1")
|
||||
|
||||
write_run("g2-benign-only", healthy_steps(), output_lines=BENIGN_NOISE + ["run complete\n"])
|
||||
|
||||
write_run(
|
||||
"g2-real-error",
|
||||
healthy_steps(),
|
||||
output_lines=BENIGN_NOISE
|
||||
+ ["error: hierarchy fetch failed: companion connection lost\n"],
|
||||
)
|
||||
|
||||
# G3: first snapshot already carries text in the login fields (clear-state
|
||||
# failed to wipe the previous session).
|
||||
write_run(
|
||||
"g3-dirty-field",
|
||||
[
|
||||
step(1, 600, login_screen("[email protected]", "leftover"), input_action("LoginEmail", "[email protected]")),
|
||||
step(2, 600, login_screen("[email protected]", "leftover")),
|
||||
],
|
||||
)
|
||||
|
||||
# G4: after typing into LoginEmail the next snapshot shows the value appended to
|
||||
# itself (append-vs-replace / double-paste regression).
|
||||
write_run(
|
||||
"g4-doubled-text",
|
||||
[
|
||||
step(1, 600, login_screen("", ""), input_action("LoginEmail", "[email protected]")),
|
||||
step(2, 600, login_screen("[email protected]@folio.app", "")),
|
||||
],
|
||||
)
|
||||
|
||||
# G5: step gaps far exceed the 2.5s ceiling, driving p95 over the limit.
|
||||
write_run("g5-slow-p95", healthy_steps(gap_ms=4000))
|
||||
|
||||
print("fixtures written")
|
||||
+1
@@ -0,0 +1 @@
|
||||
0
|
||||
Vendored
+1
@@ -0,0 +1 @@
|
||||
run complete
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
{"step": 1, "timestamp": "2026-06-06T12:00:00.600000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "[email protected]", "selector": "testTag:LoginScreen > testTag:LoginEmail"}}
|
||||
{"step": 2, "timestamp": "2026-06-06T12:00:01.200000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": ""}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}, "next_action": {"kind": "InputText", "text": "ledger123", "selector": "testTag:LoginScreen > testTag:LoginPassword"}}
|
||||
{"step": 3, "timestamp": "2026-06-06T12:00:01.800000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "ledger123"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
{"step": 4, "timestamp": "2026-06-06T12:00:02.400000+05:30", "hierarchy": {"elements": [{"resourceId": "LoginScreen", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginScreen", "class": "android.view.View", "text": ""}}, {"resourceId": "LoginEmail", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginEmail", "class": "android.widget.EditText", "text": "[email protected]"}}, {"resourceId": "LoginPassword", "class": "android.widget.EditText", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginPassword", "class": "android.widget.EditText", "text": "ledger123"}}, {"resourceId": "LoginSubmit", "class": "android.view.View", "editable": true, "bounds": {"left": 34, "top": 87, "right": 286, "bottom": 135}, "attrs": {"resource-id": "LoginSubmit", "class": "android.view.View", "text": ""}}]}}
|
||||
Reference in new issue
Block a user