fix: iOS spec driving, InputText replace semantics, native DoubleTap (#58)

* fix(hierarchy): bounds-containment fallback for scoped and path queries

Compose on iOS surfaces a testTag node as an empty leaf sibling of the
content it labels instead of as an ancestor, so descendant search under
the tagged node finds nothing and every path or scoped query returns
null. When structural search yields no match, fall back to nodes whose
bounds lie inside the scope node's bounds.

* feat(sidecar): derive iOS clickable and editable from element type

The XCTest hierarchy mapping dropped the element type, leaving no
clickable or editable flags on iOS, so the fuzzer's tap and typing
verbs never found a candidate inside the app. Map the raw
accessibility tree directly and derive clickable, editable,
scrollable, and class from the XCUIElementType raw value.

* feat(proto): add EraseText RPC for InputText replace semantics

* feat(driver): add EraseText to the device driver surface

* fix(runner): erase existing field text before InputText

InputText appended on native platforms, so repeated draws grew fields
without bound. The folio fuzz run wedged on the add-account screen:
each draw concatenated another name until the 40-character validation
error became permanent. Replace semantics also makes retried typing
idempotent. The web driver already replaced via select-all; native now
matches.

* feat(sidecar): EraseText backend support on android and ios

* fix(folio): saturation-gate account creation in the spec

The 2-3 step add-account loop outcompeted the 5-step transaction chain
at every weighted re-draw, so runs filled with account creation and
rarely exercised the balance properties. Stop offering add-account once
three accounts exist; the renormalized weights then favor the
transaction flow at every step of its chain.

* fix(folio): author spec weights to match testing intent

Revert the account saturation gate: it starved newAccountBalanceIsZero
once it tripped, and a magic account count is app-state tuning, not
intent. Instead weight the generators by what the properties need:
the transaction chain leads, account creation stays exercised, and
doubleTaps gets explicit weight everywhere because double-submission
idempotency is what the spec is testing for.

* fix(folio): lower doubleTaps weight to 5

* fix(sidecar): surface visible text on iOS static elements

Static text and button strings live in the accessibility label on
iOS, so the text attribute came through empty and every balance
extractor parsed to zero, silently disarming both folio properties.
Non-editable elements now fall back title, value, then label;
editable fields keep value-only so an empty field's caption does not
read as content.

* feat(driver): native DoubleTap RPC for a tight inter-tap gap

Composing two Tap round trips from the Go client spread the taps by
hundreds of milliseconds on iOS, wide enough for the app to navigate
between them, so double-submission races could never reproduce. The
sidecar now lands both taps back-to-back next to the device transport.

* feat(sidecar): pipeline iOS double-tap requests

Queue the second tap at the XCTest runner while the first executes.
The runner serializes handlers, so this is the tightest gap the
transport allows (~350ms per tap round trip); recorded here with
measurements for the iOS double-tap limitation.
This commit is contained in:
pj authored and GitHub committed 2026-06-05 23:42:39 +05:30
1 parent ce65cc32a4
commit 19a470121f
19 files changed
+981 -190

No files matched your search

@@ -4,8 +4,18 @@ interface DriverBackend {
fun launch(bundleId: String, clearState: Boolean, env: Map<String, String> = emptyMap())
fun terminate(bundleId: String)
fun tap(x: Int, y: Int)
// doubleTap lands two taps as close together as the platform allows.
// The default composes two taps back-to-back; backends with higher
// per-tap latency override to tighten the gap.
fun doubleTap(x: Int, y: Int) {
tap(x, y)
tap(x, y)
}
fun tapSelector(selector: String)
fun inputText(text: String)
fun eraseText(characterCount: Int)
fun swipe(fromX: Int, fromY: Int, toX: Int, toY: Int, durationMillis: Long)
fun pressKey(key: String)
fun longPress(x: Int, y: Int)
@@ -402,6 +412,16 @@ class StubDriverBackend(
runAdb(listOf("shell", "input", "text", escapeForAdbInputText(text)))
}
@Volatile var lastEraseCharacterCount: Int? = null
private set
override fun eraseText(characterCount: Int) {
lastEraseCharacterCount = characterCount
repeat(characterCount) {
runAdb(listOf("shell", "input", "keyevent", "KEYCODE_DEL"))
}
}
@Volatile var lastSwipe: SwipeRecord? = null
private set
@Volatile var lastKey: String? = null
@@ -531,6 +551,8 @@ class MaestroDriverBackend(private val serial: String?) : DriverBackend {
override fun inputText(text: String) = driver.inputText(text)
override fun eraseText(characterCount: Int) = driver.eraseText(characterCount)
override fun swipe(fromX: Int, fromY: Int, toX: Int, toY: Int, durationMillis: Long) =
driver.swipe(maestro.Point(fromX, fromY), maestro.Point(toX, toY), maxOf(durationMillis, 250L))
@@ -629,6 +651,7 @@ private fun pngHeight(bytes: ByteArray): Int {
class IosDriverBackend(private val udid: String) : DriverBackend {
private lateinit var driver: maestro.drivers.IOSDriver
private lateinit var localDevice: ios.LocalIOSDevice
private val reconnectLock = java.util.concurrent.locks.ReentrantLock()
init {
@@ -670,6 +693,7 @@ class IosDriverBackend(private val udid: String) : DriverBackend {
deviceController = simctlDevice,
insights = maestro.utils.NoopInsights,
)
localDevice = device
driver = maestro.drivers.IOSDriver(device, maestro.utils.NoopInsights)
driver.open()
warmup()
@@ -720,6 +744,19 @@ class IosDriverBackend(private val udid: String) : DriverBackend {
override fun tap(x: Int, y: Int) = withReconnect { driver.tap(maestro.Point(x, y)) }
// The second tap request is already queued at the XCTest runner while the
// first executes, so the on-device gap collapses to the runner's
// turnaround instead of a full transport round trip. Sequential requests
// leave a gap wide enough for the app to navigate between the taps.
override fun doubleTap(x: Int, y: Int): Unit = withReconnect {
val point = maestro.Point(x, y)
val firstTap = java.util.concurrent.CompletableFuture.runAsync { driver.tap(point) }
Thread.sleep(40)
driver.tap(point)
firstTap.join()
Unit
}
override fun longPress(x: Int, y: Int) = withReconnect { driver.longPress(maestro.Point(x, y)) }
override fun tapSelector(selector: String) = withReconnect {
@@ -730,6 +767,8 @@ class IosDriverBackend(private val udid: String) : DriverBackend {
override fun inputText(text: String) = withReconnect { driver.inputText(text) }
override fun eraseText(characterCount: Int) = withReconnect { driver.eraseText(characterCount) }
override fun swipe(fromX: Int, fromY: Int, toX: Int, toY: Int, durationMillis: Long) = withReconnect {
driver.swipe(maestro.Point(fromX, fromY), maestro.Point(toX, toY), maxOf(durationMillis, 250L))
}
@@ -750,7 +789,7 @@ class IosDriverBackend(private val udid: String) : DriverBackend {
override fun hierarchy(): String = withReconnect {
com.fasterxml.jackson.module.kotlin.jacksonObjectMapper()
.writeValueAsString(driver.contentDescriptor(false))
.writeValueAsString(iosAxElementToTreeNode(localDevice.viewHierarchy(false).axElement))
}
override fun recentLogs(sinceUnixMillis: Long, minLevel: String): List<LogLine> = emptyList()
@@ -784,3 +823,115 @@ private fun keyCodeToMaestro(adbKeyCode: String): maestro.KeyCode? {
else -> null
}
}
// XCUIElementType raw values that accept a tap. The XCTest hierarchy carries
// no clickable flag, so tappability is derived from the element type that
// XCUITest itself derives from the accessibility traits.
internal val IOS_CLICKABLE_ELEMENT_TYPES = setOf(
9, // button
10, // radioButton
12, // checkBox
14, // popUpButton
15, // comboBox
16, // menuButton
17, // toolbarButton
20, // key
37, // segmentedControl
40, // switch
41, // toggle
42, // link
54, // menuItem
75, // cell
79, // stepper
80, // tab
)
// XCUIElementType raw values that accept text input.
internal val IOS_EDITABLE_ELEMENT_TYPES = setOf(
45, // searchField
49, // textField
50, // secureTextField
52, // textView
)
// XCUIElementType raw values that scroll.
internal val IOS_SCROLLABLE_ELEMENT_TYPES = setOf(
26, // table
32, // collectionView
46, // scrollView
58, // webView
)
// Checkable element types and the value convention ("1" when on) mirror the
// upstream content-descriptor mapping.
internal val IOS_CHECKABLE_ELEMENT_TYPES = setOf(
12, // checkBox
40, // switch
41, // toggle
)
private val IOS_ELEMENT_TYPE_NAMES = arrayOf(
"Any", "Other", "Application", "Group", "Window", "Sheet", "Drawer",
"Alert", "Dialog", "Button", "RadioButton", "RadioGroup", "CheckBox",
"DisclosureTriangle", "PopUpButton", "ComboBox", "MenuButton",
"ToolbarButton", "Popover", "Keyboard", "Key", "NavigationBar", "TabBar",
"TabGroup", "Toolbar", "StatusBar", "Table", "TableRow", "TableColumn",
"Outline", "OutlineRow", "Browser", "CollectionView", "Slider",
"PageIndicator", "ProgressIndicator", "ActivityIndicator",
"SegmentedControl", "Picker", "PickerWheel", "Switch", "Toggle", "Link",
"Image", "Icon", "SearchField", "ScrollView", "ScrollBar", "StaticText",
"TextField", "SecureTextField", "DatePicker", "TextView", "Menu",
"MenuItem", "MenuBar", "MenuBarItem", "Map", "WebView", "IncrementArrow",
"DecrementArrow", "Timeline", "RatingIndicator", "ValueIndicator",
"SplitGroup", "Splitter", "RelevanceIndicator", "ColorWell", "HelpTag",
"Matte", "DockItem", "Ruler", "RulerMarker", "Grid", "LevelIndicator",
"Cell", "LayoutArea", "LayoutItem", "Handle", "Stepper", "Tab",
"TouchBar", "StatusItem",
)
internal fun iosElementTypeName(elementType: Int): String =
IOS_ELEMENT_TYPE_NAMES.getOrElse(elementType) { "Other" }
// iosAxElementToTreeNode maps the raw XCTest accessibility tree to the
// TreeNode JSON shape the runner parses. The upstream content-descriptor
// mapping drops the element type, leaving no way to tell buttons and text
// fields apart, so fuzz verbs found no tap or typing candidates on iOS.
internal fun iosAxElementToTreeNode(element: hierarchy.AXElement): Map<String, Any?> {
val elementType = element.elementType
val title = element.title.orEmpty()
val value = element.value.orEmpty()
val label = element.label.orEmpty()
val editable = elementType in IOS_EDITABLE_ELEMENT_TYPES
val checked = elementType in IOS_CHECKABLE_ELEMENT_TYPES && value == "1"
// text carries the element's visible string, matching what Android
// surfaces in its text attribute. Editable fields expose only their
// typed content: their label is the field caption (e.g. "Email"), and
// leaking it into text would make an empty field look filled. Static
// text and buttons carry the visible string in the label.
val text = if (editable) value else title.ifEmpty { value }.ifEmpty { label }
val attributes = linkedMapOf(
"accessibilityText" to label,
"title" to title,
"value" to value,
"text" to text,
"hintText" to element.placeholderValue.orEmpty(),
"resource-id" to element.identifier.orEmpty(),
"bounds" to element.frame.boundsString,
"class" to iosElementTypeName(elementType),
"enabled" to element.enabled.toString(),
"focused" to element.hasFocus.toString(),
"selected" to element.selected.toString(),
"checked" to checked.toString(),
"scrollable" to (elementType in IOS_SCROLLABLE_ELEMENT_TYPES).toString(),
)
return linkedMapOf(
"attributes" to attributes,
"children" to element.children.map { iosAxElementToTreeNode(it) },
"clickable" to (elementType in IOS_CLICKABLE_ELEMENT_TYPES),
"enabled" to element.enabled,
"focused" to element.hasFocus,
"checked" to checked,
"selected" to element.selected,
"editable" to editable,
)
}
@@ -4,6 +4,7 @@ import com.google.protobuf.ByteString
import dev.sanderling.driver.v1.DriverGrpc
import dev.sanderling.driver.v1.Duration
import dev.sanderling.driver.v1.Empty
import dev.sanderling.driver.v1.EraseTextRequest
import dev.sanderling.driver.v1.HealthStatus
import dev.sanderling.driver.v1.HierarchyJSON
import dev.sanderling.driver.v1.Image
@@ -53,6 +54,13 @@ class DriverService(
}
}
override fun doubleTap(request: Point, responseObserver: StreamObserver<Empty>) {
runRpc(responseObserver) {
backend.doubleTap(request.x, request.y)
Empty.getDefaultInstance()
}
}
override fun longPress(request: Point, responseObserver: StreamObserver<Empty>) {
runRpc(responseObserver) {
backend.longPress(request.x, request.y)
@@ -74,6 +82,13 @@ class DriverService(
}
}
override fun eraseText(request: EraseTextRequest, responseObserver: StreamObserver<Empty>) {
runRpc(responseObserver) {
backend.eraseText(request.characterCount)
Empty.getDefaultInstance()
}
}
override fun swipe(request: SwipeRequest, responseObserver: StreamObserver<Empty>) {
runRpc(responseObserver) {
val from = request.from
@@ -3,6 +3,7 @@ package dev.sanderling.sidecar
import dev.sanderling.driver.v1.DriverGrpc
import dev.sanderling.driver.v1.Duration
import dev.sanderling.driver.v1.Empty
import dev.sanderling.driver.v1.EraseTextRequest
import dev.sanderling.driver.v1.LaunchRequest
import dev.sanderling.driver.v1.Point
import dev.sanderling.driver.v1.PressKeyRequest
@@ -82,6 +83,42 @@ class DriverServiceTest {
assertEquals("hello world", backend.lastInputText)
}
@Test fun doubleTapDefaultComposesTwoTaps() {
// Interface delegation would bind the default doubleTap to the
// delegate, bypassing the tap override, so implement the interface
// directly.
val taps = mutableListOf<Pair<Int, Int>>()
val backend = object : DriverBackend {
override fun launch(bundleId: String, clearState: Boolean, env: Map<String, String>) {}
override fun terminate(bundleId: String) {}
override fun tap(x: Int, y: Int) { taps.add(x to y) }
override fun tapSelector(selector: String) {}
override fun inputText(text: String) {}
override fun eraseText(characterCount: Int) {}
override fun swipe(fromX: Int, fromY: Int, toX: Int, toY: Int, durationMillis: Long) {}
override fun pressKey(key: String) {}
override fun longPress(x: Int, y: Int) {}
override fun screenshot(): Triple<ByteArray, Int, Int> = Triple(byteArrayOf(), 0, 0)
override fun hierarchy(): String = "{}"
override fun recentLogs(sinceUnixMillis: Long, minLevel: String): List<LogLine> = emptyList()
override fun waitForIdle(durationMillis: Long) {}
override fun healthy(): Boolean = true
override fun metrics(bundleId: String): MetricsSample = MetricsSample(0.0, 0L, 0L)
}
val client = newClient(backend)
client.doubleTap(Point.newBuilder().setX(120).setY(340).build())
assertEquals(listOf(120 to 340, 120 to 340), taps)
}
@Test fun eraseTextForwardsCharacterCount() {
val backend = StubDriverBackend("android")
val client = newClient(backend)
client.eraseText(EraseTextRequest.newBuilder().setCharacterCount(11).build())
assertEquals(11, backend.lastEraseCharacterCount)
}
@Test fun screenshotReturnsBackendBytes() {
val backend = object : DriverBackend by StubDriverBackend("android") {
override fun screenshot(): Triple<ByteArray, Int, Int> = Triple(byteArrayOf(1, 2, 3), 1080, 2340)
@@ -15,6 +15,18 @@ class InputTextTest {
assertEquals(listOf(listOf("shell", "input", "text", "Emergency%sFund")), commands)
}
@Test fun eraseTextSendsOneDeleteKeyPerCharacter() {
val commands = mutableListOf<List<String>>()
val backend = StubDriverBackend("android") { commands.add(it) }
backend.eraseText(3)
assertEquals(
List(3) { listOf("shell", "input", "keyevent", "KEYCODE_DEL") },
commands,
)
}
@Test fun escapeForAdbInputTextSubstitutesSpaces() {
assertEquals("hello%sworld", StubDriverBackend.escapeForAdbInputText("hello world"))
}
@@ -0,0 +1,126 @@
package dev.sanderling.sidecar
import hierarchy.AXElement
import hierarchy.AXFrame
import org.junit.Test
import kotlin.test.assertEquals
class IosHierarchyTest {
private fun element(
elementType: Int,
label: String = "",
identifier: String = "",
title: String = "",
value: String = "",
placeholderValue: String = "",
children: ArrayList<AXElement> = arrayListOf(),
) = AXElement(
label,
elementType,
identifier,
0,
0L,
0,
false,
0,
false,
placeholderValue,
value,
AXFrame(10f, 20f, 100f, 50f),
true,
title,
children,
)
@Test
fun buttonIsClickableNotEditable() {
val node = iosAxElementToTreeNode(element(elementType = 9, label = "Sign in", identifier = "LoginSubmit"))
assertEquals(true, node["clickable"])
assertEquals(false, node["editable"])
@Suppress("UNCHECKED_CAST")
val attributes = node["attributes"] as Map<String, String>
assertEquals("Button", attributes["class"])
assertEquals("LoginSubmit", attributes["resource-id"])
assertEquals("Sign in", attributes["accessibilityText"])
}
@Test
fun textFieldIsEditableNotClickable() {
val node = iosAxElementToTreeNode(element(elementType = 49, value = "[email protected]"))
assertEquals(false, node["clickable"])
assertEquals(true, node["editable"])
@Suppress("UNCHECKED_CAST")
val attributes = node["attributes"] as Map<String, String>
assertEquals("TextField", attributes["class"])
assertEquals("[email protected]", attributes["text"])
assertEquals("[email protected]", attributes["value"])
}
@Test
fun titleWinsOverValueForText() {
val node = iosAxElementToTreeNode(element(elementType = 48, title = "Heading", value = "ignored"))
@Suppress("UNCHECKED_CAST")
val attributes = node["attributes"] as Map<String, String>
assertEquals("Heading", attributes["text"])
assertEquals("StaticText", attributes["class"])
}
@Test
fun staticTextFallsBackToLabelForText() {
val node = iosAxElementToTreeNode(element(elementType = 48, label = "$1,234.00"))
@Suppress("UNCHECKED_CAST")
val attributes = node["attributes"] as Map<String, String>
assertEquals("$1,234.00", attributes["text"])
}
@Test
fun buttonFallsBackToLabelForText() {
val node = iosAxElementToTreeNode(element(elementType = 9, label = "Sign in"))
@Suppress("UNCHECKED_CAST")
val attributes = node["attributes"] as Map<String, String>
assertEquals("Sign in", attributes["text"])
}
@Test
fun emptyTextFieldDoesNotLeakLabelIntoText() {
val node = iosAxElementToTreeNode(element(elementType = 49, label = "Email"))
@Suppress("UNCHECKED_CAST")
val attributes = node["attributes"] as Map<String, String>
assertEquals("", attributes["text"])
assertEquals("Email", attributes["accessibilityText"])
}
@Test
fun switchOnIsChecked() {
val node = iosAxElementToTreeNode(element(elementType = 40, value = "1"))
assertEquals(true, node["checked"])
assertEquals(true, node["clickable"])
}
@Test
fun scrollViewIsScrollable() {
val node = iosAxElementToTreeNode(element(elementType = 46))
@Suppress("UNCHECKED_CAST")
val attributes = node["attributes"] as Map<String, String>
assertEquals("true", attributes["scrollable"])
assertEquals("ScrollView", attributes["class"])
}
@Test
fun childrenAreMappedRecursively() {
val child = element(elementType = 48, title = "Inner")
val node = iosAxElementToTreeNode(element(elementType = 1, children = arrayListOf(child)))
@Suppress("UNCHECKED_CAST")
val children = node["children"] as List<Map<String, Any?>>
assertEquals(1, children.size)
@Suppress("UNCHECKED_CAST")
val childAttributes = children[0]["attributes"] as Map<String, String>
assertEquals("Inner", childAttributes["text"])
}
@Test
fun unknownElementTypeFallsBackToOther() {
assertEquals("Other", iosElementTypeName(999))
}
}
@@ -60,6 +60,7 @@ class SnapshotHandlerTest {
override fun tap(x: Int, y: Int) {}
override fun tapSelector(selector: String) {}
override fun inputText(text: String) {}
override fun eraseText(characterCount: Int) {}
override fun swipe(fromX: Int, fromY: Int, toX: Int, toY: Int, durationMillis: Long) {}
override fun pressKey(key: String) {}
override fun longPress(x: Int, y: Int) {}