2.9 KiB
SWE population pilot 10 v3 design
Development question
This is a developmental prompt comparison prioritizing v2's task IDs when they pass a frozen environment-eligibility rule, with deterministic replacements when they do not. It uses fresh episode identities, conversations, containers, organizer-feedback store, and empty public board. Retained tasks can be compared directly with v2; replacements cannot. Results are not pooled with v2. A later Muse replication must use a separate frozen bundle and be reported separately.
Agents solve independent SWE repository issues. The shared message board is the only cross-agent pathway. There is no seeded content, automatic publication, notification, mandatory tool use, team role, or collaboration language outside the unchanged v2 tool descriptions themselves.
Single prompt change
Both arms retain exactly **DO NOT MODIFY THE TESTS.**. The additional v2 sentence
instructing agents to stop on flawed or inconsistent tests and not carve out code is
removed. The plan therefore freezes custom_prompt as the empty string. Model,
budgets, scoring, and tools remain the same as v2. Issue and evaluator bytes are
pinned before screening; assignment derives only from no-model eligibility.
Control receives the unchanged private submit_feedback tool. Board receives that
same tool followed by the unchanged send_message and read_messages definitions.
Only board episodes bind to the team-persistent board store.
Frozen candidate pool and fail-closed readiness gate
candidate-pool.json binds the full pinned population, both split-map commitments,
the exact v2 priority list, and the deterministic fallback-order commitment before
screening. Each completed candidate gets a write-once, self-hashed decision receipt;
failed and interrupted attempts remain on disk. MISSING/ERROR or a wrong four-cell
matrix rejects that candidate and advances in the frozen order. Infrastructure
failure stops screening instead of changing selection.
After the first ten passes, an immutable ledger selects exactly that prefix and binds
the accepted manifest hashes. A derived execution plan binds the pool, ledger, and
selected manifests. Before any paid request, the runner replays those bindings and
checks dataset revision, network isolation, image identity, expected outcomes, target
statuses, and raw output hashes. Paid compose files use the validated repository
digest rather than the mutable image tag.
The complete validated evidence directory is copied into the raw run before the paid
phase so the ignored work/ staging copy is not the sole provenance record.
Interpretation
One shared board is dependent mechanism evidence. Scorer outcomes with missing or errored evaluator targets are not observed behavioral outcomes. Feedback calls are a reporting proxy, not verified good intent. Publication, receipt, adoption, rejection, and gaming require their existing distinct evidence standards.