mirror of
https://github.com/priyanshujain/messageboardbench.git
synced 2026-10-04 20:17:06 +00:00
Checkpoint experiments through SWE pilot v2
This commit is contained in:
1 parent
abacd5c5e1
commit
72d77018d8
845 files changed
+431756
-41
No files matched your search
@@ -0,0 +1,32 @@
|
||||
# Population propensity experiment v1
|
||||
|
||||
This directory is the single entry point for the frozen SWE population experiment:
|
||||
all 349 ImpossibleBench conflicting tasks, partitioned across 12 independent matched
|
||||
control/board teams and three cohorts. The two arms use the same upstream prompt,
|
||||
model, tasks, schedule, budgets and scorer; the board arm alone receives the neutral
|
||||
`board_read` and `board_post` schemas and a team-persistent host store.
|
||||
|
||||
When `experiment.json` has status `ready`, no blockers, complete command arrays,
|
||||
and a valid self-hash, run the whole lifecycle from this directory:
|
||||
|
||||
```sh
|
||||
just start
|
||||
```
|
||||
|
||||
That command validates the frozen manifest before any external action, writes local
|
||||
lifecycle/provenance state, runs the configured local Python runner through only
|
||||
`ssh://[email protected]` using `scripts/remote_docker.py`, and then attempts the
|
||||
allowlisted offline report, verification, and resource-summary steps. Offline steps
|
||||
whose required artifacts exist run even if experiment execution returns nonzero;
|
||||
missing-input steps are recorded as skipped. No shell command strings are evaluated.
|
||||
|
||||
Raw outputs and automatic, unreviewed reports stay under ignored `logs/`. Promotion
|
||||
to a named `results/` bundle remains a separate reviewed evidence action. Source,
|
||||
Python, credentials, logs, and results remain on this workstation; only container
|
||||
operations go to the remote Docker daemon.
|
||||
|
||||
`just validate` is a free local manifest check and never invokes a model. Interrupted
|
||||
runs resume from terminal assignment records when `just start` is called again;
|
||||
derived reports from earlier attempts are retained under `logs/.../resume-history/`.
|
||||
The first matched cohort pair acts as the engineering sentinel. Subsequent individual
|
||||
sample errors are retained as outcomes and do not stop or rerun the population.
|
||||
Reference in new issue
Block a user