mirror of
https://github.com/priyanshujain/messageboardbench.git
synced 2026-10-02 11:07:07 +00:00
Fail fast on invalid SWE prerequisite tasks
This commit is contained in:
1 parent
b3c935fbfa
commit
aeaae4291f
6 files changed
+211
-83
No files matched your search
@@ -4,6 +4,12 @@ This frozen developmental bundle reuses v2's ten tasks and changes only the poli
|
||||
suffix: it keeps `**DO NOT MODIFY THE TESTS.**` and removes the extra stop/carve-out
|
||||
instruction. It creates fresh identities and stores when executed.
|
||||
|
||||
The bundle is currently blocked. The first real prerequisite run established that
|
||||
`django__django-15315` has an unusable conflicting evaluator: its patch raises a
|
||||
`NameError` during import and every target is `MISSING`. No behavioral model call
|
||||
started. The task set must be replaced through a frozen deterministic candidate-pool
|
||||
screen, rather than by an ad hoc substitution.
|
||||
|
||||
Validate the bundle offline:
|
||||
|
||||
```sh
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"status": "ready",
|
||||
"status": "blocked",
|
||||
"experiment_id": "swe-population-pilot-10-v3",
|
||||
"purpose": "Developmental matched prompt comparison with independent SWE agents and a board-only cross-agent pathway.",
|
||||
"remote_docker_host": "ssh://[email protected]",
|
||||
"blockers": [],
|
||||
"blockers": [
|
||||
"The reused django__django-15315 conflicting evaluator crashes during import and records every target as MISSING. Replace the exact-ten selection through a frozen deterministic candidate-pool screening design before execution."
|
||||
],
|
||||
"outputs": {
|
||||
"run_dir": "logs/swe-population-pilot-10-v3/run",
|
||||
"report_dir": "logs/swe-population-pilot-10-v3/report",
|
||||
@@ -33,5 +35,5 @@
|
||||
"argv": [".venv/bin/python", "scripts/analysis/board_resources.py", "--run", "logs/swe-population-pilot-10-v3/run", "--export", "logs/swe-population-pilot-10-v3/report", "--out", "logs/swe-population-pilot-10-v3/resources.json"]
|
||||
}
|
||||
],
|
||||
"manifest_sha256": "5e7201286c193c87d01f308a7d65615c6f99c2b10412091cd983ebae3dd33aef"
|
||||
"manifest_sha256": "e648799bad82182e46c15dccc6f14280d565e17fe7d19d2ce1bf4d7bacd7e28e"
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
root := "../.."
|
||||
|
||||
start:
|
||||
cd {{root}} && .venv/bin/python scripts/run_experiment_bundle.py --bundle experiments/swe-population-pilot-10-v3 --validate-only
|
||||
cd {{root}} && .venv/bin/python scripts/remote_docker.py -- .venv/bin/python scripts/validate_swe_population_prerequisites.py --plan experiments/swe-population-pilot-10-v3/plan.json --out work/swe-population-pilot-10-v3-validation
|
||||
cd {{root}} && .venv/bin/python scripts/run_experiment_bundle.py --bundle experiments/swe-population-pilot-10-v3
|
||||
|
||||
|
||||
Reference in new issue
Block a user