mirror of
https://github.com/priyanshujain/messageboardbench.git
synced 2026-10-02 11:07:07 +00:00
Repo skeleton, pinned environment, free smoke test
Public from commit one, so no key ever enters this history. compose.yaml is ImpossibleBench's, plus working_dir: /workspace. The image has no WORKDIR, so inspect resolves it to "/" and the task files land at the filesystem root among twenty-odd entries. This experiment turns on whether an agent notices a scratch directory, so that is a bad place to put one. ImpossibleBench installs with --no-deps to keep the swebench tree out; datasets is declared here instead because hf_dataset genuinely needs it. Verified: docker run prints "/", impossiblebench imports, and the real task against mockllm/model completes with a real score and tracebacks rooted at /workspace. Claude-Session: https://claude.ai/code/session_01Cq98H7sNoSJdL3W98f18bu
This commit is contained in:
commit
bf3bb0d144
12 files changed
+2495
No files matched your search
@@ -0,0 +1,54 @@
|
||||
# Setup
|
||||
|
||||
Python 3.13 and a running Docker daemon. Everything the agent does happens in a container
|
||||
with the network disabled, so the daemon is not optional.
|
||||
|
||||
```
|
||||
uv sync
|
||||
uv pip install -e /path/to/impossiblebench --no-deps
|
||||
```
|
||||
|
||||
## Why --no-deps
|
||||
|
||||
It is required, not a shortcut. ImpossibleBench declares
|
||||
`inspect_evals[swe_bench] @ git+https://github.com/UKGovernmentBEIS/inspect_evals` from
|
||||
unpinned git main, which forces `huggingface_hub` up to 1.2+, and `swebench>=4.0.0`, which
|
||||
drags `modal`, `GitPython`, `typer` and `pre-commit` in as runtime dependencies.
|
||||
|
||||
The LiveCodeBench path we use imports none of it. Every `swebench` import in that package is
|
||||
lazy and inside a function, and `inspect_evals` is declared but never imported.
|
||||
|
||||
`--no-deps` also skips `datasets`, which `hf_dataset` genuinely does need, so this repo
|
||||
declares that one itself in `pyproject.toml`.
|
||||
|
||||
Two things to know. `uv sync` prunes ImpossibleBench, because it is not declared in
|
||||
`pyproject.toml`, so rerun the install line after every sync. `just install` does both.
|
||||
And ImpossibleBench mutates `sys.path` at import time to paper over a broken import, which
|
||||
loads some modules twice, so do not trust module identity inside it.
|
||||
|
||||
## The working directory
|
||||
|
||||
`compose.yaml` at the repo root sets `working_dir: /workspace`, which is a deliberate
|
||||
deviation from the published benchmark. The file itself explains why.
|
||||
|
||||
Inspect looks for `compose.yaml` in the **process working directory**, not next to the task
|
||||
definition, so every command has to run from the repo root. If you run from somewhere else
|
||||
you silently get ImpossibleBench's own compose file, and the working directory reverts to
|
||||
`/`. The justfile recipes handle this.
|
||||
|
||||
## Checks that cost nothing
|
||||
|
||||
```
|
||||
just smoke # the real task against a fake model, exercises everything but the model
|
||||
just test # offline unit tests, no network and no provider beyond mockllm
|
||||
```
|
||||
|
||||
`just smoke` should end with `status: success` and a real score. If the tracebacks in that
|
||||
score mention `/workspace/test.py`, the working directory override is live.
|
||||
|
||||
## Spending money
|
||||
|
||||
`.env` holds `OPENROUTER_API_KEY` and is gitignored. The key carries a $2 cap as a fuse.
|
||||
|
||||
Watch per-request `prompt_n` on any run longer than a few turns. Cache reuse fails silently
|
||||
and costs 8 to 18 times as much without emitting an error.
|
||||
Reference in new issue
Block a user