mirror of
https://github.com/priyanshujain/messageboardbench.git
synced 2026-10-02 19:17:07 +00:00
pj
bf3bb0d144
Public from commit one, so no key ever enters this history. compose.yaml is ImpossibleBench's, plus working_dir: /workspace. The image has no WORKDIR, so inspect resolves it to "/" and the task files land at the filesystem root among twenty-odd entries. This experiment turns on whether an agent notices a scratch directory, so that is a bad place to put one. ImpossibleBench installs with --no-deps to keep the swebench tree out; datasets is declared here instead because hf_dataset genuinely needs it. Verified: docker run prints "/", impossiblebench imports, and the real task against mockllm/model completes with a real score and tracebacks rooted at /workspace. Claude-Session: https://claude.ai/code/session_01Cq98H7sNoSJdL3W98f18bu
55 lines
2.3 KiB
Markdown
55 lines
2.3 KiB
Markdown
# Setup
|
|
|
|
Python 3.13 and a running Docker daemon. Everything the agent does happens in a container
|
|
with the network disabled, so the daemon is not optional.
|
|
|
|
```
|
|
uv sync
|
|
uv pip install -e /path/to/impossiblebench --no-deps
|
|
```
|
|
|
|
## Why --no-deps
|
|
|
|
It is required, not a shortcut. ImpossibleBench declares
|
|
`inspect_evals[swe_bench] @ git+https://github.com/UKGovernmentBEIS/inspect_evals` from
|
|
unpinned git main, which forces `huggingface_hub` up to 1.2+, and `swebench>=4.0.0`, which
|
|
drags `modal`, `GitPython`, `typer` and `pre-commit` in as runtime dependencies.
|
|
|
|
The LiveCodeBench path we use imports none of it. Every `swebench` import in that package is
|
|
lazy and inside a function, and `inspect_evals` is declared but never imported.
|
|
|
|
`--no-deps` also skips `datasets`, which `hf_dataset` genuinely does need, so this repo
|
|
declares that one itself in `pyproject.toml`.
|
|
|
|
Two things to know. `uv sync` prunes ImpossibleBench, because it is not declared in
|
|
`pyproject.toml`, so rerun the install line after every sync. `just install` does both.
|
|
And ImpossibleBench mutates `sys.path` at import time to paper over a broken import, which
|
|
loads some modules twice, so do not trust module identity inside it.
|
|
|
|
## The working directory
|
|
|
|
`compose.yaml` at the repo root sets `working_dir: /workspace`, which is a deliberate
|
|
deviation from the published benchmark. The file itself explains why.
|
|
|
|
Inspect looks for `compose.yaml` in the **process working directory**, not next to the task
|
|
definition, so every command has to run from the repo root. If you run from somewhere else
|
|
you silently get ImpossibleBench's own compose file, and the working directory reverts to
|
|
`/`. The justfile recipes handle this.
|
|
|
|
## Checks that cost nothing
|
|
|
|
```
|
|
just smoke # the real task against a fake model, exercises everything but the model
|
|
just test # offline unit tests, no network and no provider beyond mockllm
|
|
```
|
|
|
|
`just smoke` should end with `status: success` and a real score. If the tracebacks in that
|
|
score mention `/workspace/test.py`, the working directory override is live.
|
|
|
|
## Spending money
|
|
|
|
`.env` holds `OPENROUTER_API_KEY` and is gitignored. The key carries a $2 cap as a fuse.
|
|
|
|
Watch per-request `prompt_n` on any run longer than a few turns. Cache reuse fails silently
|
|
and costs 8 to 18 times as much without emitting an error.
|