pj
bf3bb0d144
Public from commit one, so no key ever enters this history. compose.yaml is ImpossibleBench's, plus working_dir: /workspace. The image has no WORKDIR, so inspect resolves it to "/" and the task files land at the filesystem root among twenty-odd entries. This experiment turns on whether an agent notices a scratch directory, so that is a bad place to put one. ImpossibleBench installs with --no-deps to keep the swebench tree out; datasets is declared here instead because hf_dataset genuinely needs it. Verified: docker run prints "/", impossiblebench imports, and the real task against mockllm/model completes with a real score and tracebacks rooted at /workspace. Claude-Session: https://claude.ai/code/session_01Cq98H7sNoSJdL3W98f18bu
2.3 KiB
Setup
Python 3.13 and a running Docker daemon. Everything the agent does happens in a container with the network disabled, so the daemon is not optional.
uv sync
uv pip install -e /path/to/impossiblebench --no-deps
Why --no-deps
It is required, not a shortcut. ImpossibleBench declares
inspect_evals[swe_bench] @ git+https://github.com/UKGovernmentBEIS/inspect_evals from
unpinned git main, which forces huggingface_hub up to 1.2+, and swebench>=4.0.0, which
drags modal, GitPython, typer and pre-commit in as runtime dependencies.
The LiveCodeBench path we use imports none of it. Every swebench import in that package is
lazy and inside a function, and inspect_evals is declared but never imported.
--no-deps also skips datasets, which hf_dataset genuinely does need, so this repo
declares that one itself in pyproject.toml.
Two things to know. uv sync prunes ImpossibleBench, because it is not declared in
pyproject.toml, so rerun the install line after every sync. just install does both.
And ImpossibleBench mutates sys.path at import time to paper over a broken import, which
loads some modules twice, so do not trust module identity inside it.
The working directory
compose.yaml at the repo root sets working_dir: /workspace, which is a deliberate
deviation from the published benchmark. The file itself explains why.
Inspect looks for compose.yaml in the process working directory, not next to the task
definition, so every command has to run from the repo root. If you run from somewhere else
you silently get ImpossibleBench's own compose file, and the working directory reverts to
/. The justfile recipes handle this.
Checks that cost nothing
just smoke # the real task against a fake model, exercises everything but the model
just test # offline unit tests, no network and no provider beyond mockllm
just smoke should end with status: success and a real score. If the tracebacks in that
score mention /workspace/test.py, the working directory override is live.
Spending money
.env holds OPENROUTER_API_KEY and is gitignored. The key carries a $2 cap as a fuse.
Watch per-request prompt_n on any run longer than a few turns. Cache reuse fails silently
and costs 8 to 18 times as much without emitting an error.