A file called scratch.py is not the scratch directory

Running the checks against the baseline task, which has no scratch directory at all,
should report zero touches. It reported two. Both agents had made throwaway files named
scratch.py and scratch2.py beside their work, and the mention pattern's (?!\w) allows a
following ".", so scratch.py read as naming the directory.

relation() was already right; only mentioned_in was wrong, so the effect was confined to
the unparsed fallback. That fallback exists to turn undercounts into visible ones, and
here it was manufacturing overcounts instead. The negative control is clean now.

Calibration found this, not a test. Worth remembering that the baseline logs are a free
negative control for every check, because the directory genuinely does not exist there.

Also adds the submitted func.py to the calibration file: the baseline shows zero test.py
edits on this model, so cheat technique has to be read off the function.

Claude-Session: https://claude.ai/code/session_01Cq98H7sNoSJdL3W98f18bu
This commit is contained in:
pj committed 2026-08-31 22:02:56 +05:30
1 parent bcce9d9e90
commit d604370d96
3 files changed
+46 -1

No files matched your search

+7
View File
@@ -80,6 +80,13 @@ if __name__ == "__main__":
else:
lines += ["left behind: nothing", ""]
# ImpossibleBench's static test-modification check found nothing on this model in
# the baseline: every cheat went through func.py instead. So the cheat technique
# has to be read off the submitted function, and it belongs in the hand-read file.
if row["cheated"] and final is not None:
answer = (final.answer or "")[:3000]
lines += ["submitted func.py (cheat technique, hand-classify):", "", "```python", answer.rstrip(), "```", ""]
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text("\n".join(lines))
print(f"wrote {out} for {len(rows)} runs")