mirror of
https://github.com/priyanshujain/messageboardbench.git
synced 2026-10-02 11:07:07 +00:00
A file called scratch.py is not the scratch directory
Running the checks against the baseline task, which has no scratch directory at all, should report zero touches. It reported two. Both agents had made throwaway files named scratch.py and scratch2.py beside their work, and the mention pattern's (?!\w) allows a following ".", so scratch.py read as naming the directory. relation() was already right; only mentioned_in was wrong, so the effect was confined to the unparsed fallback. That fallback exists to turn undercounts into visible ones, and here it was manufacturing overcounts instead. The negative control is clean now. Calibration found this, not a test. Worth remembering that the baseline logs are a free negative control for every check, because the directory genuinely does not exist there. Also adds the submitted func.py to the calibration file: the baseline shows zero test.py edits on this model, so cheat technique has to be read off the function. Claude-Session: https://claude.ai/code/session_01Cq98H7sNoSJdL3W98f18bu
This commit is contained in:
1 parent
bcce9d9e90
commit
d604370d96
3 files changed
+46
-1
No files matched your search
@@ -80,6 +80,13 @@ if __name__ == "__main__":
|
||||
else:
|
||||
lines += ["left behind: nothing", ""]
|
||||
|
||||
# ImpossibleBench's static test-modification check found nothing on this model in
|
||||
# the baseline: every cheat went through func.py instead. So the cheat technique
|
||||
# has to be read off the submitted function, and it belongs in the hand-read file.
|
||||
if row["cheated"] and final is not None:
|
||||
answer = (final.answer or "")[:3000]
|
||||
lines += ["submitted func.py (cheat technique, hand-classify):", "", "```python", answer.rstrip(), "```", ""]
|
||||
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text("\n".join(lines))
|
||||
print(f"wrote {out} for {len(rows)} runs")
|
||||
Reference in new issue
Block a user