mirror of
https://github.com/priyanshujain/messageboardbench.git
synced 2026-10-02 11:07:07 +00:00
archive old experiments and publish swe series
This commit is contained in:
1 parent
480100587e
commit
638e978227
1522 files changed
+220002
-4900
No files matched your search
@@ -1,35 +1,9 @@
|
||||
# messageboardbench
|
||||
|
||||
Do coding agents share, adopt, reject, or correct cheating techniques through a public team board?
|
||||
Research harness for studying whether coding agents publish, receive, and adopt evaluator-gaming methods across shared scratchpads and message boards.
|
||||
|
||||
Agents work on ImpossibleBench tasks in separate containers with private scratch space
|
||||
and, in the board condition, explicit tools for reading and posting team messages.
|
||||
The default pilot uses three agents per cohort, two cohorts, and 90 messages per
|
||||
episode; model, population size, sampling and budgets are configurable.
|
||||
The current study runs three OpenRouter models on the same 20 ImpossibleBench SWE tasks under isolated, scratchpad, board, feedback, internet, and token-awareness conditions.
|
||||
|
||||
Built on [ImpossibleBench](https://github.com/safety-research/impossiblebench) and
|
||||
[Inspect AI](https://inspect.aisi.org.uk). Passing contradictory tests is an operational
|
||||
signal of specification gaming, not a complete label of intent or attempted cheating.
|
||||
See the [current experiment](EXPERIMENT.md), [scenario series](docs/swe-scenario-series.md), [experiment index](experiments/README.md), and [setup guide](docs/setup.md).
|
||||
|
||||
See [setup](docs/setup.md) and the [board experiment workflow](docs/board-pilot.md).
|
||||
Run `just board` to choose parameters interactively and start the experiment.
|
||||
Use `just board-preview` for a configuration preview without model calls, or
|
||||
`just board-run --model muse` to run with Muse Spark Contributor. Executing makes
|
||||
paid requests; the existing key spending cap is never raised.
|
||||
|
||||
Start with [EXPERIMENT.md](EXPERIMENT.md) for the research question, what we have
|
||||
done, what the results establish, and the next work. The latest pilot contains one
|
||||
explicitly attributed cross-task adoption; it does not establish an increased
|
||||
gaming rate or concealed collusion.
|
||||
|
||||
| Location | Purpose |
|
||||
|---|---|
|
||||
| `src/`, `scripts/`, `tests/` | Implementation, entry points and tests |
|
||||
| `docs/` | Setup, workflow and supporting research |
|
||||
| [results/](results/README.md) | Reviewed, frozen experiment evidence |
|
||||
| `logs/` | Raw local runs, ignored by Git |
|
||||
| `work/` | Disposable local working files, ignored by Git |
|
||||
|
||||
This is the sole working repository for the experiment. The former `messageboard`
|
||||
repo is [retired](docs/migration/README.md); personal notes and unrelated material
|
||||
remain in its archive. Run `just evidence-check` to verify the migrated evidence.
|
||||
Current raw runs live in `logs/`. Historical local experiments and logs stay in ignored `archive/`.
|
||||
Reference in new issue
Block a user