margin/simplify/testing.md

Testing: the dev harness and @margin/test

Scope: the browser fixture harness, the Playwright configs and suites, the spec helpers, the Rust fixture corpora, and what CI runs. The production half of @margin/ipc (the call wrapper, the phase union, listen) belongs to hooks.md; this document owns the dev-mode half of the same package. Short names: margin = python/margin, calendar = python/margin-caledar, docs = rust/margin-editor, mail = rust/margin-mail, all under /Users/pj/Workspace/projects. Line numbers and counts are as of 2026-09-06.

The harness is the most valuable thing in this consolidation

guidelines/working-together.md forbids starting a dev server: the user keeps one running, Vite is on strictPort, and in Margin Mail a dev instance shares the real app data directory, so a second one is a live-data accident. That leaves one way for an agent to see whether a change works without asking the user to look: a scratch Vite on a spare port with the invoke boundary stubbed, driven by Playwright. The stub is each app's src/dev, and it is worth more than any component in @margin/ui, because without it every verification ends in "try it yourself", which the same guideline also forbids.

What each app has today:

• calendar: a 356 line src/dev, a 12 command mock, no faked events at all, and a suite happy to pass against any checkout's dev server.
• docs: the best harness in the suite, a hand-rolled Tauri shim so the real @tauri-apps/api event plugin runs in a plain tab, plus the only identity check, both locked inside tests/ where pnpm dev cannot reach them.
• mail: the largest fixture at 3,341 lines and 105 commands, events faked as window CustomEvents with an app-side bridge, and the only sane screenshot policy.
• margin: nothing. No tests/, no src/dev, no vitest, no test script (package.json:6-13), and src/ipc.ts is 43 lines calling invoke directly behind an isDesktop early return, so there is no seam a fixture could plug into.

Margin is not a migration. It builds the seam the other three already have, which is why it is last in the sequence rather than first.

The state of play

	margin	calendar	docs	mail
test / test:ui scripts	neither	both	both	both
vitest / @playwright/test	neither	3.2.4 / 1.62.1	3.2.4 / 1.62.1	3.2.4 / 1.62.1
src/dev lines	absent	356	1,040	3,341
mock case labels	none	12	35	105
spec files / test( blocks	0 / 0	10 / 131	14 / 110	24 / 253
spec lines, helpers excluded	0	2,050	3,717	6,445
colocated *.test.ts files / tests	0 / 0	12 / 211	32 / 699	8 / 81
Rust #[test]	0	79	0, all integration	531
src-tauri/tests/	no	no	8 binaries plus support	no
src-tauri/fixtures/	no	no	no	76 files

Margin has no tests of any kind, and its vite.config.ts imports from vite rather than vitest/config, so there is not even a test block to add to. It is also the app with the most design drift, 16 hex literals and 18 rgba values outside its token layer against zero in docs and mail (design-system.md); the two facts are related. Docs' six _audit*.spec.ts files have since been deleted, so its count is 14 specs, not the 19 the audit found.

@margin/test, the Playwright side

The three configs are one file with the port swapped. Byte identical across all three: testDir, fullyParallel, retries: 0 with the same comment about a test that only passes on the second go, the list reporter, the cache outputDir, both timeouts, the 1440x900 viewport, en-GB, the trace and screenshot policies, one chromium project with the identical comment on why it is not devices["Desktop Chrome"], and a webServer running pnpm dev with reuseExistingServer: true. So the whole file becomes a call:

import { marginPlaywrightConfig } from "@margin/test/playwright";

export default marginPlaywrightConfig({
  port: 1450,
  witnesses: ["src/ipc.ts", "src/dev/backend.ts", "src/dev/fixture.ts"],
});

interface MarginPlaywrightOptions {
  /** 1430 calendar, 1440 docs, 1450 mail, 1460 margin. baseURL and webServer.url come off it. */
  port: number;
  /** Source files byte-compared against the served copy before any spec runs. Empty turns the check off. */
  witnesses: string[];
  /** Defaults to Asia/Kolkata. Pass null for a suite that must read the runner's own zone. */
  timezoneId?: string | null;
  use?: PlaywrightTestConfig["use"];   // merged over the defaults, for the rare real difference
}

Two genuine differences exist and both become defaults. timezoneId: "Asia/Kolkata" is set in calendar (playwright.config.ts:34) and mail (:31), which anchor their fixtures to the browser's local day, and absent in docs, which has no clock; the factory defaults it on and docs passes timezoneId: null, because pinning the zone by accident is harmless and failing in another zone is not. The second is globalSetup in docs only (:14), and it becomes the default by way of witnesses.

The identity check is the part of docs' setup the other two most need. tests/identity.ts is 106 lines run once before any spec: for each of five witness files it fetches ${baseURL}/${witness}?raw, decodes the string literal Vite serves by hand rather than by evaluating it (:38-80, since running what an untrusted server sends to decide whether to trust it has the order wrong), and compares it byte for byte against disk. Its header (:1-15) says why: reuseExistingServer on a fixed port is what makes the suite quick and also what lets it talk to a server another checkout left behind, where a pass proves nothing and a failure sends you hunting through a file you never touched.

Calendar and mail have no such check, so both suites are currently happy to pass against anybody's copy. With four apps on four adjacent ports and one machine, that is not hypothetical.

The factory supplies globalSetup, so no app wires it, and witnesses is the only per-app input: for mail src/ipc.ts, the dev backend and the fixture; for docs the schema, the serializer and the save path it already names at identity.ts:25-31. Saved: roughly 45 lines per app, and "all three run at 1440x900 with no retries" becomes a fact rather than a coincidence.

@margin/ipc, the dev harness side

All three apps switch dev mode on identically in src/ipc.ts (calendar :192-197, docs :288-292, mail :748-752): in a DEV build with no __TAURI_INTERNALS__, call imports ./dev/mockIpc and routes there, otherwise it invokes. Mail's differs in one way that matters and wins: its catch arm posts the failure to log_note before rethrowing, skipping log_note itself so it cannot loop (ipc.ts:752-759). That is guidelines/errors-and-feedback.md in code, and the shared makeCall(loadBackend) carries it, alongside devFlag(key), the eight-line localStorage reader that exists three times today.

The typed command registry

mockCall is a switch (command) in all three, returning as unknown as T at every arm and throwing on an unknown command (calendar mockIpc.ts:131-132). There is no type link between a command name, its arguments and its return: calendar casts args to Record<string, never> (:41) and then per field. Replace it with a table:

// src/dev/backend.ts
import { defineBackend, devFlag } from "@margin/ipc/dev";

export const backend = defineBackend({
  accounts_list: () => (devFlag("marginmail-dev-empty") ? [] : accounts),
  thread_view: ({ key }: { key: string }): ThreadView => viewOf(byKey(key)),
});
export type Command = keyof typeof backend;
export const { mockCall, has, commands, emit, script } = backend;

The arg cast at every arm and the as unknown as T at every return both disappear, and a command declared in dto.rs with no handler becomes checkable in one place instead of a runtime surprise three screens later. has and commands are what makes that check writable as a vitest.

Faking events, three ways

Calendar does not fake them: App.tsx:52 is if (!isTauri) return; ahead of every listen, so menu-action, auth, sync-progress and store-changed never arrive in a browser and everything they drive is unreachable from the suite, including the auth listener its own comment calls the only thing that ever learns a mobile sign-in worked.

Mail dispatches window CustomEvents from inside the mock under the same names, with an optional delay so a state that would otherwise last one frame is observable (mockIpc.ts:237-248; narrateFirstSync at :637 walks a five-step sync). The app side is onAppEvent (App.tsx:69-77), seven lines picking listen or window.addEventListener off isTauri. Cheapest correct answer, and it leaves the Tauri arm of that fork, the arm that ships, never exercised in a browser.

Docs does the right thing. installTauriShim (tests/disk.ts:37-133) installs a hand-rolled __TAURI_INTERNALS__ at document start: invoke, transformCallback, unregisterCallback, runCallback, plugin:event|listen and |unlisten, a listener map, metadata, convertFileSrc and __TAURI_EVENT_PLUGIN_INTERNALS__.unregisterListener. Its comment (:27-36) records that it is deliberately not @tauri-apps/api/mocks, which cannot be reached from an init script, and written to the contract rather than to convenience. emit returns a delivery count per event (:66-75), so a test can tell a working subscription from a payload that fell on the floor.

Docs is the one to keep: the app runs the real @tauri-apps/api, so the branch under test is the branch that ships, and mail's onAppEvent stops being necessary. It is not reusable today for two reasons, that it lives in tests/ so a person running pnpm dev by hand gets no events, and that the backend module specifier is hard-coded at disk.ts:79.

What an app implements

// src/dev/harness.ts, imported by src/main.tsx behind import.meta.env.DEV and by the spec helper
import { installTauriShim } from "@margin/ipc/dev";
installTauriShim({ backend: "/src/dev/backend.ts" });

It must stay self-contained with no module-scope closure, because Playwright serialises it into the page through addInitScript, which is why the specifier is an argument. From the spec side:

import { emitEvent, driveFixture } from "@margin/test/harness";
const delivered = await emitEvent(page, "store-changed", "threads");  // returns the listener count
await driveFixture(page, "external.rename", ["notes/a.md", "notes/b.md"]);

driveFixture generalises docs' change/ask pair (disk.ts:135-144): the app exports an external surface of functions that mutate the fixture the way another program would and return the events the backend would have emitted, and the harness puts them on the bus. Docs' pauseWrites and resumeWrites (mockIpc.ts:679-695) stay app-specific; the mechanism does not. Mail's timed narration becomes script([{ after: 0, event, payload }, ...]), and dev flags keep their per-app names, <app>-dev-<thing> being uniform already.

The spec helpers

Calendar and mail both have tests/app.ts, 370 and 401 lines, and 10 of 10 and 23 of 24 specs import from it (the exception, mail's kit.spec.ts, is mostly static scans). These six are code identical between the two and differ only in doc comments:

helper	calendar	mail	lines
clockAt + MIDDAY	app.ts:41-53	app.ts:46-58	13
openApp	:61-90	:66-90	25, seed keys only
settle	:105-112	:99-106	8, third copy at docs caret.ts:46-53
box	:114-124	:108-118	11
openDialog	:288-293	:188-193	6
contrastOf	:306-366	:219-278	60

contrastOf is the one that took real work: it composites every translucent background between the element and the page through a 1x1 canvas, because oklch() and color-mix() otherwise read as transparent. The two apps that have it are the two that independently found the --ink-faint defect. Two more mail helpers are generic despite being written for mail: token(page, name) (app.ts:180-185) and failCommands(page, commands) (app.ts:343-360), which answers the request for the dev backend module with a shim forwarding to the real one (?real) and rejecting the named commands. That is the only way to test a failure path when the backend is in the page and not on the wire, and 18 lines that work unchanged in any of the four.

openApp becomes a factory, since only the seed keys differ:

export const openApp = makeOpenApp({ theme: "marginmail-theme", pane: "marginmail-pane" });

The __test-seeded sentinel goes with it: the seed is written once per context and not on every navigation, so a test that reloads to check what survived is not silently reset underneath itself.

Docs has no shared spec helper at all. Its 14 specs define 16 local opener functions (openReadme, openHandbook, openWriting, openFolder, open, openPreview) and all 14 inline the same margindocs-recents seed. Adopting makeOpenApp is the single biggest deletion in this document.

The Rust fixture story

Mail's src-tauri/fixtures/ is 76 files: 36 .eml messages as they come off the wire (CRLF throughout, half not UTF-8, ISO-8859-1 and ISO-2022-JP among them), 36 matching golden/*.txt and 4 autoconfig/*.xml. A corpus! macro (fixtures.rs:10-19) compiles them into the test binary as one include_bytes! const per file plus an all() returning every pair, and fixtures.rs:58-102 is itself a test: every fixture has a header/body break, no bare LF, a plausible date, a parseable From. Beside it, provider/fake.rs (612 lines, #![cfg(test)]) is an in-memory mailbox with paging, a history log and scripted failures, and the only reason the sync engine is testable without credentials.

There are two hand-maintained lists of that one directory and they have drifted. fixtures.rs:21-52 names 30 files; a second corpus! macro at mime/parse.rs:627-672 names 36. The six missing from the first are the surface-*.eml set (dark-inline-text, newsletter-background-image, newsletter-bgcolor, plain-html, sender-dark-design, wrapper-background), so every_fixture_is_a_message checks none of them. That is the exact failure docs wrote a paragraph about at src/markdown/corpus/load.ts:6-9, where naming folders explicitly left twenty adversarial files inside the corpus and outside every gate that read it. The fix there was import.meta.glob("./*/*.md"); the fix here is the same idea.

The others have nothing comparable. Docs builds its Rust fixture at runtime: src-tauri/tests/support/notes_repo.rs (339 lines) creates a real git repository per test binary, copies 12 documents out of src/markdown/corpus/real so there is one corpus and not two, and generates 13,000 files under a vendored node_modules, because several tests need a folder large enough that skipping it beats walking it. Calendar has 79 #[test]s and no fixtures, margin neither.

Worth sharing? Not yet, and this document should say so rather than pad the package list. The macro is ten lines with one consumer, a crate for it would be indirection over a single call site, and risks.md is explicit that sharing is justified by measured duplication. What is worth doing now is fixing the drift inside mail (glob the directory with include_dir, delete both lists) and writing the rule into guidelines/code-style.md: a corpus is a directory, never a list, because adding a file must put it inside every sweep. TempRepo moves into a shared crate when a second app wants a git-backed temporary directory, and none does today.

Two gaps that are not duplication

Nothing asserts anything about icons

Design-system assertions exist in three shapes and none covers icons. Mail's tests/kit.spec.ts (132 lines) renders #/kit in both palettes, asserts geometry read from custom properties (--list-w is 420px, a row 46px, an avatar 30x30), then runs three static node:fs scans: no hex literal in any stylesheet under src/ui or src/screens, every <input> carrying NO_AUTOFILL or an explicit autoComplete, no file saying "keychain". Calendar's legibility.spec.ts is runtime: contrast, no block reading "Untitled", a non-empty accessible name on every block, both themes. Docs' is a vitest: src/theme.test.ts reads three stylesheets plus index.html's pre-bundle boot script and asserts all four declare the same variable set, because a missing dark variable falls back silently to the warm light value in :root. A grep for icon assertions across all three suites returns nothing, while risks.md records five independent fixes for the same icon baseline problem and eleven implementations of a square icon button. What @margin/test ships:

expectThemesAgree(sheets, bootScript);                   // docs src/theme.test.ts, vitest, no browser
expectNoColourLiterals(root, ["src/ui", "src/screens"]); // mail kit.spec.ts:80-96, vitest, no browser
expectIconsFromSet(root, dirs);                          // no path data outside @margin/icons, vitest
expectIconGrid(icons);                                   // every path inside the 24 unit box, vitest
expectTokens(page, { "--list-w": "420px" });             // mail kit.spec.ts:58-79
expectContrast(page, selector, { min: 4.5 });            // calendar legibility.spec.ts, built on contrastOf
expectAccessibleNames(page, selector);                   // calendar legibility.spec.ts
expectIconOptical(page, ".icon-button");                 // painted glyph box centred within 1px

The first four are static scans over source and need no browser, so they run under vitest. That is what makes them the first tests Margin ever gets: they land before it has a Playwright config, a fixture or a dev harness, and they fail today on the drift design-system.md measured.

Known failures are recorded nowhere in any tree

The only markers anywhere are guide-shots.spec.ts:19-22, an intentional GUIDE_SHOTS env gate, and a test.fail at external-changes.spec.ts:320 kept so it turns red the day the defect is fixed. The four Margin Mail browser failures are written down only in guidelines/app-facts.md, which is in this plan directory and not in the repo that fails.

They are not one thing and should not be recorded as one. Three (shell.spec.ts "New for you above Previously seen", snooze.spec.ts "Back above New for you", triage.spec.ts "Mark all as seen is a link on the heading") assert Inbox group heads the app deliberately no longer draws: stale tests, and the work is rewriting them to the current design. The fourth, kit.spec.ts "every text field tells the webview not to fill it in", flags a real input in Kit.tsx and one in Settings.tsx. That is the test doing its job, and recording it as a known failure would be weakening a test to reach green, which margin-editor/docs/conventions.md:74 forbids in as many words. Fix it.

The record goes in tests/known-failures.md per app: one heading per failure naming the spec file and the test title (matched on the name, never the line number), what it asserts, why it fails and which kind it is. @margin/test ships the checker that reads it, so a suite whose failures match the file exactly exits with a distinct code and one that fails anything else does not. Without the checker the file rots into a list of excuses within a month.

CI

Playwright runs in no app's CI. Calendar's ci.yml:31 and :70 run pnpm test and cargo test; docs' :34, :58 and :78 the same plus a separate --test-threads=1 binary; mail's :49 and :92 the same plus pnpm fonts:check and node scripts/docs-check.mjs. The three justfiles agree line for line that just test-ui is pnpm test:ui, and nothing in CI calls it. What should run where:

• Every push, every app: pnpm build, pnpm test, cargo test and the new static assertions. Margin joins this list first and gets pnpm test for the first time.
• Every push, the three with a suite and then all four: pnpm test:ui, after pnpm exec playwright install --with-deps chromium. Two minutes on a runner already building the Rust side, and the only check that the harness itself still works. The identity check is a no-op there, where Playwright starts its own server, which is fine: it costs five fetches.
• Never in CI: guide-shots.spec.ts, which writes committed files, and mail's eight #[ignore] Rust tests, six of them "hits the network" in imap/discover.rs.
• The shared repo's own CI, per risks.md: check out all four apps against the candidate tag and run each one's pnpm test and pnpm test:ui. That is the only place the four are checked together, and what turns a token rename from a silent break in the app nobody was working on into a red build before the tag exists.

Worth wiring while in here: nothing runs tsc -p tests/tsconfig.json anywhere, so every spec file is type checked by an editor and nothing else. That file is the same nine options in all three, it moves into @margin/config, and the gate gains a line.

Per app, exactly what changes

1. Docs, because it owns the two pieces worth extracting. Move installTauriShim out of tests/disk.ts into @margin/ipc/dev, unchanged except for the backend specifier becoming an argument, and call it from a dev entry point as well as from addInitScript, so pnpm dev in a browser gets real Tauri events for the first time. Move identity.ts into @margin/test behind witnesses. Convert mockIpc.ts to defineBackend, keeping external. Adopt makeOpenApp across all 14 specs and delete the 16 openers. Move theme.test.ts to expectThemesAgree. Check: the suite green, in particular external-changes.spec.ts, which drives the watcher over the shim, and its test.fail at :320 still failing.

2. Mail, the biggest fixture and the most to gain. Take the shim, delete onAppEvent (App.tsx:69-77) and the CustomEvent arm of emit (mockIpc.ts:237-248), so the app runs one code path in both environments and the listen branch that ships is the branch under test. Convert 105 case labels to defineBackend, the largest single piece of work here and the one to do in slices by screen. Contribute token and failCommands, adopt the shared helpers, keep the measurement ones. Turn kit.spec.ts's scans into expectNoColourLiterals and friends, and fix the two inputs it flags rather than recording them. Rewrite the three stale group-head assertions. Fix the corpus drift. Check: 24 specs green, cargo test at 531 green, just install last.

3. Calendar, which gains events it has never had. Take the shim and delete App.tsx:52's if (!isTauri) return;, then write the first specs that drive menu-action, auth and sync-progress, none of which has ever been reachable from a browser. Adopt the shared helpers, contribute contrastOf and legibility.spec.ts's runtime assertions upward, add witnesses. Check legibility.spec.ts and touch.spec.ts, the second because its file-scope test.use is the one thing the factory must not disturb.

4. Margin, which is new work and not a migration. In this order, each step useful on its own:

1. vitest, a test script, a test block in vite.config.ts, and expectNoColourLiterals plus expectThemesAgree pointed at its stylesheets. They fail on day one against the 16 hex and 18 rgba literals in app.css, which is the point: the first test catches the drift that motivated the plan.
2. Split src/ipc.ts into a call seam using makeCall, so a fixture has somewhere to plug in. Today the 43 lines call invoke directly and each function early-returns on !isDesktop, so a browser gets silence rather than data.
3. src/dev/fixture.ts and src/dev/backend.ts: a library of two or three books with images, which is what its data model is, plus the writing-tools and PDF commands stubbed.
4. installTauriShim at the dev entry point, @playwright/test, marginPlaywrightConfig({ port: 1460 }), and a smoke spec that opens a book, types and asserts the word count. Then the rest: the export path, the proofing colours, the settings panel.

Nothing in steps 2 to 4 blocks the shared token migration, and step 1 should land before it, because design-system.md notes Margin is the app whose base sheet extraction has no suite to catch a mistake.

What stays per app

Every measurement helper. Calendar's gridFit, axis, blocks, headerDates, hourY, columnX and drag; mail's rows, groups, paneMessages, paletteRows, actionBar, checkedRows, bodyText, toast, listScroll, place and openRow; docs' putCaret and caretIsIn, whose 28-line header documents why nothing in the suite presses End to move a caret, and watchDirty, a MutationObserver installed before typing so the 500ms autosave cannot be raced. They read app-specific class names and encode app-specific timing, so sharing them would be indirection over one caller each.

The fixtures and every spec file: a calendar of 12,067 events, a folder of markdown and a mailbox of threads have nothing in common but the loading mechanism. Mail's #[cfg(test)] mod tests beside the code, with fifteen modules large enough for a sibling <module>/tests.rs, and docs' eight integration binaries are both right for their shape, and forcing either on the other buys nothing.

The screenshot policy is the one convention worth copying by hand rather than packaging. Mail writes 28 screenshots across 16 specs into a gitignored screenshots/ (.gitignore:41), so an ordinary run leaves the tree clean, while the 10 pictures that ship inside the bundle go to a committed public/guide/ behind test.skip(() => !process.env.GUIDE_SHOTS) and just guide-shots. Only mail has that two-tier rule, it is right, and it is four lines rather than a package.