What Claude reads
⛔ Bulkhead completion check failed.
You stated "All tests pass" but the evidence ledger disagrees — the latest relevant evidence is: test_failed: npm test (exit 1) /invoice.test.ts > invoice > rounds line totals half-even AssertionError: expected 10.24 to be 10.25 // Object.is equality ❯ src/billing/invoice.test.ts:31:24 FAIL src/billing/invoice.test.ts > invoice > applies credit notes before tax AssertionError: expected 88.1 to be 88.2 // Ob…. Re-run the check and show its real output, or correct your statement to match what actually happened.
This is a deterministic claim-vs-evidence check (bulkhead.yaml: verify).
Verbatim. The agent gets the full refusal and the evidence, so it can fix the work instead of arguing with it.
A check, not a lie detector.
It isn’t a model and it doesn’t judge intent. A pattern finds the claim — tests pass, committed, opened a PR — and a lookup finds the latest matching test run, commit or pull request the ledger saw this session. Same input, same verdict, and the latest evidence decides: the same claim that was refused at #8 was supported at #15, after a passing run.
Test runs it recognises: npm, pnpm, yarn and bun test scripts; vitest, jest, mocha; pytest and python -m pytest / unittest; go test, cargo test, swift test; rspec, phpunit, ctest, make test; playwright test and cypress run. Anything else counts as no evidence: noted on the receipt, not blocked by default.
What you see
🛡 Bulkhead: agent said "All tests pass" but the latest `npm test` exited 1 — sent back to work (ledger #8)
One line in your session, with the ledger entry to look up.
Independent write-up · one developer’s transcripts
Two-thirds
of one developer’s agent “done” claims had no passing test run after the final edit — 69% in the first count, 65% on the author’s stricter recount. The author is careful to call their own check “not a lie detector”.
Read the analysis on dev.to ↗Replit’s agent said a rollback was impossible. Bulkhead doesn’t check that kind of sentence yet. The ones it does check, it checks against evidence.