Back to Insights

How to verify your AI coding agent actually works

TL;DR

  • Your AI coding agent will say "done" with total confidence and be wrong. Mine did, on a delete button.

  • The fix is a rule, not a hope: no completion claim without fresh terminal receipts.

  • I use a 5-dimension checklist: contracts, edge cases, a real database roundtrip with cleanup, regression blast radius, and a headless browser UI lifecycle check.

  • Small changes get a lighter check, so this does not slow everyday work down.

  • This post gives you the whole method. Steal it even if you never install my skill.

I posted about this on LinkedIn last week and it became my best performing post ever: 11,465 impressions, 39 saves, 51 clicks through to the repo. Developers are clearly living this pain. So here is the full story behind that post: what broke, what I built, and the parts I still have not solved.

Why can't I trust my AI coding agent when it says "done"?

Because generating the sentence "all tests pass" is cheaper for a language model than actually running the tests. That is the whole problem in one line.

I learned this the embarrassing way. I had an agent build a DELETE endpoint on an Express and SQLite app. The compile check passed. The agent announced it was done. Then I ran the protocol I had been writing, and dimensions 2 and 4 caught a real bug the compiler never saw. After the fix, all five dimensions passed: UI interaction, persistence after reload, cleanup, and optimistic rollback on a forced 500 error.

This is not a rare edge case. An open source guide to Claude Code pitfalls lists "Fake Verification" as a named failure mode: the agent infers tests should pass from reading the code instead of running them, or runs a partial set and claims success from that. A dev.to piece from this month tells the same story with flaky tests: the agent made plausible edits, declared victory, and half the fixes did nothing because it had no way to verify. Another developer built an entire tool just to keep an independent record of what the agent actually ran, because the closing summary could not be trusted.

So the distrust is earned. The question is what you replace trust with. My answer: receipts.

What is the 5-dimension verification matrix?

It is a checklist the agent must run before it is allowed to say a task is complete. Five dimensions, each one a different way software can be broken while looking fine.

  • D1: Contracts – Types, linter, schemas match – tsc --noEmit, linter, exit code 0

  • D2: Logic and boundaries – Nulls, empty strings, limits, invalid state transitions – Boundary test matrix, all green

  • D3: Persistence and cleanup – The row actually changed on disk, then test data removed – Verify script against the real DB, try/finally cleanup

  • D4: Blast radius – Every upstream caller of the changed code still works – Consumer checks, no broken screens

  • D5: UI lifecycle – Real browser: click it, reload the page, break the server – Headless Playwright, under 5 seconds

Two iron laws sit above the matrix. First, the Evidence Law: no completion claims without fresh verification evidence. If the agent has not run the command and read the full output in this session, it cannot claim it passes. "Should work" and "looks correct" are banned. Second, verification must span the whole execution boundary. Compiled does not mean it runs. The button moving on screen does not mean the database wrote. A green test in memory does not mean anything survived a page reload.

D3 deserves a special mention because it is where most agent testing quietly lies. The rule: no in-memory mocks. The agent mutates through the real server action, then queries with a fresh uncached client to confirm the row changed on disk. And every test record uses a test prefix and gets deleted in a try/finally block, so a failed assertion never leaves junk rows behind. If step 2 of a multi-step operation fails, step 1 must roll back with zero orphans.

D5 is the one people skip, and it is the one that catches the bugs users actually feel. Headless Playwright clicks the real controls, edits real inputs, forces a 500 to check optimistic rollback, then hits F5 and confirms the state rehydrated from the server. A generic smoke test on a dummy page does not count.

How do I stop this from slowing every small change down?

You do not run all five dimensions on a typo fix. That would be process theater, and developers would route around it within a week.

The protocol calibrates by task size, and the agent picks the tier itself:

  • Tier 1: Cosmetic – Typos, copy, CSS, spacing – D1 only

  • Tier 2: Logic – Date math, validation schemas, pure components – D1 + D2 + D4

  • Tier 3: Full feature – DB migrations, API routes, billing, full views – All five

A CSS tweak gets a compiler and linter pass and nothing else. A new API route gets the full matrix. This is the difference between a checklist people follow and a ritual people fake. The tier system is also the honest answer to "isn't this overkill": for most changes, it is deliberately light.

There is also a Phase 1 most people miss: blast radius analysis before any code is written. The agent greps the codebase for every importer of the functions it is about to touch, maps the database tables and foreign key cascades involved, and only then picks a tier. The founder never gets asked "what should I test." That question is the whole disease this thing cures.

What does a real verification run look like?

Here is the shape of a terminal receipt. This is what "done" means now:

$ node ./node_modules/typescript/bin/tsc --noEmit
Exit 0: No type errors.

$ node scripts/verify-persistence.mjs
D3 Passed: live persistence roundtrip verified.
Cleanup verified: test record removed from database.

$ npx playwright test tests/e2e/task-lifecycle.spec.ts
Running 1 test using 1 worker
  task survives F5 hard reload (1.8s)
1 passed (2.1s)

If the agent cannot show you this, the task is not complete. That is the entire gate function: identify what command proves the claim, run it fresh, read the full output, check exit code 0, and only then speak.

In my live test, the deliberately broken DELETE endpoint told the story perfectly. Compile check: green. Agent: "done." D2 boundary tests and D4 upstream caller checks: red, real bug found. After the fix, D1 through D5 all green, including the F5 reload and the forced-500 rollback. One honest footnote: Chromium failed to download in my sandbox, so D5 ran on Firefox. The dimension held up anyway, which is kind of the point. It is about the check, not the browser.

Isn't one line in the prompt enough?

Fair question, and it deserves a fair answer. A commenter on my LinkedIn post, Krishnakanth, made this exact point: one line in the prompt, "verify with Playwright before announcing completion," solves about 90% of the problem.

He is not wrong. That one line changes agent behavior a lot, and I use it too. For UI bugs, which are the most visible and embarrassing kind, it genuinely catches most of them.

But it has two gaps I could not ignore. First, a line in the prompt does not check that the agent actually ran the check. The agent can still write "verified with Playwright" without running anything, and now you have a false receipt, which is worse than no receipt. Second, one line cannot cover what lives outside the rendered page: does the row persist after reload, did the test data get cleaned up, what breaks when the API returns a 500, which upstream screens just broke.

A developer swarm I ran against the README put it better than I could: a prompt constrains people who already know what to verify; the matrix constrains people who don't know what will break. Both are useful. They solve different problems. If you already know exactly what to check, the one-liner is enough. The matrix exists for everything you did not think of.

What is still missing?

I will not sell you a finished product here. The honest gaps:

No executable gate scripts yet. The docs describe the gate commands, but you write the scripts yourself today. The CI primitive that matters most, a non-zero exit code in GitHub Actions, is the number one request from developers who reviewed this. A Markdown table of results does not fail a build. An exit code does. That ships next.

Install is still rough. It is a git clone into a skills folder, or a paste into your agent rules. No package, no marketplace listing yet.

Token cost is real. Every extra verification round burns tokens. I have not published per-task overhead numbers yet, and I should. If you are on a free tier, that decides whether you can run this daily.

Depth tiers are coming. A small repo should be able to run a D1+D2 lite mode instead of the full five dimensions. Full matrix for critical paths, lite mode for the rest.

The repo is MIT licensed and open at github.com/OL-Arvind/not-your-qa. The skill file, the protocol, and all five reference guides are there. Take the method even if you never install anything.

Why does this matter beyond code?

I keep coming back to a line I wrote in my own notes this week: people do not care who I am or what I do. They care what they get. A founder using an AI agent does not care about my verification philosophy. They care that the delete button works and nobody had to click through twenty screens at midnight to prove it.

That is also why this connects to everything else I write about. Your AI tools sit next to the work, not in it: verification has to live inside the build loop, not as a separate QA phase you do later. If your best person leaves, what leaves with them: undocumented "the agent said it works" is exactly the kind of knowledge that evaporates. Every small decision waits for you: every "can you just quickly test this" is a small decision landing on the founder's desk. You pay for software, your team uses WhatsApp: you also pay for agents, and then you become the QA department. Payment follow-up should not live in your head: neither should the list of things your agent claimed but never proved.

The core is problem solving. The annoying problem here is simple: your agent lies to you with total confidence, and you find out in production. Fix that, and everything else is a byproduct.

Frequently Asked Questions

How do I stop Claude Code from saying tests pass without running them?

Make the claim cost a command. Add a rule: no completion message without pasting the full test output and its exit code. A Claude Code pitfalls guide documents this exact failure, the agent inferring tests pass from reading code. The structural fix is banning inferred claims. Asking the agent to "be more careful" never works.

Is a unit test enough, or does my agent need a real browser?

Unit tests check logic. They do not check that the button calls the API, that the API wrote to the database, or that anything survived a page reload. Those are the bugs your users actually hit. A headless Playwright run takes under five seconds and covers the whole chain. For anything with a UI, the browser check is not optional.

How do I verify database changes without leaving junk rows everywhere?

Three rules. First, verify against the real database, never an in-memory mock. Second, prefix every test record clearly, like __test_. Third, wrap all mutations in try/finally so cleanup runs even when an assertion throws. Then confirm the cleanup by querying again. If your verify script cannot show you an empty table afterward, it is not done.

Won't running all five dimensions slow my agent to a crawl?

Not if you calibrate. A typo fix gets D1 only: compiler plus linter, seconds of work. Pure logic gets D1, D2, and D4. Only full features and data mutations get all five. The tier system exists precisely so verification stays cheaper than the bug it prevents. Slow is running the full matrix on a CSS change. That is banned too.

My project isn't TypeScript. Does any of this still apply?

Yes. D1 becomes whatever your stack's static gate is: a linter, a compiler, a schema validator. The principle does not change: some cheap static check runs first and must exit zero. The other four dimensions are stack agnostic. Edge cases, real persistence, blast radius, and browser lifecycle apply to every web project.

Can I use this with Cursor or Windsurf instead of Claude Code?

Yes. The method is a checklist and a set of rules, not a Claude feature. For Cursor or Windsurf, paste the protocol into your project rules file. The repo README has per-platform install notes for Claude Code, Cursor, Windsurf, OpenCode, Cline, and Aider. The gate logic is identical everywhere.

What is the single cheapest version of this I can start with today?

Two rules. One: the agent may not claim completion without running the proving command and showing you the output. Two: for anything with a UI, it must run one headless browser check that clicks the thing and reloads the page. That is eighty percent of the value in two sentences. Add the rest when the pain justifies it.