Graphic: Vibecode Beratung · Based on the sources below
Your team now has Claude Code or Cursor, and the agents produce more pull requests than anyone can read carefully. The senior engineers become the bottleneck. They read diffs, run the app, find a bug, explain it back to the agent, and start again. The tool sped up writing code, but the time it takes to know whether the code is correct did not change.
In a recent essay, Addy Osmani argues that this is the real shift: with current models, the limiting factor is no longer writing code but verifying it. His answer is to stop treating review as the main quality gate and to give the agent checks it can run on its own. This article turns that idea into decisions for an engineering lead.
Why a human-only review loop does not scale
Imagine a 12-person product team where every agent change goes straight to a human reviewer. The reviewer is the only feedback the agent gets. Osmani describes this as the slowest and most expensive loop, and one that does not scale past a single person. An agent that cannot verify its own output is, in his words, essentially guessing.
The alternative is a self-verifying loop. The agent writes a change, runs a check, reads the result and tries again until the check passes. Only then does a person look at it. Osmani says he would prefer a much better feedback loop over a slightly better model.
This does not remove the human. Someone still owns what ships. But the reviewer now looks at evidence (which checks ran and what they proved) instead of only reading lines of code.
End-to-end tests as the definition of “done”
If you can invest in only one kind of test, Osmani recommends end-to-end tests. They do what a user does: sign up, add an item to the cart, check out, see the confirmation. They ignore internal structure.
This matters because agents tend to write code that passes checks but misses the point. A mock-based unit test can pass while the feature is broken, because the agent and the test share the same wrong assumption. An end-to-end test either works or it does not.
Two practical points stand out. First, “implement this feature” is vague, while “make this end-to-end test pass” is a target the agent can chase alone. Second, failure output must be useful: screenshots, traces and the name of the failed step help, while “Timeout after 30s” does not. Keep the set small and focused on high-value flows, because these tests are slower and flakier than unit tests.
Property and differential tests for what nobody thought of
Example tests say what should happen. Property-based tests say what must never happen. You state a rule, such as “a balance never becomes negative” or “money in equals money out”, and a tool like fast-check (JS/TS), Hypothesis (Python) or proptest (Rust) generates many inputs to break it. A failure is reduced to the smallest input that still fails.
For agents, this closes a specific gap: an agent invents test examples that succeed according to its own logic. A property forces the code to handle inputs the agent did not choose. Osmani suggests starting with a small, pure function, such as a parser, a formatter or a pricing calculation, writing the property in plain English first, and asking the agent for properties instead of examples.
For rewrites and migrations, differential testing treats the old system as the oracle. Run old and new side by side on random inputs or replayed traffic, and every difference is either a bug or a deliberate change you document. This pairs well with replacing a fragile system in parts, as described in refactor or rewrite.
Slow or flaky tests teach the agent the wrong lesson
Humans know which tests are flaky and rerun them. An agent learns from the loop’s incentives: if a rerun sometimes passes, retrying becomes its strategy. If the suite takes 15 minutes, it runs tests less often and batches more changes, so failures are harder to trace. Both make output worse without a visible signal.
Ranked by severity for a team rolling out agents:
- SEV-1: flaky tests in the main suite. Osmani calls a flaky test there worse than no test, because it breaks trust in every other result. Isolate or remove them.
- SEV-2: non-deterministic tests. Use seeded randomness, frozen clocks, controlled networking and isolated state.
- SEV-3: noisy output. Short, specific failure messages beat long logs about what passed.
Keep a fast inner loop for the agent, and run the heavier end-to-end and differential checks before changes are integrated.
What this means for your project
If you are introducing AI coding agents into a team, the test infrastructure is part of the rollout, not a side project. Before scaling agent use, check three things: whether your critical user flows have end-to-end tests an agent can run locally, whether your core suite is fast and deterministic, and whether lessons are captured. When an agent makes a mistake, Osmani’s advice is to write it down as a test, a hook or shared config so it does not happen twice.
Some further habits from the same essay: ask the agent to restate the success criteria before it codes, use a separate reviewer agent with fresh context, and treat generated tests with suspicion. Ask whether a test actually separates a correct implementation from one that only looks plausible. For apps that go live without such tests, a production readiness check is the place to find the gaps.
The core shift is simple: your engineers spend their judgment on defining what success looks like, and the agent does the repetitive checking.
Want to know whether your team’s setup is ready for agents that verify their own work? Book a free intro call and we will look at your AI coding rollout together.
Frequently asked questions
Why do AI coding agents need tests they can run themselves?
Without a check it can run, an agent is essentially guessing, and a human has to read diffs, run tests, find bugs and report them back. That loop is slow, costly and does not scale beyond one reviewer. With a runnable check, the agent can change, test and retry before a person looks at anything.
Which type of test helps an AI coding agent most?
If you can only pick one, end-to-end tests. They describe what a real user does and only look at the visible result, so they are hard to trick. A mock-based unit test can pass while the feature is broken because the agent and the test share the same misunderstanding.
What is property-based testing in simple terms?
You write a rule that must always hold, for example that a cart total equals the sum of its line items, and a tool such as fast-check, Hypothesis or proptest generates many inputs to break it. When it finds a failure, it reduces the input to the smallest case that still fails.
Why are flaky tests a bigger problem with AI agents?
An agent learns from the incentives of its loop. If a failing test sometimes passes on a rerun, the agent may learn to retry instead of fixing the problem. Slow suites also lead it to run tests less often and batch more changes between runs.
Do humans still need to review AI-written code?
Yes. A human still owns what ships, but the review shifts from reading every line to checking evidence and defining what success looks like. When the agent gets something wrong, the fix should be written down as a test, hook or shared config so it does not happen again.