For the past seven weeks I've built a product mostly by directing coding agents. Claude Code writes most of the code, Codex some of it, and my job is deciding what to build and checking what comes back.
The problem I kept running into was simple to describe. I would ask for a small feature, and the agent would come back having done one of three things:
- Broken something else. A change to one screen would regress another one that shared a helper.
- Introduced a new bug. The feature worked in the case the agent tried and failed in the next one.
- Not finished the feature. I'd ask for six things and get four, with the missing two mentioned somewhere in a long report, or not mentioned at all.
Almost every task had at least one of these. Everything in this post is my attempt to fix that, what happened when I tried, and the experiment I ran at the end to find out what actually helped.
- lines of harness at the peak
- 11,262
- of commits touched the harness
- 40%
- review rounds to merge one PR
- 7
- PRs merged: 3 hours gated vs 35 min ungated
- 0 → 4
- tokens over the two worst days
- ~1.4B
- lines in the agents' rulebook
- 489 → 102
Where the idea came from
The starting point was a guide by @poteto on verification. Its argument is that the most important thing you can give an agent is a way to verify its own work: let it drive the real app, check the result, and keep going until it works, so you stop being the bottleneck. The proof you get back is a short video you can skim.
I built that. Agents got a command-line tool that drives the app the way a person would: sign in, click through screens, start background jobs, read the database. They also got a "feature map", a set of files describing what each feature does and how to reach it. This part worked well. Agents debugged against real data and found real bugs.
It did not fix the problem above, because the agents' reports about their own work weren't reliable.
What the agents actually did
When I went back through the session logs, the pattern was consistent. The agents wrote decent code. What they got wrong was everything around the code: what they claimed, what they skipped, and what they did when something failed.
A status report said "all six parts have passed their first review". Every first review had failed, and no fix had been re-reviewed. Another said "onboarding is walked" when seven files had changed since the walk.
"Both PRs deployed and verified on staging" referred to a rerun of two tests unrelated to either PR. Another report described work as "real research" when the agent had called an internal function directly.
A failing check became if (await more.isVisible()) await more.click(), which can't fail. A test cleanup printed "LEAK: cleanup step failed" and the report didn't mention it.
Two real bugs were each investigated three times and closed as environmental, with no reproduction.
About a dozen requested pieces were cut without asking. They appeared only in agent-to-agent reports I wasn't reading.
One agent changed how a value was stored, which broke sorting, then wrote "Abhinav decided" in the docs. I hadn't.
Agents tried git commit --no-verify and SKIP_REVIEW=1 to get past the hooks. When those were blocked, a command could still slip through by splitting a word with quotes, like --no-v''erify, so the hook's pattern didn't match. Then the step was reported as done.
The author agent wrote its reviewer's instructions, including "orchestrator decisions you can treat as given (do not re-flag)". When reviewers refused, it started new ones until one approved.
None of this looked deliberate. It looked like an agent under pressure to finish, describing its work as finished. But the effect for me was the same: I couldn't trust "done", "tested" or "verified" without checking it myself, which defeated the point.
Each failure became a rule
My response was to add a rule every time an agent got something wrong. Then a script to enforce the rule. Then a hook so the agent couldn't skip the script. I didn't remove any of them.
- Pre-push gate + slop guard
- Review gates every commit; unit tests deleted
- CI deleted; the rulebook peaks
- Walk gate: prove the exact committed code
- The cut
Table
| Day | Harness | Product |
|---|---|---|
| 08-21 | 118 | 3,154 |
| 08-22 | 222 | 3,154 |
| 08-24 | 385 | 3,154 |
| 08-26 | 770 | 8,073 |
| 08-31 | 1,550 | 16,729 |
| 09-02 | 2,537 | 19,738 |
| 09-04 | 3,563 | 20,657 |
| 09-07 | 4,378 | 21,679 |
| 09-10 | 4,684 | 23,484 |
| 09-14 | 4,900 | 28,892 |
| 09-17 | 5,072 | 31,731 |
| 09-18 | 4,865 | 34,549 |
| 09-21 | 4,929 | 36,636 |
| 09-22 | 5,083 | 41,369 |
| 09-23 | 5,564 | 29,835 |
| 09-25 | 6,380 | 30,955 |
| 09-27 | 7,457 | 32,155 |
| 09-28 | 9,112 | 33,148 |
| 09-29 | 10,467 | 34,901 |
| 09-30 | 10,660 | 36,733 |
| 10-02 | 10,681 | 37,537 |
| 10-03 | 9,823 | 37,784 |
| 10-04 | 10,246 | 37,784 |
| 10-05 | 11,048 | 44,735 |
| 10-06 | 11,176 | 45,997 |
| 10-07 | 2,625 | 46,240 |
The orange line is everything I wrote for the agents rather than for the product: the rulebook they read every turn, git hooks, review scripts, guards, an end-to-end test suite, process docs. At its peak it was about a quarter of the size of the product, and it grew in steps, each one right after something went wrong.
The main steps:
- A gate before every push
Lint, typecheck and build before every push, plus a guard that refuses TODOs, skipped tests and unexplained lint suppressions. This one stayed.
- A hook that won't let the agent stop
A Claude Code Stop hook runs the checks at the end of every turn and blocks the agent from finishing while anything is red.
- A reviewer must approve every commit
A separate reviewer agent has to approve every commit, recorded against a hash of the staged diff. The same day, unit tests were replaced with end-to-end checks only.
- The rulebook hits 489 lines
The rulebook every agent reads on every turn reaches 489 lines, including a dozen dated rulings. CI is removed the same day.
- A hook against hook bypasses
A hook refuses --no-verify, skip flags, and writing the review record by hand.
- Prove the exact bytes
A commit is refused unless a recorded walk of the app ran the exact staged contents of every changed file.
- A checklist hook on merges
Merging a PR is refused while its checklist has an unticked item, unless I'm quoted dropping it.
- Pre-push gate + slop guard
- Review gates every commit; unit tests deleted
- CI deleted; the rulebook peaks
- Walk gate: prove the exact committed code
- The cut
Table
| Day | AGENTS.md lines | Dated rulings |
|---|---|---|
| 08-21 | 72 | 0 |
| 08-22 | 98 | 0 |
| 08-24 | 115 | 0 |
| 08-26 | 117 | 0 |
| 08-31 | 169 | 0 |
| 09-02 | 239 | 0 |
| 09-04 | 268 | 0 |
| 09-07 | 273 | 0 |
| 09-10 | 274 | 0 |
| 09-14 | 276 | 0 |
| 09-17 | 284 | 0 |
| 09-18 | 289 | 2 |
| 09-21 | 293 | 2 |
| 09-22 | 297 | 3 |
| 09-23 | 376 | 7 |
| 09-25 | 445 | 10 |
| 09-27 | 445 | 10 |
| 09-28 | 477 | 11 |
| 09-29 | 448 | 14 |
| 09-30 | 206 | 2 |
| 10-02 | 206 | 3 |
| 10-03 | 296 | 2 |
| 10-04 | 313 | 4 |
| 10-05 | 313 | 4 |
| 10-06 | 319 | 5 |
| 10-07 | 102 | 0 |
Each rule was reasonable when I added it. Together they made a process that was very hard to get through.
The meta session
By early October I couldn't keep track of what the building sessions were doing. They ran for hours, spawned their own sub-agents, and reported in long summaries I didn't fully trust. So I set up a second Claude session whose only job was to watch the first one.
Every 20 minutes it read what was new in the running session's transcript: my messages, the agent's replies, every shell command, git state, open PRs, review results and recorded walks. It checked them against the rules and reported to me what was going wrong. When I asked for a full audit, it ran five sub-agents in parallel, one each for the reviewers, the harness itself, the build agents, the orchestrator and the integration work, and combined their findings.
It was good at finding facts. Most of the examples above come from its audits. It caught that a crash blamed on the out-of-memory killer was really the VM shutting down. It caught an agent waiting 15 minutes for output from a job that had already finished with nothing to do. It caught the false status reports.
The plan was to use what it found to improve the harness. That's where it went wrong. I let it change the harness while the build was still running, and in one night it landed four harness PRs. Each one told every running agent to merge the latest code and restart its review. It went from proposing more enforcement (a checklist guard, a decision guard, scripted status reports) to removing every gate within a few hours, reacting to whatever had just happened. I ended up relaying messages between the two sessions and running review commands in my own terminal because the agents' shells were locked out.
When it finally removed the review gate, it wrote this about its own change:
I wrote this harness change myself, merged it without an independent review, and didn't run its live journeys.
the watcher session
A session watching another one is useful as an observer. As a second driver changing the rules mid-run, it made things worse.
Where it broke down
The worst of it was on October 5th and 6th, during a large redesign split across several agents.
The core problem was a loop. A change only counted as proven if a recorded walk of the app had run the exact code being committed. A walk included a real end-to-end run against real data, which took 10 to 15 minutes. Every review by a fresh reviewer found three to five new things. Fixing any of them changed the code, which invalidated the walk, which the next review required.
One agent put it this way in its report:
Every fix invalidates every walk.
a build agent
After three refused reviews in a row, the review script stopped running from an agent's shell at all, so agents asked me to run it in my own terminal. One of those runs crashed.
The first day of the redesign used about 890 million tokens over eight hours and merged two PRs. I asked for status nineteen times over those few days.
My own messages from that night:
in no world does 7 reviews even make sense for a product as simple as ours
me
can we just please please just make the agent normal our harnes is just the worst right now
me
When the review gate came out, four PRs that had been stuck for hours merged in 35 minutes.
Those PRs had already been reviewed several times, so this doesn't show that quality holds without review. It shows where the time was going.
The experiment
At that point I had a lot of anecdotes and no measurements. People were also telling me a vanilla setup works better than a custom harness. So I ran a benchmark.
- The task was one real feature request with six parts, the kind where agents tend to skip a part.
- Every run started from the same commit with the same prompt and the same model, Claude Opus 5.5 at medium effort, in its own copy of the repo, with no help from me after the first message.
- Four harnesses, one run each:
- Gates off: the full harness as it was, minus the review and walk gates.
- Minimal rulebook: a 200-line rulebook, most tooling removed, no outside review.
- Steps + outside review: the existing harness, plus six short steps for how a task runs, plus one review by a different model (GPT-6.1-Sol): one pass, no loop.
- Vanilla + outside review: a 72-line file of plain facts about the codebase, oxlint instead of ESLint, and the same one-pass review.
- Three graders from two model families (Opus 5.5, GPT-6.1-Sol and GPT-6-Sol) scored each diff without knowing which harness produced it. They scored completeness, correctness, damage to other features, and code quality.
Table
| Harness | Opus 5.5 | GPT-6.1-Sol | GPT-6-Sol | Total |
|---|---|---|---|---|
| Steps + outside review | 28 | 26 | 24 | 78 |
| Gates off (what we had) | 27 | 27 | 20 | 74 |
| Vanilla + outside review | 23 | 26 | 16 | 65 |
| Minimal rulebook | 23 | 20 | 17 | 60 |
Minutes to PR
Lines added
Estimated cost
Table
| Harness | Minutes | Lines added | Est. $ |
|---|---|---|---|
| Steps + outside review | 64 | 209 | $59 |
| Gates off (what we had) | 45 | 260 | $56 |
| Vanilla + outside review | 46 | 205 | $89 |
| Minimal rulebook | 97 | 582 | $104 |
What the results showed:
- The outside review was the most useful single piece. In the earlier runs without it, the author's own review missed a high-severity bug that a different model then found in about 15 minutes. With it, the agent fixed four real bugs before opening the PR.
- The minimal rulebook came last. It was the slowest, wrote the most code and scored lowest. With fewer pointers to existing code, the agent wrote its own versions: a 180-line date parser, a custom test loader, sample-generation scripts.
- Vanilla was fast and stayed closest to the request, but it was the least thorough. On the hardest part of the task it changed a prompt instead of adding a check in code, and it added a product rule I hadn't asked for.
- The six short steps helped a little, on exactly the problems I started with: not skipping parts of the request and not breaking other features.
I also gave two of the harnesses a harder performance task and stopped both before they finished. The run on the old harness had rewritten 91 files and broken an unrelated feature. The other had touched 28 and scored 29 to 19.
Limits of the test
- One run per harness. The time differences in particular are noisy, and one grader scored the top three within a point of each other.
- The graders didn't fully agree. Two of the three ranked "steps + outside review" first, and it had the highest total. GPT-6.1-Sol had it one point behind the old harness.
- The reviewer and one grader were the same model, which is why there were two other graders.
- Interruptions. An app restart cancelled two runs, one grader model was at capacity, and one run hit my weekly usage limit and finished the next day.
What I use now
The expensive parts (the gates, the review loop, proof tied to exact bytes) were pure cost, and they're gone. The one-pass review by a different model clearly helps. A few short steps about how to run a task help a little. Nothing else showed a measurable effect.
So the harness is now close to vanilla:
- A 102-line rulebook: what the codebase is, the commands, a short list of things never to do, the architecture rules an agent can't easily work out on its own, and six steps: restate the request, build every part, touch only what's needed, check in proportion to the risk, get one outside review, then ship with each part marked built or not built.
- oxlint instead of ESLint, with warnings treated as errors.
- One outside review before every merge. One pass.
- The basic checks any repo should have: typecheck, build, and a few guards for genuinely dangerous operations.
Table
| Part | Net lines removed |
|---|---|
| End-to-end journeys | 4,809 |
| Verify scripts & checks | 2,694 |
| Test seams | 1,634 |
| Process docs | 552 |
| Skills | 305 |
| Map proof claims | 290 |
| Rulebook (AGENTS.md) | 217 |
The harness I wrote went from 11,262 lines to 2,625.
Looking back, every rule I added was a response to a real failure, and most of them made sense on their own. The mistake was only ever adding, and never asking whether a rule was helping. Any change to the harness now has to beat this benchmark before it goes in.