Human in the Loop AI: Verify, Not Approve

An AI reviewer checked a piece of my client work in July 2026 and came back with three findings. Two were wrong. I still run it on everything.

Most advice on human-in-the-loop AI describes approval: a person looks at the output and decides whether it seems right. The gate that holds is verification: a person compares the output against something that existed before the model ran. The two feel similar from the reviewer's chair, and only one of them catches errors.

What does human in the loop actually mean?

Human in the loop AI is a workflow design in which a model produces or proposes an output and a person checks it against a reference before it takes effect. Approval asks whether the output looks right. Verification asks whether it matches something known: a source file, a reconciling total, a row checked by hand.

A useful test: if the person reviewing could not name what they compared the output to, the step was approval. Approval measures plausibility. Verification measures correctness, and it is the only version of the loop that can be run the same way twice.

The definition in three parts:

1. The model files a proposal. It never applies it.
2. A person compares the proposal to a reference chosen before the run.
3. The comparison is cheap enough to run every time, because a check nobody runs is not a gate.

Why does confidence tell you nothing?

Because a wrong finding and a right finding arrive in identical wording. Nothing in the reviewer's tone, structure, or certainty separates them, so the reader has to check each one against the source, every time.

In the July review, the reviewer returned three findings on an analysis of mine:

• It recomputed one of my figures with a narrower filter than the original analysis used, got a different number, and reported mine as the error.
• The second finding dissolved the moment I opened the file it pointed at.
• The third was a real arithmetic mistake I would have shipped to a client.

Neither miss was a hallucination. Both looked like careful work. And the wording was identical across all three; the correct finding carried no extra certainty and the wrong ones carried no less.

It was a reminder of how confidently a model can be wrong, and why reading every AI output with a critical eye is one of the keys to avoiding embarrassment afterwards.

The cost arithmetic settles what to do about it. Checking one finding against the actual file takes about a minute. Acting on a wrong finding puts a wrong number in a client-facing document. Skipping the review altogether means the real arithmetic error ships. So the rule I run now is simple: the reviewer files findings, I verify each one against the file, and it never merges anything on its own.

Since then I have built an agentic workflow that reviews every client-facing deliverable through several lenses at once. It is expensive to run in tokens, but it flags errors before they reach a client document, and that is worth the peace of mind.

What does a verification gate look like in practice?

At volume, a verification gate means having the model build a tool whose output a person can audit line by line, instead of asking the model for the answer directly. The output has to arrive in a form a human can actually check, and 1,700 rows read by eye is not that form.

The case that taught me this was a client mid-HubSpot implementation. Their contact data lived in a single Word document: 386 tables, more than 30,000 words, hand-updated for years by different employees, with no two tables formatted alike and fields missing at random. It had to become an auditable spreadsheet.

Pasting it into a chatbot failed twice. Copilot crashed on the file. Claude crashed on the same file. The approach also has a structural problem: a model asked to transcribe messy data will sometimes fill a gap instead of flagging it, and nobody can verify 1,700 rows by reading them.

What worked was a change of output, not a better prompt. I mapped the first table by hand to show the model what a correct row looked like, then had Claude Code write a Python parser built on three rules:

1. Extract faithfully. Copy what the source says, including what is wrong with it.
2. Count every anomaly. A missing field or an unreadable address is a counted event, not a guess.
3. Fix nothing silently. Every correction is flagged for a person to make.

The parser's results:

The 43 flags matter more than the 1,728 rows. A chatbot would have produced rows too, with 43 guesses inside and no way to find them.

The same design shows up in agent code, which shows the pattern is not specific to spreadsheets. Anthropic's September 2026 guide to commerce agents, by Ali Shazal and Matthew Koen, states the harness rule for its reference agent in one sentence: "No model tool call moves money or changes the business." That describes their reference implementation, not AI agents in general, and three mechanisms in the published code carry it:

• Server-issued IDs. A write accepts only an ID the server handed the model during the current session. Anthropic describes that per-session record as "the only key any write or render will accept," so an ID the model invented, or one that arrived inside a product review, is refused before it reaches any backend.
• Staged changes. In the guide's words, "Every write tool produces a staged change with a server-generated ID, and apply_change succeeds only for IDs that have been approved through a real surface." A real surface is a button in the operator's portal or a confirmation in the CLI. A yes typed into the chat approves nothing.
• Limits on the resulting state. The guide's rule is to "Enforce every limit on the resulting state rather than the request," and the reference code checks again at apply, so a run of individually legal calls cannot add up past a cap.

Set beside the parser rules, they are the same gate: the model proposes, a person or a system-of-record check verifies, and nothing applies itself.

When is the loop too expensive to run?

The loop is too expensive when the check costs more than the error it prevents, or when the reference does not exist. A gate nobody runs is not a gate, and the second failure is more common: the team never wrote down what a correct result looks like, so the reviewer has nothing to compare against and falls back to approval.

Bank of America Institute's February 2026 payments data found that roughly 3% of its customer households pay for any AI service, at a median of $20 a month. The honest counterweight from the same dataset: those payments were up 38% on the 2024 average. Adoption is early and rising, which means most teams are designing their review step now, before the habit sets.

The evidence that the checkpoint pattern survives a production workload is one operator's account, not a measurement. An agency I work with runs a content workflow with a person at every checkpoint rather than one reviewer at the end. By the owner's own count, employee time on that workflow fell by roughly 75%. Treat that as a sign the pattern holds under load, not as a benchmark to plan against.

Two rules keep the loop affordable:

• Name the reference before the run. One sentence someone else could apply: the source row, the reconciling total, the file the finding points at. If it cannot be written down, the task is a judgment call and a person reading slowly is the whole job.
• Keep the check next to the output. A check that lives in someone's head runs once. A check that lives beside the output, as a flag count or a spot-check log, runs every time.

Language models are probabilistic, so without the right constraints the failure modes are close to endless. I built a Claude Code skill for the verification misses I keep meeting: at the end of every session I type /retro and it turns that session's corrections into instructions, skills, and automations for the next one. Every recommended fix still needs my approval before it lands.

The reviewer that got two of three wrong is still in my workflow because the minute it costs to verify each finding is cheap, and the arithmetic error it caught was not.

If you want a second pair of eyes on where your review step actually verifies and where it only approves, that is what our AI transformation advisory engagements cover. And if the question is whether the workflow should be agentic at all, start with when not to use agentic workflows.