Confident and wrong

What the paper develops

Fluent AI output can enter work that matters carrying more confidence than evidence, and "have a human review it" often does not catch the problem. This paper takes a position: output that could change a decision or an action that matters should pass a five-part challenge protocol before anyone depends on it.

I have reviewed generated analysis that looked excellent: clean structure, plausible figures, and cited sources. One number felt slightly convenient, so I opened the cited report. The report existed. The number did not. The model had blended two adjacent statistics into one that said what the narrative needed, and nothing in the document gave that away. The only signal was that it agreed with us a little too well.

The hardest AI error to catch is not nonsense; nonsense filters itself. It is the answer that is coherent, specific, and shaped to what the reader expected, and wrong in a way that survives a quick read. The paper's working claim is that an answer that confirms what someone already thinks gets less scrutiny than one that argues back. That claim is an inference. It builds on the long-studied tendency called confirmation bias and on Anthropic research in which people's preference ratings favored answers that matched a user's views. No study cited here measures how closely people check an AI answer at work.

The failure itself is documented. A 2024 study asked four models from 2023 about random federal court cases. ChatGPT 4 gave answers inconsistent with legal fact on 58 percent of the questions and Llama 2 on 88 percent. Those models have been replaced, so the study shows the rate can be high where the stakes are real, not what it is today. In an unreviewed experiment with 257 medical students, AI explanations raised accuracy by 6.3 percentage points when the AI was right (73 percent of cases) and lowered it by 4.9 when it was wrong (27 percent). They helped overall and hurt exactly when the AI erred.

The parts mostly depend on each other in order. The first decides whether the rest apply and how hard. The second feeds the third, because what the attempt to break the answer turns up is the first list of claims to check. The fifth learns from what the others recorded, so it comes last. The one exception is the reviewer for high-stakes output, who is named right after the first part because that person runs the second and third.

Name the decision it could change

A one-minute use declaration before review, which is a planning estimate: which decision, affecting whom, how reversible, at what cost if it is wrong, what the requester expected the answer to say, and for what use, whether brainstorming, drafting, a recommendation, or the basis for an action. Low-stakes uses stop here. Scrutiny concentrates where the stakes do.

Try to break the answer

Before strengthening the answer, try to break it. What would make it false? What other explanation fits the same facts? Who would dispute it, and why? Break hardest where the answer matches what the requester expected. A conclusion that has only survived attempts to improve it has earned nothing yet.

Check the claims outside the answer

Follow the claims the decision rests on to ground outside the answer. Open the cited report. Re-run the calculation. Read the clause itself. Asking the model whether it is sure only consults the same witness again; everything inside the answer is testimony from that one witness.

Name a reviewer, and a route past them

Review works when a named person owns what happens to the output, with relevant expertise, protected time, and the standing to stop the work, and has a defined route for what exceeds them. Volume caps come before generation growth. A reviewer clearing dozens of items a day signs without checking.

Learn from the errors you catch

Every catch is operating evidence. Keep a light failure log, review repeated failures monthly, and give someone authority to change prompts, data, permissions, or review tiers. Otherwise the organization keeps paying for the same lesson in near misses.

Put the five questions in the interface or the work record where people actually depend on the output. Start with one workflow where the stakes are real. The early weeks may turn up catches nobody expected; that is the baseline being found. Then watch one number: how often an outside check changes a claim. It is the share of checked claims that turned out wrong or unsupported, not the workflow's overall error rate, and no source here says how large it should be. If it stays near zero in a tier, test the checkers first with planted known-bad items or a sampled re-check; loosen only if they catch them.

The protocol is the paper's own design, and no study tests it. The white paper answers four objections (distrust, better models, speed, and proof), develops each part with how it typically fails and what counters that, covers the ways a review gets faked (forms filled in by rote, checking the answer against itself, approval in name only), and gives a starter failure log.