Catching AI’s confident mistakes

A fluent answer can be completely wrong — so check the claims, not the tone.

The idea

AI writes in a calm, confident voice whether it is right or not. You can’t re-do all the work, and you don’t need to. The trick is triage: find the few claims that could actually be false — names, numbers, dates, quotes, citations — and spend your limited checking on those. Framing and opinions have nothing to verify; leave them.

Spot-check the brief

Here is a brief an AI drafted for a pitch. It reads well. But you only have time for 3 checks. Tap the sentences you would verify first, then reveal what was actually planted.

How it works

A spot-check is a routine, not a vibe. Run the same passes every time:

• Isolate checkable claims. Underline every name, number, date, direct quote, and citation. Ignore framing (“expanding quickly”) and recommendations — there is nothing there to verify.
• Rank by consequence and source. An outside statistic wearing a citation beats an internal metric you could confirm in ten seconds. Check the claims that would most embarrass you if wrong.
• Spend the budget top-down. You will never check everything. Start at the top of the ranked list and stop when the budget runs out.
• Read the tells. Suspicious precision, a citation you can’t trace, a quote with no speaker, a number that is round and confident, facts that are a little too tidy.
• Save it as a checklist. Turn the recurring passes into a short list or a tiny eval, so next time it is a habit instead of a judgement call.

When to use it

Anything you’ll send, publish, or decide on	Always run the spot-check — a confident falsehood in front of a customer or an interviewer is expensive.
Fast triage under a budget	The trade-off: verifying costs time, so a budget forces you to rank. You are buying down risk, not proving every word.
Private brainstorming or throwaway drafts	Full verification is overkill — but still glance at any name, number, or date before you repeat it out loud.

Watch out for

• Fluency read as accuracy. The confident tone is identical whether the model knows the answer or invented it. Confidence is not evidence.
• Fabricated citations. A real-looking source that doesn’t exist is the most dangerous tell, because it makes a false claim feel checked. Verify the source itself, not just the sentence.
• Plausible-but-wrong specifics. Founders, founding years, and exact figures are easy to state and easy to get subtly wrong.
• Anchoring. If you build an argument on the first number you saw, verifying it later is too late. Check load-bearing numbers before you reason from them.
• Checking the easy thing. Confirming your own dashboard metric while skipping the outside statistic is spending the budget backwards.

Worked example

You paste an AI-written company brief into your interview prep. It says the firm was “founded in 1998” and “holds 42% market share, per Forrester.” In the interview you repeat both, confidently. If the founding year is off by five years and no such Forrester figure exists, you have just been wrong — out loud, to a hiring manager. Two checks, the date and the citation, would have caught it. That is the whole game: not distrust, just a habit of verifying the handful of claims that can actually be false.

Check yourself