Why an AI Answer Can Sound Certain and Still Be Wrong

Watch the briefing

Watch the full video on YouTube

Briefing

This answer sounds certain: “The workshop is open to everyone, and registration closes Friday.” But the evidence beside it is blank. Would you forward that answer?

Before we do, separate two things: how convincing the sentence sounds, and what actually supports it. Those are different questions.

NIST's final generative AI profile identifies a risk called confabulation: a system can confidently present information that is false. Language models predict likely language; a plausible continuation is not, by itself, a fact check. Even an explanation or citation can be wrong.

Let's make that distinction visible with an invented example. No named AI tool produced this answer. The workshop, notice, and response were created for this demonstration.

Here is our fictional notice: “Workshop: Saturday, October tenth. Current members only. Register by Thursday, October eighth.”

Now put the confident answer beside it: “Open to everyone. Registration closes Friday.”

The answer contains two claims we can check. The first is about who may attend. The second is about the deadline. Don't evaluate the whole paragraph as one impression. Compare each claim with the notice.

“Open to everyone” conflicts with “current members only.” Mark that claim contradicted.

“Registration closes Friday” conflicts with “register by Thursday.” Mark that claim contradicted too.

Notice what did not help us: complete sentences, a friendly tone, and the absence of hesitation. We discovered the problem by examining evidence.

Now imagine the answer adds, “Source: workshop notice.” That label still proves nothing until we open the notice and check what it says. A source name can look reassuring while the claim beside it remains unsupported.

We can write a better response: “According to this notice, the workshop is for current members, and registration closes Thursday, October eighth.”

That answer is narrower. It tells us where the information came from and stays inside what the notice supports.

Suppose someone then asks, “Can I bring a guest?” The notice doesn't answer that question. The useful response is not to invent a guest policy. It is to say, “Guest eligibility is not stated here; check with the organizer.”

That is a third evidence status: not established. It is different from contradicted, and different from supported.

Try this on the next answer that matters. Identify the claim you might act on. Open the relevant source yourself. Check that it addresses the same question and that its date fits your situation. Then classify the claim as supported, contradicted, or not established.

This is a practical check, not a guarantee. A source can be outdated, incomplete, or mistaken, and consequential decisions may require expertise beyond one document. Nor does a hesitant answer automatically become accurate.

The takeaway is simple: fluent wording is a presentation feature, not evidence. Keep the useful explanation, but verify the claim before letting it guide an action.

A related briefing can examine how to check whether a citation supports the sentence attached to it.