If you’ve played around with GPT or any large language model (LLM), you’ve probably been impressed—and sometimes alarmed—by how confidently it generates completely fabricated nonsense. These are what we call hallucinations: when an AI outputs inaccurate or made-up information as if it were true. But to prevent them, you need to understand why they happen…
Thank you for reading this post, don't forget to subscribe!
At the core, most LLMs like GPT are trained to complete text based on patterns. They predict the next word based on what they’ve seen during training. They’re not checking a database or fact list, they’re just doing statistical guessing.
For example, ask a model “Who was the 25th president of the US?” Unless it’s specifically tuned or grounded in fact sources, it might invent an answer like “Benjamin Franklin”—because the question format looks similar to ones it has seen where historical figures were the answer, and it guesses based on pattern, not correctness.
Cause of Hallucination
What It Looks Like
No source grounding
Cites imaginary articles or books
Overconfident text generation
Presents fiction as solid fact
Ambiguous prompts
Misinterprets intent or invents context
One time I asked an LLM to generate a brief bio for a niche software engineer I knew from Twitter. The model gave me a detailed history of their career, including companies they never worked for. When I asked, “Where did you get that from?” it hallucinated a citation from a non-existent TechCrunch article. That behavior wasn’t strange—it’s what happens when it doesn’t have real knowledge but still needs to fill the gap.
In the end, hallucinations stem from LLMs having no internal concept of truth—only text patterns. That’s why prevention has to be external and intentional.
Start With Precision: Ask Fully-Scoped Prompts
Most hallucinations begin with vague or underspecified prompts. You ask for a “summary of GDPR rules” without saying which country, jurisdiction, or date the article is from, and you get a dreamy mashup of random legal-sounding text. Instead, always force the model to work within boundaries.
Here’s a real example from a test I ran using GPT-4:
Prompt A: “Summarize the GDPR.”
→ Output included outdated rules, and referred to the UK as still being part of the EU.
Prompt B: “Summarize the key rights under GDPR (as of 2023), focusing only on data subject rights in the EU.” → Output was far more accurate and didn’t bring in irrelevant post-Brexit details.
Precision also matters when asking models to generate statistics or quotes. Try prompting:
Bad: “Give me stats on C++ usage across companies.” Better: “Cite publicly available developer survey data from 2023 showing the usage of C++ in backend applications, and include the URL if known.”
If the model invents stats anyway—which happens—it’s way easier to check what URL it gave or call it out right in the chain: “Is this a real page?” That tiny nudge can stop the hallucination short.
Pro Tip: Add “Only reply if this data is verifiable via a known source” to the end of your prompt. Sounds basic, but it pushes models toward sourcing or refusing.
Ultimately, assuming the model knows what you mean is the fastest path to a confident wrong answer.
Use Retrieval Augmented Generation Whenever You Can
RAG (short for retrieval augmented generation) is a method where your prompt is combined with knowledge pulled live from a database, API, or document corpus. It’s like giving the model a cheat sheet before it tries to write an essay.
Let’s break that down: traditional prompting is just you talking to the model. RAG is you handing it a folder of receipts, then asking for a summary based only on what’s in that folder.
During testing, I ran a prompt to write product documentation for a feature no public documentation existed for yet. Using vanilla prompting, GPT invented methods and CLI flags that sounded correct. I switched to a RAG workflow (using LangChain with a local vector store of my internal PDF docs), and suddenly the AI said “No information found” when a feature wasn’t mentioned in the source context. That’s what you actually want.
To implement RAG, you’ll need:
A way to chunk + embed your source content (e.g., documents, databases)
A vector database (like Pinecone or Weaviate)
An orchestration layer (LangChain, LlamaIndex, etc.) to connect query → relevant chunk → output
Here’s a rough flow:
User Prompt → Embeds the question → Finds closest documents → Sends both prompt + docs to LLM → LLM answers only using the provided docs
This limits hallucination because if the info isn’t in the chunk, the LLM can’t guess its way into inventing it. To sum up, RAG turns language models from freelancers with great vocabularies into trained paralegals who look up the material before talking.
Force Model Honesty With Chain-of-Thought Prompts
Another way to reduce hallucinated confidence is to force the model to explain how it’s getting its answer. These are called Chain-of-Thought (CoT) prompts. They increase accuracy by asking the model not just what the answer is, but what steps it uses to think it through.
Instead of just prompting: “What year was the iPhone released?”
Try: “Think step-by-step. What was the event where Apple announced the first iPhone? When was that? What year does that make the launch?”
In my tests, models hallucinated less when I included the phrase “explain your reasoning step by step.” And even if the answer was wrong, I could see where the thinking went off-track.
This is especially useful when asking for code or instructions. Instead of:
Write Python code to send Slack notifications
Use:
Explain first what approach you’ll use to send Slack messages from Python, then generate the code only after the approach is explained.
You’ll often catch when it’s about to do something outdated or unsupported, and you can nudge it.
This also works in reverse. When the model gives you something and it feels suspicious, simply reply: “Explain how you got this result.” That alone causes occasional walk-backs where the AI goes “Actually, that might be incorrect.” 🤯
Overall, CoT prompts act like asking a friend to show their work on a math problem—they make the truth (or fiction) easier to spot.
Don’t Let It Guess Sources—Use Inline Verification
The biggest giveaway of hallucination is fake citations. The LLM crafts an academic-sounding quote, then appends a made-up URL. It’s elegant-sounding fiction wrapped in a hyperlink. To counter this, require inline verification using a prompt style that pairs each claim with its evidence:
For each fact or statistic you include in your response, provide the exact online source (URL) immediately after the claim in parentheses.
If it starts faking links (e.g., to non-existent pages on nytimes.com), follow up with:
Is this an actual working URL? Only include sources that exist. Say “No source” if none found.
You can’t always stop the first response from including lies, but you can catch them with a second pass.
When I prompted a bibliography for a rare topic (history of Macedonian typography in computing), GPT fabricated every single citation. But using the follow-up “Are these real sources?” it actually redacted its own list and offered “Sorry, these may not be real.” That’s a win.
To conclude, never trust footnotes until you force the model to double-check them itself.
Add Verifiability Prompts at the Output Level
One of the weirdest side effects of LLMs is that they’ll often say “according to research” without ever listing actual research. Even if you prompt correctly, the output can still trail off into unverifiable fluff. That’s where output-level instructions come in.
This is as simple as ending your prompt with:
“Only include data that can be cited and externally verified.”
“Do not guess if unsure—say ‘insufficient data.’”
“Summarize only from direct quotes or facts, not assumed knowledge.”
It sounds redundant if your prompt already asks for accuracy, but it actually works about 60% of the time in reining things in.
I tested the same exact request for “latest cloud adoption statistics” with and without these extra clauses. Without, GPT mixed in 2020 data and made up survey names. With the clauses, it either provided real links or returned “No confirmed statistic found.” That’s what you want: humility over fiction.
The bottom line is, forcing verification at the end stage helps clean up even a well-built initial prompt.
