Capabilities and limits

⏱ 25 min

Hook

A model tells a user, with perfect confidence and correct-looking citations, about a court case that never happened. Nobody typed a lie into it. The model was doing its one job — producing fluent, plausible text — and fluent is not the same as true. Knowing exactly where that gap opens is what separates shipping a useful feature from shipping a liability.

Concept

Recall from the first lesson that an LLM predicts likely next tokens. "Likely" is the key word: the model optimizes for text that looks like a good continuation, not text that is verified against reality. When a plausible- sounding answer and a true answer diverge, nothing in the core mechanism prefers the true one.

That gap has a name. A hallucination is a model output that is fluent and confident but factually wrong or unsupported — an invented citation, a made-up API method, a wrong date stated as fact. It is not a rare glitch to be patched away; it is a direct consequence of a system trained to produce probable text rather than to check facts. Treat fluency and confidence as unrelated to correctness, because in an LLM they are.

The most effective mitigation is grounding: giving the model the specific source text it should answer from, in the prompt, and instructing it to answer only from that text and to say when the answer is not there. Grounding does not make the model "know" more — it narrows the job from "recall the world" to "read this passage," which is a task next-token prediction is far better at. This is the seed of retrieval-augmented generation, the subject of GA-202.

A short mental model for when to reach for an LLM at all:

Situation	LLM fit	Why
Summarize or rewrite text you provide	Strong	Grounded in supplied text; low fact risk
Answer from a document you retrieve for it	Strong	Grounding narrows the task to reading
Recall exact facts, figures, or citations	Weak	Optimized for plausible, not verified, output
Exact arithmetic or precise counting	Weak	Token-level prediction is not a calculator
Decisions needing an audit trail or guarantee	Poor	Output is probabilistic and not accountable

Two more limits deserve naming because they surprise people.

First, an LLM's built-in knowledge has a training cutoff — it was trained on data up to some date and knows nothing after it unless you supply that information in the prompt. Ask about last week's events and an ungrounded model will either refuse or, worse, confabulate. Freshness comes from grounding, not from the model.

Second, capability is jagged, not uniform. A model can draft a working function and then miscount the items in a short list. Do not infer from one impressive output that the next, superficially similar task is safe. Strength on one task is weak evidence about a neighboring one.

This leads to a responsible-use stance you should carry into every later course. Use an LLM where a fluent, plausible draft over text is the goal and a human or a downstream check catches errors. Do not use one as the sole authority for high-stakes facts, safety-critical decisions, exact computation, or anything requiring a guarantee — and when output reaches users, be honest that it can be wrong. When a task needs a correct answer every time, an LLM belongs behind a verifier, not in front of the user unchecked.

🧠 Knowledge check

1. Why do LLMs hallucinate?

a) They are trained to produce plausible text, not to verify factsb) The training data is always wrongc) It only happens when the prompt has a typod) They run out of memory mid-answer

2. What does grounding a model actually do?

a) It permanently teaches the model new factsb) It supplies source text to answer from, narrowing the task from recall to readingc) It makes the model deterministicd) It removes the context-window limit

Worked example

Let me show the same question handled two ways so the difference is concrete. Suppose a user asks an internal help assistant, "What is our refund window?" I will contrast an ungrounded prompt with a grounded one, and reason about the failure surface of each — no API call needed to see the logic.

UNGROUNDED PROMPT
-----------------
System: You are a helpful assistant.
User:   What is our refund window?

Risk: the model has no access to *your* policy. It will produce a
plausible-sounding number — "30 days" — because that is a common refund
window in its training data, not because it read your rule. Confident,
fluent, and possibly wrong.

GROUNDED PROMPT
---------------
System: Answer only from the policy below. If the answer is not in it,
        say "I don't have that information."
Policy: "Refunds are accepted within 14 days of delivery for unopened
        items. Opened items are not refundable."
User:   What is our refund window?

Now the task is reading, not recalling. The supported answer — "14 days for
unopened items" — is in the text, and the instruction gives the model a safe
exit ("I don't have that information") when a question falls outside the
policy, which curbs confabulation.

Read the contrast against the concept. The ungrounded prompt asks the model to recall a company-specific fact it was never given — the exact case where hallucination is most likely, and most damaging, because the wrong answer looks right. The grounded prompt converts the same question into "find this in the supplied passage," and adds an explicit refusal path so an out-of-scope question gets an honest "I don't know" instead of an invention. Same model, same question, very different reliability — and the only change was giving it the source and permission to refuse.

Hands-on

Now provoke and then fix the behavior yourself. The exercise below opens in the Prompt Playground with a factual question the model gets confidently wrong when ungrounded.

▶ Prompt Playgroundga101-grounding-vs-hallucination

Try it in the Prompt Playground ↗

Do this: run the ungrounded prompt and note the confident answer, then add the supplied source passage and a "say so if it is not in the text" instruction and rerun. Success criteria: you can show one answer that changed and identify one question the grounded version correctly refuses to answer. The Playground saves both runs side by side.

Recap

• You can explain hallucination as a built-in consequence of optimizing for plausible rather than verified text, and stop treating confidence as correctness.
• You can describe grounding as narrowing the task to reading supplied source, with an explicit refusal path.
• You can decide when an LLM is the wrong tool — high-stakes facts, exact computation, guarantees — and state a responsible-use stance.

Next up is the lab, where you make this course's token and context ideas mechanical: you will build a naive tokenizer, a token counter, and a context- budget trimmer in plain Python, entirely offline.

🧠 Knowledge check

1. For which task is an ungrounded LLM the poorest fit?

a) Rewriting a paragraph you paste in to be more conciseb) Summarizing an article you providec) Serving as the sole authority on a safety-critical dosage figured) Drafting a friendly first version of an email

2. A user asks an ungrounded model about an event from last week and gets a detailed answer. Why is this a warning sign?

a) The model updates itself in real time, which is unsafeb) The model has a training cutoff and cannot know recent events, so detail may be confabulatedc) Recent events are always blocked by the tokenizerd) The context window is too small for dates

📌 Key takeaways

• Hallucination is built in: models optimize for plausible text, so fluency and confidence are not evidence of truth. - Grounding — supplying the source and allowing refusal — is the main mitigation and the basis of RAG. - Match the tool to the task: keep LLMs away from high-stakes facts, exact math, and guarantees, or put a verifier in front of users.