Tokens and context windows

⏱ 25 min

Hook

You paste a long document into a chatbot and it answers as if the first half never existed. You ask a model to count the letters in "strawberry" and it gets it wrong. Both look like the model is careless. Neither is: they are direct consequences of the fact that the model never sees your words at all. It sees tokens, and its memory is measured in them.

Concept

A token is the unit of text a model actually reads. It is often a whole word, but common word-pieces, punctuation, and spaces can be tokens too. Text is converted to tokens by a tokenizer before the model sees it, and the model's output tokens are converted back to text for you. The model's entire world is this stream of tokens.

Two rules of thumb are worth memorizing, then immediately qualifying. For typical English, one token is roughly four characters, and 100 tokens is roughly 75 words. These are averages, not laws: a rare word, a URL, or code can split into many more tokens than its length suggests, and different model families use different tokenizers, so the same sentence can be a different token count on two models.

This is also why the "count the letters in strawberry" trick fails. If the tokenizer packs that word into two or three tokens, the model never sees the individual letters as separate units — it sees the chunks. Asking it to count characters is asking it to report on information the tokenization threw away. Many "the model can't do basic X" complaints are really token-boundary artifacts.

The context window is the maximum number of tokens a model can consider at once — your prompt plus the conversation history plus the response it is generating, all counted together. Think of it as a fixed-size desk: everything the model is "thinking about" has to fit on the desk at the same time, and when new material arrives, something old has to slide off the edge.

Term	What it measures	Why you care
Token	One chunk of text (≈ 4 characters)	The true unit of length and cost
Context window	Max tokens the model holds at once	Hard limit on prompt + history + reply
Input tokens	Tokens you send in	Billed, and counted against the window
Output tokens	Tokens the model generates	Usually billed at a higher rate

Two consequences follow, and they are the whole practical point of this lesson.

First, length is measured in tokens, and it is a hard wall. When a conversation grows past the context window, the model cannot simply read more. Something must be dropped — usually the oldest turns — which is exactly why the long-document chatbot "forgot" the first half. It did not choose to ignore it; it fell off the desk. Managing what stays on the desk is a real engineering task, and you will build a tool for it in this course's lab.

Second, tokens are the unit of cost. Hosted APIs bill per token, typically counting input and output separately and charging more for output. This means the price of a feature is set by how many tokens flow through it, not by how many requests you make. A prompt that stuffs in a huge context on every call can be far more expensive than one that sends only what the model needs, even at the same number of requests.

🧠 Knowledge check

1. A model gives a wrong answer when asked to count the letters in a word. What is the most likely cause?

a) The model is broken and needs retrainingb) The word was split into tokens, so the model never saw individual letters as unitsc) The internet page it copied from had a typod) Counting is disabled to save cost

2. Which of these is counted against a model's context window?

a) Only the text you type as a promptb) Only the model's replyc) The prompt, the conversation history, and the reply being generated, all togetherd) The number of separate requests you have made

Worked example

Let me estimate the token cost of a feature the way you would when sizing a budget. Suppose you are building a support assistant. Each call sends a fixed instruction, the retrieved help article, and the user's question, and the model replies. I will approximate tokens with the common "≈ 4 characters per token" rule so you can do this on the back of an envelope, then price it.

# Approximate tokens from character length (the ~4-chars-per-token rule).
def approx_tokens(text):
    return max(1, round(len(text) / 4))


instruction = "You are a support assistant. Answer only from the article."
article = "A" * 4000  # stand-in for a ~4,000-character help article
question = "How do I reset my password on a shared account?"

input_text = instruction + article + question
input_tokens = approx_tokens(input_text)
output_tokens = 120  # a typical short answer

# Illustrative prices in dollars per 1,000,000 tokens (output costs more),
# the pricing unit hosted APIs use in 2026.
price_in = 0.50
price_out = 1.50
cost = (
    (input_tokens / 1_000_000) * price_in
    + (output_tokens / 1_000_000) * price_out
)

print(f"input tokens : {input_tokens}")
print(f"output tokens: {output_tokens}")
print(f"cost/call    : ${cost:.6f}")
print(f"cost/1M calls: ${cost * 1_000_000:.2f}")

input tokens : 1026
output tokens: 120
cost/call    : $0.000693
cost/1M calls: $693.00

Read the numbers. The article dominates the input — it alone is about 1,000 tokens, dwarfing the instruction and question. A single call is a fraction of a cent, which is why cost feels free while you are testing. But the same design choice, "send the whole article every call," sets the real bill: at a million calls it is roughly $693, and with a larger article or a pricier model the same structure scales to serious money. This is the lever cost engineering pulls later — send less context, or reuse it — and it is only visible because we counted in tokens, not requests.

Hands-on

Now measure a real tokenizer instead of estimating. The exercise below opens in the Prompt Playground with a token inspector: type text and watch it split into tokens with a live count.

▶ Prompt Playgroundga101-tokenizer-inspector

Try it in the Prompt Playground ↗

Do this: tokenize a plain English sentence, then tokenize a URL and a snippet of code of similar character length, and compare the token counts. Success criteria: you can state which input packed into the most tokens per character and give the reason. The Playground shows the exact boundaries so you can see where the splits land.

Recap

• You can define a token as the chunk of text a model reads and explain the rough four-characters-per-token rule and its exceptions.
• You can explain the context window as a fixed token budget covering prompt, history, and reply, and predict that overflow drops the oldest content.
• You can reason about cost in tokens, separating input from output, and see why context size drives the bill.

Next up, you will turn from how models read to where they fail: hallucination, grounding, and how to decide when an LLM is the wrong tool. Then the lab makes this lesson tangible — you will build a tokenizer and a context-budget trimmer in plain Python.

🧠 Knowledge check

1. A chatbot answers a very long pasted document as if its opening were missing. What most likely happened?

a) The model deleted the fileb) The document plus history exceeded the context window, so the oldest tokens were droppedc) The model only reads the last sentence of any input by designd) The tokenizer refused to process the document

2. Two features make the same number of API calls, but feature A sends a large context every call and feature B sends a small one. What can you say about cost?

a) They cost the same because call count is equalb) Feature A likely costs more because it sends more tokens per callc) Feature B costs more because small prompts are inefficientd) Cost depends only on the output length, so they are equal

📌 Key takeaways

• Models read tokens, not words; roughly four characters per token for English, with big exceptions for URLs, code, and rare words. - The context window is a fixed token budget over prompt, history, and reply; overflow drops the oldest tokens. - Tokens are the unit of cost — input and output are billed separately — so context size, not request count, drives the bill.