Lab 1 — Explore tokens and context windows

Objective

By the end of this lab you will have a runnable Python module, tokenlab.py, that turns the abstract ideas from this course into working code: it splits text into approximate tokens, counts them and estimates their dollar cost, and trims a conversation so it fits inside a token budget the way a real application must before every model call. It uses only the Python standard library, so it makes no network calls, needs no API key, and costs nothing to run.

Skills exercised:

• Implementing a naive tokenizer that approximates how models chunk text
• Counting tokens and estimating per-call cost from a price
• Simulating a context-window budget by trimming a conversation to fit

info

This tokenizer is a teaching approximation — it splits on words and punctuation, which is close enough to reason about token counts but will not match a specific model's tokenizer exactly. The point is the mechanism, not the exact numbers a production tokenizer would give.

Prerequisites and setup

Prerequisites: the three lessons in this module — What is an LLM?, Tokens and context windows, and Capabilities and limits. The lab leans hardest on the tokens-and-context lesson.

Setup: you need Python 3.10 or newer. There are no packages to install — everything uses the standard library (re). Verify your version and create a working folder.

macOS / Linux

python3 --version
mkdir ga101-token-lab && cd ga101-token-lab

Windows (PowerShell)

python --version
mkdir ga101-token-lab; cd ga101-token-lab

Expected output from the version command is a line like Python 3.11.6. If the command is not found, install Python from python.org before continuing. In the steps below, use python3 on macOS/Linux and python on Windows.

note

Every step adds to the one file tokenlab.py in this folder. If you close your terminal and come back, re-run cd ga101-token-lab to return to it; your file is still there and you can continue from the step you were on.

Step 1 — Build a naive tokenizer

Goal: write the tokenizer every other function will build on — split raw text into approximate tokens, and count them.

Create tokenlab.py with these two functions:

import re


def tokenize(text):
    """Naive tokenizer: split into words and standalone punctuation."""
    return re.findall(r"[A-Za-z0-9']+|[^\sA-Za-z0-9]", text)


def count_tokens(text):
    """Approximate number of tokens in a string."""
    return len(tokenize(text))

The regular expression captures two kinds of token: runs of letters, digits, and apostrophes (words and contractions), or any single non-space, non-word character (punctuation like , and !). Spaces are separators, not tokens, so they are dropped. This mirrors how real tokenizers treat words and punctuation as separate units, which is the behavior that matters for counting.

Checkpoint: tokenize a short sentence and count it.

python3 -c "import tokenlab; print(tokenlab.tokenize('Hello, world! LLMs are fun.')); print(tokenlab.count_tokens('Hello, world! LLMs are fun.'))"

You should see the token list followed by 8:

['Hello', ',', 'world', '!', 'LLMs', 'are', 'fun', '.']
8

Note that the comma, exclamation mark, and period each became their own token — that is why a count of eight tokens comes from six words. If you see a ModuleNotFoundError, you are not in the ga101-token-lab folder where tokenlab.py lives.

Step 2 — Count tokens and estimate cost

Goal: turn a token count into a dollar figure, so the cost of a prompt is a number you can compute rather than guess.

Add this function to tokenlab.py:

def estimate_cost(n_tokens, usd_per_million):
    """Dollar cost of n_tokens at a per-1,000,000-token price."""
    return (n_tokens / 1_000_000) * usd_per_million

Hosted APIs quote prices per million tokens in 2026, so the function takes a per-million price and scales it down to the token count you pass in. A single prompt is a tiny fraction of a cent; the cost only becomes real when multiplied by many calls, which is exactly the point the tokens lesson made.

Checkpoint: count a prompt and price it at $0.50 per million tokens.

python3 -c "import tokenlab; n=tokenlab.count_tokens('Summarize the quarterly sales report in three bullet points.'); print(n); print(f'{tokenlab.estimate_cost(n, 0.50):.6f}')"

You should see:

10
0.000005

The prompt is 10 tokens, costing $0.000005 at that price — trivial once, but $5.00 across a million calls. If your token count differs, re-check the tokenize regex from Step 1; the cost is derived from it.

Step 3 — Trim a conversation to a token budget

Goal: simulate what an application does before every call — keep the system instruction and the most recent turns that fit the context budget, and let the oldest turns fall off the desk.

Add the conversation data and the two functions below to tokenlab.py:

CONVERSATION = [
    {"role": "system", "content": "You are a concise support assistant."},
    {"role": "user", "content": "My printer will not connect to the wifi."},
    {"role": "assistant", "content": "Restart the printer and the router."},
    {"role": "user", "content": "I restarted both and it still fails."},
    {"role": "assistant", "content": "Check that it uses the 2.4 GHz band."},
    {"role": "user", "content": "How do I change the band on the printer?"},
]


def conversation_tokens(messages):
    """Total approximate tokens across every message in a conversation."""
    return sum(count_tokens(m["content"]) for m in messages)


def fit_to_budget(messages, max_tokens):
    """Keep the system message plus the most recent turns that fit the budget."""
    system = [m for m in messages if m["role"] == "system"]
    others = [m for m in messages if m["role"] != "system"]
    used = sum(count_tokens(m["content"]) for m in system)
    kept_reversed = []
    for m in reversed(others):
        cost = count_tokens(m["content"])
        if used + cost > max_tokens:
            break
        used += cost
        kept_reversed.append(m)
    return system + list(reversed(kept_reversed))

The strategy is deliberate and common: always keep the system message, because it sets behavior, then walk the other turns newest-first and keep each one that still fits. Older turns are dropped first — the same "oldest content slides off the desk" behavior you saw a chatbot exhibit on a long document, now under your control.

Checkpoint: measure the full conversation, then trim it to a 30-token budget.

python3 -c "import tokenlab; c=tokenlab.CONVERSATION; print('total:', tokenlab.conversation_tokens(c)); t=tokenlab.fit_to_budget(c, 30); print('kept:', len(t), 'tokens:', tokenlab.conversation_tokens(t), 'first:', t[0]['role'])"

You should see:

total: 52
kept: 3 tokens: 28
first: system

The full conversation is 52 tokens, over budget. Trimming to 30 keeps 3 messages totaling 28 tokens, and the system message is still first — dropped turns are the oldest user/assistant exchanges. If first is not system, check that fit_to_budget prepends the system list to the kept turns.

Validation

Verify the whole build against the objective: a tokenizer, a cost estimator, and a budget trimmer that all agree. Run this validation one-liner, which exercises every function and checks the results, including the edge case that the system message survives even an impossibly small budget.

python3 -c "import tokenlab as t; c=t.CONVERSATION; ok = (t.count_tokens('Hello, world! LLMs are fun.') == 8 and t.conversation_tokens(c) == 52 and len(t.fit_to_budget(c, 30)) == 3 and t.conversation_tokens(t.fit_to_budget(c, 30)) == 28 and t.fit_to_budget(c, 1)[0]['role'] == 'system'); print('PASSED - token and context lab complete' if ok else 'FAILED - re-check the functions above')"

Checkpoint: the command prints PASSED - token and context lab complete. If it prints FAILED, run the Step 1–3 checkpoints again to find which function returns the wrong value; each checkpoint isolates one function.

Teardown

Nothing is billable — this lab runs entirely on your machine and makes no network calls. Keep the ga101-token-lab folder if you want tokenlab.py as a reference for GA-201, where you will meter real API tokens; otherwise delete it with rm -rf ga101-token-lab (macOS/Linux) or Remove-Item -Recurse ga101-token-lab (Windows).

Troubleshooting

Symptom	Likely cause	Fix
ModuleNotFoundError: No module named 'tokenlab'	Running from the wrong folder	cd into ga101-token-lab, where tokenlab.py lives
Step 1 count is not 8	Regex altered or retyped	Copy the tokenize pattern exactly, including both alternatives
Step 2 prints a very large cost	Divided by 1,000 instead of 1,000,000	estimate_cost must divide n_tokens by 1_000_000
Step 3 first is not system	System message not kept first	Return system + list(reversed(kept_reversed)), system list first
Step 3 keeps too many messages	Budget check uses >= or wrong sign	Break when used + cost > max_tokens, before appending
Validation prints FAILED	One function returns a wrong value	Re-run each step checkpoint to isolate the mismatched function