From data to a first model

⏱ 30 min

Hook

Someone asks you to predict whether a student passes, and your instinct is to reach for a fancy model. But if 70% of students pass, a one-line rule that always guesses "pass" is already right 70% of the time — and any model you build has to beat that to be worth anything. The most useful first move in machine learning is not a model at all. It is a baseline you must outperform.

Concept

A baseline is the simplest rule that makes a prediction, and it exists to give every real model something honest to beat. The standard baseline for classification is the majority-class predictor: it ignores the features entirely and always guesses the label that appears most often in the training data. If a model cannot beat that, the features are not helping and the model is not earning its complexity.

Your first real model in this school is nearest neighbor. The idea is one sentence: to label a new example, find the training example closest to it and copy that neighbor's label. "Closest" means smallest distance between feature vectors, and the everyday distance is Euclidean distance — the straight-line distance you get from the Pythagorean theorem. For two feature vectors $a = (a_1, a_2)$ and $b = (b_1, b_2)$:

$$d(a, b) = \sqrt{(a_1 - b_1)^2 + (a_2 - b_2)^2}$$

Nearest neighbor makes no assumptions about the shape of the data and needs no training step beyond storing the examples, which is exactly why it is a great first model: you can implement it from scratch and understand every line.

You measure both baseline and model the same way, on the held-out test set from the last lesson, using accuracy — the fraction of test examples the predictor labels correctly:

$$\text{accuracy} = \frac{\text{correct predictions}}{\text{total predictions}}$$

The workflow ties the whole course together: split the data, compute the majority-class baseline's accuracy, then compute the model's accuracy, and keep the model only if it wins.

from collections import Counter


def majority_label(train):
    counts = Counter(label for _, label in train)
    return counts.most_common(1)[0][0]


train = [
    ((2.0,), "fail"), ((3.0,), "fail"), ((7.0,), "pass"),
    ((8.0,), "pass"), ((6.5,), "pass"), ((2.5,), "fail"),
]
print("baseline predicts:", majority_label(train))

baseline predicts: fail

Term	Plain meaning
Baseline	Simplest rule to beat; majority class for classification
Nearest neighbor	Copy the label of the closest training example
Euclidean distance	Straight-line distance between two feature vectors
Accuracy	Correct predictions divided by total predictions

🧠 Knowledge check

1. What does a majority-class baseline predict for a new example?

a) The label of the nearest training exampleb) The label that appears most often in the training data, ignoring the featuresc) A random label each timed) The average of all the feature values

2. How does a nearest-neighbor classifier decide the label of a new example?

a) It averages every training labelb) It finds the training example with the smallest distance to the new example and copies that labelc) It always predicts the majority classd) It fits a straight line through the data first

Worked example

Let me build a one-nearest-neighbor predictor from scratch on a tiny study dataset, then check it against the baseline — the exact comparison that decides whether a model is worth keeping. Each example is a one-number feature vector (hours_studied,) labeled pass or fail.

import math
from collections import Counter

train = [
    ((2.0,), "fail"), ((3.0,), "fail"), ((7.0,), "pass"),
    ((8.0,), "pass"), ((6.5,), "pass"), ((2.5,), "fail"),
]
test = [((7.5,), "pass"), ((2.2,), "fail"), ((6.0,), "pass")]


def majority_label(samples):
    return Counter(label for _, label in samples).most_common(1)[0][0]


def nearest_label(train, features):
    closest = min(train, key=lambda sample: math.dist(sample[0], features))
    return closest[1]


baseline = majority_label(train)
baseline_hits = sum(1 for _, label in test if baseline == label)
nn_hits = sum(1 for feats, label in test if nearest_label(train, feats) == label)

print("baseline accuracy:", round(baseline_hits / len(test), 2))
print("1-NN accuracy:    ", round(nn_hits / len(test), 2))

baseline accuracy: 0.33
1-NN accuracy:     1.0

Walk through the decisions. The baseline picks fail because three of the six training rows are fail — the barest majority — and on the test set it is right only once, for 0.33 accuracy. The nearest-neighbor predictor uses math.dist, the standard library's Euclidean distance, to find the closest training example to each test point, and copies its label; every test point lands nearer to a correct neighbor, so it scores 1.0. The model clears the baseline decisively, so here it earns its place. Had they tied, I would keep the baseline: never pay for complexity that does not buy accuracy.

Hands-on

Your turn, with a different dataset and a small twist. The exercise below gives you two-feature examples (not one) labeled with two classes, plus a train/test split. Compute the majority-class baseline accuracy, then build a one-nearest-neighbor predictor using Euclidean distance and compute its accuracy, and report both. It runs in Python Arena, so there is nothing to install.

▶ Python Arenaai101-beat-the-baseline

Open in Python Arena ↗

Success criteria: your baseline accuracy and your 1-NN accuracy match the expected values on the hidden test set, and your nearest-neighbor function uses a correct Euclidean distance over both features. The Arena grades both numbers automatically.

Recap

• You can state a majority-class baseline and compute its accuracy, giving every model an honest bar to clear.
• You can implement a nearest-neighbor predictor from scratch using Euclidean distance over feature vectors.
• You can measure a predictor with held-out accuracy and decide, on the numbers, whether a model beats its baseline.

Next up: the lab. You will pull these three lessons together — load a real CSV, split it, and build a k-nearest-neighbor classifier that votes among several neighbors — into one runnable predictor you measure end to end.

🧠 Knowledge check

1. Your nearest-neighbor model scores exactly the same accuracy as the majority-class baseline. What is the sound decision?

a) Always keep the more complex modelb) Keep the baseline — do not pay for complexity that does not improve accuracyc) Delete the dataset and start overd) Report the higher of the two numbers as the model score

2. Why must accuracy be measured on the held-out test set rather than the training set?

a) The training set has no labelsb) Test accuracy estimates performance on unseen data; training accuracy can reflect memorizationc) Accuracy can only be computed on smaller setsd) The test set is always easier to predict

📌 Key takeaways

• Start with a majority-class baseline; every real model has to beat it to be worth its complexity. - Nearest neighbor copies the label of the closest training example, using Euclidean distance over feature vectors. - Judge models by held-out accuracy, and keep the simplest one that wins.