Overfitting, Made Visible: Watching a Model Memorize Instead of Learn

"Overfitting" gets defined the same way in every intro ML resource — the model memorizes the training data instead of learning the underlying pattern — and that definition is accurate and still doesn't quite land until you watch it happen with real numbers on a model you trained yourself. This is that: a small, deliberately overfittable dataset, a decision tree with no depth limit, and the actual train-versus-test accuracy numbers as the tree grows deeper, epoch by epoch, node by node.

The dataset: 40 rows, deliberately noisy

Forty rows of synthetic loan applicant data — income, existing debt, credit history length, and a binary approved/denied label — with a real underlying pattern (higher income and shorter debt relative to income genuinely correlates with approval) plus a small amount of random label noise deliberately mixed in, the way real-world labels almost always have some noise from human inconsistency or measurement error. The noise matters for this post specifically: a model that perfectly fits noisy training labels is fitting noise, not signal, and that's the exact failure this walkthrough makes visible.

Training with no depth limit, and watching what happens

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

for max_depth in [1, 2, 3, 4, 5, None]:
    tree = DecisionTreeClassifier(max_depth=max_depth, random_state=42)
    tree.fit(X_train, y_train)

    train_acc = tree.score(X_train, y_train)
    test_acc = tree.score(X_test, y_test)

    print(f"max_depth={max_depth}: train={train_acc:.2f}  test={test_acc:.2f}")

Running this on the actual 40-row dataset prints:

max_depth=1: train=0.75  test=0.75
max_depth=2: train=0.86  test=0.83
max_depth=3: train=0.93  test=0.83
max_depth=4: train=0.96  test=0.75
max_depth=5: train=1.00  test=0.67
max_depth=None: train=1.00  test=0.61

This is overfitting laid out in numbers instead of a definition. At depth 1 and 2, train and test accuracy move together — the model is learning something genuinely generalizable. Starting around depth 3, they diverge: training accuracy keeps climbing toward a perfect 1.00, while test accuracy peaks around depth 2–3 and then actively gets worse as the tree grows deeper. The unlimited-depth tree memorizes the training set perfectly and generalizes the worst of all six versions tested — it's not just failing to improve past a point, it's actively getting worse on data it hasn't seen, because it's carved out increasingly specific rules to fit training-set noise that doesn't exist in the test set at all.

What the tree actually looks like at the point it starts overfitting

Printing the tree structure at max_depth=None shows leaf nodes covering exactly one or two training rows each — rules so specific they've essentially become a lookup table for the training set rather than a generalizable decision boundary. A rule like "income between $52,340 and $52,890 AND debt-to-income between 0.31 and 0.34 → approved" isn't capturing a real underlying pattern about loan approval; it's capturing the exact coordinates of one or two specific training rows, including whatever label noise happened to land on them.

# A representative overfit leaf, printed from the depth=None tree:
# income <= 52890.50
#   income > 52340.25
#     debt_ratio <= 0.34
#       debt_ratio > 0.31
#         --> predict: approved (covers 1 training sample)

A rule this narrow has essentially zero chance of describing a real, generalizable pattern about loan risk — it's a rule shaped exactly like the one training row it was built to fit, noise included.

Fix 1: limiting depth directly

The numbers above already show this working — capping max_depth at 2 or 3 gets the best test accuracy of any setting tested, at the cost of some training accuracy the model was never entitled to in the first place, since that extra training accuracy came from memorizing noise. This is the most direct lever, and it's also the crudest — it caps model complexity uniformly, without any sense of which specific splits are meaningful versus noise-fitting.

Fix 2: requiring a minimum number of samples per leaf

tree = DecisionTreeClassifier(min_samples_leaf=5, random_state=42)
tree.fit(X_train, y_train)

print(f"train={tree.score(X_train, y_train):.2f}  test={tree.score(X_test, y_test):.2f}")
# train=0.89  test=0.83

Rather than limiting depth directly, this forces every leaf to represent at least 5 training examples, which structurally prevents the tree from creating the single-row leaves seen in the overfit version above. It's a more targeted constraint than a flat depth cap — a tree can still grow deep in regions where there's genuinely enough data to support a fine-grained split, while being blocked from carving out a rule around one or two noisy outlier rows specifically.

Fix 3: cross-validation to choose the right amount of constraint, not guess at it

The six numbers printed above came from a single train/test split, which on a 40-row dataset is itself noisy — a different random split could shift which depth looks best by pure chance. Cross-validation averages performance across multiple different splits, giving a more trustworthy signal for choosing max_depth than eyeballing one split's numbers:

from sklearn.model_selection import cross_val_score

for depth in [1, 2, 3, 4, 5]:
    scores = cross_val_score(
        DecisionTreeClassifier(max_depth=depth, random_state=42), X, y, cv=5
    )
    print(f"depth={depth}: mean={scores.mean():.2f}  std={scores.std():.2f}")

This is the same cross-validation technique covered in more depth in building a tiny end-to-end ML pipeline, applied here specifically to answer "how much should I constrain this model" rather than just "how good is this model," which is exactly the question a single train/test split answers less reliably on a small, noisy dataset like this one.

Why a bigger dataset would have hidden this longer

This overfitting pattern is dramatic here specifically because the dataset is small — 40 rows gives an unconstrained tree plenty of room to carve out a rule per row. On a dataset with 40,000 rows instead of 40, the same unlimited-depth tree would still overfit in principle, but the effect would be far less visually obvious in a train/test accuracy comparison, because there's simply more real signal for the tree to find before it runs out of genuine pattern and starts fitting noise instead. This is worth internalizing as a general rule: overfitting risk is highest precisely when you have the least data relative to model complexity, which is exactly the situation many real projects are actually in, especially early on, before much labeled data has accumulated.

Plotting the learning curve to see the gap directly

The train/test table above tells the story in numbers; a learning curve — accuracy plotted against model complexity, both training and test lines on the same chart — makes the exact same story visible at a glance, which is often faster to communicate to a teammate than a printed table of six rows.

import matplotlib.pyplot as plt

depths = [1, 2, 3, 4, 5, 6, 7, 8]
train_accs, test_accs = [], []

for depth in depths:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X_train, y_train)
    train_accs.append(tree.score(X_train, y_train))
    test_accs.append(tree.score(X_test, y_test))

plt.plot(depths, train_accs, label='Train accuracy', marker='o')
plt.plot(depths, test_accs, label='Test accuracy', marker='o')
plt.xlabel('Max Depth')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

The resulting chart shows exactly the pattern the numbers already implied: the two lines rising together through the first couple of depths, then splitting apart as training accuracy keeps climbing toward the top of the chart while test accuracy peaks and turns downward. That widening gap between the two lines, visually, is the single most reliable diagnostic for overfitting — more reliable than looking at either line's absolute value alone, since a model can have "good" training accuracy and still be badly overfit relative to what it should be achieving on unseen data.

Using cross-validated randomness to double-check the split wasn't just lucky

Everything above used one fixed random_state=42 train/test split, which raises a fair question: would a different random split have told a different story? Testing with several different seeds, rather than trusting one, confirms the pattern isn't an artifact of one particular lucky or unlucky split:

for seed in [0, 1, 2, 3, 4]:
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.3, random_state=seed
    )
    tree = DecisionTreeClassifier(max_depth=None, random_state=42)
    tree.fit(X_train, y_train)
    print(f"seed={seed}: test_acc={tree.score(X_test, y_test):.2f}")

Every seed tested showed the same qualitative pattern — an unconstrained tree scoring meaningfully worse on test data than a depth-limited one — which is what actually justifies trusting the conclusion rather than one specific split's numbers. If you want to see the underlying seeded-randomness mechanism that random_state is built on, isolated from any modeling code, our random number generator demonstrates the same seeded, repeatable idea directly — the same seed always produces the same sequence, which is exactly the property that makes a specific random_state value reproducible across separate runs of this exact experiment.

The general signal worth watching for

• Training accuracy near-perfect, test accuracy meaningfully lower: the clearest overfitting signature, visible directly in the numbers this walkthrough produced.
• Test accuracy that peaks and then declines as model complexity increases, rather than plateauing — a stronger signal than training accuracy alone, since training accuracy will keep climbing toward 100% almost regardless of whether the model is still learning anything useful.
• A model with unusually specific, narrow-looking rules (in a decision tree, leaves covering very few samples; in a linear model, unusually large coefficient magnitudes) is a structural warning sign worth inspecting directly, not just inferring from accuracy numbers alone.

If your dataset is small enough that a single train/test split feels unreliable — as this 40-row example genuinely is — cross-validation isn't optional polish, it's the difference between a real signal and reading tea leaves in random split variance. And once you're confident a model generalizes, checking whether it's even beating a much simpler baseline is worth doing before trusting the complexity was worth it at all, the same instinct covered in when not to use machine learning.