Cross-Validation Done Right: Why a Single Train/Test Split Lies to You

Running the same model against the same dataset, with only the random seed for the train/test split changed, produced 91% accuracy on one run and 84% on another — an 7-point swing from nothing but which specific rows happened to land in the test set. That gap is the entire argument for cross-validation over a single split, and it's worth seeing the actual numbers before the mechanism, because the size of the gap is what makes the problem impossible to ignore rather than a theoretical concern.

Reproducing the swing

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

for seed in [1, 2, 3, 4, 5]:
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=seed
    )
    model = RandomForestClassifier(random_state=42).fit(X_train, y_train)
    accuracy = accuracy_score(y_test, model.predict(X_test))
    print(f"Seed {seed}: {accuracy:.3f}")

# Seed 1: 0.910
# Seed 2: 0.867
# Seed 3: 0.843
# Seed 4: 0.895
# Seed 5: 0.881

Nothing about the model or the data changed between these five runs — only which specific rows ended up in the 20% test set changed, purely by chance from a different random seed each time. On a dataset of a few hundred rows, that's enough to shift accuracy by nearly seven percentage points depending on whether the test set happened to contain a disproportionate share of harder or easier examples purely by chance.

Why a single split is this sensitive to luck

A test set of, say, 100 rows out of 500 total is a genuinely small sample, and any small sample carries sampling variance — the specific rows that land in it by chance won't perfectly represent the full distribution of the underlying data, especially for a dataset with any class imbalance or with certain patterns that only show up in a subset of examples. A single split reports one specific sample's accuracy as if it were the model's true, general performance, when it's actually one draw from a distribution of possible accuracies the model could show depending on which rows happened to be held out.

What k-fold cross-validation actually does differently

Instead of one split, k-fold cross-validation divides the data into k roughly equal parts ("folds"), trains the model k separate times, each time holding out a different fold as the test set and training on the remaining k-1 folds, then averages the resulting k scores:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(
    RandomForestClassifier(random_state=42), X, y, cv=5, scoring='accuracy'
)

print(scores)
# [0.891, 0.874, 0.902, 0.858, 0.885]
print(f"Mean: {scores.mean():.3f}, Std: {scores.std():.3f}")
# Mean: 0.882, Std: 0.015

Every single row in the dataset gets used as test data exactly once, across the five folds, and as training data four times. The reported mean (0.882) is a genuinely more stable estimate than any single split's number, because it's averaged across five different held-out samples rather than trusting one. The standard deviation (0.015) is at least as valuable as the mean itself — it directly quantifies how much the estimate would plausibly vary on yet another different split, which a single accuracy number can never tell you on its own.

How many folds, and why 5 or 10 is the usual default

• More folds (e.g., 10) means each training set is larger (90% of the data per fold rather than 80%), which generally gives a less biased estimate of true model performance, at the cost of more total training runs and more compute time.
• Fewer folds (e.g., 3) is faster but each training set is smaller, and the estimate tends to have higher variance between folds as a result.
• 5 and 10 are common defaults because they balance this trade-off reasonably well for typical dataset sizes — there's no universally correct number, and for a very large dataset, even 3-fold can produce a stable estimate, while a genuinely small dataset benefits from more folds specifically because each individual fold represents a larger, more informative fraction of the limited data available.

The mistake that undoes cross-validation's benefit: leaking preprocessing across folds

A genuinely common bug undermines cross-validation's whole point without throwing any error: fitting a preprocessing step — a scaler, an imputer — on the full dataset before cross-validation runs, rather than fitting it fresh inside each fold using only that fold's training data.

# Wrong: scaler sees the full dataset, including future test folds,
# before cross-validation even starts — this leaks information
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # fit on everything, including what will be "held out"
scores = cross_val_score(RandomForestClassifier(), X_scaled, y, cv=5)

# Right: scaling happens fresh inside each fold, via a Pipeline,
# using only that fold's training data
from sklearn.pipeline import Pipeline

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', RandomForestClassifier(random_state=42)),
])
scores = cross_val_score(pipeline, X, y, cv=5)

The broken version's scaler computes its mean and standard deviation using every row in the dataset, including rows that will later be held out as "test" data within each fold — meaning each fold's supposedly unseen test data already influenced the preprocessing the model was trained under, a subtle form of information leakage that inflates the reported score without an obvious symptom pointing to why. Wrapping preprocessing and the model together in a Pipeline, then cross-validating the pipeline as a single unit, ensures every preprocessing step is refit from scratch on only each fold's own training data — exactly matching what would happen on genuinely new, unseen data in production.

A real before-and-after from this exact bug

Running the leaky version against the same dataset used above reported a mean cross-validation accuracy of 0.911 — notably higher than the correctly-piped version's 0.882. That 3-point gap is entirely explained by leakage, not a genuinely better model, and it's a dangerous gap specifically because it's invisible without comparing the two approaches directly: the leaky version doesn't throw an error, doesn't look obviously wrong, and reports a more flattering number that a team under deadline pressure has every incentive to simply accept and move on from.

What this changes about reporting a model's performance

• Report the mean and standard deviation across folds, not a single accuracy number — the standard deviation communicates the estimate's actual reliability in a way a lone number can't.
• Always preprocess inside a Pipeline, cross-validated as one unit, never fit preprocessing on the full dataset before splitting into folds.
• A single train/test split is still fine for quick iteration during early experimentation, where speed matters more than a precise estimate — the issue is trusting a single split's number as a final, reportable result rather than a rough, provisional signal.

Getting a genuinely reliable performance number matters more than it might seem, especially before deciding a model is ready to ship at all — which connects directly to the broader judgment call covered in when not to use machine learning, since an inflated, leaky cross-validation score can make a model look ready for production when it genuinely isn't. And if the metric you're cross-validating against is accuracy on an imbalanced dataset specifically, that number can mislead in a completely different way covered in precision, recall, and the mistake most people make evaluating models.

Stratified k-fold: a small change that matters for imbalanced classes

Plain k-fold cross-validation splits data into folds without regard to class balance, which is fine when classes are roughly even and becomes a real problem on an imbalanced dataset — a fold could, purely by chance, end up with almost none of the minority class, producing a wildly unstable score for that specific fold that has nothing to do with model quality and everything to do with an unlucky split.

from sklearn.model_selection import StratifiedKFold, cross_val_score

# Plain KFold: fold composition is left to chance
plain_scores = cross_val_score(pipeline, X, y, cv=5)

# StratifiedKFold: each fold preserves the same class proportions
# as the full dataset, every time
stratified_scores = cross_val_score(
    pipeline, X, y, cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
)

StratifiedKFold explicitly ensures each fold contains roughly the same proportion of each class as the overall dataset, rather than leaving that to chance — for any classification task with class imbalance, this isn't an optional refinement, it's close to a requirement for getting a stable, trustworthy estimate at all. cross_val_score actually defaults to stratified splitting automatically for classification tasks in recent scikit-learn versions, but it's worth knowing explicitly what's happening under the hood rather than trusting a default without understanding why it exists.

What to do when cross-validation itself is too slow

For a model expensive enough that even 5-fold cross-validation takes a meaningful amount of time — a large neural network, a slow-to-train ensemble — running full cross-validation for every single experimental change during active development isn't always practical. A reasonable middle ground: use a single, fixed validation split (not test set) during rapid iteration, reserving full k-fold cross-validation for confirming a final candidate model before it's seriously considered for deployment, rather than for every minor experiment along the way. This isn't a contradiction of everything above — it's an acknowledgment that a single split's instability is a worse trade-off for a final, reported number than it is for a quick internal signal during iteration, where speed matters more than precision and a rough sense of "did this change help or hurt" is enough to guide the next step.

• During active experimentation: a single, consistent validation split is a reasonable speed trade-off, as long as everyone on the team understands it's a rough signal, not a final number.
• Before reporting a final number or comparing candidate models seriously: full k-fold (or better, repeated k-fold, running the whole process multiple times with different random splits) gives the estimate this decision actually deserves.
• Never mix the two without being explicit about which was used — a number from a single split and a number from 5-fold cross-validation are not directly comparable, and presenting them side by side without that context invites exactly the kind of false confidence this whole post is about.