Data Leakage: The Silent Bug That Makes a Model Look Better Than It Is

A customer-churn prediction model scored 94% accuracy in testing, which was good enough that the number itself should have prompted more suspicion than it did. In production, predicting churn for customers the model had genuinely never seen during training, its real-world performance barely beat guessing the majority class. The gap between those two numbers had a name and a specific cause: data leakage, and finding it meant re-examining every single feature in the training data for one very particular kind of mistake.

What data leakage actually is

Data leakage happens when information that wouldn't actually be available at prediction time — in the real world, at the moment you'd actually need the prediction — ends up in the training data anyway, letting the model learn a shortcut that has nothing to do with genuinely predicting the outcome. The model isn't lying about its test-set performance; it's accurately measuring how well it does at a task that's subtly easier than the real task, because the test set has the same leaked information baked into it as the training set does. That's what makes leakage so dangerous: the model's own reported accuracy actively hides the problem rather than revealing it.

The specific feature that caused it

The churn dataset included a column called days_since_last_support_ticket, included with the reasonable-sounding logic that recent support activity might correlate with dissatisfaction and predict churn. The actual data generation process, it turned out, populated that field based on support tickets logged up through the point each customer's outcome (churned or retained) was recorded — which meant that for customers who churned, a support ticket specifically about cancelling their account was frequently the most recent one in the system, making days_since_last_support_ticket a near-direct proxy for "did this customer already contact support to cancel," rather than a genuine early-warning signal available before the churn decision had effectively already happened.

# The leak, made explicit:
# A customer's "days_since_last_support_ticket" at the time the outcome
# was recorded already reflects a cancellation-related ticket for churned
# customers — the model is partly just detecting "did a cancel ticket exist,"
# not predicting churn before it happens.

df['days_since_last_support_ticket'] = (
    outcome_recorded_date - last_ticket_date_up_to_outcome
).dt.days

Why standard cross-validation didn't catch it

This is the part that made the bug genuinely dangerous rather than obviously wrong: the leaked feature was leaked consistently across both the training and test data, since both were built by the same flawed data-generation process. Standard k-fold cross-validation checks whether a model generalizes to data it hasn't seen during a particular training fold — it does not check whether every fold shares the same underlying leakage, and when every fold does, cross-validation reports a stable, trustworthy-looking, and completely wrong accuracy figure. This is the same class of problem covered from a different angle in cross-validation done right: why a single train/test split lies to you — proper cross-validation protects against overfitting to one particular split of the data, but it offers no protection at all against a feature that's leaked identically into every split.

How the leak actually got found

The discovery didn't come from a cleverer validation technique — it came from feature importance analysis, which showed days_since_last_support_ticket as by a wide margin the single most predictive feature in the model, disproportionately more influential than every other feature combined. That result alone was suspicious enough to warrant investigation: a single feature dominating a model's predictions that heavily is a common tell for leakage, not necessarily proof of it, and the actual confirmation came from manually tracing how that specific column got populated in the data pipeline and finding the timing overlap between "support ticket about cancelling" and "the churn outcome itself."

The fix, and the honest accuracy number it produced

The fix was rebuilding the feature to only use support ticket data from a clearly defined earlier window — support activity from more than 30 days before the outcome was recorded, deliberately excluding the window where a cancellation-related ticket would already reflect the outcome rather than predict it. Retrained on the corrected feature, the model's cross-validated accuracy dropped from 94% to 71% — a large, uncomfortable-looking drop, and also the first honest number the model had produced. Deployed against genuinely new data afterward, that 71% held up closely, within a few points, which the original 94% figure never had a chance of doing in the first place.

The second leak the same audit turned up, in a completely different feature

Investigating the support-ticket feature prompted a full audit of every other feature in the dataset for the same class of problem, which turned up a second, unrelated leak in a feature called account_age_at_outcome — a customer's account age, calculated as the time between account creation and the outcome date. That sounds harmless, and mostly was, except for one specific subgroup: customers who churned and later created a new account under a different email, whose original account's "age at outcome" was being calculated using the churn date correctly, but whose retained-customer comparisons elsewhere in the pipeline occasionally pulled the wrong account record for a small number of duplicate-account cases, inflating a small number of retained customers' apparent tenure. The effect on overall model accuracy was much smaller than the support-ticket leak — an estimated 1-2 percentage points, not 23 — but it's worth including here specifically because it illustrates that a single leakage audit rarely surfaces only one problem; once the habit of checking "would this value actually be knowable at prediction time" gets applied systematically rather than to just the one feature that triggered the investigation, it tends to find more than expected.

A checklist for catching leakage before it reaches production

1. For every feature, ask: would this exact value have actually been available at the real moment a prediction would be made? Not "is this data technically true" but "is this specific value knowable before the outcome, in a live deployment, not just in a historical dataset built after the fact."
2. Check timestamps explicitly for any feature derived from an event log — support tickets, account changes, login activity — and confirm the data-generation logic uses only events strictly before some defined prediction point, not events up through the outcome date.
3. Be suspicious of any single feature with unusually high, dominant importance — not proof of leakage on its own, but a strong enough signal to warrant tracing that specific feature's origin by hand.
4. Rebuild the train/test split around a real point in time whenever the data has a time dimension — train only on data available before a cutoff date, test only on outcomes after it, which surfaces many leakage patterns that a random split hides.
5. Treat a suspiciously high accuracy number as a bug report, not a success — 94% accuracy on a genuinely hard prediction problem (churn is a hard problem) should prompt investigation before celebration, not after a production rollout reveals the gap.

Why "suspiciously good" is a more useful instinct than any specific technique

The single most useful change from this experience wasn't a new validation method — it was developing real suspicion toward any accuracy number that looks unusually strong for how hard the underlying prediction problem actually is. Churn, fraud, and similar behavioral-prediction problems are hard by nature, and a model clearing 90%+ on one of them, without an unusually rich and genuinely predictive feature set to explain it, is now treated as a prompt to go looking for what's actually driving that number before trusting it, rather than as a result to be pleased with. This connects directly to a broader evaluation habit covered in precision, recall, and the mistake most people make evaluating models — a single aggregate metric, however good it looks, isn't a substitute for actually understanding why the model is making the predictions it's making.

What changed in the team's process afterward

Every new feature added to a training dataset now requires an explicit, written answer to "when would this value actually be known in a real, live prediction scenario," reviewed as a normal part of feature engineering rather than left implicit. It's a small amount of added friction on every new feature, and it's meaningfully less costly than discovering, after a model is already deployed and making real business decisions, that its headline accuracy number was measuring something other than what everyone believed it was measuring.