Handling Imbalanced Datasets: Techniques That Actually Move the Metric

A fraud-detection model reporting 97% accuracy sounds like a success story until you check what it's actually catching: on a dataset where fraudulent transactions made up roughly 3% of the total, the model achieved that headline accuracy by predicting "not fraud" almost every time, correctly identifying only 40% of actual fraud cases. Four techniques were tested against the same dataset to see which ones genuinely moved that recall number, and the results weren't uniform — one technique that looks reasonable on paper made things measurably worse.

Why accuracy is the wrong headline metric here

With 97% of transactions legitimate, a model that predicts "not fraud" unconditionally, for every single transaction with no exceptions, scores 97% accuracy while catching zero fraud — a completely useless model by any practical measure, reporting an impressive-sounding number. This is the specific reason accuracy is a misleading primary metric on an imbalanced dataset, and it's why every technique below is evaluated on recall for the minority class specifically (of all the actual fraud cases, what fraction did the model correctly catch), not overall accuracy.

Baseline: the model as originally trained

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test)))
#               precision    recall  f1-score
#     not fraud      0.98      0.99      0.98
#         fraud      0.71      0.40      0.51

0.40 recall on the fraud class means the model misses 60% of actual fraud cases — a real, costly gap for the actual business problem being solved, hidden entirely behind the flattering 97%-ish overall accuracy number.

Technique 1: class weights

model = LogisticRegression(max_iter=1000, class_weight='balanced').fit(X_train, y_train)
#               precision    recall  f1-score
#     not fraud      0.99      0.94      0.97
#         fraud      0.44      0.81      0.57

class_weight='balanced' tells the model to penalize a misclassified minority-class example more heavily during training than a misclassified majority-class example, roughly in inverse proportion to each class's frequency — recall on fraud jumped from 0.40 to 0.81, a genuinely large improvement, at the cost of precision dropping from 0.71 to 0.44, meaning more legitimate transactions now get incorrectly flagged as fraud. Whether that trade-off is worth it depends entirely on the actual cost of each error type in the real business context — a missed fraud case versus a legitimate customer's transaction getting flagged for review — which is a business judgment call, not something the metric alone can answer.

Technique 2: SMOTE (synthetic oversampling)

from imblearn.over_sampling import SMOTE

X_resampled, y_resampled = SMOTE(random_state=42).fit_resample(X_train, y_train)
model = LogisticRegression(max_iter=1000).fit(X_resampled, y_resampled)
#               precision    recall  f1-score
#     not fraud      0.99      0.93      0.96
#         fraud      0.40      0.79      0.53

SMOTE generates synthetic minority-class examples by interpolating between real minority-class points in feature space, rather than simply duplicating existing ones, then trains on the resulting more balanced dataset. The result here (0.79 recall) was close to but slightly below class weighting's 0.81 — genuinely competitive, but not clearly better for this specific dataset, while adding real complexity: SMOTE must be applied only to the training fold, never before cross-validation splits (the exact same leakage risk covered in preprocessing pipelines generally), and it meaningfully increases training set size and training time.

Technique 3: random undersampling — the one that quietly made things worse

from imblearn.under_sampling import RandomUnderSampler

X_resampled, y_resampled = RandomUnderSampler(random_state=42).fit_resample(X_train, y_train)
model = LogisticRegression(max_iter=1000).fit(X_resampled, y_resampled)
#               precision    recall  f1-score
#     not fraud      0.99      0.88      0.93
#         fraud      0.29      0.83      0.43

Undersampling discards majority-class examples until the classes are balanced, rather than generating new minority-class examples. Recall (0.83) looked competitive with the other techniques — and precision collapsed to 0.29, the worst of any approach tested, because throwing away the majority of legitimate-transaction training examples cost the model real information about what normal transactions actually look like, making it substantially more prone to false alarms. On a dataset already limited in total size, discarding data is a genuinely risky trade that looked fine on the recall number alone and clearly worse once precision was checked alongside it — this is exactly the kind of technique that "moves the metric" you're watching while quietly damaging one you're not.

Technique 4: adjusting the decision threshold instead of the training data

probabilities = model.predict_proba(X_test)[:, 1]
predictions_at_threshold = (probabilities >= 0.3).astype(int)  # default is 0.5

print(classification_report(y_test, predictions_at_threshold))
#               precision    recall  f1-score
#     not fraud      0.99      0.95      0.97
#         fraud      0.52      0.76      0.62

Rather than changing the training process at all, this leaves the original baseline model untouched and simply lowers the probability threshold required to classify a transaction as fraud — from the default 0.5 down to 0.3. This achieved the best overall F1-score of any technique tested (0.62), with a more balanced precision/recall trade-off than either class weighting or SMOTE, and it's the cheapest to implement and iterate on by a wide margin, since it requires no retraining at all — just picking a different point along the same trained model's existing probability output, which can be tuned after the fact without touching the training pipeline.

Side-by-side comparison

What actually shipped, and why

Threshold adjustment shipped as the primary fix, specifically because it required no retraining pipeline changes and could be tuned live against the actual cost trade-off the fraud team cared about — a false positive costs a customer support ticket and a brief hold on a transaction; a false negative costs actual fraud losses — by simply moving the threshold up or down and observing the resulting precision/recall trade-off directly, without a full retrain-and-redeploy cycle for every adjustment. Class weighting was kept as a secondary lever, tested alongside threshold adjustment rather than instead of it, since the two aren't mutually exclusive and combining them produced a marginal further improvement over either alone.

What this generalizes to

• Never trust accuracy alone on an imbalanced dataset — check precision and recall for the minority class specifically, since that's usually the class that actually matters for the real business problem.
• Test multiple techniques against the same held-out data rather than assuming any one approach — SMOTE, class weighting, and undersampling are not interchangeable, and the "obviously reasonable" one (undersampling, in this case) was the clear worst performer once precision was checked.
• Threshold adjustment is worth trying before more complex resampling techniques, since it's cheaper to implement, faster to iterate on, and in this case outperformed both SMOTE and undersampling on the metric that mattered most.

If you're building the surrounding pipeline this kind of model lives in, building a tiny end-to-end ML pipeline covers the rest of that structure, and the evaluation-metric confusion at the heart of this post is worth pairing with precision, recall, and the mistake most people make evaluating models for the deeper mechanics of why accuracy alone misleads on exactly this kind of dataset.

A fifth option worth naming: collecting more minority-class data

Every technique above works with the data already available, and it's worth stating plainly that none of them is a substitute for the actual highest-leverage fix when it's feasible at all: getting more real minority-class examples. SMOTE's synthetic examples are interpolations between existing points, not genuinely new information, and class weighting and threshold adjustment both work with the same limited real signal the original data already contains, just weighted or thresholded differently — none of them can teach the model something about fraud patterns that no example in the training data actually demonstrates. For the fraud team in this example, a parallel, longer-term effort to specifically flag and label more confirmed fraud cases from manual review queues, feeding back into future training data, was pursued alongside the four techniques above, not instead of them — the two are complementary, not competing solutions to the same underlying data scarcity problem.

Combining techniques, and why more isn't automatically better

It's tempting to stack class weighting, SMOTE, and threshold adjustment together, reasoning that if each helps individually, all three together should help more. Testing that combination against this same dataset showed a smaller improvement over threshold adjustment alone than the individual numbers might suggest — recall reached 0.84, only marginally above threshold adjustment's 0.76 on its own, while precision dropped further to 0.31, worse than threshold adjustment alone. The techniques aren't fully independent: class weighting and SMOTE are both already pushing the model's decision boundary toward the minority class in overlapping ways, and stacking a third adjustment (the lowered threshold) on top of two techniques already doing similar work pushed the trade-off further than actually intended, rather than compounding cleanly. The lesson: test combinations explicitly rather than assuming techniques that each help individually simply stack additively — they often don't, and the actual combined effect needs to be measured directly, the same way each technique was measured individually above.

Deciding on a metric before touching any technique

None of the four techniques compared above can be judged as "better" in isolation — each one shifts the precision/recall trade-off differently, and which trade-off is actually preferable depends entirely on the real cost of each error type in the specific business context, a decision that has to happen before comparing techniques, not after picking whichever one reports the highest single number. For the fraud team, a missed fraud case (false negative) was judged roughly four times costlier than a false alarm (false positive) requiring manual review, which is why recall was weighted more heavily than precision throughout this comparison. A different team, in a context where false alarms carry a heavier real cost — a medical screening tool where an unnecessary follow-up procedure is itself costly and stressful — might reasonably weight the same four techniques' results in the opposite direction, and would likely land on a different final choice using the exact same underlying numbers.