Feature Engineering That Actually Moved the Needle: A Real Before/After

Feature engineering advice tends to stay abstract — "create meaningful features from raw data" — without showing what that concretely looks like or how much it actually moves a real number. This is three specific feature engineering changes applied to a real churn-prediction dataset, each with the actual F1 score before and after, plus a fourth change that looked reasonable going in and made the model measurably worse, which is at least as instructive as the ones that helped.

The baseline: raw columns, no engineering

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

# Raw columns: signup_date, last_login_date, monthly_charges,
# total_charges, contract_type, support_tickets_count
baseline_model = RandomForestClassifier(random_state=42)
baseline_scores = cross_val_score(baseline_model, X_raw, y, cv=5, scoring='f1')

print(f"Baseline F1: {baseline_scores.mean():.3f}")
# Baseline F1: 0.612

Two raw date columns, two raw numeric columns, a categorical column, and a count — fed directly into a random forest with no transformation beyond basic encoding. This is the number every change below gets measured against.

Change 1: turning two dates into a duration

signup_date and last_login_date as raw dates carry almost no directly usable signal for a tree-based model — a specific calendar date doesn't meaningfully generalize across different customers who signed up on different dates. The actual signal buried in those two columns is how long it's been since the customer last logged in, and how long they've been a customer overall:

df['days_since_last_login'] = (reference_date - df['last_login_date']).dt.days
df['account_age_days'] = (reference_date - df['signup_date']).dt.days
df = df.drop(columns=['signup_date', 'last_login_date'])

scores = cross_val_score(RandomForestClassifier(random_state=42), X_with_durations, y, cv=5, scoring='f1')
print(f"With duration features: {scores.mean():.3f}")
# With duration features: 0.687

This was the single largest individual improvement of everything tried — turning two low-signal date columns into two duration features the model could actually use meaningfully. It makes intuitive sense in hindsight: "how many days since this person last logged in" is a genuinely predictive churn signal in a way that "what specific calendar date did they last log in" mostly isn't, since the model has no way to generalize across dates it hasn't seen in training without the transformation doing that generalization work for it first.

Change 2: a ratio feature instead of two raw numbers

monthly_charges and total_charges individually are both reasonable signals, but their ratio — effectively, tenure-adjusted spending — captures something neither one does alone:

df['avg_charge_per_month'] = df['total_charges'] / df['account_age_days'].clip(lower=1)

scores = cross_val_score(RandomForestClassifier(random_state=42), X_with_ratio, y, cv=5, scoring='f1')
print(f"With ratio feature added: {scores.mean():.3f}")
# With ratio feature added: 0.701

The .clip(lower=1) matters more than it looks like it should — without it, a brand-new customer with an account age of zero days produces a division by zero, which either crashes the pipeline or silently produces an infinite or NaN value depending on the library, either of which corrupts training if it slips through unnoticed. This is a small, easy-to-miss detail with a real correctness cost, not just a style preference.

Change 3: bucketing support ticket count instead of using it raw

df['ticket_bucket'] = pd.cut(
    df['support_tickets_count'],
    bins=[-1, 0, 2, 5, float('inf')],
    labels=['none', 'low', 'medium', 'high']
)

scores = cross_val_score(RandomForestClassifier(random_state=42), X_with_buckets, y, cv=5, scoring='f1')
print(f"With bucketed tickets: {scores.mean():.3f}")
# With bucketed tickets: 0.706

A smaller gain than the previous two, but a real one — the raw count treats the difference between 0 and 1 ticket the same as the difference between 20 and 21, when the actual churn-relevant signal is closer to "did this person have a rough support experience at all" (a coarse category) rather than the exact count. Bucketing is a deliberate trade: it throws away some precision in exchange for a representation that may match the underlying relationship better, and here it paid off slightly, though this is exactly the kind of change worth verifying with a real before/after number rather than assuming it helps just because it sounds reasonable.

Change 4: the one that looked promising and made things worse

Adding a feature for "day of week the customer signed up" seemed like a plausible signal — maybe customers who sign up on a weekend behave differently than weekday sign-ups. Testing it directly rather than trusting the intuition:

df['signup_day_of_week'] = df['signup_date'].dt.dayofweek

scores = cross_val_score(RandomForestClassifier(random_state=42), X_with_dow, y, cv=5, scoring='f1')
print(f"With signup day-of-week: {scores.mean():.3f}")
# With signup day-of-week: 0.691

That's worse than the 0.706 from the previous step, not better. With a 7-category feature carrying little genuine signal on a dataset of this size, the model spent some of its limited capacity finding spurious patterns in day-of-week that don't generalize — a small-scale version of the exact overfitting mechanism covered in watching a model overfit, where added complexity without added genuine signal makes generalization worse, not better. This feature got dropped from the final model specifically because it was tested and measured, not kept on the assumption that more features are automatically better.

The full before/after summary

The pattern worth noticing: the biggest win came from the most fundamental transformation (raw dates into meaningful durations), and each subsequent change delivered a smaller improvement — a common shape in feature engineering work, where the first few genuinely well-targeted features do most of the work and later additions face diminishing, occasionally negative, returns.

Checking feature importance to confirm the reasoning, not just the score

A rising F1 score confirms a change helped overall, but it doesn't confirm the change helped for the reason you assumed it did. Checking the trained model's feature importances after the final version closed that gap:

final_model = RandomForestClassifier(random_state=42)
final_model.fit(X_train_final, y_train)

importances = sorted(
    zip(X_train_final.columns, final_model.feature_importances_),
    key=lambda x: -x[1]
)

for name, importance in importances:
    print(f"{name}: {importance:.3f}")

The output confirmed days_since_last_login as the single most important feature by a real margin — matching the hypothesis that motivated adding it in the first place, rather than the improvement coming from some unrelated interaction nobody had actually reasoned about. This step matters because a model can improve for reasons different from the ones you assumed, and checking feature importance against your original hypothesis is a cheap way to confirm your mental model of the data actually matches what the model learned, not just that the accuracy number went up for some reason.

A change that helped accuracy but hurt something else worth checking

One more test worth running before calling any of this finished: does an improved F1 score come with a corresponding change in precision-versus-recall balance that matters for how the model gets used? Checking precision and recall separately, not just the combined F1:

from sklearn.metrics import precision_score, recall_score

for name, preds in [('Baseline', baseline_preds), ('Final', final_preds)]:
    p = precision_score(y_test, preds)
    r = recall_score(y_test, preds)
    print(f"{name}: precision={p:.3f}  recall={r:.3f}")

In this case, both precision and recall improved roughly proportionally alongside the F1 gain, which is the reassuring outcome — but it's worth checking explicitly rather than assuming, since it's entirely possible for an F1 improvement to hide a precision-recall trade-off that matters more for the actual business use case than the single blended number suggests, a trap covered in more depth in precision, recall, and the mistake most people make evaluating models.

Order matters when applying these changes

It's worth noting the order these four changes were tried in wasn't arbitrary — the duration transformation came first specifically because it addressed the columns carrying the most obviously wasted signal in their raw form, and each subsequent change was tested against the current best version rather than against the original baseline in isolation. Testing every candidate feature against the original untouched baseline, rather than against the current best version, can produce misleading results when two features interact or partially overlap in what they capture — a feature that looks like a strong improvement in isolation might add little or nothing once a related, already-added feature already captures most of the same signal.

What made these specific changes work

• Every change targeted a specific, named hypothesis about the underlying pattern ("recency of engagement matters," "spending relative to tenure matters"), not a generic "let's add more features" instinct.
• Every change was measured against the same cross-validated baseline, not just intuited as probably helpful — which is exactly what caught the day-of-week feature actually hurting despite sounding reasonable going in.
• Domain reasoning came before the code, not after — knowing that "days since last login" is a plausible churn signal came from understanding the actual business problem, not from mechanically transforming every column and seeing what stuck.

If you're extracting features from free-text fields rather than structured columns like these — support ticket descriptions, for instance — a quick pass through a word counter on a sample of that text is a fast first check on whether length or word-count patterns are even worth engineering into a feature before building anything more elaborate around them. A support ticket description that's consistently much longer or shorter for customers who eventually churn versus those who don't is exactly the kind of cheap, easy-to-check signal worth testing before reaching for anything resembling real natural language processing, and it costs almost nothing to check first, before committing real engineering time and effort to a much heavier text-modeling approach that might not even turn out to be necessary at all in the end.

None of the four changes here required exotic technique — a duration calculation, a ratio, a bucketing function, and one tested-and-rejected idea. What made them add up to a real, cumulative F1 improvement was treating each one as a specific, testable hypothesis rather than a general "throw more features at it" strategy, and being willing to measure and discard the one that didn't actually help despite sounding entirely plausible going into the test.