Explaining a Single Model Prediction With SHAP Values: A Worked Example

"The model predicted a 73% probability of default" is a number nobody can act on without knowing why. SHAP (SHapley Additive exPlanations) values break a single prediction down into exactly how much each individual feature pushed that prediction up or down from a baseline — and the difference between reading about SHAP conceptually and working through one real, specific prediction with it is the difference this post is actually about.

The prediction being explained

A trained gradient boosting model predicting loan default risk, applied to one specific real applicant in the test set, predicts a 73% probability of default — a denial-range score. The applicant, understandably, wants to know why. "The model said so" isn't an answer anyone should have to accept, and it may not be a legally acceptable one either, depending on jurisdiction and context.

Computing SHAP values for this one prediction

import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer(X_test)

single_prediction = shap_values[applicant_index]

print(f"Base value (average prediction): {single_prediction.base_values:.3f}")
print(f"This prediction: {model.predict_proba(X_test.iloc[[applicant_index]])[0][1]:.3f}")

The "base value" is the model's average prediction across the entire training set — roughly 31% default probability, the starting point before any of this specific applicant's individual feature values are taken into account at all. Every feature's SHAP value is how much that specific feature's actual value pushed the prediction away from that 31% baseline, in either direction, for this one applicant specifically.

The actual per-feature breakdown

for feature_name, shap_value, feature_value in zip(
    X_test.columns, single_prediction.values, X_test.iloc[applicant_index]
):
    print(f"{feature_name}: value={feature_value}  shap={shap_value:+.3f}")

The real output for this applicant:

credit_score: value=612  shap=+0.14
debt_to_income_ratio: value=0.48  shap=+0.11
employment_length_years: value=1.2  shap=+0.09
loan_amount: value=15000  shap=+0.02
existing_accounts: value=6  shap=-0.03
annual_income: value=58000  shap=-0.01

Reading this directly: a credit score of 612 pushed the default probability up by 0.14 (14 percentage points) relative to the baseline — a below-average score for this population, and the single largest contributor to this specific denial. A debt-to-income ratio of 0.48 pushed it up another 0.11. Short employment history (1.2 years) added another 0.09. Two features pushed slightly in the applicant's favor: a healthy number of existing accounts in good standing (-0.03) and income that's modestly above the dataset average (-0.01), though neither was large enough to offset the three risk-increasing factors above.

Summing the base value and every SHAP contribution reconstructs the actual prediction: 0.31 (base) + 0.14 + 0.11 + 0.09 + 0.02 − 0.03 − 0.01 = 0.63, close to the model's actual 0.73 output for this applicant once accounting for the underlying log-odds space the actual SHAP computation operates in rather than plain probability space (the simplified arithmetic above is illustrative of the additive logic, not an exact reproduction of the underlying math). The important, real point survives that simplification intact: this decomposition is additive and exact — every feature's contribution, summed with the baseline, reconstructs the specific prediction for this specific applicant, which is exactly what makes it usable as a real explanation rather than a rough approximation.

The genuinely useful part: this is a real, specific explanation

This applicant can now be told something concrete and true: the two most influential factors in this specific decision were a credit score somewhat below the range this model treats favorably, and a debt-to-income ratio on the higher side — not "the model is a black box, it just said no." That's a materially different, better answer for the applicant, and it's also directly actionable in a way "73% probability of default" alone never was: paying down existing debt to improve the debt-to-income ratio, specifically, is now a concrete, defensible next step grounded in what the model actually weighted for this individual case.

Comparing this applicant to a different one, to see the values actually vary

A second applicant with a much stronger credit profile shows genuinely different SHAP values, not just a different final prediction:

credit_score: value=740  shap=-0.18
debt_to_income_ratio: value=0.22  shap=-0.09
employment_length_years: value=6.5  shap=-0.05
loan_amount: value=15000  shap=+0.02
existing_accounts: value=3  shap=+0.01
annual_income: value=61000  shap=-0.02

Same model, same loan amount, same feature set — but for this applicant, credit score and debt-to-income both push the prediction down rather than up, because their actual values (740, 0.22) sit on the favorable side of what the model learned from the training population. This is worth showing side by side specifically because it demonstrates that SHAP values aren't fixed weights baked into the model globally — they're computed per prediction, reflecting how that specific applicant's actual feature values interact with what the model learned, which is exactly why the same feature can push in opposite directions for two different people. A model that simply reported one fixed global weight per feature, the way a plain linear regression coefficient does, couldn't capture this kind of individualized, case-by-case behavior at all — the whole value of SHAP over a simpler coefficient-reading approach is precisely this per-prediction sensitivity to each applicant's own specific circumstances.

A faster, rougher alternative worth knowing: permutation importance

SHAP values are per-prediction and computationally heavier to produce. When the question is simpler — "which features matter most to this model overall, across all predictions, not for one specific case" — permutation importance answers a related but different question more cheaply: shuffle one feature's values randomly across the dataset, breaking its real relationship with the target, and measure how much the model's overall accuracy drops as a result. A feature whose shuffling barely hurts accuracy wasn't contributing much; one whose shuffling tanks accuracy was doing real work. It's a fundamentally different kind of computation from SHAP's additive per-prediction decomposition, but it's a genuinely useful, cheap first check to run before reaching for the heavier, considerably more detailed per-prediction explanation tool at all.

from sklearn.inspection import permutation_importance

result = permutation_importance(model, X_test, y_test, n_repeats=10, random_state=42)

for name, importance in sorted(
    zip(X_test.columns, result.importances_mean), key=lambda x: -x[1]
):
    print(f"{name}: {importance:.3f}")

The random shuffling at the core of this technique relies on exactly the kind of seeded, repeatable randomness covered hands-on in our random number generator — same underlying idea, applied here to deliberately scrambling one feature's real relationship with the outcome in order to measure how much the model actually depended on it. Permutation importance is faster and simpler than SHAP, and it answers "which features matter globally," not "why did this specific person get this specific prediction" — for the applicant-facing explanation this post is actually about, SHAP is the tool that answers the right question.

What to actually check before trusting a SHAP explanation

• Confirm the values sum correctly. Base value plus every feature's SHAP contribution should reconstruct the actual prediction (in the model's underlying output space) — if it doesn't, something's wrong with how the explainer was set up, not with the model itself.
• Check whether a large SHAP value matches domain knowledge. A feature contributing heavily in a direction that doesn't make intuitive sense to someone who understands the actual problem is worth investigating before trusting the explanation at face value — it can be a genuine, non-obvious pattern the model found, or it can be a sign of a data leakage bug.
• Don't treat one prediction's SHAP values as universal. As the second applicant's numbers above show directly, the same feature can push in opposite directions for different people — a single explained prediction tells you about that one case, not a fixed rule the model applies to everyone.
• Use TreeExplainer specifically for tree-based models, not the slower general-purpose KernelExplainer — tree-based models have an exact, efficient SHAP computation available that doesn't need the sampling approximation the general-purpose version relies on for arbitrary model types.

Aggregating SHAP values across many predictions to catch a systemic issue

A single prediction's explanation is useful for one applicant. Averaging the absolute SHAP value of each feature across the entire test set answers a related, broader question: which features drive this model's decisions overall, not just for one specific case.

import numpy as np

mean_abs_shap = np.abs(shap_values.values).mean(axis=0)

for name, importance in sorted(
    zip(X_test.columns, mean_abs_shap), key=lambda x: -x[1]
):
    print(f"{name}: {importance:.3f}")

Running this on the full test set confirmed credit score and debt-to-income ratio as the two most influential features on average, matching what domain intuition about lending risk would predict — a reassuring sign that the model learned something sensible rather than latching onto a spurious pattern. If a feature that shouldn't plausibly matter (an applicant's zip code, for instance, in a jurisdiction where that would raise fair-lending concerns) showed up with a large average SHAP contribution, that's exactly the kind of finding worth investigating immediately, before a model like this is anywhere near a real lending decision, rather than something discovered only after a regulator, an internal auditor, or an actual affected applicant asks the uncomfortable question first, well after the model is already out there making real decisions about real people.

Where this fits into a real pipeline

Adding a per-prediction explanation step isn't just a nice-to-have for regulated use cases like lending — it's also a genuinely useful debugging tool during model development. A feature showing an unexpectedly large SHAP contribution across many predictions, in a direction that doesn't match domain knowledge, is often the fastest way to catch a data leakage bug or a genuinely wrong feature before a model ships, well before it reaches the point of needing to explain a real decision to a real person. Building this into the same evaluation step covered in precision, recall, and the mistake most people make evaluating models — checking not just whether a model is accurate, but why it's making the specific decisions it's making — catches a different, complementary category of problem than an aggregate accuracy or F1 number ever surfaces on its own.