Skip to main content

How to Log SHAP Explanations with mlflow.models.evaluate

· 7 min read
Jagdish Kumawat
Founder @ Dewiride

Metrics say how well a model does; they don't say why. SHAP values explain each prediction column by column: how much each input pushed it up or down. mlflow.models.evaluate computes them and logs three plots by itself, as soon as the shap package is installed. This guide installs it, evaluates a diabetes model and reads the plots, with real output and UI from MLflow 3.18.0.

How SHAP Fits into an Evaluation​

The reference for mlflow.models.evaluate: "For both the "regressor" and "classifier" model types, the default evaluator generates model summary plots and feature importance plots using SHAP." It's on by default, through the option log_model_explainability, but shap isn't one of mlflow's requirements, so without it MLflow skips the step with a warning. The limits it lists: "all feature values must be numeric, and each feature column must only contain scalar values."

Prerequisites​

  • An MLflow tracking server at http://127.0.0.1:8080. MLflow's tracking server guide shows how to start one.
  • mlflow and scikit-learn in your project, for example with uv add mlflow scikit-learn pandas.

Step 1: Install shap​

In your project:

Terminal
uv add shap

uv adds shap 0.54.1 and two small packages it uses, slicer and tqdm. How to Add a Package to a Python Project with uv explains the command.

Step 2: Evaluate a Model​

The example fits a linear regression to scikit-learn's diabetes data, which predicts how far each patient's disease progressed after a year from ten measurements: age, sex, body mass index (bmi), blood pressure (bp) and six blood serum values, s1 to s6. It logs the model, evaluates it on the test rows, and prints the names of the artifacts:

The explain_model.py script in a macOS-style code window: the diabetes data, a LinearRegression logged in a run named explain-linear, mlflow.models.evaluate on the test rows with targets progression and model_type regressor, and a loop printing each artifact's name.

explain_model.py
import mlflow
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("shap-explanations-demo")

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
model = LinearRegression().fit(X_train, y_train)

with mlflow.start_run(run_name="explain-linear"):
info = mlflow.sklearn.log_model(model, name="model", input_example=X_train.head(3))
result = mlflow.models.evaluate(
info.model_uri,
X_test.assign(progression=y_test),
targets="progression",
model_type="regressor",
)

for name in result.artifacts:
print(name)
Terminal
uv run python explain_model.py
Output (last lines)
2026/11/21 17:55:57 INFO mlflow.models.evaluation.default_evaluator: Testing metrics on first row...
Background dataset has 111 samples but max_samples=100. Subsampling to 100 samples for SHAP value computation. To use all samples, set max_samples=111 when initializing the masker.
2026/11/21 17:55:57 INFO mlflow.models.evaluation.evaluators.shap: Shap explainer ExactExplainer is used.
🏃 View run explain-linear at: http://127.0.0.1:8080/#/experiments/15/runs/7ff3e0426fc143dc8878ce5d0d8ff413
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/15
shap_beeswarm_plot
shap_summary_plot
shap_feature_importance_plot

The end of the output in a terminal: Testing metrics on first row, shap's note that it subsamples 100 of the 111 test rows, Shap explainer ExactExplainer is used, the run's link, and the three artifacts shap_beeswarm_plot, shap_summary_plot and shap_feature_importance_plot.

  • No SHAP warning, and three artifacts, where a regressor without shap gets none.
  • Shap explainer ExactExplainer is used: MLflow lets shap pick its algorithm.
  • Background dataset has 111 samples but max_samples=100: shap's note that it uses 100 of the 111 test rows as its background data. MLflow's own option, explainability_nsamples, is "The number of sample rows to use for computing model explainability insights. Default value is 2000."

With shap installed, a Downloading artifacts progress bar, 7/7, also appeared before the evaluation started.

Step 3: Read the Feature Importance Plot​

shap_feature_importance_plot.png ranks the columns by their average effect on the predictions, the mean absolute SHAP value:

The shap_feature_importance_plot.png file: horizontal bars for mean absolute SHAP value, from s5 at 25.54, s1 at 21.58 and bmi at 21.03, through bp 12.31, s2 10.2, sex 9.78 and s4 4.66, down to age 1.63, s6 0.86 and s3 0.33.

s5, s1 and bmi move a prediction by about 21 to 26 points on average; age, s6 and s3 barely move it at all.

Step 4: Read the Beeswarm Plot​

Open the run, explain-linear, go to Artifacts and click shap_beeswarm_plot.png. Each dot is one test patient; its position is how much that column pushed the prediction, and its colour is the column's value, red high and blue low:

The run's shap_beeswarm_plot.png on the Artifacts tab: one row of dots per column from s5 down to s3, with SHAP values from about minus 70 to plus 85 on the x-axis; for s5 and bmi the red dots sit on the right, for s1 on the left, and for sex the dots form two tight groups.

  • s5 and bmi: red dots on the right, so high values push the prediction up.
  • s1: red dots on the left, so in this model a high value pushes it down.
  • sex: two tight groups, one per value, as a column with two values gives.

The third file, shap_summary_plot.png, shows the same SHAP values in a slightly different layout.

Common Mistakes​

MistakeWhat happensDo this instead
Expecting explanations for data with columns of wordsMLflow's check fails before SHAP runsTurn explainability off for that model: Fixed: Cannot interpret StringDtype
Reading SHAP values as causesThey describe the model, not the worldRead them as "what this model relies on"

Troubleshooting​

SymptomLikely causeFix
SHAP or matplotlib package is not installedNo shap in the projectuv add shap: Fixed: SHAP or matplotlib package is not installed
FutureWarning: The NumPy global RNG was seeded by calling np.random.seedshap's own summary_plot, called by MLflow, after shap's PermutationExplainer seeded NumPy's global generator (shap issue #4607); it appeared when a classifier was explainedNothing to change in your code

FAQ​

Does every model get the same explainer?​

No: MLflow lets shap choose. This linear regression got ExactExplainer; the tumour classifier of the SHAP fixes post got PermutationExplainer. The option explainability_algorithm picks one yourself.

Can I turn the explanations off again?​

Yes: evaluator_config={"log_model_explainability": False} skips them, with shap installed or not.

Conclusion​

With shap installed, mlflow.models.evaluate logged three SHAP plots for the diabetes model: s5, s1 and bmi move its predictions most, high s5 and bmi push them up, and high s1 pushes them down.

Additional Resources​