Skip to main content

How to Evaluate a Regression Model with mlflow.models.evaluate

· 6 min read
Jagdish Kumawat
Founder @ Dewiride

A regression model predicts a number, so its test scores are about distance: how far off it is on average, at worst, and how much of the spread it explains. mlflow.models.evaluate with model_type="regressor" computes them all in one call and logs them to the run, linked to the model. This guide evaluates a model of diabetes progression, with real output from MLflow 3.18.0.

Syntax​

Syntax
result = mlflow.models.evaluate(
model_uri,
data,
targets="target_column",
model_type="regressor",
)
PartWhat it means
model_uriThe model to evaluate, such as models:/m-...
dataA pandas DataFrame with the model's input columns and the true values
targetsThe name of the column of true values in data
model_type"regressor", so MLflow computes the regression metrics
resultAn EvaluationResult; result.metrics holds the scores

The reference for mlflow.models.evaluate lists what a regressor gets: "example_count, mean_absolute_error, mean_squared_error, root_mean_squared_error, sum_on_target, mean_on_target, r2_score, max_error, mean_absolute_percentage_error."

Prerequisites​

  • An MLflow tracking server at http://127.0.0.1:8080. MLflow's tracking server guide shows how to start one.
  • mlflow and scikit-learn in your project, for example with uv add mlflow scikit-learn pandas.

Step 1: Log and Evaluate the Model​

The example fits a linear regression to three quarters of scikit-learn's diabetes data, which predicts how far each patient's disease progressed after a year. It logs the model and evaluates it on the other quarter:

The evaluate_regressor.py script in a macOS-style code window: the diabetes data split in four, a LinearRegression, log_model inside a run named evaluate-linear, mlflow.models.evaluate on the test rows with targets progression and model_type regressor, and every metric printed.

evaluate_regressor.py
import mlflow
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("evaluate-regressor-demo")

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
model = LinearRegression().fit(X_train, y_train)

with mlflow.start_run(run_name="evaluate-linear"):
info = mlflow.sklearn.log_model(model, name="model", input_example=X_train.head(3))
result = mlflow.models.evaluate(
info.model_uri,
X_test.assign(progression=y_test),
targets="progression",
model_type="regressor",
)

for name, value in result.metrics.items():
print(f"{name}: {value:.4f}")
Terminal
uv run python evaluate_regressor.py
Output (last lines)
2026/11/21 17:52:02 INFO mlflow.tracking.fluent: Active model is set to the logged model with ID: m-efdb7e4071df4f1ba3e11d2a89539c4c
2026/11/21 17:52:02 INFO mlflow.tracking.fluent: Use `mlflow.set_active_model` to set the active model to a different one if needed.
2026/11/21 17:52:02 INFO mlflow.models.evaluation.default_evaluator: Testing metrics on first row...
2026/11/21 17:52:02 WARNING mlflow.models.evaluation.evaluators.shap: SHAP or matplotlib package is not installed, so model explainability insights will not be logged.
🏃 View run evaluate-linear at: http://127.0.0.1:8080/#/experiments/6/runs/c405c4547aa14fd7a61ed5e9c431d06b
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/6
score: 0.3594
example_count: 111.0000
mean_absolute_error: 45.1206
mean_squared_error: 3180.1596
root_mean_squared_error: 56.3929
sum_on_target: 16957.0000
mean_on_target: 152.7658
r2_score: 0.3594
max_error: 162.4418
mean_absolute_percentage_error: 0.3796

The end of the output in a terminal: the SHAP warning, the run's link, and ten metrics, from score 0.3594 and mean_absolute_error 45.1206 to mean_absolute_percentage_error 0.3796.

Step 2: Read the Metrics​

On the 111 test patients (example_count), whose true values average 152.8 (mean_on_target):

  • mean_absolute_error 45.12: on average, a prediction is 45 points off.
  • root_mean_squared_error 56.39, the square root of mean_squared_error 3180.16: like the MAE, but large errors count more.
  • max_error 162.44: the worst single prediction.
  • mean_absolute_percentage_error 0.3796: about 38% off, relative to the true value.
  • r2_score 0.3594, and score, the same from scikit-learn's own model.score: the model explains about a third of the variation. How to Calculate MAE, RMSE and R² in scikit-learn explains these three.
  • sum_on_target 16957, the sum of the true values, which with example_count gives their mean.

A regressor gets no plots: the five plots MLflow draws are classifier plots, and its regression plots come from SHAP, which isn't installed here, as the WARNING line says. How to Log SHAP Explanations with mlflow.models.evaluate adds them.

Common Mistakes​

MistakeWhat happensDo this instead
Reading MAPE when true values can be near zeroTiny true values blow the percentage upRead the MAE and RMSE, in the target's own units
Evaluating on the rows the model trained onThe errors look smaller than they will beKeep test rows aside with train_test_split

Troubleshooting​

SymptomLikely causeFix
SHAP or matplotlib package is not installedNo shap packageFixed: SHAP or matplotlib package is not installed
No files on the run's Artifacts tabA regressor without SHAP logs metrics onlyInstall shap, or add your own plot: How to Add a Custom Plot to an MLflow Evaluation

FAQ​

Which of these metrics should I watch?​

The MAE for a plain "how far off", the RMSE when big misses hurt more, and R² to compare models on the same data.

Can I add a metric MLflow doesn't compute?​

Yes, with make_metric: How to Add a Custom Metric to an MLflow Evaluation with make_metric.

Conclusion​

mlflow.models.evaluate(..., model_type="regressor") scored the diabetes model on 111 test patients: 45 points off on average, 162 at worst, R² 0.36, all logged to the run and linked to the model.

Additional Resources​