Skip to main content

How to Log Test Metrics Automatically with MLflow Autologging

· 7 min read
Jagdish Kumawat
Founder @ Dewiride

Autologging's own metrics are measured on the training data. What matters is the score on data the model never saw, and MLflow can log that by itself too: with autologging on, when you call a sklearn.metrics function on a model's predictions, MLflow logs the result to the model's run. MLflow calls these post training metrics. This guide logs a test error without a log call, then shows the import order that silently breaks it, with real output from MLflow 3.18.0.

Syntax​

Syntax
mlflow.sklearn.autolog()
from sklearn.metrics import <metric> # after autolog()

predictions = model.predict(X_test)
<metric>(y_test, predictions) # logged as "<metric>_X_test"
PartWhat it means
mlflow.sklearn.autolog()Turns on autologging, post training metrics included (log_post_training_metrics=True by default)
from sklearn.metrics import ... after itMLflow's reference: "Autologging must be enabled before scikit-learn metric APIs are imported from sklearn.metrics"
model.predict(X_test)Predictions MLflow can trace back to the model, and so to its run
The metric's name<metric function>_<variable name>: here mean_absolute_error_X_test, after the variable passed to predict

Prerequisites​

Step 1: Log a Test Metric Without a Log Call​

The example trains a linear regression on three quarters of scikit-learn's diabetes data and measures its mean absolute error (MAE) on the other quarter. Create post_training.py. The sklearn imports come after autolog() on purpose, which is why they carry # noqa: E402, a note that silences the linter's rule about imports at the top:

The post_training.py script in a macOS-style code window: it sets the tracking server and the experiment post-training-demo, calls mlflow.sklearn.autolog with log_models False, then imports load_diabetes, LinearRegression, mean_absolute_error and train_test_split, splits the data, and inside a run named after-autolog fits the model, predicts on X_test and prints the MAE.

post_training.py
import mlflow

mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("post-training-demo")
mlflow.sklearn.autolog(log_models=False)

from sklearn.datasets import load_diabetes # noqa: E402
from sklearn.linear_model import LinearRegression # noqa: E402
from sklearn.metrics import mean_absolute_error # noqa: E402
from sklearn.model_selection import train_test_split # noqa: E402

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

with mlflow.start_run(run_name="after-autolog"):
model = LinearRegression().fit(X_train, y_train)
predictions = model.predict(X_test)
print("MAE:", mean_absolute_error(y_test, predictions))
Terminal
uv run python post_training.py
Output
2026/11/21 11:02:38 INFO mlflow.tracking.fluent: Experiment with name 'post-training-demo' does not exist. Creating a new experiment.
MAE: 45.120563074396195
🏃 View run after-autolog at: http://127.0.0.1:8080/#/experiments/10/runs/a33c5c462fc04a219c719124d3395414
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/10

A terminal running uv run python post_training.py: the experiment is created, MAE: 45.120563074396195 is printed, and the run after-autolog is linked.

The script only printed the MAE. Open the run: next to autologging's five training metrics is a sixth, the test MAE:

The after-autolog run's Metrics (6) table: the five training metrics, training_mean_squared_error 2780.08, training_mean_absolute_error 42.85, training_r2_score 0.555, training_root_mean_squared_error 52.73 and training_score 0.555, plus mean_absolute_error_X_test 45.12.

The run also got a file, metric_info.json, that records which call produced the metric:

metric_info.json
{
"mean_absolute_error_X_test": "mean_absolute_error(y_true=y_test, y_pred=predictions)"
}

The test MAE, 45.12, is a little worse than the training MAE, 42.85: the model does slightly better on the rows it learned from.

Step 2: See What the Import Order Does​

The same script with the sklearn.metrics import at the top, before autolog(), as linters prefer, is post_training_early_import.py. A third version, post_training_module.py, imports the module, import sklearn.metrics, at the top and calls sklearn.metrics.mean_absolute_error(...):

The post_training_module.py script in a macOS-style code window: all imports at the top, including import sklearn.metrics, then autolog with log_models False, and inside a run named module-import it prints sklearn.metrics.mean_absolute_error of the test predictions.

post_training_module.py
import mlflow
import sklearn.metrics
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("post-training-demo")
mlflow.sklearn.autolog(log_models=False)

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

with mlflow.start_run(run_name="module-import"):
model = LinearRegression().fit(X_train, y_train)
predictions = model.predict(X_test)
print("MAE:", sklearn.metrics.mean_absolute_error(y_test, predictions))
Terminal
uv run python post_training_early_import.py
uv run python post_training_module.py
Output
MAE: 45.120563074396195
🏃 View run imported-before-autolog at: http://127.0.0.1:8080/#/experiments/10/runs/32720073d0734a638e7ba3d3176917d3
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/10
MAE: 45.120563074396195
🏃 View run module-import at: http://127.0.0.1:8080/#/experiments/10/runs/6cb2838615474634826c30e7cfd034f7
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/10

Two terminal runs: post_training_early_import.py prints MAE 45.120563074396195 and links the run imported-before-autolog; post_training_module.py prints the same MAE and links the run module-import.

Both printed the same MAE, but only one logged it:

RunHow mean_absolute_error was reachedmean_absolute_error_X_test logged?
after-autologfrom sklearn.metrics import ... after autolog()Yes
imported-before-autologfrom sklearn.metrics import ... before autolog()No, with no warning
module-importimport sklearn.metrics at the top, sklearn.metrics.mean_absolute_error(...)Yes

autolog() replaces the functions in sklearn.metrics with versions that log. A from ... import done earlier keeps a direct reference to the old function, so MLflow never sees the call; sklearn.metrics.mean_absolute_error looks the function up when it's called, after the switch. Importing the module is the simplest way to keep your imports at the top.

Common Mistakes​

MistakeWhat happensDo this instead
from sklearn.metrics import ... before autolog()Nothing is logged, silentlyImport the module, import sklearn.metrics, or import after autolog()
Passing a slice or copy of the predictions, such as predict_proba(X_test)[:, 1]Not logged: MLflow only recognizes the object predict returnedLog that metric yourself with mlflow.log_metric
Passing data.X_test instead of a variableThe metric's name ends in unknown_datasetPut the data in a variable first; its name becomes the suffix

Troubleshooting​

SymptomLikely causeFix
The metric prints but isn't in the runThe import order, or a derived prediction objectSee the table above
mean_absolute_error-2_X_test appearsThe same metric was called twice on the same dataExpected: MLflow numbers repeated calls

FAQ​

Can I turn this off?​

Yes: mlflow.sklearn.autolog(log_post_training_metrics=False).

What about model.score?​

It's logged too, as <ModelClass>_score_<dataset>, such as LinearRegression_score_X_test.

Conclusion​

With autologging on and mean_absolute_error imported after autolog(), one call on the test predictions logged mean_absolute_error_X_test, 45.12, with no log call. Import sklearn.metrics as a module, pass the object predict returned, and keep the data in a named variable.

Additional Resources​