Skip to main content

How to Log Your Own Metrics in an MLflow Autologged Run

· 5 min read
Jagdish Kumawat
Founder @ Dewiride

Autologging records what it knows: the model's settings and its scores on the training data. Your own numbers, such as a score on a test set, and your own labels still need a log call, and they belong in the same run. The way to get them there is to start the run yourself, so autologging and your calls share it. This guide adds a test score and a tag to an autologged run, with real output from MLflow 3.18.0.

Syntax​

Syntax
mlflow.sklearn.autolog()

with mlflow.start_run(run_name="<name>"):
model.fit(X_train, y_train)
mlflow.log_metric("<name>", value)
mlflow.set_tag("<key>", "<value>")
PartWhat it means
with mlflow.start_run(...)You start the run. Autologging logs into it instead of starting its own, and doesn't end it
mlflow.log_metric(...), mlflow.set_tag(...)Your own values, logged to the same open run

MLflow's reference calls this the default: with exclusive=False, "autologged content is logged to the active fluent run, which may be user-created".

Prerequisites​

Step 1: Train, Then Log a Test Score​

The example trains a ridge regression, a linear model with a little regularization, on three quarters of scikit-learn's diabetes data, then logs its R² on the other quarter as test_r2, and a tag naming the data. Create autolog_plus_manual.py:

The autolog_plus_manual.py script in a macOS-style code window: it calls mlflow.sklearn.autolog with log_models False, splits the diabetes data, and inside a run named ridge fits a Ridge model with alpha 1.0, logs the metric test_r2 from r2_score on the test predictions, and sets the tag data to diabetes.

autolog_plus_manual.py
import mlflow
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split

mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("manual-autolog-demo")
mlflow.sklearn.autolog(log_models=False)

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

with mlflow.start_run(run_name="ridge"):
model = Ridge(alpha=1.0).fit(X_train, y_train)
mlflow.log_metric("test_r2", r2_score(y_test, model.predict(X_test)))
mlflow.set_tag("data", "diabetes")
Terminal
uv run python autolog_plus_manual.py
Output
2026/11/21 11:03:07 INFO mlflow.tracking.fluent: Experiment with name 'manual-autolog-demo' does not exist. Creating a new experiment.
🏃 View run ridge at: http://127.0.0.1:8080/#/experiments/12/runs/f608fce8f4a54faa989f28ab1daa77ac
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/12

A terminal running uv run python autolog_plus_manual.py: the experiment manual-autolog-demo is created and one run, ridge, is linked.

One run, ridge: no second run was started for the fit.

Step 2: See Both Kinds of Values in One Run​

The run's metrics hold autologging's five training scores and your test_r2:

The ridge run's Metrics (6) table: training_mean_squared_error 3360.98, training_mean_absolute_error 49.35, training_r2_score 0.4625, training_root_mean_squared_error 57.97, training_score 0.4625, and test_r2 0.3570.

And its tags hold autologging's estimator_name and estimator_class next to your data:

The ridge run's About this run panel: Source autolog_plus_manual.py, Datasets with dataset (f7e5a261) Train and one more, and Tags estimator_name Ridge, estimator_class sklearn.linear_model._ridge.Ridge and data diabetes.

test_r2, 0.3570, is lower than training_r2_score, 0.4625: the usual gap between the data a model learned from and data it didn't. Logging both is what shows the gap.

Common Mistakes​

MistakeWhat happensDo this instead
Calling fit before start_runAutologging starts and ends its own run, and your later values land in a second runFit inside your with mlflow.start_run() block
Logging after the with blockNo run is open, so MLflow starts a new one for the valueLog inside the block
Naming your metric like autologging's, such as training_scoreYour value and autologging's share one nameUse your own names, such as test_r2

Troubleshooting​

SymptomLikely causeFix
Two runs instead of oneThe model was fitted outside your runMove fit into the with block
Your metric is in the run, but autologging's aren'tAutologging was off, or set with exclusive=TrueCall mlflow.sklearn.autolog() before the run, without exclusive=True

FAQ​

Could autologging log the test score by itself?​

Yes, in some cases: How to Log Test Metrics Automatically with MLflow Autologging explains when, and why the import order matters. Logging it yourself is explicit and always works.

Yes: start the parent run yourself, run the search inside it, and log your test score on the parent after fit. The child runs are created inside it as usual.

Conclusion​

Starting the run with mlflow.start_run(run_name="ridge") put autologging's training metrics and tags, and the test_r2 metric and data tag logged by hand, into one run. Start the run yourself, fit inside it, and log your own values before the block ends.

Additional Resources​