Skip to main content

MLflow Tutorial for Machine Learning in Python — Part 3: Log Artifacts and Datasets

· 30 min read
Jagdish Kumawat
Founder @ Dewiride

Part 2's runs record numbers: scores at every step and the computer's load. Two questions those numbers can't answer: which data did this run learn from, and what did it produce besides scores? In this part every run records its training, validation and test data as MLflow datasets, and saves three files with itself: which features the forest relies on, every setting it used, and every prediction it got wrong.

This is Part 3 of a 10-part series.

  1. Part 1: Install MLflow and Track Your First Experiment
  2. Part 2: Log Metrics Over Time and System Metrics
  3. Part 3: Log Artifacts and Datasets (you are here)
  4. Part 4: Search and Filter Runs (coming soon)
  5. Part 5: Autologging and Hyperparameter Tuning with Nested Runs (coming soon)
  6. Part 6: Log, Load and Validate Models (coming soon)
  7. Part 7: Evaluate Models and Set Quality Gates (coming soon)
  8. Part 8: Register, Version and Promote Models with the Model Registry (coming soon)
  9. Part 9: Serve a Model as a REST API and in Docker (coming soon)
  10. Part 10: Production Checklist and Recap (coming soon)

Every command and output on this page is real, captured while building the project.

What You'll Build in This Part​

You'll keep working on the house price project and add:

  • datasets: each run records the three parts of the data it used, training, validation and test, with a fingerprint of their contents (the digest), their source, their columns and their size;
  • a feature importance table, logged with mlflow.log_table(), which the UI can show for several runs side by side;
  • the forest's complete settings, logged as a JSON file with mlflow.log_dict();
  • a CSV file of every test prediction and its error, biggest misses first, logged with mlflow.log_artifact();
  • comparisons in the MLflow UI: which runs used the same data, and how their files differ.

Two words for this part. An artifact is any file stored with a run, such as Part 1's chart. A dataset, in MLflow, is a record of the data a run used, not the data itself.

What This Part Costs​

Nothing. Everything runs on your own computer, and there are no new packages.

Prerequisites​

  • Part 2 finished: the mlflow-server folder with its runs, and the dewiride-house-price-prediction project, committed to Git. Part 2 builds on Part 1 for both.

Step 1: Start the Tracking Server​

Open a terminal in the mlflow-server folder and start the server, as before:

Terminal 1 (in mlflow-server)
uv run mlflow server --port 8080

Wait for Application startup complete, leave it running, and open a second terminal in the dewiride-house-price-prediction folder for everything else.

Step 2: Note Where the Data Comes From​

A dataset record includes its source: where the data came from, so anyone can find the original later. scikit-learn's fetch_california_housing() downloads the California Housing data from one fixed address, so that address is the source. Open src/house_price_prediction/data.py and add it below TARGET:

src/house_price_prediction/data.py (below TARGET)
TARGET = "MedHouseVal"
# The file scikit-learn downloads the California Housing data from.
SOURCE_URL = "https://ndownloader.figshare.com/files/5976036"

The complete file:

src/house_price_prediction/data.py
"""Load the California Housing data and split it for training, validation and testing."""

from dataclasses import dataclass

import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

TARGET = "MedHouseVal"
# The file scikit-learn downloads the California Housing data from.
SOURCE_URL = "https://ndownloader.figshare.com/files/5976036"


@dataclass(frozen=True)
class DataSplit:
"""The features and target values for training, validation and testing."""

X_train: pd.DataFrame
X_val: pd.DataFrame
X_test: pd.DataFrame
y_train: pd.Series
y_val: pd.Series
y_test: pd.Series


def load_housing_data() -> pd.DataFrame:
"""Return the dataset as one table. The first call downloads it; later calls read a cache."""
return fetch_california_housing(as_frame=True).frame


def split_data(
frame: pd.DataFrame, test_size: float, validation_size: float, random_state: int
) -> DataSplit:
"""Hold back `test_size` of the rows for the final test, then `validation_size` of the rest
to watch the model while it trains. The model learns from what is left."""
features = frame.drop(columns=TARGET)
target = frame[TARGET]
X_rest, X_test, y_rest, y_test = train_test_split(
features, target, test_size=test_size, random_state=random_state
)
X_train, X_val, y_train, y_val = train_test_split(
X_rest, y_rest, test_size=validation_size, random_state=random_state
)
return DataSplit(X_train, X_val, X_test, y_train, y_val, y_test)

Step 3: Describe Each Part of the Split as a Dataset​

mlflow.data.from_pandas() turns a pandas table into an MLflow dataset. It computes three things from the table, without storing the table itself:

  • a digest, an 8-character fingerprint of the contents: the same data gives the same digest in every run;
  • a schema, each column with its type;
  • a profile, the number of rows and of values.

The split keeps each part's features and target values apart, in X_train and y_train and so on. A dataset should hold both, with the target marked, so each part is joined back into one table first. Create src/house_price_prediction/datasets.py:

src/house_price_prediction/datasets.py
"""Describe the data each run trains, validates and tests on, so MLflow can record it."""

import mlflow.data
import pandas as pd
from mlflow.data.pandas_dataset import PandasDataset

from house_price_prediction.data import SOURCE_URL, TARGET, DataSplit


def as_dataset(features: pd.DataFrame, target: pd.Series, name: str) -> PandasDataset:
"""One part of the split as an MLflow dataset: its features with the target column added."""
return mlflow.data.from_pandas(
features.join(target), source=SOURCE_URL, targets=TARGET, name=name
)


def split_datasets(data: DataSplit) -> dict[str, PandasDataset]:
"""The three parts of the split, each under the context MLflow records it with."""
return {
"training": as_dataset(data.X_train, data.y_train, "california-housing-train"),
"validation": as_dataset(data.X_val, data.y_val, "california-housing-validation"),
"testing": as_dataset(data.X_test, data.y_test, "california-housing-test"),
}

The datasets.py module in a macOS-style code window: an as_dataset function that joins the target to the features and returns mlflow.data.from_pandas with the figshare source URL, the MedHouseVal target and a name, and a split_datasets function that returns a dictionary of three datasets keyed training, validation and testing.

  • features.join(target) adds the target column to the features, matching rows by their labels: How to Join a pandas Series to a DataFrame by Index.
  • source=SOURCE_URL records the download address. Because it starts with https://, MLflow stores it as a web source, and the UI offers to open it.
  • targets=TARGET marks MedHouseVal as the column the model predicts.
  • name= gives each part a readable name; without it, every dataset would be called dataset.
  • The dictionary's keys, training, validation and testing, are the contexts: what the run used each part for. They're the labels MLflow's own examples use.

Step 4: Log the Datasets with Each Run​

mlflow.log_input() attaches a dataset to the active run, with its context. In src/house_price_prediction/train.py, import the new module:

src/house_price_prediction/train.py (imports)
from house_price_prediction.datasets import split_datasets

Then log the three datasets right after the run's tags, before training starts:

src/house_price_prediction/train.py (after set_tags)
mlflow.set_tags({"model_type": "random_forest", "dataset": "california_housing"})
for context, dataset in split_datasets(data).items():
mlflow.log_input(dataset, context=context)

Three calls, one per part. Only the description travels to the tracking server; the data stays on your computer.

Step 5: Turn the Trained Forest into Two Tables​

Two tables are worth keeping with every run: which inputs the forest relies on, and where its predictions go wrong. Create src/house_price_prediction/reports.py:

src/house_price_prediction/reports.py
"""Tables that show what the trained model relies on and where it goes wrong."""

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor


def feature_importances(model: RandomForestRegressor) -> pd.DataFrame:
"""How much each feature helped the forest's splits, most important first."""
table = pd.DataFrame(
{"feature": model.feature_names_in_, "importance": model.feature_importances_}
)
return table.sort_values("importance", ascending=False, ignore_index=True)


def prediction_errors(
X_test: pd.DataFrame, y_test: pd.Series, predictions: np.ndarray
) -> pd.DataFrame:
"""Every test row's true value, the model's prediction and its error, then the row's
features. The biggest errors come first."""
table = pd.DataFrame(
{y_test.name: y_test, "Prediction": predictions, "Error": predictions - y_test}
).join(X_test)
return table.sort_values("Error", key=abs, ascending=False, ignore_index=True)

The reports.py module in a macOS-style code window: a feature_importances function that builds a DataFrame of the forest's feature names and importances sorted largest first, and a prediction_errors function that builds a DataFrame of the true value, the prediction and the error, joins the test features and sorts by the error's absolute size, largest first.

feature_importances reads two attributes of the trained forest: feature_names_in_, the column names, and feature_importances_, one share per feature that adds up to 1. How to Get Feature Importances from a Random Forest in scikit-learn explains what the shares mean. It pairs them in a table, built from a dictionary, and sorts it largest first, renumbering the rows.

prediction_errors builds one row per test district: the true value, the prediction and the error (prediction minus true value), followed by the district's features, so a spreadsheet shows the important columns first. key=abs sorts by the error's size whatever its sign, so the worst misses, too high or too low, come first.

The function is named prediction_errors, not test_predictions, for a practical reason: pytest collects any function whose name starts with test_ as a test once a test file imports it.

Step 6: Save the Table, the Settings and the CSV with the Run​

Import the rest at the top of train.py:

src/house_price_prediction/train.py (imports)
import tempfile
from pathlib import Path

from house_price_prediction.reports import feature_importances, prediction_errors

Then, after Part 1's chart, save three files:

src/house_price_prediction/train.py (after log_figure)
mlflow.log_figure(figure, "plots/predicted_vs_actual.png")
mlflow.log_table(feature_importances(model), "evaluation/feature_importances.json")
mlflow.log_dict(model.get_params(), "model_params.json")
with tempfile.TemporaryDirectory() as folder:
csv_path = Path(folder) / "test_predictions.csv"
errors = prediction_errors(data.X_test, data.y_test, predictions)
errors.to_csv(csv_path, index=False)
mlflow.log_artifact(csv_path, artifact_path="evaluation")

Each uses a different logging function, chosen for what happens to the file next:

LineWhat it savesWhy this function
mlflow.log_table(...)The importance table, as evaluation/feature_importances.jsonMLflow marks it as a table: the UI shows it as rows and columns, compares it across runs, and mlflow.load_table() reads it back as a DataFrame
mlflow.log_dict(model.get_params(), ...)All 18 of the forest's settings, as model_params.jsonParameters hold the few settings you search by; this keeps every one, defaults included. How to See Every Setting of a scikit-learn Model with get_params
The with blockThe prediction errors, as evaluation/test_predictions.csvlog_artifact uploads a file that already exists, here a CSV anyone can open in a spreadsheet

The CSV needs a file on disk first. tempfile.TemporaryDirectory() creates a fresh folder that Python deletes, file and all, when the block ends, so nothing is left in your project: How to Create a Temporary Folder in Python with tempfile. Path(folder) / "test_predictions.csv" builds the file's path, and to_csv(..., index=False) writes the table without its row numbers. artifact_path="evaluation" puts the file in the run's evaluation folder, next to the table.

The complete file:

src/house_price_prediction/train.py
"""Train a house price model and record the run in MLflow."""

import argparse
import logging
import tempfile
from collections.abc import Sequence
from pathlib import Path

import mlflow
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, r2_score, root_mean_squared_error

from house_price_prediction.config import get_settings
from house_price_prediction.data import load_housing_data, split_data
from house_price_prediction.datasets import split_datasets
from house_price_prediction.plots import predicted_vs_actual
from house_price_prediction.reports import feature_importances, prediction_errors

logger = logging.getLogger("house_price_prediction")


def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
"""Read the training options from the command line."""
parser = argparse.ArgumentParser(
prog="train",
description="Train a model that predicts house prices, and track the run in MLflow.",
)
parser.add_argument(
"--n-estimators", type=int, default=100, help="number of trees in the forest (default: 100)"
)
parser.add_argument(
"--max-depth",
type=int,
default=None,
help="how deep each tree may grow (default: no limit)",
)
parser.add_argument(
"--trees-per-step",
type=int,
default=10,
help="trees added between two validation scores (default: 10)",
)
parser.add_argument(
"--test-size",
type=float,
default=0.2,
help="share of the rows kept back for testing (default: 0.2)",
)
parser.add_argument(
"--validation-size",
type=float,
default=0.2,
help="share of the remaining rows used to score the model while it grows (default: 0.2)",
)
parser.add_argument(
"--random-state",
type=int,
default=42,
help="seed that makes the split and the model repeatable (default: 42)",
)
parser.add_argument(
"--run-name", default=None, help="name shown in the MLflow UI (default: MLflow picks one)"
)
args = parser.parse_args(argv)
if args.n_estimators < 1:
parser.error("--n-estimators must be 1 or more")
if args.max_depth is not None and args.max_depth < 1:
parser.error("--max-depth must be 1 or more")
if args.trees_per_step < 1:
parser.error("--trees-per-step must be 1 or more")
if not 0 < args.test_size < 1:
parser.error("--test-size must be between 0 and 1")
if not 0 < args.validation_size < 1:
parser.error("--validation-size must be between 0 and 1")
return args


def tree_counts(n_estimators: int, trees_per_step: int) -> list[int]:
"""The forest's size after each step: 10, 20, ... up to `n_estimators`."""
counts = list(range(trees_per_step, n_estimators, trees_per_step))
return [*counts, n_estimators]


def main(argv: Sequence[str] | None = None) -> int:
"""Grow the forest step by step, score each step on the validation rows, then measure the
finished model on the test rows. Everything goes into one MLflow run."""
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
datefmt="%Y/%m/%d %H:%M:%S",
)
args = parse_args(argv)
settings = get_settings()

mlflow.set_tracking_uri(settings.mlflow_tracking_uri)
mlflow.set_experiment(settings.mlflow_experiment_name)
mlflow.set_system_metrics_sampling_interval(settings.system_metrics_interval)

logger.info("Loading the California Housing data")
data = split_data(load_housing_data(), args.test_size, args.validation_size, args.random_state)

with mlflow.start_run(
run_name=args.run_name, log_system_metrics=settings.log_system_metrics
) as run:
mlflow.log_params(
{
"n_estimators": args.n_estimators,
"max_depth": args.max_depth,
"trees_per_step": args.trees_per_step,
"test_size": args.test_size,
"validation_size": args.validation_size,
"random_state": args.random_state,
}
)
mlflow.set_tags({"model_type": "random_forest", "dataset": "california_housing"})
for context, dataset in split_datasets(data).items():
mlflow.log_input(dataset, context=context)

model = RandomForestRegressor(
max_depth=args.max_depth,
random_state=args.random_state,
n_jobs=-1,
warm_start=True,
)
logger.info(
"Training on %d rows, scoring on %d validation rows", len(data.X_train), len(data.X_val)
)
for n_trees in tree_counts(args.n_estimators, args.trees_per_step):
model.set_params(n_estimators=n_trees)
model.fit(data.X_train, data.y_train)
val_predictions = model.predict(data.X_val)
val_metrics = {
"val_mae": mean_absolute_error(data.y_val, val_predictions),
"val_rmse": root_mean_squared_error(data.y_val, val_predictions),
"val_r2": r2_score(data.y_val, val_predictions),
}
mlflow.log_metrics(val_metrics, step=n_trees)
logger.info("%4d trees: validation RMSE %.4f", n_trees, val_metrics["val_rmse"])

predictions = model.predict(data.X_test)
metrics = {
"mae": mean_absolute_error(data.y_test, predictions),
"rmse": root_mean_squared_error(data.y_test, predictions),
"r2": r2_score(data.y_test, predictions),
}
mlflow.log_metrics(metrics)
figure = predicted_vs_actual(data.y_test.to_numpy(), predictions)
mlflow.log_figure(figure, "plots/predicted_vs_actual.png")
mlflow.log_table(feature_importances(model), "evaluation/feature_importances.json")
mlflow.log_dict(model.get_params(), "model_params.json")
with tempfile.TemporaryDirectory() as folder:
csv_path = Path(folder) / "test_predictions.csv"
errors = prediction_errors(data.X_test, data.y_test, predictions)
errors.to_csv(csv_path, index=False)
mlflow.log_artifact(csv_path, artifact_path="evaluation")

logger.info(
"Run %s: MAE %.3f, RMSE %.3f, R2 %.3f",
run.info.run_name,
metrics["mae"],
metrics["rmse"],
metrics["r2"],
)
return 0

Step 7: Train Three Forests​

Three runs make the comparisons in the next steps meaningful: Part 2's default forest, a shallower one trained on the same data, and the default forest trained on a different split, from another --random-state:

Terminal 2
uv run train --run-name seed-42-100-trees
Output
2026/11/19 16:10:14 INFO house_price_prediction: Loading the California Housing data
2026/11/19 16:10:14 INFO mlflow.system_metrics.system_metrics_monitor: Skip logging GPU metrics. Set logger level to DEBUG for more details.
2026/11/19 16:10:14 INFO mlflow.system_metrics.system_metrics_monitor: Started monitoring system metrics.
2026/11/19 16:10:14 INFO house_price_prediction: Training on 13209 rows, scoring on 3303 validation rows
2026/11/19 16:10:14 INFO house_price_prediction: 10 trees: validation RMSE 0.5490
2026/11/19 16:10:14 INFO house_price_prediction: 20 trees: validation RMSE 0.5417
2026/11/19 16:10:14 INFO house_price_prediction: 30 trees: validation RMSE 0.5363
2026/11/19 16:10:14 INFO house_price_prediction: 40 trees: validation RMSE 0.5320
2026/11/19 16:10:15 INFO house_price_prediction: 50 trees: validation RMSE 0.5308
2026/11/19 16:10:15 INFO house_price_prediction: 60 trees: validation RMSE 0.5300
2026/11/19 16:10:15 INFO house_price_prediction: 70 trees: validation RMSE 0.5301
2026/11/19 16:10:15 INFO house_price_prediction: 80 trees: validation RMSE 0.5287
2026/11/19 16:10:15 INFO house_price_prediction: 90 trees: validation RMSE 0.5283
2026/11/19 16:10:15 INFO house_price_prediction: 100 trees: validation RMSE 0.5276
2026/11/19 16:10:15 INFO house_price_prediction: Run seed-42-100-trees: MAE 0.337, RMSE 0.516, R2 0.797
🏃 View run seed-42-100-trees at: http://127.0.0.1:8080/#/experiments/1/runs/175b697b4d2c4370b6d9f210d64e20be
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/1
2026/11/19 16:10:15 INFO mlflow.system_metrics.system_metrics_monitor: Stopping system metrics monitoring...
2026/11/19 16:10:15 INFO mlflow.system_metrics.system_metrics_monitor: Successfully terminated system metrics monitoring!

The same scores as Part 2's curve-100-trees, MAE 0.337, RMSE 0.516 and R² 0.797: same data, same seed, same forest. The new logging adds no lines to the output. Now the other two:

Terminal 2
uv run train --run-name seed-42-depth-8 --max-depth 8
uv run train --run-name seed-7-100-trees --random-state 7
Output
2026/11/19 16:10:18 INFO house_price_prediction: Loading the California Housing data
2026/11/19 16:10:18 INFO mlflow.system_metrics.system_metrics_monitor: Skip logging GPU metrics. Set logger level to DEBUG for more details.
2026/11/19 16:10:18 INFO mlflow.system_metrics.system_metrics_monitor: Started monitoring system metrics.
2026/11/19 16:10:18 INFO house_price_prediction: Training on 13209 rows, scoring on 3303 validation rows
2026/11/19 16:10:18 INFO house_price_prediction: 10 trees: validation RMSE 0.5939
2026/11/19 16:10:18 INFO house_price_prediction: 20 trees: validation RMSE 0.5922
2026/11/19 16:10:19 INFO house_price_prediction: 30 trees: validation RMSE 0.5888
2026/11/19 16:10:19 INFO house_price_prediction: 40 trees: validation RMSE 0.5875
2026/11/19 16:10:19 INFO house_price_prediction: 50 trees: validation RMSE 0.5875
2026/11/19 16:10:19 INFO house_price_prediction: 60 trees: validation RMSE 0.5867
2026/11/19 16:10:19 INFO house_price_prediction: 70 trees: validation RMSE 0.5872
2026/11/19 16:10:19 INFO house_price_prediction: 80 trees: validation RMSE 0.5862
2026/11/19 16:10:19 INFO house_price_prediction: 90 trees: validation RMSE 0.5861
2026/11/19 16:10:19 INFO house_price_prediction: 100 trees: validation RMSE 0.5857
2026/11/19 16:10:19 INFO house_price_prediction: Run seed-42-depth-8: MAE 0.403, RMSE 0.586, R2 0.738
🏃 View run seed-42-depth-8 at: http://127.0.0.1:8080/#/experiments/1/runs/1bc28a52be544b08a01bd133014a36cb
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/1
2026/11/19 16:10:19 INFO mlflow.system_metrics.system_metrics_monitor: Stopping system metrics monitoring...
2026/11/19 16:10:19 INFO mlflow.system_metrics.system_metrics_monitor: Successfully terminated system metrics monitoring!
2026/11/19 16:10:22 INFO house_price_prediction: Loading the California Housing data
2026/11/19 16:10:22 INFO mlflow.system_metrics.system_metrics_monitor: Skip logging GPU metrics. Set logger level to DEBUG for more details.
2026/11/19 16:10:22 INFO mlflow.system_metrics.system_metrics_monitor: Started monitoring system metrics.
2026/11/19 16:10:22 INFO house_price_prediction: Training on 13209 rows, scoring on 3303 validation rows
2026/11/19 16:10:22 INFO house_price_prediction: 10 trees: validation RMSE 0.5359
2026/11/19 16:10:22 INFO house_price_prediction: 20 trees: validation RMSE 0.5206
2026/11/19 16:10:23 INFO house_price_prediction: 30 trees: validation RMSE 0.5182
2026/11/19 16:10:23 INFO house_price_prediction: 40 trees: validation RMSE 0.5151
2026/11/19 16:10:23 INFO house_price_prediction: 50 trees: validation RMSE 0.5157
2026/11/19 16:10:23 INFO house_price_prediction: 60 trees: validation RMSE 0.5153
2026/11/19 16:10:23 INFO house_price_prediction: 70 trees: validation RMSE 0.5148
2026/11/19 16:10:23 INFO house_price_prediction: 80 trees: validation RMSE 0.5144
2026/11/19 16:10:23 INFO house_price_prediction: 90 trees: validation RMSE 0.5142
2026/11/19 16:10:23 INFO house_price_prediction: 100 trees: validation RMSE 0.5132
2026/11/19 16:10:23 INFO house_price_prediction: Run seed-7-100-trees: MAE 0.344, RMSE 0.526, R2 0.794
🏃 View run seed-7-100-trees at: http://127.0.0.1:8080/#/experiments/1/runs/71fd7f2ea29145f6adf4d43e99c46a9d
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/1
2026/11/19 16:10:23 INFO mlflow.system_metrics.system_metrics_monitor: Stopping system metrics monitoring...
2026/11/19 16:10:23 INFO mlflow.system_metrics.system_metrics_monitor: Successfully terminated system metrics monitoring!
  • The depth-8 forest scores worse, as in Part 2: test RMSE 0.586.
  • The seed-7 forest has a better validation score than the seed-42 one, 0.5132 against 0.5276, but a worse test score, 0.526 against 0.516. It wasn't scored on the same rows: --random-state 7 split the data differently. The datasets will show it.

Your times and run IDs will differ.

Step 8: See Which Data Each Run Used​

Open the house-price-prediction experiment in the MLflow UI at http://127.0.0.1:8080. The runs table has a Dataset column; with three datasets per run it's cut short, so turn on Expand rows above the table. Widen the column by dragging its right edge if the labels are cut off:

The runs table of the house-price-prediction experiment with Expand rows on: seed-7-100-trees lists california-housing-train (340131a0) Training, california-housing-validation (1b3920f4) Validation and california-housing-test (c0004229) Testing; seed-42-depth-8 and seed-42-100-trees both list california-housing-train (f3c40721), california-housing-validation (eba3edbe) and california-housing-test (81820aec); the older Part 1 and Part 2 runs show a dash.

  • The two seed-42 runs show the same three digests, f3c40721, eba3edbe and 81820aec: they learned from, were checked against and were tested on identical data. Their scores can be compared fairly, and the depth limit is the only difference.
  • The seed-7 run shows three different digests. Its better validation score came from different validation rows, so it says nothing about which forest is better.
  • Parts 1 and 2's runs show -: they logged no datasets.

The Datasets button in the toolbar lists every dataset by name and digest; tick one to show only the runs that used it. How to Check Whether Two MLflow Runs Used the Same Data does the same check in Python.

Click a dataset's name, such as california-housing-train (f3c40721) in the seed-42-100-trees row:

The Data details panel for seed-42-100-trees: 3 datasets used, the training, validation and test sets with their digests and contexts; the training set is selected and shows num rows 13209, num elements 118881, digest f3c40721, source type HTTP, the figshare URL, an Open dataset button, and a schema of nine double columns from MedInc to MedHouseVal.

The panel shows what the run recorded about its training data: 13,209 rows, 118,881 values (13,209 rows of 9 columns), the digest, the source with a link to it, and all nine columns with their type, double. Open dataset opens the source address in a new tab, which downloads the original archive. The run's own page lists the same datasets under About this run:

The About this run section of the seed-42-100-trees run page: created by you, experiment ID 1, status Finished, the run ID, a duration of 1.6s, source train, and under Datasets california-housing-train (f3c40721) with a Training label and a +2 button for the other two.

Click +2 for the validation and test sets.

Step 9: Open the Run's Files​

On the seed-42-100-trees run page, click Artifacts. Next to Part 1's plots folder are the new files: evaluation with the table and the CSV, and model_params.json. Expand evaluation and click feature_importances.json:

The Artifacts tab of seed-42-100-trees with evaluation/feature_importances.json selected, View as table ticked: MedInc 0.5277642886, AveOccup 0.1352437888, Latitude 0.0877231228, Longitude 0.0876805398, HouseAge 0.0568124492, AveRooms 0.0442092858, Population 0.030875788 and AveBedrms 0.0296907371.

Because log_table saved it, the preview has View as table. The median income (MedInc) accounts for 53% of the forest's error reduction, household size (AveOccup) for 14%, and the location for about 9% per coordinate.

Click model_params.json:

The Artifacts tab with model_params.json selected, 432 bytes: a JSON object of 18 settings, from bootstrap true, ccp_alpha 0 and criterion squared_error through max_depth null, n_estimators 100, n_jobs -1 and random_state 42 to warm_start true.

All 18 settings, including the 13 the code never sets, such as max_features and min_samples_leaf. The run's parameters still hold the six chosen on the command line. The file also shows warm_start: true and n_jobs: -1 from the code, and n_estimators: 100, the forest's final size after the last stage.

Click test_predictions.csv:

The Artifacts tab with evaluation/test_predictions.csv selected, 517.6KB, previewing the first 500 rows: MedHouseVal, Prediction, Error, MedInc and HouseAge columns, starting with true values of 5.00001 predicted as 1.64971 and 1.93147, errors of -3.3503 and -3.06854.

The CSV has all 4,128 test districts, and the preview shows the first 500, biggest misses first. Six of the ten biggest misses are districts whose true value is 5.00001, the highest value in the test rows, which the forest predicts far lower. That's the kind of pattern this file is for: open it in a spreadsheet, or with pandas, to see where the model fails.

Step 10: Compare the Files Across Runs​

Two places compare files between runs. For tables logged with log_table, the experiment's runs page has a third view: click the Artifact evaluation button next to the table and chart buttons. Table picks evaluation/feature_importances.json, Group by feature and Compare importance:

The Artifact evaluation view of the house-price-prediction experiment: the feature importance table compared across seed-7-100-trees, seed-42-depth-8 and seed-42-100-trees, each column headed by the run and its training dataset; MedInc reads 0.5316, 0.6432 and 0.5278, and AveOccup 0.1326, 0.1370 and 0.1352.

The shallow forest leans even harder on income, 64% against 53%, and less on the location. Only runs that logged the table get a column. How to Compare a Logged Table Across Runs in the MLflow UI covers the view's other controls.

For any other file, select runs in the runs table and click Compare. Near the bottom, the Artifacts section lists the files at the top level of the runs, model_params.json here; click it:

The Artifacts section of the compare page for seed-42-100-trees and seed-42-depth-8: model_params.json shown side by side, identical except max_depth, null on the left and 8 on the right.

The two seed-42 forests differ in exactly one setting, max_depth. The section lists only top-level files, which is one reason model_params.json sits outside the evaluation folder.

Step 11: Save Your Work with Git​

Terminal 2
git status --short --untracked-files=all
Output
M src/house_price_prediction/data.py
M src/house_price_prediction/train.py
?? src/house_price_prediction/datasets.py
?? src/house_price_prediction/reports.py

Two changed files and two new ones. The CSV was written to a temporary folder, so nothing else appears. Commit them:

Terminal 2
git add .
git commit -m "Record each run's datasets and evaluation files"
Output
[master c4224c2] Record each run's datasets and evaluation files
4 files changed, 62 insertions(+)
create mode 100644 src/house_price_prediction/datasets.py
create mode 100644 src/house_price_prediction/reports.py

Your commit ID will differ.

Your Folders Now Look Like This​

source/
source/
├── mlflow-server/
│ ├── .venv/
│ ├── mlartifacts/
│ ├── mlflow.db
│ ├── pyproject.toml
│ └── uv.lock
└── dewiride-house-price-prediction/
├── .venv/
├── src/
│ └── house_price_prediction/
│ ├── __init__.py
│ ├── config.py
│ ├── curve.py
│ ├── data.py
│ ├── datasets.py
│ ├── plots.py
│ ├── reports.py
│ └── train.py
├── .env
├── .env.example
├── .gitignore
├── .python-version
├── README.md
├── pyproject.toml
└── uv.lock

The files went to mlflow-server/mlartifacts/1/<run ID>/artifacts/, next to Part 1's chart. The dataset records went into mlflow.db, with no copy of the data.

Do It with the CLI​

The same files are a command away. The mlflow command reads the server's address from MLFLOW_TRACKING_URI, which your project's .env file already holds; --env-file .env loads it, the same way in every shell. List a run's files, using your own run ID:

Terminal 2
uv run mlflow --env-file .env artifacts list --run-id 175b697b4d2c4370b6d9f210d64e20be
Output
Loaded environment variables from: .env
[{
"path": "evaluation",
"is_dir": true
}, {
"path": "model_params.json",
"is_dir": false,
"file_size": 432
}, {
"path": "plots",
"is_dir": true
}]

One level at a time. Look inside evaluation:

Terminal 2
uv run mlflow --env-file .env artifacts list --run-id 175b697b4d2c4370b6d9f210d64e20be --artifact-path evaluation
Output
Loaded environment variables from: .env
[{
"path": "evaluation/feature_importances.json",
"is_dir": false,
"file_size": 254
}, {
"path": "evaluation/test_predictions.csv",
"is_dir": false,
"file_size": 530018
}]

And download the folder next to your project, so the files stay out of Git:

Terminal 2
uv run mlflow --env-file .env artifacts download --run-id 175b697b4d2c4370b6d9f210d64e20be --artifact-path evaluation --dst-path ../downloads
Output
Loaded environment variables from: .env

/Users/you/source/downloads/evaluation

The command prints where it put the folder. How to Download Artifacts from an MLflow Run shows the same from Python. Delete the downloads folder when you no longer need it.

Stop the Server When You're Done​

Go to the first terminal and press Ctrl+C. Your runs, datasets and files stay in mlflow.db and mlartifacts/, and uv run mlflow server --port 8080 in the mlflow-server folder brings them back.

Keep both folders for Part 4. If you ever want to remove everything the series created, stop the server first, then delete the mlflow-server and dewiride-house-price-prediction folders, and the scikit_learn_data folder in your home folder.

Common Mistakes​

MistakeWhat happensDo this instead
Comparing runs' scores without checking their datasetsRuns tested on different rows look better or worse for the wrong reason, like seed-7-100-trees hereCheck that the digests match first
Expecting the dataset record to contain the dataThe run holds a description: digest, schema, row count and sourceKeep the data where source points, or log the file itself as an artifact
Logging the importance table with log_artifactThe UI shows plain JSON, and the table can't be compared across runsUse log_table for tables
Writing the CSV into the project folderIt shows up in git status and gets committed by accidentWrite it into a temporary folder, as in Step 6
Putting a file you want to compare inside a folderThe compare page lists only top-level filesLog it at the top level, as model_params.json is

Troubleshooting​

SymptomLikely causeFix
The Dataset column shows - for new runslog_input wasn't reached, or runs were trained with the old codeCheck Step 4's lines and train again
MlflowException: The specified pandas DataFrame does not contain the specified targets column 'MedHouseVal'.The target wasn't joined to the featuresPass features.join(target) to from_pandas, as in Step 3
ValueError: Invalid artifact file path ... '.json' or '.parquet' extensionlog_table was given another extensionName the table .json
pytest prints Pandas4Warning: For backward compatibility, 'str' dtypes are included by select_dtypesMLflow's log_table with pandas 3, in testsFixed: Pandas4Warning in MLflow log_table
pytest reports ERROR for a function called test_predictionsA function whose name starts with test_ was imported into a test file, so pytest collected it as a testName it something else, such as prediction_errors

FAQ​

Why not store the data itself with every run?​

The training set is 13,209 rows, and real projects have far bigger ones; copying it into every run would fill the artifact store. The digest is enough to tell whether two runs used the same data, and the source says where to get it. When a dataset can't be fetched again, log the file as an artifact as well.

Does the digest catch every change to the data?​

For a pandas table, MLflow fingerprints the text and number columns of the first 10,000 rows, plus the row count and the column names. The training set has 13,209 rows, so a change in its last 3,209 rows that keeps the row count wouldn't change it. How to Make an MLflow Dataset Digest Cover Every Row shows how to fingerprint every row.

Can I read the feature importances of several runs in Python?​

Yes: mlflow.load_table("evaluation/feature_importances.json", extra_columns=["tags.mlflow.runName"]) stacks them from every run of the experiment, labelled by run name: How to Load a Logged Table from MLflow with load_table.

Why do the CSV's predictions have so many decimals?​

Each prediction is the average of 100 trees, written with every digit Python keeps. The values are exact; round them in your spreadsheet if you prefer.

Conclusion​

Every run now records its data and keeps its files. mlflow.data.from_pandas() and mlflow.log_input() attach the training, validation and test sets with their digests, so the UI shows at a glance that the two seed-42 runs used identical data and the seed-7 run didn't. log_table saved the feature importances as a table the UI compares across runs, log_dict kept all 18 of the forest's settings, and log_artifact uploaded a CSV of every test prediction, which showed that the forest misses most on the highest-valued districts.

In Part 4 you'll search runs by their parameters, metrics, tags and datasets, in the UI and from Python.

Additional Resources​


← Previous: Part 2: Log Metrics Over Time and System Metrics

Next → Part 4: Search and Filter Runs (coming soon)