Skip to main content

MLflow Tutorial for Machine Learning in Python — Part 2: Log Metrics Over Time and System Metrics

· 38 min read
Jagdish Kumawat
Founder @ Dewiride

Part 1 logged each score once, after training was over. That hides the question every model raises: was more training still helping? In this part the forest grows in stages, MLflow records its score after every stage, and the curves show where extra trees stop paying off. MLflow also records how hard the computer worked while it trained.

This is Part 2 of a 9-part series.

  1. Part 1: Install MLflow and Track Your First Experiment
  2. Part 2: Log Metrics Over Time and System Metrics (you are here)
  3. Part 3: Log Artifacts and Datasets (coming soon)
  4. Part 4: Autologging and Hyperparameter Tuning with Nested Runs (coming soon)
  5. Part 5: Log, Load and Validate Models (coming soon)
  6. Part 6: Evaluate Models and Set Quality Gates (coming soon)
  7. Part 7: Register, Version and Promote Models with the Model Registry (coming soon)
  8. Part 8: Serve a Model as a REST API and in Docker (coming soon)
  9. Part 9: Production Checklist and Recap (coming soon)

Every command and output on this page is real, captured while building the project.

What You'll Build in This Part​

You'll keep working on Part 1's house price project and add:

  • a validation set, a third slice of the data, to score the model while it trains without touching the test rows;
  • a forest that grows in stages, 10 trees at a time, with three validation scores logged after each stage at a step equal to the number of trees so far;
  • learning curves in the MLflow UI, for one run and for three runs side by side;
  • a curve command that reads a run's curve back from MLflow and names its best step;
  • system metrics: the computer's CPU, memory, disk and network use, sampled every second during each run.

What This Part Costs​

Nothing. Everything still runs on your own computer, and the one new package, psutil, is free and open source.

Prerequisites​

  • Part 1 finished: the mlflow-server folder with its runs, and the dewiride-house-price-prediction project, committed to Git. Part 1 builds both.
  • The same tools as Part 1: Python 3.15 and uv 0.13.9 or newer.

Each technique this part uses also has a short guide of its own, linked where it comes up.

Step 1: Start the Tracking Server​

Open a terminal in the mlflow-server folder and start the server, as in Part 1:

Terminal 1 (in mlflow-server)
uv run mlflow server --port 8080

Wait until it prints Application startup complete, and leave it running. Open a second terminal in the dewiride-house-price-prediction folder for everything else.

Step 2: Split Off a Validation Set​

Part 1 split the rows in two: 80% to train on and 20% to test. To watch the score while the forest grows, you need rows the model doesn't learn from, and the test rows can't do that job: once you choose the number of trees by looking at the test score, that score is no longer a fair check on data the model has never seen. So this part takes a third slice out of the training rows, the validation set, and keeps the test rows for one final score at the end. How to Split Data into Training, Validation and Test Sets with scikit-learn explains the idea on its own.

Open src/house_price_prediction/data.py. Give DataSplit two more fields, for the validation rows and their prices:

src/house_price_prediction/data.py (the DataSplit class)
@dataclass(frozen=True)
class DataSplit:
"""The features and target values for training, validation and testing."""

X_train: pd.DataFrame
X_val: pd.DataFrame
X_test: pd.DataFrame
y_train: pd.Series
y_val: pd.Series
y_test: pd.Series

Split twice. Replace split_data with:

src/house_price_prediction/data.py (replace split_data)
def split_data(
frame: pd.DataFrame, test_size: float, validation_size: float, random_state: int
) -> DataSplit:
"""Hold back `test_size` of the rows for the final test, then `validation_size` of the rest
to watch the model while it trains. The model learns from what is left."""
features = frame.drop(columns=TARGET)
target = frame[TARGET]
X_rest, X_test, y_rest, y_test = train_test_split(
features, target, test_size=test_size, random_state=random_state
)
X_train, X_val, y_train, y_val = train_test_split(
X_rest, y_rest, test_size=validation_size, random_state=random_state
)
return DataSplit(X_train, X_val, X_test, y_train, y_val, y_test)
  • The first split is Part 1's, with the same test_size and random_state, so the test set is the same 4,128 rows as before.
  • The second split takes validation_size of the remaining 16,512 rows for validation: with 0.2, that's 3,303 rows, which leaves 13,209 to train on.

The complete file, with its first line updated too:

src/house_price_prediction/data.py
"""Load the California Housing data and split it for training, validation and testing."""

from dataclasses import dataclass

import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

TARGET = "MedHouseVal"


@dataclass(frozen=True)
class DataSplit:
"""The features and target values for training, validation and testing."""

X_train: pd.DataFrame
X_val: pd.DataFrame
X_test: pd.DataFrame
y_train: pd.Series
y_val: pd.Series
y_test: pd.Series


def load_housing_data() -> pd.DataFrame:
"""Return the dataset as one table. The first call downloads it; later calls read a cache."""
return fetch_california_housing(as_frame=True).frame


def split_data(
frame: pd.DataFrame, test_size: float, validation_size: float, random_state: int
) -> DataSplit:
"""Hold back `test_size` of the rows for the final test, then `validation_size` of the rest
to watch the model while it trains. The model learns from what is left."""
features = frame.drop(columns=TARGET)
target = frame[TARGET]
X_rest, X_test, y_rest, y_test = train_test_split(
features, target, test_size=test_size, random_state=random_state
)
X_train, X_val, y_train, y_val = train_test_split(
X_rest, y_rest, test_size=validation_size, random_state=random_state
)
return DataSplit(X_train, X_val, X_test, y_train, y_val, y_test)

Step 3: Add the System Metrics Settings​

System metrics are MLflow's record of the computer itself while a run works: how busy the processor is, how much memory is in use, and how much the disk and network move. They show whether training is starved for memory or using every core. MLflow samples them every 10 seconds by default, but each training run here takes a few seconds, and a run that ends before its first sample is logged records nothing. So the project samples every second, and both choices become settings you can change without touching the code. How to Turn On System Metrics Logging in MLflow and How to Change the System Metrics Sampling Interval in MLflow cover both settings on their own.

Open src/house_price_prediction/config.py and add two settings at the end of the Settings class:

src/house_price_prediction/config.py (add to the Settings class)
log_system_metrics: bool = Field(
default=True,
description="Record the computer's CPU, memory, disk and network use during each run.",
)
system_metrics_interval: float = Field(
default=1.0,
ge=0.1,
description="Seconds between two system metrics samples. A run records nothing unless it "
"lasts longer than one interval, so keep this below your shortest run.",
)
  • pydantic-settings reads them from the environment variables HOUSE_PRICES_LOG_SYSTEM_METRICS and HOUSE_PRICES_SYSTEM_METRICS_INTERVAL: the class's env_prefix, HOUSE_PRICES_, plus the field's name. How to Load Settings from Environment Variables and a .env File with pydantic-settings explains the prefix.
  • ge=0.1 refuses an interval shorter than 0.1 seconds. psutil, the package MLflow reads CPU use with, recommends at least that much time between two readings for accuracy.

The complete file:

src/house_price_prediction/config.py
"""Settings, read from environment variables and the .env file."""

from functools import lru_cache

from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
"""Where the project sends its MLflow runs, and what each run records about the computer."""

model_config = SettingsConfigDict(
env_prefix="HOUSE_PRICES_",
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
)

mlflow_tracking_uri: str = Field(
default="http://127.0.0.1:8080",
validation_alias="MLFLOW_TRACKING_URI",
description="The address of the MLflow tracking server.",
)
mlflow_experiment_name: str = Field(
default="house-price-prediction",
min_length=1,
validation_alias="MLFLOW_EXPERIMENT_NAME",
description="The MLflow experiment that groups this project's runs.",
)
log_system_metrics: bool = Field(
default=True,
description="Record the computer's CPU, memory, disk and network use during each run.",
)
system_metrics_interval: float = Field(
default=1.0,
ge=0.1,
description="Seconds between two system metrics samples. A run records nothing unless it "
"lasts longer than one interval, so keep this below your shortest run.",
)


@lru_cache
def get_settings() -> Settings:
"""Read the settings once, then reuse the same object everywhere."""
return Settings()

Then document the new settings for whoever uses the project next. Add these lines at the end of .env.example:

dewiride-house-price-prediction/.env.example (add at the end)
# Record the computer's CPU, memory, disk and network use during each run (true or false).
HOUSE_PRICES_LOG_SYSTEM_METRICS=true
# Seconds between two system metrics samples (0.1 or more). A run records none unless it lasts
# longer than one interval, so keep this below your shortest training run.
HOUSE_PRICES_SYSTEM_METRICS_INTERVAL=1

Your own .env needs no change: a setting that's missing there takes its default from config.py, and the defaults are the same values.

Step 4: Grow the Forest in Stages and Log a Score at Every Step​

This is the heart of the part. A random forest averages the answers of many decision trees. Part 1 grew all its trees in one go and scored the finished forest once. Now the forest grows 10 trees, gets scored on the validation rows, grows 10 more, gets scored again, and so on. Two settings make that work:

  • warm_start=True tells scikit-learn to keep the trees it already has when you raise n_estimators and call fit again, so each stage adds trees instead of starting over: How to Use warm_start to Add Trees to a Random Forest in scikit-learn.
  • step= in mlflow.log_metrics() records each score together with a whole number you choose. A metric logged at several steps becomes a curve: MLflow keeps every value with its step, and the UI draws them as a line. Here the step is the number of trees, so the curve reads in trees. How to Log Metrics in MLflow introduced step.

Open src/house_price_prediction/train.py and change it piece by piece.

Two new options. In parse_args, add --trees-per-step below --max-depth, and --validation-size below --test-size:

src/house_price_prediction/train.py (add below --max-depth)
parser.add_argument(
"--trees-per-step",
type=int,
default=10,
help="trees added between two validation scores (default: 10)",
)
src/house_price_prediction/train.py (add below --test-size)
parser.add_argument(
"--validation-size",
type=float,
default=0.2,
help="share of the remaining rows used to score the model while it grows (default: 0.2)",
)

Check both. The checks at the end of parse_args become:

src/house_price_prediction/train.py (the checks at the end of parse_args)
args = parser.parse_args(argv)
if args.n_estimators < 1:
parser.error("--n-estimators must be 1 or more")
if args.max_depth is not None and args.max_depth < 1:
parser.error("--max-depth must be 1 or more")
if args.trees_per_step < 1:
parser.error("--trees-per-step must be 1 or more")
if not 0 < args.test_size < 1:
parser.error("--test-size must be between 0 and 1")
if not 0 < args.validation_size < 1:
parser.error("--validation-size must be between 0 and 1")
return args

How to Add Command-Line Options to a Python Script with argparse explains options, types and parser.error().

The forest's size after each stage. Add this function below parse_args:

src/house_price_prediction/train.py (add below parse_args)
def tree_counts(n_estimators: int, trees_per_step: int) -> list[int]:
"""The forest's size after each step: 10, 20, ... up to `n_estimators`."""
counts = list(range(trees_per_step, n_estimators, trees_per_step))
return [*counts, n_estimators]

For 100 trees, 10 at a time, it returns [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]. When n_estimators isn't a multiple of the step, the last stage is the full forest, so no tree is left out.

System metrics, set up before the run. In main(), add one line below mlflow.set_experiment(...), pass the validation share to split_data, and turn system metrics on in start_run:

src/house_price_prediction/train.py (in main)
mlflow.set_tracking_uri(settings.mlflow_tracking_uri)
mlflow.set_experiment(settings.mlflow_experiment_name)
mlflow.set_system_metrics_sampling_interval(settings.system_metrics_interval)

logger.info("Loading the California Housing data")
data = split_data(load_housing_data(), args.test_size, args.validation_size, args.random_state)

with mlflow.start_run(
run_name=args.run_name, log_system_metrics=settings.log_system_metrics
) as run:
  • mlflow.set_system_metrics_sampling_interval() must come before start_run, because each run's monitor reads the interval when the run starts.
  • log_system_metrics=True starts that monitor for this run, and stops it when the run ends.

Two more params. Add trees_per_step and validation_size to mlflow.log_params(), so every run records them:

src/house_price_prediction/train.py (the log_params call)
mlflow.log_params(
{
"n_estimators": args.n_estimators,
"max_depth": args.max_depth,
"trees_per_step": args.trees_per_step,
"test_size": args.test_size,
"validation_size": args.validation_size,
"random_state": args.random_state,
}
)

The training loop. Replace the lines from model = RandomForestRegressor( down to predictions = model.predict(data.X_test) with:

src/house_price_prediction/train.py (replace the model and the single fit)
model = RandomForestRegressor(
max_depth=args.max_depth,
random_state=args.random_state,
n_jobs=-1,
warm_start=True,
)
logger.info(
"Training on %d rows, scoring on %d validation rows", len(data.X_train), len(data.X_val)
)
for n_trees in tree_counts(args.n_estimators, args.trees_per_step):
model.set_params(n_estimators=n_trees)
model.fit(data.X_train, data.y_train)
val_predictions = model.predict(data.X_val)
val_metrics = {
"val_mae": mean_absolute_error(data.y_val, val_predictions),
"val_rmse": root_mean_squared_error(data.y_val, val_predictions),
"val_r2": r2_score(data.y_val, val_predictions),
}
mlflow.log_metrics(val_metrics, step=n_trees)
logger.info("%4d trees: validation RMSE %.4f", n_trees, val_metrics["val_rmse"])

predictions = model.predict(data.X_test)
  • The forest is created without n_estimators. The loop sets it before each fit: 10, then 20, and so on.
  • warm_start=True makes each fit grow only the new trees.
  • The validation scores are named val_mae, val_rmse and val_r2, so they never mix with the test scores, mae, rmse and r2, which are logged once at the end exactly as in Part 1.
  • step=n_trees files each score under the forest's size. Runs that add 10 or 20 trees at a time still line up on the same axis.

The complete file:

The complete train.py in a macOS-style code window: the command-line options including --trees-per-step and --validation-size, the tree_counts function, and main(), which sets the system metrics sampling interval, starts a run with system metrics on, grows a warm-started random forest in stages and logs val_mae, val_rmse and val_r2 at each step, then logs the test metrics and the chart.

src/house_price_prediction/train.py
"""Train a house price model and record the run in MLflow."""

import argparse
import logging
from collections.abc import Sequence

import mlflow
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, r2_score, root_mean_squared_error

from house_price_prediction.config import get_settings
from house_price_prediction.data import load_housing_data, split_data
from house_price_prediction.plots import predicted_vs_actual

logger = logging.getLogger("house_price_prediction")


def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
"""Read the training options from the command line."""
parser = argparse.ArgumentParser(
prog="train",
description="Train a model that predicts house prices, and track the run in MLflow.",
)
parser.add_argument(
"--n-estimators", type=int, default=100, help="number of trees in the forest (default: 100)"
)
parser.add_argument(
"--max-depth",
type=int,
default=None,
help="how deep each tree may grow (default: no limit)",
)
parser.add_argument(
"--trees-per-step",
type=int,
default=10,
help="trees added between two validation scores (default: 10)",
)
parser.add_argument(
"--test-size",
type=float,
default=0.2,
help="share of the rows kept back for testing (default: 0.2)",
)
parser.add_argument(
"--validation-size",
type=float,
default=0.2,
help="share of the remaining rows used to score the model while it grows (default: 0.2)",
)
parser.add_argument(
"--random-state",
type=int,
default=42,
help="seed that makes the split and the model repeatable (default: 42)",
)
parser.add_argument(
"--run-name", default=None, help="name shown in the MLflow UI (default: MLflow picks one)"
)
args = parser.parse_args(argv)
if args.n_estimators < 1:
parser.error("--n-estimators must be 1 or more")
if args.max_depth is not None and args.max_depth < 1:
parser.error("--max-depth must be 1 or more")
if args.trees_per_step < 1:
parser.error("--trees-per-step must be 1 or more")
if not 0 < args.test_size < 1:
parser.error("--test-size must be between 0 and 1")
if not 0 < args.validation_size < 1:
parser.error("--validation-size must be between 0 and 1")
return args


def tree_counts(n_estimators: int, trees_per_step: int) -> list[int]:
"""The forest's size after each step: 10, 20, ... up to `n_estimators`."""
counts = list(range(trees_per_step, n_estimators, trees_per_step))
return [*counts, n_estimators]


def main(argv: Sequence[str] | None = None) -> int:
"""Grow the forest step by step, score each step on the validation rows, then measure the
finished model on the test rows. Everything goes into one MLflow run."""
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
datefmt="%Y/%m/%d %H:%M:%S",
)
args = parse_args(argv)
settings = get_settings()

mlflow.set_tracking_uri(settings.mlflow_tracking_uri)
mlflow.set_experiment(settings.mlflow_experiment_name)
mlflow.set_system_metrics_sampling_interval(settings.system_metrics_interval)

logger.info("Loading the California Housing data")
data = split_data(load_housing_data(), args.test_size, args.validation_size, args.random_state)

with mlflow.start_run(
run_name=args.run_name, log_system_metrics=settings.log_system_metrics
) as run:
mlflow.log_params(
{
"n_estimators": args.n_estimators,
"max_depth": args.max_depth,
"trees_per_step": args.trees_per_step,
"test_size": args.test_size,
"validation_size": args.validation_size,
"random_state": args.random_state,
}
)
mlflow.set_tags({"model_type": "random_forest", "dataset": "california_housing"})

model = RandomForestRegressor(
max_depth=args.max_depth,
random_state=args.random_state,
n_jobs=-1,
warm_start=True,
)
logger.info(
"Training on %d rows, scoring on %d validation rows", len(data.X_train), len(data.X_val)
)
for n_trees in tree_counts(args.n_estimators, args.trees_per_step):
model.set_params(n_estimators=n_trees)
model.fit(data.X_train, data.y_train)
val_predictions = model.predict(data.X_val)
val_metrics = {
"val_mae": mean_absolute_error(data.y_val, val_predictions),
"val_rmse": root_mean_squared_error(data.y_val, val_predictions),
"val_r2": r2_score(data.y_val, val_predictions),
}
mlflow.log_metrics(val_metrics, step=n_trees)
logger.info("%4d trees: validation RMSE %.4f", n_trees, val_metrics["val_rmse"])

predictions = model.predict(data.X_test)
metrics = {
"mae": mean_absolute_error(data.y_test, predictions),
"rmse": root_mean_squared_error(data.y_test, predictions),
"r2": r2_score(data.y_test, predictions),
}
mlflow.log_metrics(metrics)
figure = predicted_vs_actual(data.y_test.to_numpy(), predictions)
mlflow.log_figure(figure, "plots/predicted_vs_actual.png")

logger.info(
"Run %s: MAE %.3f, RMSE %.3f, R2 %.3f",
run.info.run_name,
metrics["mae"],
metrics["rmse"],
metrics["r2"],
)
return 0

Step 5: Add a curve Command​

The MLflow UI will draw the curves, but a curve is also data you'll want in code: for a report, a test, or to pick the best number of trees. Create src/house_price_prediction/curve.py and build it in two pieces.

The options: the run's name, and which validation score to show:

src/house_price_prediction/curve.py
"""Show how a run's validation score changed as its forest grew, read back from MLflow."""

import argparse
from collections.abc import Sequence

from mlflow import MlflowClient

from house_price_prediction.config import get_settings

LOWER_IS_BETTER = {"val_mae", "val_rmse"}


def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
"""Read the run's name and the score to show from the command line."""
parser = argparse.ArgumentParser(
prog="curve",
description="Print a run's validation score after each training step, and the best step.",
)
parser.add_argument("run_name", help="the run's name, as shown in the MLflow UI")
parser.add_argument(
"--metric",
default="val_rmse",
choices=["val_rmse", "val_mae", "val_r2"],
help="the score to show (default: val_rmse)",
)
return parser.parse_args(argv)

choices makes argparse refuse anything but the three validation scores.

Find the run and read its history. Add below parse_args:

src/house_price_prediction/curve.py (add below parse_args)
def main(argv: Sequence[str] | None = None) -> int:
"""Find the newest run with the given name and print its metric history, step by step."""
args = parse_args(argv)
settings = get_settings()
client = MlflowClient(tracking_uri=settings.mlflow_tracking_uri)

experiment = client.get_experiment_by_name(settings.mlflow_experiment_name)
if experiment is None:
print(f"No experiment called {settings.mlflow_experiment_name!r}")
return 1
runs = client.search_runs([experiment.experiment_id], order_by=["attributes.start_time DESC"])
run = next((r for r in runs if r.info.run_name == args.run_name), None)
if run is None:
print(f"No run called {args.run_name!r} in {experiment.name!r}")
return 1

history = sorted(client.get_metric_history(run.info.run_id, args.metric), key=lambda m: m.step)
if not history:
print(f"Run {args.run_name!r} has no {args.metric!r} history")
return 1
for point in history:
print(f"{point.step:5d} trees {args.metric} {point.value:.4f}")
pick = min if args.metric in LOWER_IS_BETTER else max
best = pick(history, key=lambda m: m.value)
print(f"Best {args.metric}: {best.value:.4f} at {best.step} trees")
return 0
  • MlflowClient is MLflow's lower-level client. It talks to the same tracking server as the training code, read from the same settings.
  • The runs come newest first, and the first one with the requested name wins. MLflow doesn't require run names to be unique, so a name you reuse finds your latest run.
  • get_metric_history() returns every value the metric was logged with, each with its step: How to Get the History of a Metric in MLflow. The run's summary, by contrast, keeps only the last value.
  • The best step is the lowest value for the two errors and the highest for R².
  • It returns 1 when the experiment, the run or the history is missing, so a script that calls it can tell that it failed.

The complete file is the two pieces together:

src/house_price_prediction/curve.py
"""Show how a run's validation score changed as its forest grew, read back from MLflow."""

import argparse
from collections.abc import Sequence

from mlflow import MlflowClient

from house_price_prediction.config import get_settings

LOWER_IS_BETTER = {"val_mae", "val_rmse"}


def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
"""Read the run's name and the score to show from the command line."""
parser = argparse.ArgumentParser(
prog="curve",
description="Print a run's validation score after each training step, and the best step.",
)
parser.add_argument("run_name", help="the run's name, as shown in the MLflow UI")
parser.add_argument(
"--metric",
default="val_rmse",
choices=["val_rmse", "val_mae", "val_r2"],
help="the score to show (default: val_rmse)",
)
return parser.parse_args(argv)


def main(argv: Sequence[str] | None = None) -> int:
"""Find the newest run with the given name and print its metric history, step by step."""
args = parse_args(argv)
settings = get_settings()
client = MlflowClient(tracking_uri=settings.mlflow_tracking_uri)

experiment = client.get_experiment_by_name(settings.mlflow_experiment_name)
if experiment is None:
print(f"No experiment called {settings.mlflow_experiment_name!r}")
return 1
runs = client.search_runs([experiment.experiment_id], order_by=["attributes.start_time DESC"])
run = next((r for r in runs if r.info.run_name == args.run_name), None)
if run is None:
print(f"No run called {args.run_name!r} in {experiment.name!r}")
return 1

history = sorted(client.get_metric_history(run.info.run_id, args.metric), key=lambda m: m.step)
if not history:
print(f"Run {args.run_name!r} has no {args.metric!r} history")
return 1
for point in history:
print(f"{point.step:5d} trees {args.metric} {point.value:.4f}")
pick = min if args.metric in LOWER_IS_BETTER else max
best = pick(history, key=lambda m: m.value)
print(f"Best {args.metric}: {best.value:.4f} at {best.step} trees")
return 0

Register the command. In pyproject.toml, add a curve line to [project.scripts], below train:

dewiride-house-price-prediction/pyproject.toml (the scripts section)
[project.scripts]
train = "house_price_prediction.train:main"
curve = "house_price_prediction.curve:main"

Step 6: Add psutil​

MLflow reads the computer's CPU and memory use through psutil, but doesn't install it for you. Its documentation says: "We explicitly don't include psutil in MLflow's dependencies". Add it to the project:

Terminal 2
uv add psutil
Output
Resolved 93 packages in 2.11s
Built house-price-prediction @ file:///Users/you/source/dewiride-house-price-prediction
Prepared 1 package in 7ms
Uninstalled 1 package in 1ms
Installed 2 packages in 2ms
~ house-price-prediction==0.1.0 (from file:///Users/you/source/dewiride-house-price-prediction)
+ psutil==7.2.2
  • + psutil==7.2.2: the new package, now listed in pyproject.toml and locked in uv.lock.
  • ~ house-price-prediction: uv rebuilt your own project, because pyproject.toml changed. That's what makes the new curve command available.

Without psutil, a run with system metrics on doesn't even start: How to Check CPU and Memory Usage in Python with psutil introduces the package itself.

Step 7: Train Three Forests​

Train the default forest first, 100 trees in steps of 10:

Terminal 2
uv run train --run-name curve-100-trees
Output
2026/11/18 10:48:29 INFO house_price_prediction: Loading the California Housing data
2026/11/18 10:48:30 INFO mlflow.system_metrics.system_metrics_monitor: Skip logging GPU metrics. Set logger level to DEBUG for more details.
2026/11/18 10:48:30 INFO mlflow.system_metrics.system_metrics_monitor: Started monitoring system metrics.
2026/11/18 10:48:30 INFO house_price_prediction: Training on 13209 rows, scoring on 3303 validation rows
2026/11/18 10:48:30 INFO house_price_prediction: 10 trees: validation RMSE 0.5490
2026/11/18 10:48:30 INFO house_price_prediction: 20 trees: validation RMSE 0.5417
2026/11/18 10:48:30 INFO house_price_prediction: 30 trees: validation RMSE 0.5363
2026/11/18 10:48:30 INFO house_price_prediction: 40 trees: validation RMSE 0.5320
2026/11/18 10:48:30 INFO house_price_prediction: 50 trees: validation RMSE 0.5308
2026/11/18 10:48:30 INFO house_price_prediction: 60 trees: validation RMSE 0.5300
2026/11/18 10:48:30 INFO house_price_prediction: 70 trees: validation RMSE 0.5301
2026/11/18 10:48:31 INFO house_price_prediction: 80 trees: validation RMSE 0.5287
2026/11/18 10:48:31 INFO house_price_prediction: 90 trees: validation RMSE 0.5283
2026/11/18 10:48:31 INFO house_price_prediction: 100 trees: validation RMSE 0.5276
2026/11/18 10:48:31 INFO house_price_prediction: Run curve-100-trees: MAE 0.337, RMSE 0.516, R2 0.797
🏃 View run curve-100-trees at: http://127.0.0.1:8080/#/experiments/1/runs/3f7155effd274ec884c35e62025969a8
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/1
2026/11/18 10:48:31 INFO mlflow.system_metrics.system_metrics_monitor: Stopping system metrics monitoring...
2026/11/18 10:48:31 INFO mlflow.system_metrics.system_metrics_monitor: Successfully terminated system metrics monitoring!

Line by line:

  1. Skip logging GPU metrics: MLflow couldn't start a monitor for an NVIDIA or AMD graphics card. That's expected on a Mac; elsewhere, GPU monitoring needs extra packages.
  2. Started monitoring system metrics: the run's monitor is sampling the computer every second.
  3. Training on 13209 rows, scoring on 3303 validation rows: the three-way split from Step 2.
  4. One line per stage. The validation error falls fast at first, from 0.5490 at 10 trees to 0.5308 at 50, then barely moves: 0.5276 at 100.
  5. The test scores: MAE 0.337, RMSE 0.516, R² 0.797. Part 1's 100-tree forest scored 0.328, 0.506 and 0.804 on the same test rows. The difference is the data it learned from: 13,209 rows instead of 16,512, because the validation rows came out of the training rows. That's the price of a validation set.
  6. Stopping system metrics monitoring: the run ended, and its monitor with it.

Now two more forests: 300 trees, 20 at a time, and a forest whose trees may only grow 8 levels deep:

Terminal 2
uv run train --n-estimators 300 --trees-per-step 20 --run-name curve-300-trees
uv run train --max-depth 8 --run-name curve-depth-8
Output
2026/11/18 10:49:48 INFO house_price_prediction: Loading the California Housing data
2026/11/18 10:49:48 INFO mlflow.system_metrics.system_metrics_monitor: Skip logging GPU metrics. Set logger level to DEBUG for more details.
2026/11/18 10:49:48 INFO mlflow.system_metrics.system_metrics_monitor: Started monitoring system metrics.
2026/11/18 10:49:48 INFO house_price_prediction: Training on 13209 rows, scoring on 3303 validation rows
2026/11/18 10:49:49 INFO house_price_prediction: 20 trees: validation RMSE 0.5417
2026/11/18 10:49:49 INFO house_price_prediction: 40 trees: validation RMSE 0.5320
2026/11/18 10:49:49 INFO house_price_prediction: 60 trees: validation RMSE 0.5300
2026/11/18 10:49:49 INFO house_price_prediction: 80 trees: validation RMSE 0.5287
2026/11/18 10:49:49 INFO house_price_prediction: 100 trees: validation RMSE 0.5276
2026/11/18 10:49:50 INFO house_price_prediction: 120 trees: validation RMSE 0.5266
2026/11/18 10:49:50 INFO house_price_prediction: 140 trees: validation RMSE 0.5261
2026/11/18 10:49:50 INFO house_price_prediction: 160 trees: validation RMSE 0.5254
2026/11/18 10:49:50 INFO house_price_prediction: 180 trees: validation RMSE 0.5254
2026/11/18 10:49:50 INFO house_price_prediction: 200 trees: validation RMSE 0.5254
2026/11/18 10:49:51 INFO house_price_prediction: 220 trees: validation RMSE 0.5250
2026/11/18 10:49:51 INFO house_price_prediction: 240 trees: validation RMSE 0.5253
2026/11/18 10:49:51 INFO house_price_prediction: 260 trees: validation RMSE 0.5248
2026/11/18 10:49:51 INFO house_price_prediction: 280 trees: validation RMSE 0.5244
2026/11/18 10:49:52 INFO house_price_prediction: 300 trees: validation RMSE 0.5243
2026/11/18 10:49:52 INFO house_price_prediction: Run curve-300-trees: MAE 0.334, RMSE 0.512, R2 0.800
🏃 View run curve-300-trees at: http://127.0.0.1:8080/#/experiments/1/runs/66d6ec9ec6ad4f10bb337e57549c3d4f
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/1
2026/11/18 10:49:52 INFO mlflow.system_metrics.system_metrics_monitor: Stopping system metrics monitoring...
2026/11/18 10:49:52 INFO mlflow.system_metrics.system_metrics_monitor: Successfully terminated system metrics monitoring!
2026/11/18 10:49:56 INFO house_price_prediction: Loading the California Housing data
2026/11/18 10:49:56 INFO mlflow.system_metrics.system_metrics_monitor: Skip logging GPU metrics. Set logger level to DEBUG for more details.
2026/11/18 10:49:56 INFO mlflow.system_metrics.system_metrics_monitor: Started monitoring system metrics.
2026/11/18 10:49:56 INFO house_price_prediction: Training on 13209 rows, scoring on 3303 validation rows
2026/11/18 10:49:56 INFO house_price_prediction: 10 trees: validation RMSE 0.5939
2026/11/18 10:49:56 INFO house_price_prediction: 20 trees: validation RMSE 0.5922
2026/11/18 10:49:56 INFO house_price_prediction: 30 trees: validation RMSE 0.5888
2026/11/18 10:49:56 INFO house_price_prediction: 40 trees: validation RMSE 0.5875
2026/11/18 10:49:56 INFO house_price_prediction: 50 trees: validation RMSE 0.5875
2026/11/18 10:49:56 INFO house_price_prediction: 60 trees: validation RMSE 0.5867
2026/11/18 10:49:56 INFO house_price_prediction: 70 trees: validation RMSE 0.5872
2026/11/18 10:49:56 INFO house_price_prediction: 80 trees: validation RMSE 0.5862
2026/11/18 10:49:57 INFO house_price_prediction: 90 trees: validation RMSE 0.5861
2026/11/18 10:49:57 INFO house_price_prediction: 100 trees: validation RMSE 0.5857
2026/11/18 10:49:57 INFO house_price_prediction: Run curve-depth-8: MAE 0.403, RMSE 0.586, R2 0.738
🏃 View run curve-depth-8 at: http://127.0.0.1:8080/#/experiments/1/runs/60a6a374e9724a86bd4a3b401de03142
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/1
2026/11/18 10:49:57 INFO mlflow.system_metrics.system_metrics_monitor: Stopping system metrics monitoring...
2026/11/18 10:49:57 INFO mlflow.system_metrics.system_metrics_monitor: Successfully terminated system metrics monitoring!
  • The 300-tree forest matches the 100-tree one stage for stage, 0.5417 at 20 trees and 0.5276 at 100, because both start from the same random_state and grow the same trees in the same order. The next 200 trees take the error from 0.5276 to 0.5243.
  • The depth-8 forest is worse at every stage, and more trees hardly help: 0.5939 at 10 trees, 0.5857 at 100. Shallow trees can't capture enough detail, however many there are.

Your times and run IDs will differ.

Step 8: See One Run's Curves​

Open the curve-300-trees link from the output and click Model metrics:

The Model metrics tab of the curve-300-trees run in MLflow 3.18.0: bar charts for mae at 0.33, r2 at 0.80 and rmse at 0.51, and line charts for val_mae and val_rmse falling steeply over the first 60 trees and then flattening, and val_r2 rising the same way, from step 20 to 300.

  • The three val_ metrics are lines, with Step, the number of trees, along the bottom. MLflow drew a line because each was logged at 15 steps.
  • mae, r2 and rmse are bars. They were logged once, so there's nothing to draw a line through.
  • The shape is the lesson: the error drops steeply over the first 60 trees, then flattens. Most of what a forest learns, it learns early.

Step 9: Compare the Curves of Three Runs​

Open the house-price-prediction experiment and click the chart icon above the runs table, as in Part 1. Part 1's four runs have no curves, so hide them from the charts: click the eye icon next to each of their names. The three new runs remain:

The chart view of the house-price-prediction experiment with curve-depth-8 (green), curve-300-trees (orange) and curve-100-trees (blue) visible and Part 1's runs hidden: bar charts for mae, r2 and rmse, and line charts where the blue and orange val_rmse curves fall from 0.55 and run on together to 300 trees near 0.525, while the green curve stays near 0.59.

What the curves say, without a single extra run:

  • Blue and orange overlap up to 100 trees. They're the same forest, scored every 10 trees in one run and every 20 in the other.
  • Orange shows what trees 100 to 300 add: almost nothing, a 0.6% lower error for three times the trees.
  • Green sits apart. Limiting the depth costs more than any number of trees can win back.

So 100 unlimited trees, Part 1's default, stays a sound choice, and now the curves show why. The X-axis can also show time instead of steps: How to Change the X-Axis of a Metric Chart in the MLflow UI.

Step 10: Read a Curve from Python​

The same curve, from the command you added in Step 5:

Terminal 2
uv run curve curve-300-trees
Output
20 trees val_rmse 0.5417
40 trees val_rmse 0.5320
60 trees val_rmse 0.5300
80 trees val_rmse 0.5287
100 trees val_rmse 0.5276
120 trees val_rmse 0.5266
140 trees val_rmse 0.5261
160 trees val_rmse 0.5254
180 trees val_rmse 0.5254
200 trees val_rmse 0.5254
220 trees val_rmse 0.5250
240 trees val_rmse 0.5253
260 trees val_rmse 0.5248
280 trees val_rmse 0.5244
300 trees val_rmse 0.5243
Best val_rmse: 0.5243 at 300 trees

Every value the run logged came back, in step order. The best is at 300 trees, but only by a little, which is why the curve matters more than the single best number. Ask for R² instead, on the depth-8 run:

Terminal 2
uv run curve curve-depth-8 --metric val_r2
Output
10 trees val_r2 0.7444
20 trees val_r2 0.7458
30 trees val_r2 0.7488
40 trees val_r2 0.7499
50 trees val_r2 0.7499
60 trees val_r2 0.7505
70 trees val_r2 0.7501
80 trees val_r2 0.7510
90 trees val_r2 0.7510
100 trees val_r2 0.7514
Best val_r2: 0.7514 at 100 trees

For R², higher is better, so the command picked the highest value. The shallow forest never explains more than about 75% of the variation.

Step 11: See the System Metrics​

Open the curve-300-trees run again and click System metrics:

The System metrics tab of the curve-300-trees run, with eight charts against time between 10:49:50 and 10:49:52: cpu_utilization_percentage rising from 0 to about 87 and staying there, disk_available_megabytes, disk_usage_megabytes and disk_usage_percentage, and network_receive_megabytes and network_transmit_megabytes rising slowly.

  • Eight charts, with time along the bottom, one for each system/ metric MLflow recorded.
  • CPU went from 0 to about 87%. The first CPU sample of every run is 0.0: psutil's first reading in each new thread is "a meaningless 0.0 value", and each run's monitor is a new thread. After that, the forest kept the processor busy on all cores, because n_jobs=-1 uses them all. The figure covers the whole computer, not only this program.
  • Three samples, for a run that lasted 3.7 seconds at one sample per second. The monitor logs a sample only when a whole interval has passed and the run is still going, so the sample taken in a run's last second is never logged.

The curve-depth-8 run lasted 1.0 second on this computer, and its System metrics tab says System metrics (0) and "No charts in this section": it ended before its first sample could be logged. Fixed: "No charts in this section" on the MLflow System Metrics Tab walks through the cause. To record its system metrics, train it again with a sample every 0.2 seconds. The new setting from Step 3 changes the interval for this one command:

Terminal 2
HOUSE_PRICES_SYSTEM_METRICS_INTERVAL=0.2 uv run train --max-depth 8 --run-name curve-depth-8-sampled
Output
2026/11/18 10:58:40 INFO house_price_prediction: Loading the California Housing data
2026/11/18 10:58:40 INFO mlflow.system_metrics.system_metrics_monitor: Skip logging GPU metrics. Set logger level to DEBUG for more details.
2026/11/18 10:58:40 INFO mlflow.system_metrics.system_metrics_monitor: Started monitoring system metrics.
2026/11/18 10:58:40 INFO house_price_prediction: Training on 13209 rows, scoring on 3303 validation rows
2026/11/18 10:58:40 INFO house_price_prediction: 10 trees: validation RMSE 0.5939
2026/11/18 10:58:40 INFO house_price_prediction: 20 trees: validation RMSE 0.5922
2026/11/18 10:58:40 INFO house_price_prediction: 30 trees: validation RMSE 0.5888
2026/11/18 10:58:40 INFO house_price_prediction: 40 trees: validation RMSE 0.5875
2026/11/18 10:58:40 INFO house_price_prediction: 50 trees: validation RMSE 0.5875
2026/11/18 10:58:40 INFO house_price_prediction: 60 trees: validation RMSE 0.5867
2026/11/18 10:58:40 INFO house_price_prediction: 70 trees: validation RMSE 0.5872
2026/11/18 10:58:40 INFO house_price_prediction: 80 trees: validation RMSE 0.5862
2026/11/18 10:58:40 INFO house_price_prediction: 90 trees: validation RMSE 0.5861
2026/11/18 10:58:41 INFO house_price_prediction: 100 trees: validation RMSE 0.5857
2026/11/18 10:58:41 INFO house_price_prediction: Run curve-depth-8-sampled: MAE 0.403, RMSE 0.586, R2 0.738
🏃 View run curve-depth-8-sampled at: http://127.0.0.1:8080/#/experiments/1/runs/a5c92ce28557481999f160318456da7a
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/1
2026/11/18 10:58:41 INFO mlflow.system_metrics.system_metrics_monitor: Stopping system metrics monitoring...
2026/11/18 10:58:41 INFO mlflow.system_metrics.system_metrics_monitor: Successfully terminated system metrics monitoring!

The same scores as curve-depth-8, and this time the run's System metrics tab has all eight charts, five samples each:

The System metrics tab of the curve-depth-8-sampled run, System metrics (8), with cpu_utilization_percentage rising from 0 to about 85 over five samples within one second, and disk_available_megabytes and disk_usage_megabytes against time.

How long a run takes depends on your computer. If your curve-depth-8 run lasted longer than a second, its tab has charts already.

Step 12: Save Your Work with Git​

See what changed since Part 1's commit:

Terminal 2
git status --short --untracked-files=all
Output
M .env.example
M pyproject.toml
M src/house_price_prediction/config.py
M src/house_price_prediction/data.py
M src/house_price_prediction/train.py
M uv.lock
?? src/house_price_prediction/curve.py

M marks a changed file and ?? a new one: the five files you edited, the lock file that uv add updated, and the new curve.py. How to See What Changed in a Git Repository Before You Commit explains the codes. Commit them:

Terminal 2
git add .
git commit -m "Log validation curves and system metrics"
Output
[master e6b8df4] Log validation curves and system metrics
7 files changed, 151 insertions(+), 14 deletions(-)
create mode 100644 src/house_price_prediction/curve.py

Your commit ID will differ.

Your Folders Now Look Like This​

source/
source/
├── mlflow-server/
│ ├── .venv/
│ ├── mlartifacts/
│ ├── mlflow.db
│ ├── pyproject.toml
│ └── uv.lock
└── dewiride-house-price-prediction/
├── .venv/
├── src/
│ └── house_price_prediction/
│ ├── __init__.py
│ ├── config.py
│ ├── curve.py
│ ├── data.py
│ ├── plots.py
│ └── train.py
├── .env
├── .env.example
├── .gitignore
├── .python-version
├── README.md
├── pyproject.toml
└── uv.lock

The step metrics and system metrics went where every metric goes: into the metrics table of mlflow-server/mlflow.db, one row per value, each with its step.

Stop the Server When You're Done​

Go to the first terminal and press Ctrl+C. Your runs stay in mlflow.db and mlartifacts/, and uv run mlflow server --port 8080 in the mlflow-server folder brings them back.

Keep both folders for Part 3. If you ever want to remove everything the series created, stop the server first, then delete the mlflow-server and dewiride-house-price-prediction folders, and the scikit_learn_data folder in your home folder.

Common Mistakes​

MistakeWhat happensDo this instead
Choosing the number of trees by the test scoreThe test score stops being a fair check on unseen dataWatch a validation set, and score the test set once at the end
Logging a value at every stage without step=Every value is stored at step 0, so there's no curve to readPass step=, such as the number of trees or the epoch
Leaving out warm_start=TrueEach stage grows a whole new forest, so training takes far longerSet warm_start=True and raise n_estimators between fits
Setting the sampling interval after start_runThe run's monitor keeps the interval it started withCall mlflow.set_system_metrics_sampling_interval() first
Keeping MLflow's 10-second default for runs of a few secondsThe System metrics tab stays emptySample more often than your shortest run lasts
Comparing Part 2's test scores with Part 1's and blaming the codeThe difference comes from the smaller training setCompare runs that learned from the same rows

Troubleshooting​

SymptomLikely causeFix
MlflowException: Failed to start system metrics monitoring as package `psutil` is not installed.psutil is missinguv add psutil (Step 6)
System metrics (0), "No charts in this section"The run ended within one sampling intervalA shorter interval: Fixed: "No charts in this section" on the MLflow System Metrics Tab
No run called 'curve-300' in 'house-price-prediction'No run has exactly that nameCopy the name from the runs table
curve: error: argument --metric: invalid choice: 'rmse', maybe you meant 'val_rmse'?curve only reads the validation scoresUse val_rmse, val_mae or val_r2
Part 1's runs crowd the charts as bars with no curveThey logged each score onceHide them with the eye icon next to their names

FAQ​

Why not watch the test score while training?​

Because then you'd be choosing the model with the test rows, and their score would no longer tell you how the model does on data it has never seen. The validation rows take that role, and the test rows stay untouched until the end.

What can a step be?​

Any whole number you choose. MLflow stores it with each value and draws the curve along it. Common choices are the epoch for neural networks, the number of trees for forests and boosting, or simply a counter.

Does logging at every step slow training down?​

Each log_metrics call sends one request to the tracking server, here 10 to 15 per run. A loop with thousands of steps can log every tenth or hundredth step instead.

Why does the CPU chart always start at 0?​

The first CPU reading in each new thread is 0.0, and every run's monitor is a new thread. Read the chart from its second point. How to Turn On System Metrics Logging in MLflow has the details.

Can I monitor a GPU?​

MLflow's documentation lists extra packages for NVIDIA and AMD graphics cards. Without one, MLflow prints Skip logging GPU metrics and records the other eight metrics, as here.

Conclusion​

Your forest now grows in stages, and after each stage mlflow.log_metrics(..., step=n_trees) records three validation scores, scored on rows the model never trains on. The MLflow UI turned them into learning curves, which showed that trees beyond 100 add almost nothing and that a depth limit costs more than trees can win back. The curve command read the same curve with get_metric_history(), and system metrics recorded the processor, memory, disk and network for every run long enough to be sampled.

In Part 3 you'll attach whole files and the exact dataset each run used, and search runs from Python.

Additional Resources​


← Previous: Part 1: Install MLflow and Track Your First Experiment

Next → Part 3: Log Artifacts and Datasets (coming soon)