Skip to main content

How to Change the System Metrics Sampling Interval in MLflow

· 8 min read
Jagdish Kumawat
Founder @ Dewiride

By default, MLflow samples the computer's CPU, memory, disk and network every 10 seconds. That suits a training job that runs for an hour, but a run that lasts a minute gets about five points, and a run shorter than 10 seconds gets none. Two functions change how often MLflow samples and how many samples it averages into each point. This guide tries three settings on the same run, with real output from MLflow 3.18.0.

Syntax​

Syntax
import mlflow

mlflow.set_system_metrics_sampling_interval(<seconds>)
mlflow.set_system_metrics_samples_before_logging(<samples>)
PartWhat it means
set_system_metrics_sampling_interval(<seconds>)"Every interval seconds, the system metrics will be collected. By default interval=10."
set_system_metrics_samples_before_logging(<samples>)"Every time samples samples have been collected, the system metrics will be logged to mlflow. By default samples=1." Each logged point is the average of those samples
NonePassing None to either function goes back to the default

Both functions apply to every run the program starts after the call. MLflow's documentation sums up how they combine: "The actual logging time window is the product of" the two.

Prerequisites​

Step 1: Try Three Settings on the Same Run​

The example runs the same six-second busy loop three times, sampling every second, every two seconds, and every second with three samples averaged into each point. Create sampling_demo.py:

The sampling_demo.py script in a macOS-style code window: a busy_run function that keeps the CPU busy for six seconds in an MLflow run with system metrics on and then prints how many CPU samples were logged, called three times after setting the sampling interval to 1, to 2, and to 1 with three samples before logging.

sampling_demo.py
import logging
import time

import mlflow
from mlflow import MlflowClient

logging.getLogger("mlflow").setLevel(logging.WARNING) # keep the output short
mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("sampling-demo")


def busy_run(name: str, seconds: int = 6) -> None:
with mlflow.start_run(run_name=name, log_system_metrics=True) as run:
end = time.time() + seconds
while time.time() < end:
sum(i * i for i in range(10_000))
history = MlflowClient().get_metric_history(run.info.run_id, "system/cpu_utilization_percentage")
values = [point.value for point in sorted(history, key=lambda point: point.step)]
print(f"{name}: {len(values)} samples logged, CPU % {values}")


mlflow.set_system_metrics_sampling_interval(1)
busy_run("every-second")

mlflow.set_system_metrics_sampling_interval(2)
busy_run("every-2-seconds")

mlflow.set_system_metrics_sampling_interval(1)
mlflow.set_system_metrics_samples_before_logging(3)
busy_run("average-of-3")
LineWhat it does
logging.getLogger("mlflow").setLevel(logging.WARNING)Hides MLflow's start and stop messages, so the output shows only the results
busy_runStarts a run with system metrics on, keeps one CPU core busy for six seconds, then reads back every CPU sample the run logged
get_metric_history(...)Returns every logged point of system/cpu_utilization_percentage: How to Get the History of a Metric in MLflow
The last five linesChange the settings between runs: every second, every two seconds, then every second averaged over three samples

Step 2: Run It​

Terminal
uv run python sampling_demo.py
Output
🏃 View run every-second at: http://127.0.0.1:8080/#/experiments/4/runs/bb621b879fb344f1af8a2680b0e58dbd
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/4
every-second: 4 samples logged, CPU % [0.0, 40.7, 50.4, 44.8]
🏃 View run every-2-seconds at: http://127.0.0.1:8080/#/experiments/4/runs/1b22fd515e004276b06e8c23ed5f6bde
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/4
every-2-seconds: 2 samples logged, CPU % [0.0, 45.5]
🏃 View run average-of-3 at: http://127.0.0.1:8080/#/experiments/4/runs/06e0c9dced7a477da3eef91d452580d2
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/4
average-of-3: 1 samples logged, CPU % [46.9]

A terminal running uv run python sampling_demo.py: the every-second run logged 4 CPU samples starting at 0.0, the every-2-seconds run logged 2, and the average-of-3 run logged 1 point of 46.9.

Step 3: Read the Results​

Every second: 4 points in a six-second run, not 6. MLflow's monitor takes a sample, waits one interval, checks with the tracking server that the run is still going, and only then logs the sample. When the run ends, the sample it's waiting on is dropped. So a run gets fewer points than its length divided by the interval.

Every two seconds: 2 points. Half the samples, for the same run.

Three samples averaged: 1 point. One point every three seconds, each the average of three samples. The first average includes the first sample, which is always 0.0: psutil, which MLflow uses to read CPU use, returns "a meaningless 0.0 value" the first time it's called in a thread, and every run starts a new one.

The rule that follows: a run records a point only if it's still running one full window after the window began, where the window is the interval times the samples. A run shorter than one window records nothing at all, and its System metrics tab shows "No charts in this section": Fixed: "No charts in this section" on the MLflow System Metrics Tab.

Step 4: Set It Without Changing the Code​

set_system_metrics_sampling_interval() stores its value in an environment variable MLflow reads, MLFLOW_SYSTEM_METRICS_SAMPLING_INTERVAL. Set it yourself to change the interval for one program without editing it:

Terminal
MLFLOW_SYSTEM_METRICS_SAMPLING_INTERVAL=1 uv run python train.py

Here train.py stands for your own script. A call to the function in the code overrides the variable, because the function sets the same variable.

Common Mistakes​

MistakeWhat happensDo this instead
Keeping the 10-second default for runs that last secondsThe run ends before anything is loggedSample more often than your shortest run lasts
Setting the interval after start_runThe run already started its monitor with the old settingCall the functions before start_run
Averaging many samples on a short runFewer points, and the first one includes the meaningless 0.0Keep samples_before_logging at 1 for short runs
Expecting run length ÷ interval pointsThe last sample is dropped when the run endsExpect one or two fewer

Troubleshooting​

SymptomLikely causeFix
System metrics (0), "No charts in this section"The run was shorter than one interval times the samplesA shorter interval, or fewer samples per point
Every run's CPU chart starts at 0psutil's first reading in each run's monitor threadExpected: read from the second point on
No system metrics whatever the intervalSystem metrics aren't turned on, or psutil is missingHow to Turn On System Metrics Logging in MLflow

FAQ​

What interval should I use?​

One that gives your typical run a useful number of points: for a run of a few seconds, a second or less; for an hour-long run, the default 10 seconds gives about 360 points. Each sample also asks the tracking server about the run, so very short intervals mean more requests.

Can the interval be less than a second?​

Yes: MLflow reads the interval as a decimal number of seconds. psutil's documentation recommends at least 0.1 seconds between CPU readings for accuracy.

Why average samples at all?​

To smooth out spikes on long runs, and to log fewer points. On a run that lasts seconds, it mostly removes the detail you wanted.

Conclusion​

mlflow.set_system_metrics_sampling_interval() set how often MLflow samples the computer, and mlflow.set_system_metrics_samples_before_logging() how many samples it averages into each point. On the same six-second run, sampling every second logged 4 points, every two seconds 2, and three-sample averages 1. A point is logged only if the run is still going a full window later, so pick an interval shorter than your runs.

Additional Resources​