Skip to main content

How to Compare a Model with a Baseline in MLflow

· 7 min read
Jagdish Kumawat
Founder @ Dewiride

A model can clear fixed minimum scores and still be no better than what you already have. MLflow's thresholds can also compare: min_absolute_change and min_relative_change make a candidate beat a baseline model by a margin, using two evaluations on the same data. This guide compares a logistic regression with a baseline that guesses at random, both ways round, with real output from MLflow 3.18.0.

Syntax​

Syntax
thresholds = {
"metric_name": MetricThreshold(min_absolute_change=0.1, greater_is_better=True),
"other_metric": MetricThreshold(min_relative_change=0.1, greater_is_better=True),
}
mlflow.validate_evaluation_results(thresholds, candidate_result, baseline_result)
PartWhat it means
min_absolute_changeThe candidate must beat the baseline by at least this much: 0.1 means 0.1 higher
min_relative_changeThe candidate must beat the baseline by at least this share of the baseline's score: 0.1 means 10% higher
greater_is_betterWhich way is better; with False, the candidate must be that much lower
candidate_result, baseline_resultWhat mlflow.models.evaluate returned for each model, on the same data

The reference for MetricThreshold: with min_absolute_change, "If greater is better for the metric, metric value has to be >= baseline model metric value + min_absolute_change to pass the validation." It must be "A positive number": the candidate always has to improve. A threshold can also hold a threshold, a minimum the candidate must reach either way.

Prerequisites​

  • An MLflow tracking server at http://127.0.0.1:8080. MLflow's tracking server guide shows how to start one.
  • mlflow and scikit-learn in your project, for example with uv add mlflow scikit-learn pandas.

Step 1: Evaluate Both Models and Compare​

The candidate is a logistic regression that tells malignant tumours from benign ones. The baseline is scikit-learn's DummyClassifier with strategy="stratified", which guesses at random, as often as each answer appears in the training data: the floor any real model must clear. Both are evaluated on the same test rows, each in its own run, then compared both ways:

The baseline.py script in a macOS-style code window: a logistic regression candidate and a stratified DummyClassifier baseline, an evaluate helper that logs and evaluates a model in its own run, a roc_auc threshold with min_absolute_change 0.1 and an accuracy_score threshold with min_relative_change 0.1, and a loop that validates candidate against baseline and the other way round.

baseline.py
import mlflow
from mlflow.models import MetricThreshold
from mlflow.models.evaluation.validation import ModelValidationFailedException
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("baseline-demo")

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
y = y == 0 # True when the tumour is malignant
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
test = X_test.assign(malignant=y_test)

candidate = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
# The baseline guesses at random, as often as each answer appears in the training data.
baseline = DummyClassifier(strategy="stratified", random_state=0).fit(X_train, y_train)


def evaluate(model, run_name):
with mlflow.start_run(run_name=run_name):
info = mlflow.sklearn.log_model(model, name="model", input_example=X_train.head(3))
return mlflow.models.evaluate(
info.model_uri, test, targets="malignant", model_type="classifier"
)


candidate_result = evaluate(candidate, "candidate")
baseline_result = evaluate(baseline, "baseline")

thresholds = {
# At least 0.1 higher than the baseline's ROC AUC.
"roc_auc": MetricThreshold(min_absolute_change=0.1, greater_is_better=True),
# At least 10% higher than the baseline's accuracy.
"accuracy_score": MetricThreshold(min_relative_change=0.1, greater_is_better=True),
}

for label, first, second in [
("candidate vs baseline", candidate_result, baseline_result),
("baseline vs candidate", baseline_result, candidate_result),
]:
try:
mlflow.validate_evaluation_results(thresholds, first, second)
print(f"{label}: passed")
except ModelValidationFailedException as error:
print(f"{label}: failed\n{error.message}")

The exception comes from mlflow.models.evaluation.validation: Fixed: module 'mlflow.exceptions' has no attribute 'ModelValidationFailedException' explains why.

Terminal
uv run python baseline.py
Output (last lines)
🏃 View run baseline at: http://127.0.0.1:8080/#/experiments/11/runs/279c7f483750446990c8b5f1df27198b
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/11
2026/11/21 17:54:53 INFO mlflow.models.evaluation.validation: Validating candidate model metrics against baseline
2026/11/21 17:54:53 INFO mlflow.models.evaluation.validation: Model validation passed!
candidate vs baseline: passed
2026/11/21 17:54:53 INFO mlflow.models.evaluation.validation: Validating candidate model metrics against baseline
baseline vs candidate: failed
Metric roc_auc minimum absolute change check failed: candidate model roc_auc = 0.5669811320754717, baseline model roc_auc = 0.9914046121593291, roc_auc minimum absolute change threshold = 0.1.
Metric accuracy_score minimum relative change check failed: candidate model accuracy_score = 0.6013986013986014, baseline model accuracy_score = 0.958041958041958, accuracy_score minimum relative change threshold = 0.1.

The end of the output in a terminal: candidate vs baseline passed; baseline vs candidate failed, with roc_auc 0.567 against 0.991 failing the minimum absolute change of 0.1, and accuracy_score 0.601 against 0.958 failing the minimum relative change of 0.1.

Step 2: Read the Two Results​

  • Candidate against baseline: passed. Its ROC AUC, 0.991, is far more than 0.1 above the baseline's, and its accuracy, 0.958, far more than 10% above.
  • Baseline against candidate: failed, on both checks. MLflow calls the first result the "candidate", so the messages read the guesser's 0.567 ROC AUC against the logistic regression's 0.991, and its 0.601 accuracy against 0.958. The message lists every check that failed, one line each.

A random guesser's ROC AUC near 0.5 is the usual floor; a model that can't beat it by a margin isn't using its data.

Common Mistakes​

MistakeWhat happensDo this instead
Evaluating the two models on different dataThe comparison measures the data as much as the modelsEvaluate both on the same rows
min_absolute_change=0 to mean "not worse"Could not instantiate MetricThreshold class: min_absolute_change parameter must be a positive number.Use a small positive margin, such as 0.01
A comparison threshold without a baseline resultModel comparison failed: metric ... was missing from the evaluation result of the baseline model.Pass the baseline's result, or use threshold alone

Troubleshooting​

SymptomLikely causeFix
Metric ... minimum absolute change check failedThe candidate didn't beat the baseline by the marginRead the two scores in the message
Model comparison failed: metric ... was missing from the evaluation result of the baseline model.No baseline result, or one evaluated without that metricPass a baseline evaluated the same way

FAQ​

What makes a good baseline?​

Either the model in use, so a new one has to be better before it replaces it, or a simple guesser like this one, so a model has to be better than chance.

Can I mix comparisons and minimum scores?​

Yes: one MetricThreshold can hold threshold and min_absolute_change together, and the model must meet both.

Conclusion​

With min_absolute_change on ROC AUC and min_relative_change on accuracy, mlflow.validate_evaluation_results let the logistic regression past a random-guessing baseline and stopped the comparison the other way round, naming both failed checks.

Additional Resources​