How to Compare a Model with a Baseline in MLflow
A model can clear fixed minimum scores and still be no better than what you already have. MLflow's thresholds can also compare: min_absolute_change and min_relative_change make a candidate beat a baseline model by a margin, using two evaluations on the same data. This guide compares a logistic regression with a baseline that guesses at random, both ways round, with real output from MLflow 3.18.0.
Syntax
thresholds = {
"metric_name": MetricThreshold(min_absolute_change=0.1, greater_is_better=True),
"other_metric": MetricThreshold(min_relative_change=0.1, greater_is_better=True),
}
mlflow.validate_evaluation_results(thresholds, candidate_result, baseline_result)
| Part | What it means |
|---|---|
min_absolute_change | The candidate must beat the baseline by at least this much: 0.1 means 0.1 higher |
min_relative_change | The candidate must beat the baseline by at least this share of the baseline's score: 0.1 means 10% higher |
greater_is_better | Which way is better; with False, the candidate must be that much lower |
candidate_result, baseline_result | What mlflow.models.evaluate returned for each model, on the same data |
The reference for MetricThreshold: with min_absolute_change, "If greater is better for the metric, metric value has to be >= baseline model metric value + min_absolute_change to pass the validation." It must be "A positive number": the candidate always has to improve. A threshold can also hold a threshold, a minimum the candidate must reach either way.
Prerequisites
- An MLflow tracking server at
http://127.0.0.1:8080. MLflow's tracking server guide shows how to start one. mlflowandscikit-learnin your project, for example withuv add mlflow scikit-learn pandas.
Step 1: Evaluate Both Models and Compare
The candidate is a logistic regression that tells malignant tumours from benign ones. The baseline is scikit-learn's DummyClassifier with strategy="stratified", which guesses at random, as often as each answer appears in the training data: the floor any real model must clear. Both are evaluated on the same test rows, each in its own run, then compared both ways:

import mlflow
from mlflow.models import MetricThreshold
from mlflow.models.evaluation.validation import ModelValidationFailedException
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("baseline-demo")
X, y = load_breast_cancer(return_X_y=True, as_frame=True)
y = y == 0 # True when the tumour is malignant
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
test = X_test.assign(malignant=y_test)
candidate = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
# The baseline guesses at random, as often as each answer appears in the training data.
baseline = DummyClassifier(strategy="stratified", random_state=0).fit(X_train, y_train)
def evaluate(model, run_name):
with mlflow.start_run(run_name=run_name):
info = mlflow.sklearn.log_model(model, name="model", input_example=X_train.head(3))
return mlflow.models.evaluate(
info.model_uri, test, targets="malignant", model_type="classifier"
)
candidate_result = evaluate(candidate, "candidate")
baseline_result = evaluate(baseline, "baseline")
thresholds = {
# At least 0.1 higher than the baseline's ROC AUC.
"roc_auc": MetricThreshold(min_absolute_change=0.1, greater_is_better=True),
# At least 10% higher than the baseline's accuracy.
"accuracy_score": MetricThreshold(min_relative_change=0.1, greater_is_better=True),
}
for label, first, second in [
("candidate vs baseline", candidate_result, baseline_result),
("baseline vs candidate", baseline_result, candidate_result),
]:
try:
mlflow.validate_evaluation_results(thresholds, first, second)
print(f"{label}: passed")
except ModelValidationFailedException as error:
print(f"{label}: failed\n{error.message}")
The exception comes from mlflow.models.evaluation.validation: Fixed: module 'mlflow.exceptions' has no attribute 'ModelValidationFailedException' explains why.
uv run python baseline.py
🏃 View run baseline at: http://127.0.0.1:8080/#/experiments/11/runs/279c7f483750446990c8b5f1df27198b
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/11
2026/11/21 17:54:53 INFO mlflow.models.evaluation.validation: Validating candidate model metrics against baseline
2026/11/21 17:54:53 INFO mlflow.models.evaluation.validation: Model validation passed!
candidate vs baseline: passed
2026/11/21 17:54:53 INFO mlflow.models.evaluation.validation: Validating candidate model metrics against baseline
baseline vs candidate: failed
Metric roc_auc minimum absolute change check failed: candidate model roc_auc = 0.5669811320754717, baseline model roc_auc = 0.9914046121593291, roc_auc minimum absolute change threshold = 0.1.
Metric accuracy_score minimum relative change check failed: candidate model accuracy_score = 0.6013986013986014, baseline model accuracy_score = 0.958041958041958, accuracy_score minimum relative change threshold = 0.1.

Step 2: Read the Two Results
- Candidate against baseline: passed. Its ROC AUC, 0.991, is far more than 0.1 above the baseline's, and its accuracy, 0.958, far more than 10% above.
- Baseline against candidate: failed, on both checks. MLflow calls the first result the "candidate", so the messages read the guesser's 0.567 ROC AUC against the logistic regression's 0.991, and its 0.601 accuracy against 0.958. The message lists every check that failed, one line each.
A random guesser's ROC AUC near 0.5 is the usual floor; a model that can't beat it by a margin isn't using its data.
Common Mistakes
| Mistake | What happens | Do this instead |
|---|---|---|
| Evaluating the two models on different data | The comparison measures the data as much as the models | Evaluate both on the same rows |
min_absolute_change=0 to mean "not worse" | Could not instantiate MetricThreshold class: min_absolute_change parameter must be a positive number. | Use a small positive margin, such as 0.01 |
| A comparison threshold without a baseline result | Model comparison failed: metric ... was missing from the evaluation result of the baseline model. | Pass the baseline's result, or use threshold alone |
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Metric ... minimum absolute change check failed | The candidate didn't beat the baseline by the margin | Read the two scores in the message |
Model comparison failed: metric ... was missing from the evaluation result of the baseline model. | No baseline result, or one evaluated without that metric | Pass a baseline evaluated the same way |
FAQ
What makes a good baseline?
Either the model in use, so a new one has to be better before it replaces it, or a simple guesser like this one, so a model has to be better than chance.
Can I mix comparisons and minimum scores?
Yes: one MetricThreshold can hold threshold and min_absolute_change together, and the model must meet both.
Conclusion
With min_absolute_change on ROC AUC and min_relative_change on accuracy, mlflow.validate_evaluation_results let the logistic regression past a random-guessing baseline and stopped the comparison the other way round, naming both failed checks.
Additional Resources
- MetricThreshold —
min_absolute_changeandmin_relative_change. - mlflow.validate_evaluation_results — comparing a candidate with a baseline.
