Skip to main content

How to Set Minimum Scores for a Model with mlflow.validate_evaluation_results

· 6 min read
Jagdish Kumawat
Founder @ Dewiride

An evaluation tells you a model's scores; a threshold says which scores are good enough. MLflow checks an evaluation against minimum scores with MetricThreshold and mlflow.validate_evaluation_results, and raises an exception naming each metric that fell short, which is the start of a quality gate. This guide checks a tumour classifier against accuracy and recall thresholds, with real output from MLflow 3.18.0.

Syntax​

Syntax
from mlflow.models import MetricThreshold
from mlflow.models.evaluation.validation import ModelValidationFailedException

thresholds = {
"metric_name": MetricThreshold(threshold=value, greater_is_better=True),
}
try:
mlflow.validate_evaluation_results(candidate_result=result, validation_thresholds=thresholds)
except ModelValidationFailedException as error:
...
PartWhat it means
"metric_name"A metric of the evaluation, built-in or your own
thresholdThe score the metric has to reach
greater_is_betterTrue: the metric must be at or above threshold. False: at or below, for errors and losses
resultWhat mlflow.models.evaluate returned
ModelValidationFailedExceptionRaised when any threshold is missed; its message names each one

The reference for MetricThreshold: "If greater is better for the metric, the metric value has to be >= threshold to pass validation. Otherwise, the metric value has to be <= threshold to pass the validation." The exception is imported from mlflow.models.evaluation.validation, where MLflow 3.18.0 defines it; the evaluation guide's mlflow.exceptions.ModelValidationFailedException doesn't exist: Fixed: module 'mlflow.exceptions' has no attribute 'ModelValidationFailedException'.

Prerequisites​

  • An MLflow tracking server at http://127.0.0.1:8080. MLflow's tracking server guide shows how to start one.
  • mlflow and scikit-learn in your project, for example with uv add mlflow scikit-learn pandas.

Step 1: Check the Model Against Two Thresholds​

The model is a logistic regression that tells malignant tumours from benign ones, evaluated on test rows. The thresholds ask for 0.95 accuracy and 0.95 recall, the share of malignant tumours found:

The thresholds.py script in a macOS-style code window: the classifier logged and evaluated, a thresholds dictionary with accuracy_score and recall_score at 0.95, and validate_evaluation_results in a try block that prints Model meets all thresholds or Validation failed with the reason.

thresholds.py
import mlflow
from mlflow.models import MetricThreshold
from mlflow.models.evaluation.validation import ModelValidationFailedException
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("thresholds-demo")

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
y = y == 0 # True when the tumour is malignant
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)

with mlflow.start_run(run_name="check-thresholds"):
info = mlflow.sklearn.log_model(model, name="model", input_example=X_train.head(3))
result = mlflow.models.evaluate(
info.model_uri,
X_test.assign(malignant=y_test),
targets="malignant",
model_type="classifier",
)

thresholds = {
"accuracy_score": MetricThreshold(threshold=0.95, greater_is_better=True),
"recall_score": MetricThreshold(threshold=0.95, greater_is_better=True),
}

try:
mlflow.validate_evaluation_results(
candidate_result=result,
validation_thresholds=thresholds,
)
print("Model meets all thresholds")
except ModelValidationFailedException as e:
print(f"Validation failed: {e}")
Terminal
uv run python thresholds.py
Output (last lines)
🏃 View run check-thresholds at: http://127.0.0.1:8080/#/experiments/10/runs/a52a5f6f76c14d6cb924e1fb8d7ca0c6
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/10
2026/11/21 17:54:14 INFO mlflow.models.evaluation.validation: Validating candidate model metrics against baseline
Validation failed: Metric recall_score value threshold check failed: candidate model recall_score = 0.9433962264150944, recall_score threshold = 0.95.

The end of the output in a terminal: the run's link, Validating candidate model metrics against baseline, and Validation failed: Metric recall_score value threshold check failed, with recall 0.9434 against the threshold 0.95.

Accuracy, 0.958, cleared 0.95, so the message names only recall: 0.9434, below 0.95. MLflow's line says "against baseline" even without a baseline; it checks the thresholds on their own here.

Step 2: Lower the Recall Threshold​

In thresholds.py, change the recall threshold to 0.90:

thresholds.py (the recall threshold)
"recall_score": MetricThreshold(threshold=0.90, greater_is_better=True),
Terminal
uv run python thresholds.py
Output (last lines)
🏃 View run check-thresholds at: http://127.0.0.1:8080/#/experiments/10/runs/7a3571e389d8487db14624b1e1c5002b
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/10
2026/11/21 18:35:55 INFO mlflow.models.evaluation.validation: Validating candidate model metrics against baseline
2026/11/21 18:35:55 INFO mlflow.models.evaluation.validation: Model validation passed!
Model meets all thresholds

The end of the output in a terminal: the run's link, Validating candidate model metrics against baseline, Model validation passed!, and Model meets all thresholds.

Both thresholds met: MLflow logs Model validation passed! and returns without raising.

Common Mistakes​

MistakeWhat happensDo this instead
except mlflow.exceptions.ModelValidationFailedExceptionAttributeError the first time a model failsImport it from mlflow.models.evaluation.validation
greater_is_better=True for an error or a lossA model passes when its error is too highgreater_is_better=False
A plain number instead of a MetricThresholdMlflowException: The validation thresholds argument must be a dictionary that maps strings to MetricThreshold objects.Wrap each value in MetricThreshold(...)

Troubleshooting​

SymptomLikely causeFix
Metric ... value threshold check failedThe model missed that thresholdThe message gives the score and the threshold
Metric validation failed: metric ... was missing from the evaluation result of the candidate modelA metric name the evaluation didn't logUse a name from result.metrics

FAQ​

Can a custom metric have a threshold?​

Yes: use the name you gave it in make_metric: How to Add a Custom Metric to an MLflow Evaluation with make_metric.

How do I make a script fail when a threshold is missed?​

Return a non-zero exit code from the except block: How to Make a Python Script Exit with an Error Code.

Conclusion​

MetricThreshold(threshold=0.95, greater_is_better=True) on recall made mlflow.validate_evaluation_results raise ModelValidationFailedException, naming recall 0.9434; at 0.90 the same model passed.

Additional Resources​