How to Set Minimum Scores for a Model with mlflow.validate_evaluation_results
An evaluation tells you a model's scores; a threshold says which scores are good enough. MLflow checks an evaluation against minimum scores with MetricThreshold and mlflow.validate_evaluation_results, and raises an exception naming each metric that fell short, which is the start of a quality gate. This guide checks a tumour classifier against accuracy and recall thresholds, with real output from MLflow 3.18.0.
Syntax
from mlflow.models import MetricThreshold
from mlflow.models.evaluation.validation import ModelValidationFailedException
thresholds = {
"metric_name": MetricThreshold(threshold=value, greater_is_better=True),
}
try:
mlflow.validate_evaluation_results(candidate_result=result, validation_thresholds=thresholds)
except ModelValidationFailedException as error:
...
| Part | What it means |
|---|---|
"metric_name" | A metric of the evaluation, built-in or your own |
threshold | The score the metric has to reach |
greater_is_better | True: the metric must be at or above threshold. False: at or below, for errors and losses |
result | What mlflow.models.evaluate returned |
ModelValidationFailedException | Raised when any threshold is missed; its message names each one |
The reference for MetricThreshold: "If greater is better for the metric, the metric value has to be >= threshold to pass validation. Otherwise, the metric value has to be <= threshold to pass the validation." The exception is imported from mlflow.models.evaluation.validation, where MLflow 3.18.0 defines it; the evaluation guide's mlflow.exceptions.ModelValidationFailedException doesn't exist: Fixed: module 'mlflow.exceptions' has no attribute 'ModelValidationFailedException'.
Prerequisites
- An MLflow tracking server at
http://127.0.0.1:8080. MLflow's tracking server guide shows how to start one. mlflowandscikit-learnin your project, for example withuv add mlflow scikit-learn pandas.
Step 1: Check the Model Against Two Thresholds
The model is a logistic regression that tells malignant tumours from benign ones, evaluated on test rows. The thresholds ask for 0.95 accuracy and 0.95 recall, the share of malignant tumours found:

import mlflow
from mlflow.models import MetricThreshold
from mlflow.models.evaluation.validation import ModelValidationFailedException
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("thresholds-demo")
X, y = load_breast_cancer(return_X_y=True, as_frame=True)
y = y == 0 # True when the tumour is malignant
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
with mlflow.start_run(run_name="check-thresholds"):
info = mlflow.sklearn.log_model(model, name="model", input_example=X_train.head(3))
result = mlflow.models.evaluate(
info.model_uri,
X_test.assign(malignant=y_test),
targets="malignant",
model_type="classifier",
)
thresholds = {
"accuracy_score": MetricThreshold(threshold=0.95, greater_is_better=True),
"recall_score": MetricThreshold(threshold=0.95, greater_is_better=True),
}
try:
mlflow.validate_evaluation_results(
candidate_result=result,
validation_thresholds=thresholds,
)
print("Model meets all thresholds")
except ModelValidationFailedException as e:
print(f"Validation failed: {e}")
uv run python thresholds.py
🏃 View run check-thresholds at: http://127.0.0.1:8080/#/experiments/10/runs/a52a5f6f76c14d6cb924e1fb8d7ca0c6
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/10
2026/11/21 17:54:14 INFO mlflow.models.evaluation.validation: Validating candidate model metrics against baseline
Validation failed: Metric recall_score value threshold check failed: candidate model recall_score = 0.9433962264150944, recall_score threshold = 0.95.

Accuracy, 0.958, cleared 0.95, so the message names only recall: 0.9434, below 0.95. MLflow's line says "against baseline" even without a baseline; it checks the thresholds on their own here.
Step 2: Lower the Recall Threshold
In thresholds.py, change the recall threshold to 0.90:
"recall_score": MetricThreshold(threshold=0.90, greater_is_better=True),
uv run python thresholds.py
🏃 View run check-thresholds at: http://127.0.0.1:8080/#/experiments/10/runs/7a3571e389d8487db14624b1e1c5002b
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/10
2026/11/21 18:35:55 INFO mlflow.models.evaluation.validation: Validating candidate model metrics against baseline
2026/11/21 18:35:55 INFO mlflow.models.evaluation.validation: Model validation passed!
Model meets all thresholds

Both thresholds met: MLflow logs Model validation passed! and returns without raising.
Common Mistakes
| Mistake | What happens | Do this instead |
|---|---|---|
except mlflow.exceptions.ModelValidationFailedException | AttributeError the first time a model fails | Import it from mlflow.models.evaluation.validation |
greater_is_better=True for an error or a loss | A model passes when its error is too high | greater_is_better=False |
A plain number instead of a MetricThreshold | MlflowException: The validation thresholds argument must be a dictionary that maps strings to MetricThreshold objects. | Wrap each value in MetricThreshold(...) |
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
Metric ... value threshold check failed | The model missed that threshold | The message gives the score and the threshold |
Metric validation failed: metric ... was missing from the evaluation result of the candidate model | A metric name the evaluation didn't log | Use a name from result.metrics |
FAQ
Can a custom metric have a threshold?
Yes: use the name you gave it in make_metric: How to Add a Custom Metric to an MLflow Evaluation with make_metric.
How do I make a script fail when a threshold is missed?
Return a non-zero exit code from the except block: How to Make a Python Script Exit with an Error Code.
Conclusion
MetricThreshold(threshold=0.95, greater_is_better=True) on recall made mlflow.validate_evaluation_results raise ModelValidationFailedException, naming recall 0.9434; at 0.90 the same model passed.
Additional Resources
- MetricThreshold —
threshold,min_absolute_changeandmin_relative_change. - mlflow.validate_evaluation_results — the check itself.
