Skip to main content

Fixed: "AttributeError: module 'mlflow.exceptions' has no attribute 'ModelValidationFailedException'"

· 6 min read
Jagdish Kumawat
Founder @ Dewiride

I checked a model against minimum scores with mlflow.validate_evaluation_results, catching the failure the way MLflow's evaluation guide does, with except mlflow.exceptions.ModelValidationFailedException. When the model fell short, Python raised AttributeError: module 'mlflow.exceptions' has no attribute 'ModelValidationFailedException' instead of printing my message. In MLflow 3.18.0 the class lives in mlflow.models.evaluation.validation; importing it from there fixed it.

What I Was Doing​

I was writing How to Set Minimum Scores for a Model with mlflow.validate_evaluation_results, and I followed the Model Validation example of MLflow's evaluation guide, which ends with except mlflow.exceptions.ModelValidationFailedException as e:. The model is a logistic regression on scikit-learn's breast cancer data, and the thresholds ask for 0.95 accuracy and 0.95 recall:

thresholds.py
import mlflow
from mlflow.models import MetricThreshold
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("thresholds-demo")

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
y = y == 0 # True when the tumour is malignant
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)

with mlflow.start_run(run_name="check-thresholds"):
info = mlflow.sklearn.log_model(model, name="model", input_example=X_train.head(3))
result = mlflow.models.evaluate(
info.model_uri,
X_test.assign(malignant=y_test),
targets="malignant",
model_type="classifier",
)

thresholds = {
"accuracy_score": MetricThreshold(threshold=0.95, greater_is_better=True),
"recall_score": MetricThreshold(threshold=0.95, greater_is_better=True),
}

try:
mlflow.validate_evaluation_results(
candidate_result=result,
validation_thresholds=thresholds,
)
print("Model meets all thresholds")
except mlflow.exceptions.ModelValidationFailedException as e:
print(f"Validation failed: {e}")
Terminal
uv run python thresholds.py

What I Expected​

Validation failed: and the reason: this model's recall is 0.9434, below 0.95.

What I Got​

Two tracebacks. The first is the validation failure I wanted to catch; the second, raised while Python handled it, is the AttributeError:

Output (excerpt)
2026/11/21 17:53:50 INFO mlflow.models.evaluation.validation: Validating candidate model metrics against baseline
Traceback (most recent call last):
File "/Users/you/source/mlflow-demo/thresholds.py", line 32, in <module>
mlflow.validate_evaluation_results(
...
mlflow.models.evaluation.validation.ModelValidationFailedException: Metric recall_score value threshold check failed: candidate model recall_score = 0.9433962264150944, recall_score threshold = 0.95.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/you/source/mlflow-demo/thresholds.py", line 37, in <module>
except mlflow.exceptions.ModelValidationFailedException as e:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'mlflow.exceptions' has no attribute 'ModelValidationFailedException'

The end of the output in a terminal: mlflow.models.evaluation.validation.ModelValidationFailedException with the recall_score threshold message, then During handling of the above exception, another exception occurred, and AttributeError: module mlflow.exceptions has no attribute ModelValidationFailedException.

What I Checked​

Where the class really is. The first traceback names it in full: mlflow.models.evaluation.validation.ModelValidationFailedException. MLflow 3.18.0's mlflow/exceptions.py defines MlflowException and others, but not this one, and mlflow/models/__init__.py doesn't export it either.

What the reference says. mlflow.validate_evaluation_results says "If the candidate results do not meet the validation thresholds, an ModelValidationFailedException will be raised", without a module. Its signature does name the module of its thresholds: validation_thresholds: dict[str, mlflow.models.evaluation.validation.MetricThreshold].

Why it only failed now. Python looks up the class in an except clause only when an exception reaches that clause. With every threshold met, the script prints Model meets all thresholds and never touches the missing name; the mistake waits for the first model that fails.

Why It Happens​

The guide's example names the exception through mlflow.exceptions, where MLflow 3.18.0 doesn't define it. The class is defined, and raised, in mlflow.models.evaluation.validation, next to MetricThreshold.

The Fix​

Import the class from the module that defines it, and catch it by name:

The fixed thresholds.py in a macOS-style code window: ModelValidationFailedException imported from mlflow.models.evaluation.validation next to MetricThreshold, and the except clause catching ModelValidationFailedException.

thresholds.py (the import and the except clause)
from mlflow.models import MetricThreshold
from mlflow.models.evaluation.validation import ModelValidationFailedException

# ...

try:
mlflow.validate_evaluation_results(
candidate_result=result,
validation_thresholds=thresholds,
)
print("Model meets all thresholds")
except ModelValidationFailedException as e:
print(f"Validation failed: {e}")

After the Fix​

Terminal
uv run python thresholds.py
Output (last lines)
🏃 View run check-thresholds at: http://127.0.0.1:8080/#/experiments/10/runs/f666a5485585448cae6360c910122fc5
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/10
2026/11/21 18:58:12 INFO mlflow.models.evaluation.validation: Validating candidate model metrics against baseline
Validation failed: Metric recall_score value threshold check failed: candidate model recall_score = 0.9433962264150944, recall_score threshold = 0.95.

The end of the fixed script's output in a terminal: Validating candidate model metrics against baseline, then the script's own line, Validation failed: Metric recall_score value threshold check failed, with recall 0.9434 against the threshold 0.95.

The failure is caught, and the script prints which threshold the model missed.

Common Mistakes​

MistakeWhat happensDo this instead
Testing the script only with a model that passesThe except clause never runs, so the mistake stays hiddenAlso run it with a threshold the model misses
except Exception to get past itEvery other error is caught as a failed gate tooCatch ModelValidationFailedException

Troubleshooting​

SymptomLikely causeFix
AttributeError: module 'mlflow.exceptions' has no attribute 'ModelValidationFailedException'The class named through mlflow.exceptionsImport it from mlflow.models.evaluation.validation
MlflowException: The validation thresholds argument must be a dictionary that maps strings to MetricThreshold objects.Thresholds that aren't MetricThreshold objectsWrap each one in MetricThreshold(...)

FAQ​

Can I catch MlflowException instead?​

It works, since ModelValidationFailedException is a subclass of MlflowException, but it also catches MLflow's other errors, such as thresholds of the wrong type. Catching the exact class keeps a failed gate apart from a broken one.

Does the gate still fail without the except clause?​

Yes: validate_evaluation_results raises, and an uncaught exception stops the script with a traceback and exit code 1.

Conclusion​

The evaluation guide's example catches mlflow.exceptions.ModelValidationFailedException, which MLflow 3.18.0 doesn't define, so the except line itself failed the first time a model missed a threshold. Importing ModelValidationFailedException from mlflow.models.evaluation.validation made the failure catchable, and the script printed which threshold was missed. I reported the guide's example to MLflow as issue #28014.

Additional Resources​