Skip to main content

How to Evaluate Predictions You Already Have with MLflow

· 6 min read
Jagdish Kumawat
Founder @ Dewiride

Sometimes the predictions already exist: a model that runs somewhere else, an old system you want to compare with, or a spreadsheet of answers. mlflow.models.evaluate can score those without a model. Put the predictions and the true answers side by side in a DataFrame, and MLflow computes the metrics and logs them to a run. This guide scores a spam filter's answers, with real output from MLflow 3.18.0.

Syntax​

Syntax
result = mlflow.models.evaluate(
data=frame,
predictions="prediction_column",
targets="target_column",
model_type="classifier",
)
PartWhat it means
dataA pandas DataFrame holding the predictions and the true answers
predictionsThe name of the column of predictions
targetsThe name of the column of true answers
model_type"classifier" or "regressor", which picks the metrics

No model argument. The reference for mlflow.models.evaluate: if model is omitted, "a static dataset will be used for evaluation instead of a model. In this case, the data argument must be a Pandas DataFrame or an mlflow PandasDataset that contains model outputs, and the predictions argument must be the name of the column in data that contains model outputs."

Prerequisites​

  • An MLflow tracking server at http://127.0.0.1:8080. MLflow's tracking server guide shows how to start one.
  • mlflow and pandas in your project, for example with uv add mlflow scikit-learn pandas.

Step 1: Evaluate the Predictions​

Ten emails: what a spam filter said about each one, and what each one really was:

The evaluate_predictions.py script in a macOS-style code window: a DataFrame of ten emails with predicted_spam and is_spam columns of True and False, mlflow.models.evaluate with data, predictions predicted_spam, targets is_spam and model_type classifier inside a run, and every metric printed.

evaluate_predictions.py
import mlflow
import pandas as pd

mlflow.set_tracking_uri("http://127.0.0.1:8080")
mlflow.set_experiment("evaluate-predictions-demo")

# Ten emails: what a spam filter said about each one, and what each one really was.
emails = pd.DataFrame(
{
"predicted_spam": [True, False, True, True, False, False, True, False, False, True],
"is_spam": [True, False, False, True, False, True, True, False, False, True],
}
)

with mlflow.start_run(run_name="spam-filter-predictions"):
result = mlflow.models.evaluate(
data=emails,
predictions="predicted_spam",
targets="is_spam",
model_type="classifier",
)

for name, value in result.metrics.items():
print(f"{name}: {value:.4f}")
Terminal
uv run python evaluate_predictions.py
Output
2026/11/21 17:52:05 INFO mlflow.tracking.fluent: Experiment with name 'evaluate-predictions-demo' does not exist. Creating a new experiment.
2026/11/21 17:52:07 INFO mlflow.models.evaluation.evaluators.classifier: The evaluation dataset is inferred as binary dataset, positive label is True, negative label is False.
2026/11/21 17:52:07 INFO mlflow.models.evaluation.default_evaluator: Testing metrics on first row...
2026/11/21 17:52:07 WARNING mlflow.models.evaluation.evaluators.shap: SHAP or matplotlib package is not installed, so model explainability insights will not be logged.
🏃 View run spam-filter-predictions at: http://127.0.0.1:8080/#/experiments/7/runs/12cd66abb5c041ed949ba6dfa3e0292d
🧪 View experiment at: http://127.0.0.1:8080/#/experiments/7
true_negatives: 4.0000
false_positives: 1.0000
false_negatives: 1.0000
true_positives: 4.0000
example_count: 10.0000
accuracy_score: 0.8000
recall_score: 0.8000
precision_score: 0.8000
f1_score: 0.8000

The output in a terminal: the evaluation's INFO lines, the SHAP warning, the run's link, and nine metrics: 4 true negatives, 1 false positive, 1 false negative, 4 true positives, 10 examples, and accuracy, recall, precision and F1 all 0.8.

Step 2: See What a Table of Answers Can and Can't Give​

The filter was right on 8 of the 10 emails: it caught 4 of the 5 spam emails and wrongly flagged 1 of the 5 others, so accuracy, recall, precision and F1 are all 0.8.

Nine metrics, not the 13 a model gets, and on the run one plot, confusion_matrix.png, not five. ROC AUC, the precision-recall AUC, log loss and the curves all need the probability behind each answer, and a column of True and False doesn't have it.

The evaluation guide suggests you "include a column with probabilities for metrics like ROC-AUC". In MLflow 3.18.0 that column isn't read: I added one, prediction_proba, and the run got the same nine metrics. For those metrics, evaluate the model itself, which hands MLflow its probabilities. I reported the guide's error to MLflow as issue #28014.

The WARNING line shows MLflow's SHAP step runs even here; Fixed: SHAP or matplotlib package is not installed covers it.

Common Mistakes​

MistakeWhat happensDo this instead
Expecting ROC AUC from answers aloneIt isn't computed, even with a probability column addedEvaluate the model itself: How to Evaluate a Classification Model with mlflow.models.evaluate

Troubleshooting​

SymptomLikely causeFix
Only nine metrics and one plotPredictions without probabilitiesExpected for answers alone
SHAP or matplotlib package is not installedNo shap packageFixed: SHAP or matplotlib package is not installed

FAQ​

Can I compare these answers with a model's?​

Yes: evaluate both on the same rows, then compare the metrics, or run a quality gate on them: How to Compare a Model with a Baseline in MLflow.

Does it work for numbers too?​

The evaluation guide's Dataset Evaluation section lists model_type as "classifier" or "regressor" for this kind of evaluation, so predictions of numbers take model_type="regressor".

Conclusion​

mlflow.models.evaluate(data=emails, predictions="predicted_spam", targets="is_spam", model_type="classifier") scored ten answers without a model: 4 of 5 spam emails caught, 1 false alarm, 0.8 on every rate, and a confusion matrix on the run.

Additional Resources​