Skip to main content

Fixed: MLflow Autolog's training_recall_score Is the Same as training_accuracy_score

· 6 min read
Jagdish Kumawat
Founder @ Dewiride

On a yes/no model, MLflow's autologging logged a training_recall_score of 0.9044006069802731, the very same number as training_accuracy_score. That's no coincidence: without pos_label, autologging averages precision, recall and F1 over both classes, weighted by how common each is, and weighted recall always equals accuracy. With mlflow.sklearn.autolog(pos_label=1), it logs the scores of the "yes" class instead, which told a very different story: a recall of 0.27.

What I Was Doing​

I was building the term deposit project of the MLflow series: a model that predicts whether a bank customer signs up for a term deposit, 1 for yes and 0 for no. Only about 1 call in 9 ends in a yes. I trained it with a GridSearchCV search and mlflow.sklearn.autolog() on MLflow 3.18.0, then opened the search's run in the MLflow UI.

What I Expected​

Precision and recall that describe how well the model finds the customers who say yes, the ones the bank wants to call.

What I Got​

The run's Metrics (9) table: training_precision_score 0.8893, training_recall_score 0.9044006069802731, training_f1_score 0.8851, training_accuracy_score 0.9044006069802731, training_log_loss 0.2624, training_roc_auc 0.8278, training_score 0.8278, best_cv_score 0.7998 and test_roc_auc 0.8145.

  • training_recall_score was 0.9044006069802731, identical to training_accuracy_score down to the last digit.
  • training_precision_score, 0.8893, and training_f1_score, 0.8851, were close to it.

A model that finds 90% of the yeses, with a ROC AUC of only 0.83, didn't add up.

What I Checked​

MLflow's reference. mlflow.sklearn.autolog has a pos_label parameter: "If given, used as the positive label to compute binary classification training metrics such as precision, recall, f1, etc." It doesn't say what happens when it isn't given.

MLflow's source. The function that computes these metrics, _get_classifier_metrics in mlflow/sklearn/utils.py, says it: "By default, when pos_label is not specified (passed in as None), we set average to weighted to compute the weighted score of these metrics. When the pos_label is specified (not None), we set average to binary."

What weighted means. scikit-learn computes the score for each class, here once for "no" and once for "yes", then averages them, weighting each by its number of rows. With 89% of the rows being "no", the average is mostly the "no" class's score.

Why It Happens​

For recall, the weighted average is more than close to accuracy: it's equal. Each class's recall is the share of its rows predicted correctly. Weighting it by the class's share of all rows turns it back into a count of correct predictions over all rows, and adding the classes gives correct predictions over all rows: accuracy. So training_recall_score was accuracy under another name, and precision and F1 were dominated by the easy "no" class.

The Fix​

Tell autologging which class is the positive one. In the project's tune.py, the answers are 1 for yes, so:

The fixed autolog call in tune.py in a macOS-style code window: a comment that MLflow records a child run for each of the best candidates and the precision, recall and F1 of the yes answers, 1, and that the search doesn't save the models it tries, then mlflow.sklearn.autolog with max_tuning_runs, pos_label 1 and log_models False.

src/term_deposit_prediction/tune.py (excerpt)
# Let MLflow record the search: a child run for each of the best candidates, and the
# precision, recall and F1 of the "yes" answers (1). The search only picks the settings,
# so it doesn't save the models it tries.
mlflow.sklearn.autolog(max_tuning_runs=args.max_tuning_runs, pos_label=1, log_models=False)

After the Fix​

The same search, with pos_label=1:

The run's Metrics (9) table after the fix: training_precision_score 0.6949, training_recall_score 0.2699, training_f1_score 0.3888, training_accuracy_score still 0.9044006069802731, and the other metrics unchanged.

MetricWeighted (before)Yes class, pos_label=1 (after)
training_precision_score0.88930.6949
training_recall_score0.90440.2699
training_f1_score0.88510.3888
training_accuracy_score0.90440.9044

Accuracy and the other metrics didn't change; precision, recall and F1 now describe the yes class. And they tell the real story: when the model says yes, it's right 69% of the time, but it says yes for only 27% of the customers who did sign up, at the default cut-off of 0.5. Weighted, that 27% was hidden behind the 98% of noes the model got right.

Common Mistakes​

MistakeWhat happensDo this instead
Reading weighted recall on an imbalanced problem as the positive class's recallA rare class's poor recall hides behind the common classSet pos_label
Setting pos_label for a model that isn't a yes/no modelMLflow's reference: "If used for multi-label model, the training metrics calculation will fail and the training metrics won't be logged"Only set it for yes/no models

Troubleshooting​

SymptomLikely causeFix
training_recall_score equals training_accuracy_scoreWeighted averages, pos_label not setmlflow.sklearn.autolog(pos_label=1)
Precision and recall look too good on a rare classThe common class dominates the weighted averageSame fix

FAQ​

Which metric should I trust on an imbalanced problem?​

ROC AUC judges the ranking, whatever the cut-off; precision and recall of the positive class judge the yes/no answers at your cut-off. Accuracy flatters any model that mostly answers with the common class.

Does pos_label change the model?​

No. It only changes how MLflow computes the logged precision, recall and F1. The model and its predictions are the same.

Conclusion​

training_recall_score equaled training_accuracy_score because autologging averages recall over both classes when pos_label isn't set, and weighted recall is accuracy. mlflow.sklearn.autolog(pos_label=1) made it log the yes class's precision, recall and F1, 0.6949, 0.2699 and 0.3888, which showed how few of the yeses the model finds at the default cut-off.

Additional Resources​