Skip to main content

How to Calculate Balanced Accuracy in scikit-learn

· 4 min read
Jagdish Kumawat
Founder @ Dewiride

When one class is rare, accuracy flatters a model: if 8 emails in 10 are fine, a filter that never flags anything is still 80% accurate. Balanced accuracy averages how well the model does on each class, so the common class can't hide mistakes on the rare one. This guide compares the two on two spam filters, with real output from scikit-learn 1.9.1.

Syntax​

Syntax
from sklearn.metrics import balanced_accuracy_score

score = balanced_accuracy_score(y_true, y_pred)
PartWhat it means
y_trueThe true classes
y_predThe classes the model predicted, in the same order
scoreThe average of the recall of each class: 1 is perfect, 0.5 is what a model that always gives the same answer gets on two classes

scikit-learn's user guide defines it: "It is the macro-average of recall scores per class", and "for balanced datasets, the score is equal to accuracy."

Step 1: Compare Accuracy and Balanced Accuracy​

Ten emails, two of them spam (1), and two filters: one that catches one of the two spam emails, and one that never flags anything:

The balanced_accuracy.py script in a macOS-style code window: ten true labels with two spam, a spam filter that catches one of them, a filter that always says no, and for each the accuracy, the balanced accuracy and the recall per class.

balanced_accuracy.py
from sklearn.metrics import accuracy_score, balanced_accuracy_score, recall_score

# Ten emails, two of them spam (1), and what two filters said about them.
actual = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1]
spam_filter = [0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
always_no = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

for name, predicted in [("spam_filter", spam_filter), ("always_no", always_no)]:
print(name)
print(" accuracy: ", accuracy_score(actual, predicted))
print(" balanced accuracy:", balanced_accuracy_score(actual, predicted))
print(" recall per class: ", recall_score(actual, predicted, average=None))
Terminal
uv run python balanced_accuracy.py
Output
spam_filter
accuracy: 0.9
balanced accuracy: 0.75
recall per class: [1. 0.5]
always_no
accuracy: 0.8
balanced accuracy: 0.5
recall per class: [1. 0.]

The output of balanced_accuracy.py in a terminal: spam_filter has accuracy 0.9, balanced accuracy 0.75 and recall per class 1 and 0.5; always_no has accuracy 0.8, balanced accuracy 0.5 and recall per class 1 and 0.

  • spam_filter: 90% accurate, but it found only half the spam. recall_score(..., average=None) gives each class's recall, [1. 0.5], and balanced accuracy is their average, 0.75.
  • always_no: still 80% accurate without catching a single spam email. Balanced accuracy shows it for what it is: 0.5, the score of a model that ignores its input.

Common Mistakes​

MistakeWhat happensDo this instead
Judging a model on accuracy when one class is rareA model that ignores the rare class looks goodLook at balanced accuracy, or each class's recall
Reading balanced accuracy like accuracy0.5 sounds like "half right", but it's the floor of a do-nothing model on two classesCompare it with 0.5, not with 0

Troubleshooting​

SymptomLikely causeFix
UserWarning: A single label was found in 'y_true' and 'y_pred'Only one class in the rows scoredScore more rows; inside MLflow's evaluation, see Fixed: A single label was found

FAQ​

Is balanced accuracy the same as macro-averaged recall?​

Yes, the user guide defines it that way, and recall_score(y_true, y_pred, average="macro") gives the same number.

When is balanced accuracy equal to accuracy?​

When the classes are the same size, as the user guide says.

Conclusion​

On imbalanced data, accuracy rewarded a filter that never flagged spam with 0.8. balanced_accuracy_score, the average of each class's recall, gave it 0.5 and gave the filter that caught half the spam 0.75.

Additional Resources​