Skip to main content

How to Calculate ROC AUC in scikit-learn

· 5 min read
Jagdish Kumawat
Founder @ Dewiride

ROC AUC measures how well a model ranks: the chance that it gives a randomly picked positive row a higher score than a randomly picked negative one. 1.0 is a perfect ranking and 0.5 is no better than a coin. Because it judges the ranking, not a yes/no cut-off, it suits problems where positives are rare. scikit-learn computes it with roc_auc_score; this guide scores a tiny example and a real model, with real output from scikit-learn 1.9.1.

Syntax​

Syntax
score = roc_auc_score(y_true, y_score)
PartWhat it means
y_trueThe true labels, such as 0 and 1
y_scoreOne score per row, higher meaning "more likely positive". For a scikit-learn classifier, the probability of the positive class: model.predict_proba(X)[:, 1]

For a yes/no problem, the documentation says y_score "corresponds to an array of shape (n_samples,)" whose probability estimates are those "of the class with the greater label": column 1 of predict_proba.

Step 1: Score a Tiny Example​

Four rows, two of each class, with the scores a model might give them. Create roc_auc.py:

The roc_auc.py script in a macOS-style code window: y_true is 0, 0, 1, 1, y_score is 0.1, 0.4, 0.35, 0.8, and it prints roc_auc_score of the two.

roc_auc.py
from sklearn.metrics import roc_auc_score

y_true = [0, 0, 1, 1]
y_score = [0.1, 0.4, 0.35, 0.8]

print(roc_auc_score(y_true, y_score))
Terminal
uv run python roc_auc.py
Output
0.75

There are 2 × 2 = 4 positive-negative pairs. The model ranked three of them right, and one wrong: the positive scored 0.35, below the negative's 0.4. Three out of four is 0.75.

Step 2: Score a Real Model​

Create roc_auc_model.py, which trains a gradient boosting classifier on scikit-learn's breast cancer data and scores its test set twice: from probabilities, and from the 0/1 labels predict returns:

The roc_auc_model.py script in a macOS-style code window: it splits the breast cancer data 75/25 with stratify, fits a HistGradientBoostingClassifier, and prints roc_auc_score from predict_proba column 1 and from predict.

roc_auc_model.py
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=0, stratify=y
)
model = HistGradientBoostingClassifier(random_state=0).fit(X_train, y_train)

print("From probabilities:", round(roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]), 4))
print("From labels: ", round(roc_auc_score(y_test, model.predict(X_test)), 4))
Terminal
uv run python roc_auc_model.py
Output
From probabilities: 0.9866
From labels: 0.9306

A terminal running uv run python roc_auc_model.py: From probabilities 0.9866 and From labels 0.9306.

From probabilities, 0.9866. From labels, 0.9306: with only 0 and 1 as scores, many positive-negative pairs get the same score, each such tie counts as half right, and the finer ranking the probabilities held is lost. Always pass probabilities.

Common Mistakes​

MistakeWhat happensDo this instead
Passing model.predict(X)A lower score that ignores the ranking: 0.9306 instead of 0.9866 heremodel.predict_proba(X)[:, 1]
Passing all of predict_proba(X)ValueError: y should be a 1d array, got an array of shape (169, 2) instead.Take column 1: [:, 1]
A test set with only one classUndefinedMetricWarning: Only one class is present in y_true. ROC AUC score is not defined in that case., and the result is nanSplit with stratify=y so both classes are present

Troubleshooting​

SymptomLikely causeFix
The score is nanOnly one class in y_trueCheck the labels; split with stratify
The score is below 0.5The scores point the wrong way: column 0 instead of 1, or a "lower is more likely" scoreUse column 1 of predict_proba

FAQ​

Is a high ROC AUC enough?​

It says the model ranks well, not that its yes/no answers are good: those depend on the cut-off you choose. Look at precision and recall at your cut-off too.

Can GridSearchCV tune for it?​

Yes: GridSearchCV(..., scoring="roc_auc"): How to Tune Hyperparameters with GridSearchCV in scikit-learn.

Conclusion​

roc_auc_score([0, 0, 1, 1], [0.1, 0.4, 0.35, 0.8]) returned 0.75, three of four pairs ranked right, and a real model scored 0.9866 from its probabilities but only 0.9306 from its labels. Pass column 1 of predict_proba, and make sure the test set holds both classes.

Additional Resources​