Skip to main content

How to Plot a Calibration Curve in scikit-learn

· 5 min read
Jagdish Kumawat
Founder @ Dewiride

A classifier's predict_proba gives numbers between 0 and 1, but are they real chances? If a model says 0.8 for a hundred cases, about eighty of them should turn out positive. A calibration curve checks exactly that. This guide plots one for two logistic regressions, a plain one and one trained with balanced class weights, with real output from scikit-learn 1.9.1.

Syntax​

Syntax
from sklearn.calibration import CalibrationDisplay

CalibrationDisplay.from_estimator(model, X_test, y_test, n_bins=10, name="label", ax=ax)
PartWhat it means
modelA fitted binary classifier with predict_proba
X_test, y_testRows the model didn't train on, and their true classes
n_binsHow many groups to split the predicted probabilities into
name, axThe curve's name in the legend, and the Matplotlib axes to draw on, so several curves can share one plot

scikit-learn's user guide explains what a good model looks like: "a well calibrated (binary) classifier should classify the samples such that among the samples to which it gave a predict_proba value close to, say, 0.8, approximately 80% actually belong to the positive class." On the plot, the x-axis "represents the average predicted probability in each bin", and the y-axis is "the fraction of positives" in it.

Step 1: Plot Two Models​

The example makes 5,000 rows, about 15 in 100 of them positive, and fits a logistic regression twice, once with class_weight="balanced":

The calibration_demo.py script in a macOS-style code window: 5,000 made-up rows with weights 0.85 and 0.15, a plain LogisticRegression and a balanced one, both drawn on the same axes with CalibrationDisplay.from_estimator and saved as calibration_curve.png.

calibration_demo.py
import matplotlib.pyplot as plt
from sklearn.calibration import CalibrationDisplay
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# 5,000 made-up rows; about 15 in 100 belong to class 1.
X, y = make_classification(n_samples=5000, weights=[0.85, 0.15], random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)

plain = LogisticRegression().fit(X_train, y_train)
balanced = LogisticRegression(class_weight="balanced").fit(X_train, y_train)

fig, ax = plt.subplots(figsize=(6, 5))
CalibrationDisplay.from_estimator(plain, X_test, y_test, n_bins=10, name="plain", ax=ax)
CalibrationDisplay.from_estimator(balanced, X_test, y_test, n_bins=10, name="balanced", ax=ax)
fig.savefig("calibration_curve.png")
print("Saved calibration_curve.png")
Terminal
uv run python calibration_demo.py
Output
Saved calibration_curve.png

The saved plot:

The saved calibration_curve.png: a dotted diagonal labelled perfectly calibrated, a blue curve for the plain model that stays near the diagonal, and an orange curve for the balanced model far below it, with the fraction of positives on the y-axis and the mean predicted probability on the x-axis.

  • The dotted diagonal is a perfectly calibrated model: the share of positives equals the probability it predicted.
  • The plain model, in blue, stays close to it: its probabilities can be read as chances.
  • The balanced model, in orange, runs far below it: rows it gave about 0.75 were positive only about 3 times in 10. Balanced class weights push its probabilities up, so they're higher than the real chances.

Common Mistakes​

MistakeWhat happensDo this instead
Plotting on the training rowsThe curve looks better than the model isUse rows the model didn't train on
Reading a balanced-weight model's probabilities as chancesThey overstate how likely a positive isCheck the curve first, and treat them as scores

Troubleshooting​

SymptomLikely causeFix
A jagged curveFew rows in some binsMore test rows, or fewer bins
Each curve in its own figureNo shared axCreate one with plt.subplots() and pass ax=ax to each call

FAQ​

Does ranking quality change with calibration?​

Not necessarily: a model can rank cases well, with a high ROC AUC, and still give probabilities that are too high or too low. The curve checks the probabilities' values, not their order.

Can I fix a badly calibrated model?​

scikit-learn's CalibratedClassifierCV re-calibrates a classifier; its user guide page covers how.

Conclusion​

CalibrationDisplay.from_estimator plotted both models against the diagonal. The plain logistic regression's probabilities followed it; the balanced one's ran well above the real share of positives, which matters whenever its probabilities are read as chances.

Additional Resources​