Skip to main content

How to Calculate Log Loss in scikit-learn

· 4 min read
Jagdish Kumawat
Founder @ Dewiride

Accuracy, precision and recall only look at a classifier's final answers. Log loss scores the probabilities behind them: it's small when the model is confident and right, and it grows fast when the model is confident and wrong. This guide scores three sets of probabilities for the same four cases, with real output from scikit-learn 1.9.1.

Syntax​

Syntax
from sklearn.metrics import log_loss

loss = log_loss(y_true, y_proba)
PartWhat it means
y_trueThe true classes
y_probaThe probabilities the model gave each class, such as the output of predict_proba: one row per case, one column per class
lossThe average loss per case: 0 is perfect, lower is better

The reference for log_loss describes y_proba as "Predicted probabilities, as returned by a classifier’s predict_proba method", and notes "Added in version 1.9: y_pred was renamed to y_proba."

Step 1: Score Three Sets of Probabilities​

Four cases, two of each class, and the probabilities from three imaginary models: one confident and right, one that always says 50/50, and one confident and wrong:

The log_loss_demo.py script in a macOS-style code window: four true labels, and three models' probabilities as pairs of chance of 0 and chance of 1, with the log loss of each printed.

log_loss_demo.py
from sklearn.metrics import log_loss

# Four cases, and three models' probabilities for them: [chance of 0, chance of 1].
actual = [1, 0, 1, 0]
models = {
"confident and right": [[0.1, 0.9], [0.9, 0.1], [0.2, 0.8], [0.8, 0.2]],
"unsure": [[0.5, 0.5], [0.5, 0.5], [0.5, 0.5], [0.5, 0.5]],
"confident and wrong": [[0.9, 0.1], [0.1, 0.9], [0.8, 0.2], [0.2, 0.8]],
}
for name, probabilities in models.items():
print(f"{name:<20} log loss = {log_loss(actual, probabilities):.4f}")
Terminal
uv run python log_loss_demo.py
Output
confident and right log loss = 0.1643
unsure log loss = 0.6931
confident and wrong log loss = 1.9560

The output of log_loss_demo.py in a terminal: confident and right 0.1643, unsure 0.6931, confident and wrong 1.9560.

  • 0.1643 for the confident, right model.
  • 0.6931 for the model that always says 50/50: that's the natural log of 2, the score of a coin toss on two classes, a useful line to compare against.
  • 1.9560 for the confident, wrong model: twelve times the first, for probabilities that are just as confident.

Common Mistakes​

MistakeWhat happensDo this instead
Passing the predicted classes instead of the probabilitiesEach answer counts as 100% sure: one mistake in four cases gave a loss of 9.01Pass predict_proba's output
log_loss(y_true=..., y_pred=...)FutureWarning: y_pred was renamed to y_proba in version 1.9 and will be removed in 1.11Pass the probabilities second, or as y_proba=

Troubleshooting​

SymptomLikely causeFix
FutureWarning: y_pred was renamed to y_proba in version 1.9 and will be removed in 1.11. Use y_proba instead.The old keyword namey_proba=
A loss far above 0.69Confident wrong answers, or classes passed instead of probabilitiesLook at the cases with the biggest error

FAQ​

Is lower log loss always better?​

Yes: 0 means the model gave every true class a probability of 1. Compare it with 0.69, the coin toss on two classes, to see whether a model's probabilities are worth anything.

Why use log loss if I already have accuracy?​

Accuracy can't tell a 51% guess from a 99% one. Log loss can, which matters when you use the probabilities themselves, for example to rank cases or to estimate how many will happen.

Conclusion​

log_loss gave 0.1643 to confident right probabilities, 0.6931 to a coin toss and 1.9560 to confident wrong ones: it rewards probabilities that are both sure and right.

Additional Resources​