Skip to main content

How to Train a Gradient Boosting Classifier with HistGradientBoostingClassifier

· 5 min read
Jagdish Kumawat
Founder @ Dewiride

Gradient boosting builds many small decision trees one after another, each correcting the mistakes of the trees before it, and it's among the strongest models for tables of data. HistGradientBoostingClassifier is scikit-learn's fast version: it groups each column's values into bins first, so it trains quickly even on large tables. This guide trains it on a medical yes/no problem, with real output from scikit-learn 1.9.1.

Syntax​

Syntax
model = HistGradientBoostingClassifier(learning_rate=0.1, max_leaf_nodes=31, random_state=seed)
model.fit(X_train, y_train)
PartWhat it means
learning_rateHow much each new tree corrects the trees before it. Smaller is slower but often more careful. Default 0.1
max_leaf_nodesThe most leaves each tree may have: how detailed one tree can get. Default 31
max_iter (not shown)The most trees to build. Default 100
random_stateMakes the run repeatable where randomness is involved, such as early stopping's validation split

Step 1: Train and Score a Model​

The example is scikit-learn's breast cancer dataset: 569 tumors described by 30 measurements, each labelled as malignant (0) or benign (1). Create hist_gradient_boosting.py:

The hist_gradient_boosting.py script in a macOS-style code window: it loads the breast cancer data, splits it 75/25 with stratify, fits a HistGradientBoostingClassifier with random_state 0, and prints n_iter_, the rounded test accuracy and the first three predicted probabilities.

hist_gradient_boosting.py
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=0, stratify=y
)

model = HistGradientBoostingClassifier(random_state=0)
model.fit(X_train, y_train)

print("Trees built: ", model.n_iter_)
print("Test accuracy:", round(model.score(X_test, y_test), 4))
print("First 3 probabilities:", model.predict_proba(X_test[:3]).round(3).tolist())
Terminal
uv run python hist_gradient_boosting.py
Output
Trees built: 100
Test accuracy: 0.9371
First 3 probabilities: [[0.0, 1.0], [1.0, 0.0], [1.0, 0.0]]

A terminal running uv run python hist_gradient_boosting.py: Trees built 100, Test accuracy 0.9371, and the first three probabilities, 0.0 and 1.0, then 1.0 and 0.0 twice.

Step 2: Read the Output​

  • n_iter_, 100, is the number of boosting rounds it ran: all 100 allowed by max_iter. Early stopping is off for this small dataset: its default, "auto", only turns it on when there are more than 10,000 rows.
  • Test accuracy, 0.9371, is the share of the 143 test tumors classified correctly.
  • predict_proba gives each row's probability of each class, in the order of model.classes_, here 0 then 1. The first tumor is benign with a probability that rounds to 1.0.

Step 3: Know Its Two Habits​

  • It handles missing values itself. scikit-learn's documentation: "This estimator has native support for missing values (NaNs)". Each split learns which way rows with a missing value should go, so you don't have to fill the gaps first.
  • It needs dense input. A sparse matrix, such as the default output of OneHotEncoder, fails: set sparse_output=False on the encoder in front of it.

Common Mistakes​

MistakeWhat happensDo this instead
Giving it a sparse matrixTypeError: Sparse data was passed for X, but dense data is required. Use '.toarray()' to convert to a dense numpy array.OneHotEncoder(sparse_output=False), or .toarray()
Scaling the columns firstTrees don't care about scale, so it adds a step for nothingLeave numbers as they are
Filling missing values with 0 out of habitReal zeros and gaps look the same to the modelLeave them as NaN and let the model learn from them

Troubleshooting​

SymptomLikely causeFix
n_iter_ is lower than max_iterEarly stopping ended training: the default turns it on above 10,000 rowsExpected; early_stopping=False to always build max_iter trees
Results change from run to runEarly stopping's validation split is randomSet random_state

FAQ​

How is it different from GradientBoostingClassifier?​

Same idea, different speed: the histogram version bins the values first and adds missing-value support. Its documentation says it "is much faster than GradientBoostingClassifier for big datasets (n_samples >= 10 000)".

Which settings should I tune first?​

learning_rate and max_leaf_nodes have the most effect: How to Tune Hyperparameters with GridSearchCV in scikit-learn shows how.

Conclusion​

HistGradientBoostingClassifier(random_state=0) built 100 trees and classified 93.71% of the test tumors correctly. Feed it dense data, leave missing values as NaN, and expect early stopping to cut training short on tables above 10,000 rows.

Additional Resources​