How to Train a Gradient Boosting Classifier with HistGradientBoostingClassifier
Gradient boosting builds many small decision trees one after another, each correcting the mistakes of the trees before it, and it's among the strongest models for tables of data. HistGradientBoostingClassifier is scikit-learn's fast version: it groups each column's values into bins first, so it trains quickly even on large tables. This guide trains it on a medical yes/no problem, with real output from scikit-learn 1.9.1.
Syntax
model = HistGradientBoostingClassifier(learning_rate=0.1, max_leaf_nodes=31, random_state=seed)
model.fit(X_train, y_train)
| Part | What it means |
|---|---|
learning_rate | How much each new tree corrects the trees before it. Smaller is slower but often more careful. Default 0.1 |
max_leaf_nodes | The most leaves each tree may have: how detailed one tree can get. Default 31 |
max_iter (not shown) | The most trees to build. Default 100 |
random_state | Makes the run repeatable where randomness is involved, such as early stopping's validation split |
Step 1: Train and Score a Model
The example is scikit-learn's breast cancer dataset: 569 tumors described by 30 measurements, each labelled as malignant (0) or benign (1). Create hist_gradient_boosting.py:

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=0, stratify=y
)
model = HistGradientBoostingClassifier(random_state=0)
model.fit(X_train, y_train)
print("Trees built: ", model.n_iter_)
print("Test accuracy:", round(model.score(X_test, y_test), 4))
print("First 3 probabilities:", model.predict_proba(X_test[:3]).round(3).tolist())
uv run python hist_gradient_boosting.py
Trees built: 100
Test accuracy: 0.9371
First 3 probabilities: [[0.0, 1.0], [1.0, 0.0], [1.0, 0.0]]

Step 2: Read the Output
n_iter_, 100, is the number of boosting rounds it ran: all 100 allowed bymax_iter. Early stopping is off for this small dataset: its default,"auto", only turns it on when there are more than 10,000 rows.- Test accuracy, 0.9371, is the share of the 143 test tumors classified correctly.
predict_probagives each row's probability of each class, in the order ofmodel.classes_, here 0 then 1. The first tumor is benign with a probability that rounds to 1.0.
Step 3: Know Its Two Habits
- It handles missing values itself. scikit-learn's documentation: "This estimator has native support for missing values (NaNs)". Each split learns which way rows with a missing value should go, so you don't have to fill the gaps first.
- It needs dense input. A sparse matrix, such as the default output of
OneHotEncoder, fails: setsparse_output=Falseon the encoder in front of it.
Common Mistakes
| Mistake | What happens | Do this instead |
|---|---|---|
| Giving it a sparse matrix | TypeError: Sparse data was passed for X, but dense data is required. Use '.toarray()' to convert to a dense numpy array. | OneHotEncoder(sparse_output=False), or .toarray() |
| Scaling the columns first | Trees don't care about scale, so it adds a step for nothing | Leave numbers as they are |
| Filling missing values with 0 out of habit | Real zeros and gaps look the same to the model | Leave them as NaN and let the model learn from them |
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
n_iter_ is lower than max_iter | Early stopping ended training: the default turns it on above 10,000 rows | Expected; early_stopping=False to always build max_iter trees |
| Results change from run to run | Early stopping's validation split is random | Set random_state |
FAQ
How is it different from GradientBoostingClassifier?
Same idea, different speed: the histogram version bins the values first and adds missing-value support. Its documentation says it "is much faster than GradientBoostingClassifier for big datasets (n_samples >= 10 000)".
Which settings should I tune first?
learning_rate and max_leaf_nodes have the most effect: How to Tune Hyperparameters with GridSearchCV in scikit-learn shows how.
Conclusion
HistGradientBoostingClassifier(random_state=0) built 100 trees and classified 93.71% of the test tumors correctly. Feed it dense data, leave missing values as NaN, and expect early stopping to cut training short on tables above 10,000 rows.
Additional Resources
- HistGradientBoostingClassifier — its settings, early stopping and missing values.
