How to Use warm_start to Add Trees to a Random Forest in scikit-learn
A random forest normally grows all its trees in one call to fit. With warm_start=True, it keeps the trees it already has, and the next fit adds only the ones you asked for on top. That lets you score the forest at 25, 50, 75 and 100 trees without starting over each time. This guide shows how, with real output from scikit-learn 1.9.1.
Syntax
from sklearn.ensemble import RandomForestRegressor
forest = RandomForestRegressor(warm_start=True, random_state=<seed>)
forest.set_params(n_estimators=<total trees>)
forest.fit(X_train, y_train)
| Part | What it means |
|---|---|
warm_start=True | In scikit-learn's words, "reuse the solution of the previous call to fit and add more estimators to the ensemble". Without it, every fit grows "a whole new forest" |
set_params(n_estimators=<total trees>) | How many trees the forest should have after the next fit: the total, not how many to add |
forest.fit(X_train, y_train) | Grows only the missing trees, then stops |
forest.estimators_ | The list of trees grown so far |
random_state=<seed> | Makes the trees repeatable, stage after stage |
scikit-learn's ensemble guide says "RandomForest, Extra-Trees and RandomTreesEmbedding estimators all support warm_start=True", and its glossary says the option is for fitting "the same estimator with the same data but varying parameters". So keep the training data the same from one stage to the next.
Prerequisites
- A uv project with scikit-learn and pandas: see How to Add a Package to a Python Project with uv.
- A random forest you know: How to Train a Random Forest Regression Model with scikit-learn explains
RandomForestRegressorand its options. - A validation set to score each stage on, kept apart from the training rows: How to Split Data into Training, Validation and Test Sets with scikit-learn.
Step 1: Grow the Forest in Four Stages
The example predicts California house values. Create warm_start_forest.py:

import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import root_mean_squared_error
from sklearn.model_selection import train_test_split
X, y = fetch_california_housing(return_X_y=True, as_frame=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
forest = RandomForestRegressor(warm_start=True, random_state=42, n_jobs=-1)
for n_trees in [25, 50, 75, 100]:
forest.set_params(n_estimators=n_trees)
forest.fit(X_train, y_train)
rmse = root_mean_squared_error(y_val, forest.predict(X_val))
print(f"{len(forest.estimators_):3} trees: validation RMSE {rmse:.4f}")
all_at_once = RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1)
all_at_once.fit(X_train, y_train)
same = np.allclose(forest.predict(X_val), all_at_once.predict(X_val))
print("Same predictions as 100 trees grown at once:", same)
| Line | What it does |
|---|---|
RandomForestRegressor(warm_start=True, random_state=42, n_jobs=-1) | A forest that keeps its trees between fits. n_jobs=-1 uses every processor core |
forest.set_params(n_estimators=n_trees) | Raises the target: 25, then 50, 75 and 100 trees |
forest.fit(X_train, y_train) | The first call grows 25 trees; each later call grows only the next 25 |
len(forest.estimators_) | How many trees the forest has right now |
root_mean_squared_error(y_val, ...) | Scores the forest on the validation rows after each stage. Lower is better |
| The last four lines | Grow the same 100 trees in a single call and check the two forests predict the same values |
Step 2: Run It
uv run python warm_start_forest.py
25 trees: validation RMSE 0.5127
50 trees: validation RMSE 0.5070
75 trees: validation RMSE 0.5083
100 trees: validation RMSE 0.5063
Same predictions as 100 trees grown at once: True
The forest grew by 25 trees at each stage, and the score barely moved after 50 trees: a useful thing to know before paying for 500. Plotted over the stages, these scores are a training curve, which MLflow can record and draw for you.
Step 3: Understand Why It Matches a Forest Grown in One Go
The staged forest isn't just similar to one grown in a single call; it has the same trees. scikit-learn's source code, in sklearn/ensemble/_forest.py, skips ahead in the random numbers when it adds trees to a warm-started forest, so that each new tree gets "the random state we would have got if we hadn't used a warm_start."
The check uses np.allclose, which allows for the last-digit rounding of decimal numbers, and not np.array_equal. With n_jobs=-1, even the same forest can return predictions that differ in the 15th decimal place from one call to the next: Fixed: scikit-learn Random Forest Predictions Change Slightly Between Calls with n_jobs=-1 shows why.
Common Mistakes
| Mistake | What happens | Do this instead |
|---|---|---|
Calling fit again without raising n_estimators | UserWarning: Warm-start fitting without increasing n_estimators does not fit new trees. The forest keeps the trees it had | Raise n_estimators before each fit |
Setting n_estimators lower than the trees it has | ValueError: n_estimators=30 must be larger or equal to len(estimators_)=50 when warm_start==True | Start a new forest, or keep the total at least as high |
Treating n_estimators as "trees to add" | set_params(n_estimators=25) four times grows 25 trees, not 100 | Set the running total: 25, 50, 75, 100 |
| Changing the training data between stages | Earlier trees learned from other rows than the later ones | Keep X_train and y_train the same for every stage |
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| The score never changes from stage to stage | warm_start is True but n_estimators isn't going up, so no trees are added | Check len(forest.estimators_) after each fit |
np.array_equal says the staged and one-go predictions differ | Parallel prediction adds the trees' outputs in a different order each time | Compare with np.allclose, or predict with n_jobs=1 |
FAQ
Is growing in stages slower than growing all trees at once?
The forest ends up growing the same 100 trees either way; each stage only adds new ones. What staging adds is the scoring between stages. Training four separate forests of 25, 50, 75 and 100 trees would grow 250 trees instead.
Do other models have warm_start?
Several scikit-learn models do, and what it reuses differs from model to model. Check each model's reference page before relying on it.
Can I remove trees again?
Not with warm_start: asking for fewer trees than the forest has raises the ValueError above. Train a new forest with the size you want.
Conclusion
RandomForestRegressor(warm_start=True) kept its trees between calls to fit, so raising n_estimators from 25 to 100 in steps grew 25 new trees each time and let the script score every stage. The finished forest predicted the same values as 100 trees grown at once, because scikit-learn gives warm-started trees the same random seeds.
Additional Resources
- RandomForestRegressor — the reference, including
warm_start. - Ensembles: Fitting additional trees — scikit-learn's guide to
warm_startfor forests. - Glossary: warm_start — what the option means across scikit-learn.
Related
- How to Split Data into Training, Validation and Test Sets with scikit-learn
- How to Train a Random Forest Regression Model with scikit-learn
- Fixed: scikit-learn Random Forest Predictions Change Slightly Between Calls with n_jobs=-1
- MLflow Tutorial for Machine Learning in Python — Part 2: Log Metrics Over Time and System Metrics
