Skip to main content

How to Use warm_start to Add Trees to a Random Forest in scikit-learn

· 7 min read
Jagdish Kumawat
Founder @ Dewiride

A random forest normally grows all its trees in one call to fit. With warm_start=True, it keeps the trees it already has, and the next fit adds only the ones you asked for on top. That lets you score the forest at 25, 50, 75 and 100 trees without starting over each time. This guide shows how, with real output from scikit-learn 1.9.1.

Syntax​

Syntax
from sklearn.ensemble import RandomForestRegressor

forest = RandomForestRegressor(warm_start=True, random_state=<seed>)
forest.set_params(n_estimators=<total trees>)
forest.fit(X_train, y_train)
PartWhat it means
warm_start=TrueIn scikit-learn's words, "reuse the solution of the previous call to fit and add more estimators to the ensemble". Without it, every fit grows "a whole new forest"
set_params(n_estimators=<total trees>)How many trees the forest should have after the next fit: the total, not how many to add
forest.fit(X_train, y_train)Grows only the missing trees, then stops
forest.estimators_The list of trees grown so far
random_state=<seed>Makes the trees repeatable, stage after stage

scikit-learn's ensemble guide says "RandomForest, Extra-Trees and RandomTreesEmbedding estimators all support warm_start=True", and its glossary says the option is for fitting "the same estimator with the same data but varying parameters". So keep the training data the same from one stage to the next.

Prerequisites​

Step 1: Grow the Forest in Four Stages​

The example predicts California house values. Create warm_start_forest.py:

The warm_start_forest.py script in a macOS-style code window: it splits the California Housing data, creates a RandomForestRegressor with warm_start=True, raises n_estimators to 25, 50, 75 and 100 and fits each time while printing the validation RMSE, then grows 100 trees in one go and compares the two forests' predictions with np.allclose.

warm_start_forest.py
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import root_mean_squared_error
from sklearn.model_selection import train_test_split

X, y = fetch_california_housing(return_X_y=True, as_frame=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

forest = RandomForestRegressor(warm_start=True, random_state=42, n_jobs=-1)
for n_trees in [25, 50, 75, 100]:
forest.set_params(n_estimators=n_trees)
forest.fit(X_train, y_train)
rmse = root_mean_squared_error(y_val, forest.predict(X_val))
print(f"{len(forest.estimators_):3} trees: validation RMSE {rmse:.4f}")

all_at_once = RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1)
all_at_once.fit(X_train, y_train)
same = np.allclose(forest.predict(X_val), all_at_once.predict(X_val))
print("Same predictions as 100 trees grown at once:", same)
LineWhat it does
RandomForestRegressor(warm_start=True, random_state=42, n_jobs=-1)A forest that keeps its trees between fits. n_jobs=-1 uses every processor core
forest.set_params(n_estimators=n_trees)Raises the target: 25, then 50, 75 and 100 trees
forest.fit(X_train, y_train)The first call grows 25 trees; each later call grows only the next 25
len(forest.estimators_)How many trees the forest has right now
root_mean_squared_error(y_val, ...)Scores the forest on the validation rows after each stage. Lower is better
The last four linesGrow the same 100 trees in a single call and check the two forests predict the same values

Step 2: Run It​

Terminal
uv run python warm_start_forest.py
Output
25 trees: validation RMSE 0.5127
50 trees: validation RMSE 0.5070
75 trees: validation RMSE 0.5083
100 trees: validation RMSE 0.5063
Same predictions as 100 trees grown at once: True

The forest grew by 25 trees at each stage, and the score barely moved after 50 trees: a useful thing to know before paying for 500. Plotted over the stages, these scores are a training curve, which MLflow can record and draw for you.

Step 3: Understand Why It Matches a Forest Grown in One Go​

The staged forest isn't just similar to one grown in a single call; it has the same trees. scikit-learn's source code, in sklearn/ensemble/_forest.py, skips ahead in the random numbers when it adds trees to a warm-started forest, so that each new tree gets "the random state we would have got if we hadn't used a warm_start."

The check uses np.allclose, which allows for the last-digit rounding of decimal numbers, and not np.array_equal. With n_jobs=-1, even the same forest can return predictions that differ in the 15th decimal place from one call to the next: Fixed: scikit-learn Random Forest Predictions Change Slightly Between Calls with n_jobs=-1 shows why.

Common Mistakes​

MistakeWhat happensDo this instead
Calling fit again without raising n_estimatorsUserWarning: Warm-start fitting without increasing n_estimators does not fit new trees. The forest keeps the trees it hadRaise n_estimators before each fit
Setting n_estimators lower than the trees it hasValueError: n_estimators=30 must be larger or equal to len(estimators_)=50 when warm_start==TrueStart a new forest, or keep the total at least as high
Treating n_estimators as "trees to add"set_params(n_estimators=25) four times grows 25 trees, not 100Set the running total: 25, 50, 75, 100
Changing the training data between stagesEarlier trees learned from other rows than the later onesKeep X_train and y_train the same for every stage

Troubleshooting​

SymptomLikely causeFix
The score never changes from stage to stagewarm_start is True but n_estimators isn't going up, so no trees are addedCheck len(forest.estimators_) after each fit
np.array_equal says the staged and one-go predictions differParallel prediction adds the trees' outputs in a different order each timeCompare with np.allclose, or predict with n_jobs=1

FAQ​

Is growing in stages slower than growing all trees at once?​

The forest ends up growing the same 100 trees either way; each stage only adds new ones. What staging adds is the scoring between stages. Training four separate forests of 25, 50, 75 and 100 trees would grow 250 trees instead.

Do other models have warm_start?​

Several scikit-learn models do, and what it reuses differs from model to model. Check each model's reference page before relying on it.

Can I remove trees again?​

Not with warm_start: asking for fewer trees than the forest has raises the ValueError above. Train a new forest with the size you want.

Conclusion​

RandomForestRegressor(warm_start=True) kept its trees between calls to fit, so raising n_estimators from 25 to 100 in steps grew 25 new trees each time and let the script score every stage. The finished forest predicted the same values as 100 trees grown at once, because scikit-learn gives warm-started trees the same random seeds.

Additional Resources​