Skip to main content

How to Keep the Class Balance in a Train-Test Split with stratify

· 5 min read
Jagdish Kumawat
Founder @ Dewiride

When only 1 row in 10 is a "yes", a random split can put 3 of them in the test set, or none at all, and the test score then says more about luck than about the model. train_test_split(..., stratify=y) splits so that both parts keep the same share of each class. This guide splits 100 rows with 10 yeses five times each way, with real output from scikit-learn 1.9.1.

Syntax​

Syntax
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=share, random_state=seed, stratify=y
)
PartWhat it means
stratify=ySplit so that each class keeps its share in both parts. Pass the labels, usually the same y
test_size=shareThe share of rows that goes to the test set, such as 0.2
random_state=seedMakes the split repeatable

How to Split Data into Training and Test Sets with scikit-learn covers the rest of train_test_split.

Step 1: Compare Plain and Stratified Splits​

The example is 100 rows, 10 labelled 1 and 90 labelled 0. A 20% test set should hold 2 of the 1s. Create stratify_split.py, which splits with five different seeds, without and with stratify, and counts the 1s in each test set:

The stratify_split.py script in a macOS-style code window: 10 labels of 1 and 90 of 0, and a loop over seeds 0 to 4 that splits 100 rows 80/20 once plainly and once with stratify set to the labels, printing how many 1s each test set got.

stratify_split.py
from sklearn.model_selection import train_test_split

labels = [1] * 10 + [0] * 90 # 10 "yes" rows out of 100
rows = list(range(100))

print("seed yes in plain test yes in stratified test")
for seed in range(5):
plain = train_test_split(rows, labels, test_size=0.2, random_state=seed)[3]
kept = train_test_split(rows, labels, test_size=0.2, random_state=seed, stratify=labels)[3]
print(f"{seed:>4} {sum(plain):>17} {sum(kept):>22}")

[3] takes the fourth result of train_test_split, the test labels; their sum is the number of 1s.

Terminal
uv run python stratify_split.py
Output
seed yes in plain test yes in stratified test
0 3 2
1 0 2
2 3 2
3 2 2
4 1 2

A terminal running uv run python stratify_split.py: a table of seeds 0 to 4 where the plain test set got 3, 0, 3, 2 and 1 yes rows and the stratified test set got 2 every time.

Step 2: Read the Comparison​

  • Plain splits gave the test set 3, 0, 3, 2 and 1 yeses. With seed 1, the test set had none: no score on it could tell whether the model finds yeses at all.
  • Stratified splits gave exactly 2 every time: the same 10% as the whole data.

The rarer a class, the more this matters: with only 10 yeses, one or two rows more or less in the test set change its share of yeses a lot.

Common Mistakes​

MistakeWhat happensDo this instead
Stratifying a regression target, such as pricesAlmost every value is its own "class", and the split failsUse stratify for class labels only
A class with a single rowValueError: The least populated classes in y have only 1 member, which is too few. The minimum number of groups for any class cannot be less than 2.Collect more rows of that class, or merge it with another
Splitting again later without stratifyA validation split can lose the balance the first split keptPass stratify to every split

Troubleshooting​

SymptomLikely causeFix
ValueError: The least populated classes in y have only 1 member...One class has a single rowSee the row above
Test scores jump between seedsFew rows of a rare class, split unevenlystratify=y

FAQ​

Does cross-validation stratify too?​

For classifiers, yes: cross_val_score and GridSearchCV with cv=5 use StratifiedKFold, which keeps the balance in every fold.

Should I always stratify classification splits?​

It never hurts a classification split, and it matters most when a class is rare.

Conclusion​

Over five seeds, plain splits put between 0 and 3 of the 10 yeses into the test set, while stratify=labels put exactly 2 every time. Pass stratify=y whenever you split class labels, and especially when one class is rare.

Additional Resources​