Skip to main content

How to Split Data into Training, Validation and Test Sets with scikit-learn

· 7 min read
Jagdish Kumawat
Founder @ Dewiride

A test set tells you how good a finished model is, but only if you never use it while building the model. To check the model while you build it, you need a third part of the data, a validation set. scikit-learn has no single function for that, so you call train_test_split twice. This guide shows how, with real output from scikit-learn 1.9.1.

Syntax​

Syntax
from sklearn.model_selection import train_test_split

X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=<test share>, random_state=<seed>)
X_train, X_val, y_train, y_val = train_test_split(
X_rest, y_rest, test_size=<validation share of the rest>, random_state=<seed>
)
PartWhat it means
First callKeeps <test share> of all rows apart as the test set. Everything else is "the rest"
Second callSplits the rest again. Its test_size is the validation set's share of the rest, not of all rows
random_state=<seed>A whole number that makes the shuffle, and so the split, the same every time
What each call returnsThe two parts of X, then the two parts of y, in that order

To turn the share you want of all rows into the second call's test_size, divide by what's left after the test set: validation share ÷ (1 − test share). For a 60/20/20 split, that's 0.20 ÷ 0.80 = 0.25.

train_test_split itself is explained in How to Split Data into Training and Test Sets with scikit-learn.

Prerequisites​

Step 1: Write the Split​

Create three_way_split.py:

The three_way_split.py script in a macOS-style code window: it loads the California Housing data, keeps 20% of all rows for the test set with a first train_test_split, keeps 25% of the remaining rows for the validation set with a second one, then prints each part's size and checks that no row is in two parts.

three_way_split.py
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

X, y = fetch_california_housing(return_X_y=True, as_frame=True)

# 1. Keep 20% of all rows for the final test.
X_rest, X_test, y_rest, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# 2. Keep 25% of the remaining rows (20% of all of them) for validation.
X_train, X_val, y_train, y_val = train_test_split(
X_rest, y_rest, test_size=0.25, random_state=42
)

for name, part in [("Training", X_train), ("Validation", X_val), ("Test", X_test)]:
print(f"{name + ' rows:':17} {len(part):5} ({len(part) / len(X):.0%})")

shared = (set(X_train.index) & set(X_val.index)) | (set(X_train.index) & set(X_test.index))
shared |= set(X_val.index) & set(X_test.index)
print("Rows in more than one part:", len(shared))
LineWhat it does
fetch_california_housing(return_X_y=True, as_frame=True)Loads the features as a table, X, and the house values as a column, y
First train_test_splitTakes 20% of all rows for the test set, X_test and y_test, and keeps the other 80% as X_rest and y_rest
Second train_test_splitTakes 25% of those 80%, which is 20% of all rows, for validation. The remaining 60% is the training set
The for loopPrints each part's size and its share of all rows
sharedCollects any row number that appears in two parts. There should be none

Step 2: Run It​

Terminal
uv run python three_way_split.py
Output
Training rows: 12384 (60%)
Validation rows: 4128 (20%)
Test rows: 4128 (20%)
Rows in more than one part: 0

Three parts of 60%, 20% and 20%, and no row in more than one of them. Run it again and you get the same rows in each part, because both calls use random_state=42.

What Each Part Is For​

scikit-learn's cross-validation guide explains why the third part exists: "training proceeds on the training set, after which evaluation is done on the validation set, and when the experiment seems to be successful, final evaluation can be done on the test set."

PartUse it toUse it how often
TrainingFit the modelEvery time you train
ValidationCompare settings, or watch a model improve while it trainsAs often as you like
TestReport how good the final model isOnce, at the end

The guide also gives the reason not to tune on the test set: the settings "can be tweaked until the estimator performs optimally", so "knowledge about the test set can 'leak' into the model". The validation set takes that risk instead, and the test set stays a fair judge.

Common Mistakes​

MistakeWhat happensDo this instead
Using the share you want of all rows in the second call, such as test_size=0.2 for 20%The validation set gets 20% of the rest, which is 16% of all rows, and training keeps 64%Divide by what's left: 0.20 ÷ 0.80 = 0.25
Choosing settings by their test scoreThe test score stops telling you how the model does on new dataChoose with the validation set, and score the test set once at the end
Splitting X and y in separate callsFeatures and target end up mismatched, row by rowPass both to the same train_test_split call

Troubleshooting​

SymptomLikely causeFix
ValueError: The 'test_size' parameter of train_test_split must be a float in the range (0.0, 1.0), an int in the range [1, inf) or None. Got 1.0 instead.A share of 1 (or 0) leaves nothing on one sideUse a share between 0 and 1, such as 0.25
Different rows in each part on every runNo random_statePass the same random_state to both calls

FAQ​

Do I need a validation set if I use cross-validation?​

No. The same guide says that with cross-validation "the validation set is no longer needed", and that "A test set should still be held out for final evaluation."

What split should I use?​

There's no fixed rule. 60/20/20 is a common choice when you have plenty of rows; with fewer rows, keep more for training and use cross-validation instead of a validation set.

Why random_state=42?​

Any whole number works. What matters is using one, so the split is the same every time you run the script.

Conclusion​

Two calls to train_test_split give three parts: the first keeps the test set apart, and the second takes the validation set out of what's left. A second test_size of 0.25 turned the remaining 80% into a 60/20/20 split, 12,384, 4,128 and 4,128 rows, with no row in two parts.

Additional Resources​