Skip to main content

How to Tune Hyperparameters with GridSearchCV in scikit-learn

· 5 min read
Jagdish Kumawat
Founder @ Dewiride

A model's settings, its hyperparameters, change how well it learns, and the best values are rarely obvious. GridSearchCV tries every combination you list, scores each one with cross-validation, and keeps the best. This guide tunes the number of neighbors of a k-nearest-neighbors classifier on the iris flowers, with real output from scikit-learn 1.9.1.

Syntax​

Syntax
search = GridSearchCV(estimator, param_grid, cv=folds, scoring=metric)
search.fit(X, y)
PartWhat it means
estimatorThe model to tune, created with the settings you're not tuning
param_gridA dictionary from setting name to the values to try. Every combination is one candidate
cv=foldsHow many parts to split the data into for cross-validation: each candidate is trained on all parts but one and scored on the one left out, once per part
scoring=metric (optional)What "best" means, such as "accuracy" or "roc_auc". By default, the model's own score method
search.fit(X, y)Runs the whole search, then trains the best candidate again on all of X and y

After fit, best_params_ holds the winning settings, best_score_ its average cross-validation score, cv_results_ the results of every candidate, and best_estimator_ the winning model, ready to use.

The example tries four values of n_neighbors, how many nearby flowers a k-nearest-neighbors classifier looks at before it decides, with 5-fold cross-validation: 4 candidates × 5 folds = 20 fits. Create grid_search.py:

The grid_search.py script in a macOS-style code window: it loads the iris data, builds a GridSearchCV over a KNeighborsClassifier with n_neighbors 1, 5, 15 and 45 and cv 5, fits it, prints best_params_ and the rounded best_score_, then prints three columns of cv_results_ as a table.

grid_search.py
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.model_selection import GridSearchCV
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(return_X_y=True)

search = GridSearchCV(KNeighborsClassifier(), {"n_neighbors": [1, 5, 15, 45]}, cv=5)
search.fit(X, y)

print("Best settings:", search.best_params_)
print("Best score: ", round(search.best_score_, 4))
results = pd.DataFrame(search.cv_results_)
print(results[["param_n_neighbors", "mean_test_score", "rank_test_score"]].to_string(index=False))
Terminal
uv run python grid_search.py
Output
Best settings: {'n_neighbors': 5}
Best score: 0.9733
param_n_neighbors mean_test_score rank_test_score
1 0.960000 3
5 0.973333 1
15 0.966667 2
45 0.940000 4

A terminal running uv run python grid_search.py: best settings n_neighbors 5, best score 0.9733, and a table of the four candidates with their mean test scores, 0.96, 0.973333, 0.966667 and 0.94, and their ranks 3, 1, 2 and 4.

Step 2: Read the Results​

  • best_params_, {'n_neighbors': 5}, is the candidate with the highest average score.
  • best_score_, 0.9733, is that average: the share of flowers classified correctly, averaged over the 5 folds.
  • cv_results_ has a row per candidate. mean_test_score is each one's average and rank_test_score orders them, 1 being the best. Looking only at too few neighbors (1) or too many (45) scored worse than 5 or 15.

The scores here are on the same data the search chose with, so they flatter the winner a little. To know how the chosen model does on new data, keep a test set out of the search, then score search.best_estimator_ on it once: How to Split Data into Training and Test Sets with scikit-learn.

Common Mistakes​

MistakeWhat happensDo this instead
Misspelling a setting in param_grid, "n_neighbours"ValueError: Invalid parameter 'n_neighbours' for estimator KNeighborsClassifier(). Valid parameters are: [...]Copy the name from the error's list, or from get_params()
A huge gridEvery combination is trained once per fold: 5 values × 5 values × 5 folds is 125 fitsStart coarse, then search near the best values
Judging the final model by best_score_It's the score the search optimized, so it's optimisticScore best_estimator_ on a test set kept out of the search

Troubleshooting​

SymptomLikely causeFix
The search takes very longToo many candidates, or a slow modelFewer values, fewer folds, or n_jobs=-1 to use every CPU core
Several candidates share rank 1Their average scores are equalAny of them is fine; prefer the simpler one

FAQ​

Is GridSearchCV the same as cross-validation?​

It uses cross-validation, once per candidate. For classifiers, cv=5 splits with StratifiedKFold, which keeps the share of each class equal in every fold.

How do I tune a model inside a pipeline?​

Name the setting with the step's name, two underscores and the setting: How to Set the Parameters of a Pipeline Step in scikit-learn.

Yes, with one line: How to Log a GridSearchCV Search in MLflow with Parent and Child Runs.

Conclusion​

GridSearchCV(KNeighborsClassifier(), {"n_neighbors": [1, 5, 15, 45]}, cv=5) tried four candidates and picked n_neighbors=5, with an average accuracy of 0.9733. Read best_params_ for the winner and cv_results_ for every candidate, and judge the final model on data the search never saw.

Additional Resources​