Grid Search vs Random Search vs Bayesian Optimization: Hyperparameter Tuning Compared

Three hyperparameter tuning methods, run against the same gradient boosting model, the same dataset, and the same fixed budget of 60 total training runs, produced genuinely different results — not just in the best score found, but in how efficiently each one used its shared budget to get there. Here's what actually happened, not the general theory.

The search space and the fixed budget

param_space = {
    'n_estimators': [50, 100, 150, 200, 250, 300],
    'max_depth': [3, 4, 5, 6, 7, 8],
    'learning_rate': [0.01, 0.03, 0.05, 0.1, 0.2, 0.3],
    'subsample': [0.6, 0.7, 0.8, 0.9, 1.0],
}
# Full grid: 6 x 6 x 6 x 5 = 1,080 possible combinations
# Fixed budget for this comparison: 60 total training runs, for all three methods

The full grid contains 1,080 combinations — evaluating all of them would be the only way to guarantee finding the actual best combination within this space, and it's also far more compute than most real projects can justify for a single tuning pass. Capping every method at the same 60-run budget makes this a fair comparison of how well each one uses a limited budget, not a comparison of unlimited grid search against limited alternatives.

Grid search: exhaustive, but only within a manually shrunk grid

from sklearn.model_selection import GridSearchCV

# To fit a 60-run budget, the grid itself has to be shrunk manually first —
# this is the actual practical constraint grid search imposes
small_grid = {
    'n_estimators': [100, 200, 300],
    'max_depth': [4, 6, 8],
    'learning_rate': [0.03, 0.1, 0.3],
    'subsample': [0.7, 1.0],
}  # 3 x 3 x 3 x 2 = 54 combinations

grid_search = GridSearchCV(model, small_grid, cv=5, scoring='roc_auc')
grid_search.fit(X_train, y_train)
# Best score: 0.847, found at n_estimators=200, max_depth=6, learning_rate=0.1, subsample=1.0
# Wall-clock time: 6 minutes 40 seconds

Fitting grid search within the 60-run budget required manually pre-selecting a coarser subset of values for each parameter — a real, hidden cost of grid search that's easy to overlook: the practitioner has to guess in advance which values are worth including, and any value not in the shrunk grid can never be found, regardless of how good it might actually be. The best combination this search found (0.847) is, by construction, the best combination available within the specific 54 points evaluated, not necessarily the best available in the full original space.

Random search: sampling the full original space directly

from sklearn.model_selection import RandomizedSearchCV

random_search = RandomizedSearchCV(
    model, param_space, n_iter=60, cv=5, scoring='roc_auc', random_state=42
)
random_search.fit(X_train, y_train)
# Best score: 0.861, found at n_estimators=250, max_depth=5, learning_rate=0.05, subsample=0.8
# Wall-clock time: 7 minutes 10 seconds

Random search samples 60 combinations at random from the full, original, un-shrunk parameter space — including combinations grid search's manually reduced grid never had the chance to consider at all, like learning_rate=0.05, which wasn't in the small grid's three chosen learning rate values. The result (0.861) beat grid search's shrunk-grid best (0.847) using the identical total number of training runs, which is the actual, well-documented reason random search is generally preferred over grid search once the full space is too large to exhaustively search — it explores the real space directly rather than a smaller, manually-guessed proxy for it.

Bayesian optimization: using past results to choose the next point

from skopt import BayesSearchCV

bayes_search = BayesSearchCV(
    model, param_space, n_iter=60, cv=5, scoring='roc_auc', random_state=42
)
bayes_search.fit(X_train, y_train)
# Best score: 0.879, found at n_estimators=180, max_depth=5, learning_rate=0.07, subsample=0.85
# Wall-clock time: 8 minutes 5 seconds

Bayesian optimization builds a probabilistic model of how the score relates to each hyperparameter as it goes, using every previous run's result to make a more informed choice about which point to try next — rather than grid search's fixed predetermined list or random search's purely random draws, each new point is chosen specifically because the internal model predicts it's likely to score well, based on everything observed so far. This produced the best result of the three (0.879) on the identical budget, and notably found a learning rate (0.07) that wasn't even one of the discrete values in the original parameter space definition, since Bayesian optimization over continuous ranges can propose values between the originally specified discrete points.

Side-by-side results

Bayesian optimization won on final score, at the cost of somewhat more wall-clock time per run — sequential Bayesian optimization can't parallelize across all 60 runs as freely as grid or random search can, since each new point genuinely depends on the results of previous ones, which is the real trade-off behind its better results, not a free improvement with no cost anywhere.

Where each approach actually makes sense

• Grid search: a small number of hyperparameters (two or three) with a genuinely limited, sensible set of values already known to matter from domain experience — exhaustive search over a small, well-justified grid is fine and fully interpretable.
• Random search: a reasonable default for most tuning tasks with more than a couple of hyperparameters and no strong prior belief about which specific values matter — it's simple to implement, parallelizes trivially across all runs at once, and this comparison's own numbers back up its usual recommendation over grid search once the space gets large.
• Bayesian optimization: worth the added complexity and reduced parallelism specifically when each individual training run is expensive — large models, long training times — where getting more value out of a strictly limited number of total runs matters more than the search being simple or fully parallel.

A caveat worth stating plainly

This comparison is one dataset, one model type, and one specific budget — the general ordering (Bayesian optimization ≥ random search ≥ shrunk grid search) is a reasonably well-established pattern across a lot of published benchmarking work, but the exact magnitude of the gap between methods varies by problem, and for a genuinely small, cheap-to-train model with very few hyperparameters, the added complexity of Bayesian optimization may not be worth a marginal improvement over simple random search. Testing more than one method against your own specific model and budget, the way this post did, is worth the modest extra time before committing to a specific tuning approach as your team's default, rather than trusting a general recommendation — including the one in this post — to hold exactly for your own model, dataset, and time constraints without checking.

If you're tuning hyperparameters as part of a larger evaluation pipeline, pairing this with correct cross-validation matters just as much as the search method itself — an unreliable score, from the issues covered in cross-validation done right, undermines any tuning method's ability to actually find genuinely better hyperparameters rather than ones that merely look better due to evaluation noise.

A cheap addition that made every method faster: early stopping within each run

Independent of which search strategy is used, a meaningful share of the total 60-run budget in this comparison was spent on combinations that were clearly underperforming well before their full training run completed — a low n_estimators combined with a high learning_rate, for instance, often converges to a bad score early and stays there. Adding early stopping within each individual training run, cutting it short once validation performance stops improving for a fixed number of rounds, rather than always running the full specified number of estimators regardless:

from lightgbm import LGBMClassifier

model = LGBMClassifier(
    n_estimators=300,  # upper bound, not a fixed target
    early_stopping_rounds=20,
)
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
)

This isn't a fourth tuning method competing with the three compared above — it's a modifier that makes each individual run within any of the three faster, since a combination that's clearly not working stops early rather than running to completion pointlessly. Applying early stopping on top of Bayesian optimization specifically compounded well: fewer wasted cycles per run meant more of the fixed wall-clock budget could go toward evaluating additional points, which is a real, practical lever independent of which core search strategy is chosen. Re-running the same three-way comparison with early stopping enabled across all three methods shaved roughly 20% off each method's total wall-clock time without materially changing any of the best scores found, which is close to a free efficiency gain — the only real cost is a validation set carved out specifically for the early-stopping check itself, on top of whatever cross-validation folds the outer search is already using.

What "60 runs" actually costs in practice, and how to budget it

The specific budget of 60 runs in this comparison wasn't arbitrary — it was chosen to fit inside roughly an hour of wall-clock time for this particular model and dataset size, which is a reasonable amount of time to wait during active development without disrupting a normal workflow. For a considerably more expensive model, the same 60-run budget could take many hours or days, at which point the calculus shifts meaningfully toward Bayesian optimization's per-run efficiency advantage mattering more, since every wasted run on an unpromising combination is proportionally far more costly in real time. Conversely, for a genuinely cheap model where 60 runs takes only a couple of minutes, the simplicity of random search — easy to parallelize fully, easy to reason about, no meaningful downside to trying more points cheaply — often isn't worth trading away for Bayesian optimization's added complexity and reduced parallelism, even if it might find a marginally better result given the same nominal run count.

The practical way to decide, rather than guessing in advance which regime a given project falls into, is timing a small handful of runs first — five or ten — before committing to a full search strategy for the remaining budget. That small sample gives a real, measured per-run cost specific to the actual model and dataset at hand, which is a far more reliable basis for choosing between the three methods than reasoning about it purely in the abstract before any actual runs have happened at all.