School of Business
Reducing cross-validation variance through seed blocking in hyperparameter tuning
Hyperparameter tuning plays a crucial role in optimizing the performance of predictive learners. Cross-validation (CV) is a widely adopted technique for estimating the error of different hyperparameter settings. Repeated cross-validation (RCV) is commonly employed to reduce the variability of CV errors. This study investigates the efficacy of blocking cross-validation partitions and algorithm initialization seeds during hyperparameter tuning. The proposed approach, termed Controlled Cross-Validation (CCV), reduces variability in error estimates, enabling fairer and more reliable comparisons of predictive model performance. We provide both theoretical and empirical evidence to demonstrate that this blocking approach lowers the variance of the estimates compared to RCV. Our experiments indicate that the algorithm’s internal random behavior often does not significantly affect CV error variability. We present extensive examples using real-world datasets to compare the effectiveness and efficiency of blocking the CV partitions when tuning the hyperparameters of different supervised predictive learning algorithms.