Sage Journals: Discover world-class research

Abstract

We present a new procedure for enhanced variable selection for component-wise gradient boosting. Statistical boosting is a computational approach that emerged from machine learning, which allows to fit regression models in the presence of high-dimensional data. Furthermore, the algorithm can lead to data-driven variable selection. In practice, however, the final models typically tend to include too many variables in some situations. This occurs particularly for low-dimensional data ( $p < n$ ), where we observe a slow overfitting behavior of boosting. As a result, more variables get included into the final model without altering the prediction accuracy. Many of these false positives are incorporated with a small coefficient and therefore have a small impact, but lead to a larger model. We try to overcome this issue by giving the algorithm the chance to deselect base-learners with minor importance. We analyze the impact of the new approach on variable selection and prediction performance in comparison to alternative methods including boosting with earlier stopping as well as twin boosting. We illustrate our approach with data of an ongoing cohort study for chronic kidney disease patients, where the most influential predictors for the health-related quality of life measure are selected in a distributional regression approach based on beta regression.

Keywords

Beta regression generalized additive models for location scale and shape model-based boosting variable selection earlier stopping

Get full access to this article

View all access options for this article.

References

Fan

. A selective overview of variable selection in high dimensional feature space. Stat Sin 2010; 20: 101–148.

Chen

Chatterjee

Landi

, et al. A penalized regression framework for building polygenic risk models based on summary statistics from genome-wide association studies and incorporating external information. J Am Stat Assoc 2020; 0: 1–11.

Choi

Mak

TSH

OReilly

. Tutorial: a guide to performing polygenic risk score analyses. Nat Protoc 2020; 15: 2759–2772.

Steyerberg

Vergouwe

. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J 2014; 35: 1925–1931.

Sauerbrei

Perperoglou

Schmid

, et al. State of the art in selection of variables and functional forms in multivariable analysisoutstanding issues. Diagnostic Prognostic Res 2020; 4: 1–18.

Bühlmann

Hothorn

. Boosting algorithms: Regularization, prediction and model fitting. Stat Sci 2007; 22: 477–505.

Mayr

Hofner

Waldmann

, et al. An update on statistical boosting in biomedicine. Comput Math Methods Med 2017; 2017: 1–12.

Bühlmann

Gertheiss

Hieke

, et al. Discussion of the evolution of boosting algorithms and extending statistical boosting. Methods Inf Med 2014; 53: 436–445.

Mayr

Weinhold

Hofner

, et al. The betaboost package – a software tool for modelling bounded outcome variables in potentially high-dimensional epidemiological data. Int J Epidemiol 2018; 47: 1383–1388.

10.

Bühlmann

Hothorn

. Twin boosting: improved feature selection and prediction. Stat Comput 2010; 20: 119–138.

11.

Zou

. The adaptive LASSO and its Oracle properties. J Am Stat Assoc 2006; 101: 1418–1429.

12.

Breiman

Friedman

Stone

, et al. Classification and Regression Trees. Boca Raton: CRC Press, 1984.

13.

Ellenbach

Boulesteix

Bischl

, et al. Improved outcome prediction across data sources through robust parameter tuning. J Classif, 2021; 38: 212–231.

14.

Van Calster

van Smeden

De Cock

, et al. Regression shrinkage methods for clinical prediction models do not guarantee improved performance: Simulation study. Stat Methods Med Res 2020; 29: 3166–3178.

15.

Zhou S. Thresholding procedures for high dimensional variable selection and statistical estimation. In Proceedings of the 22nd International Conference on Neural Information Processing Systems (NIPS’09). Curran Associates Inc., Red Hook, NY, USA, 2009. 2304–2312.

16.

Weinstein

Bogdan

, et al. A Power Analysis for Knockoffs with the Lasso Coefficient-Difference Statistic; 2020. arXiv preprint arXiv:2007.15346.

17.

Freund, Y. Boosting a weak learning algorithm by majority. Information and Computation 1995; 12: 256–285.

18.

Freund Y and Schapire RE. Experiments with a New Boosting Algorithm. In: Proceedings of the Thirteenth International Conference on International Conference on Machine Learning. Bari, Italie: Morgan Kaufmann Publishers Inc, 1996. pp.148–156.

19.

Friedman

Hastie

Tibshirani

. Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors). Ann Stat 2000; 28: 337–407.

20.

Friedman

. Greedy function approximation: A gradient boosting machine. Ann Stat 2001; 29: 1189–1232.

21.

Mayr

Binder

Gefeller

, et al. The evolution of boosting algorithms. Methods Inf Med 2014; 53: 419–427.

22.

Mayr

Binder

Gefeller

, et al. Extending statistical boosting. Methods Inf Med 2014; 53: 428–435.

23.

Hofner

Mayr

Robinzonov

, et al. Model-based boosting in R: A hands-on tutorial using the R package mboost. Comput Stat 2014; 29: 3–35.

24.

Mayr

Hofner

Schmid

. The importance of knowing when to stop: A sequential stopping rule for component-wise gradient boosting. Methods Inf Med 2012; 51: 178–186.

25.

Friedman

Hastie

Tibshirani

. Regularization paths for generalized linear models via coordinate descent. J Stat Softw 2010; 33: 1–22.

26.

Hastie T, Tibshirani R and Friedman JH. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer, 2009.

27.

Thomas J, Hepp T, Mayr A and Bischl B. Probing for Sparse and Fast Variable Selection with model-based boosting. Computational and Mathematical Methods in Medicine 2017; 2017: 1421409.

28.

Hofner

Boccuto

Göker

. Controlling false discoveries in high-dimensional situations: Boosting with stability selection. BMC Bioinformatics 2015; 16: 144.

29.

Wood SN. Generalized Additive Models: An Introduction with R. 2nd ed. Boco Raton: CRC Press, 2017.

30.

Rigby

Stasinopoulos

. Generalized additive models for location, scale and shape. Journal of the Royal Statistical Society: Series C (Applied Statistics) 2005; 54: 507–554.

31.

Mayr

Fenske

Hofner

, et al. Generalized additive models for location, scale and shape for high dimensional data—a flexible approach based on boosting. Journal of the Royal Statistical Society: Series C (Applied Statistics) 2012; 61: 403–427.

32.

Thomas

Mayr

Bischl

, et al. Gradient boosting for distributional regression: faster tuning and improved variable selection via noncyclical updates. Stat Comput 2017; 28: 1–15.

33.

Schmid

Hothorn

. Boosting additive models using component-wise P-splines. Comput Stat Data Anal 2008; 53: 298–311.

34.

R Core Team.: R: A Language and Environment for Statistical Computing. Vienna, Austria; 2019.

35.

Hothorn

Bühlmann

Kneib

, et al. mboost: Model-Based Boosting; 2020. R package version 2.9-2.

36.

Wang

. bst: Gradient Boosting; 2020. R package version 0.3-23.

37.

Ferrari

Cribari-Neto

. Beta regression or modeling rates and proportions. J Appl Stat 2004; 31: 799–815.

38.

Hunger

Baumert

Holle

. Analysis of SF-6D index data: is beta regression appropriate? Value Health 2011; 14: 759–767.

39.

Hunger

Döring

Holle

. Longitudinal beta regression models for analyzing health-related quality of life scores over time. BMC Med Res Methodol 2012; 12: 144.

40.

Eckardt

Bärthlein

Baid-Agrawal

, et al. The German chronic kidney disease (GCKD) study: design and methods. Nephrol Dial Transplant 2012; 27: 1454–1460.

41.

Meinshausen

Bühlmann

. Stability selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2010; 72: 417–473.

42.

Shah

Samworth

. Variable selection with error control: another look at stability selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2013; 75: 55–80.

43.

Hofner

Boccuto

Göker

. Controlling false discoveries in high-dimensional situations: boosting with stability selection. BMC Bioinformatics 2015; 16: 144.

44.

Mayr

Hofner

Schmid

. Boosting the discriminatory power of sparse survival models via optimization of the concordance index and stability selection. BMC Bioinformatics 2016; 17: 288.

45.

Wyatt

Altman

. Prognostic models: clinically useful or quickly forgotten? Br Med J 1995; 311: 1539–1541.

46.

Murdoch WJ, Singh C, Kumbier K, Abbasi-Asl R, and Yu B. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences 2019; 116: 22071–22080.

47.

Stasinopoulos

Rigby

Heller

, et al. Definitions, methods, and applications in interpretable machine learning. CRC Press, 2017.

48.

Hothorn

. Invited discussion on “Meinshausen and Bühlmann: Stability selection”. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2010; 72: 463–464.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.78 MB