Sage Journals: Discover world-class research

Abstract

Ensemble learning via manipulating the training set is an effective technique for improving classification accuracy. In this work, we investigate the strategy how to combine learning set resampling method and random subspace method applied in high-dimensional domains. We propose a new procedure, Bag of Little Bootstraps on Features (BLBF), which works by combining the results of bootstrapping multiple feature subsets of the original dataset using the random subspace method. Our empirical experiments on various high-dimensional datasets demonstrate that our proposed approach outperforms the state-of-the-art instance-based resampling learning algorithm BLB and its two relevant variants, in terms of classification performance. In addition, we also investigate the effect of hyperparameters on classification performance, which shows that the parameters can be easily set while maintaining a good performance.

Keywords

Ensemble learning bag of little bootstraps on features high-dimensional data classification

Get full access to this article

View all access options for this article.

References

Bay

S.D.

, Nearest neighbor classification from multiple feature subsets, Intelligent Data Analysis 3(3) (1999), 191-209.

Bickel

P.J.

, Götze

and van Zwet

W.R.

, Resampling fewer than n observations: Gains, losses, and remedies for losses, Statistica Sinica 7 (1997), 1-31.

Breiman

, Bagging predictors, Machine Learning 24(2) (1996), 123-140.

Bryll

, Gutierrez-Osuna

and Quek

, Attribute bagging: Improving accuracy of classifier ensembles by using random feature subsets, Pattern Recognition 36(6) (2003), 1291-1302.

Cai

, He

, Han

and Huang

T.S.

, Graph regularized nonnegative matrix factorization for data representation, IEEE Transactions on Pattern Analysis and Machine Intelligence 33(8) (2011), 1548-1560.

Cai

, He

, Zhang

W.V.

and Han

, Regularized locality preserving indexing via spectral regression, in: Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management, ACM (2007), 741-750.

Cai

, Mei

, Han

and Zhai

, Modeling hidden topics on document manifold, in: Proceedings of the 17th ACM Conference on Information and Knowledge Management, ACM (2008), 911-920.

Claesen

, De Smet

, Suykens

J.A.K.

and De Moor

, Ensemblesvm: A library for ensemble learning using support vector machines, Journal of Machine Learning Research 15(1) (2014), 141-145.

Efron

and Tibshirani

R.J.

, An Introduction to the Bootstrap, volume 57, Chapman and Hall, 1993.

10.

Hansen

L.K.

and Salamon

, Neural network ensembles, IEEE Transactions on Pattern Analysis and Machine Intelligence 12(10) (1990), 993-1001.

11.

T.K.

, The random subspace method for constructing decision forests, IEEE Transactions on Pattern Analysis and Machine Intelligence 20(8) (1998), 832-844.

12.

Hosmer

D.W.

and Lemeshow

, Applied logistic regression, Wiley, New York, 2 edition, 2000.

13.

Huang

G.-B.

, Wang

D.H.

and Lan

, Extreme learning machines: A survey, International Journal of Machine Learning and Cybernetics 2(2) (2011), 107-122.

14.

Huang

G.-B.

, Zhu

Q.-Y.

and Siew

C.-K.

, Extreme learning machine: A new learning scheme of feedforward neural networks, in: Proceedings of the International Joint Conference on Neural Networks (IJCNN2004), IEEE 2 (2004), 985-990.

15.

Huang

G.-B.

, Zhu

Q.-Y.

and Siew

C.-K.

, Extreme learning machine: Theory and applications, Neurocomputing 70(1) (2006), 489-501.

16.

Kleiner

, Talwalkar

, Sarkar

and Jordan

M.I.

, A scalable bootstrap for massive data, Journal of the Royal Statistical Society: Series B (Statistical Methodology) 76(4) (2014), 795-816.

17.

Komarek

, Logistic regression for data mining and high-dimensional classification, Robotics Institute (2004), 222.

18.

Linghu

and Sun

B.-Y.

, Constructing effective svm ensembles for image classification, in: 2010 3rd International Symposium on Knowledge Acquisition and Modeling, IEEE (2010), 80-83.

19.

Mordelet

and Vert

J.-P.

, A bagging svm to learn from positive and unlabeled examples, Pattern Recognition Letters 37 (2014), 201-209.

20.

Neter

, Kutner

M.H.

, Nachtsheim

C.J.

and Wasserman

, Applied Linear Statistical Models, Irwin, Chicago, 4 edition, 1996.

21.

Pan

and Tang

, Ensemble methods in bank direct marketing, in: 11th International Conference on Service Systems and Service Management, IEEE (2014), 1-5.

22.

Politis

D.N.

, Romano

J.P.

and Wolf

, Subsampling, Springer, 1999.

23.

Ramakrishnan

, Mirzaei

and Shanker

, Corporate default prediction with data mining techniques, Management 4(1) (2015).

24.

Ryan

T.P.

, Modern Regression Methods, Wiley, New York, 2 edition, 2008.

25.

Tian

and Meng

, A new modeling method based on bagging elm for day-ahead electricity price prediction, in: IEEE Fifth International Conference on Bio-Inspired Computing: Theories and Applications (BIC-TA), IEEE (2010), 1076-1079.

26.

Valentini

and Dietterich

T.G.

, Low bias bagged support vector machines, in: Proceedings of the 20th International Conference on Machine Learning (ICML-2003) (2003), 752-759.

27.

Vapnik

V.N.

, Statistical Learning Theory, volume 2, New York: Wiley, 1998.

28.

Wang

, He

, Shang

, Zhuang

and Shi

, Extreme learning machine ensemble classifier for large-scale data, in: Proceedings of ELM-2014, Springer 1 (2015), 151-161.

29.

Wang

, Mathew

, Chen

, Xi

, Ma

and Lee

, Empirical analysis of support vector machine ensemble classifiers, Expert Systems with Applications 36(3) (2009), 6466-6476.

30.

, Geng

and Zhou

, A short term load forecasting based on bagging-elm algorithm, in: Proceedings of the 2012 International Conference on Information Technology and Software Engineering, Springer (2013), 507-514.

31.

and Suganthan

P.N.

, Empirical comparison of bagging-based ensemble classifiers, in: 15th International Conference on Information Fusion, IEEE (2012), 917-924.