Sage Journals: Discover world-class research

Abstract

Background

The popularity of a large number of microarray applications has in cancer research led to the development of predictive or prognostic gene expression profiles. However, the diversity of microarray platforms has made the full validation of such profiles and their related gene lists across studies difficult and, at the level of classification accuracies, rarely validated in multiple independent datasets. Frequently, while the individual genes between such lists may not match, genes with same function are included across such gene lists. Development of such lists does not take into account the fact that genes can be grouped together as metagenes (MGs) based on common characteristics such as pathways, regulation, or genomic location. Such MGs might be used as features in building a predictive model applicable for classifying independent data. It is, therefore, demanding to systematically compare independent validation of gene lists or classifiers based on metagene or individual gene (SG) features.

Methods

In this study we compared the performance of either metagene- or single gene-based feature sets and classifiers using random forest and two support vector machines for classifier building. The performance within the same dataset, feature set validation performance, and validation performance of entire classifiers in strictly independent datasets were assessed by 10 times repeated 10-fold cross validation, leave-one-out cross validation, and one-fold validation, respectively. To test the significance of the performance difference between MG- and SG-features/classifiers, we used a repeated down-sampled binomial test approach.

Results

MG- and SG-feature sets are transferable and perform well for training and testing prediction of metastasis outcome in strictly independent data sets, both between different and within similar microarray platforms, while classifiers had a poorer performance when validated in strictly independent datasets. The study showed that MG- and SG-feature sets perform equally well in classifying independent data. Furthermore, SG-classifiers significantly outperformed MG-classifier when validation is conducted between datasets using similar platforms, while no significant performance difference was found when validation was performed between different platforms.

Conclusion

Prediction of metastasis outcome in lymph node–negative patients by MG- and SG-classifiers showed that SG-classifiers performed significantly better than MG-classifiers when validated in independent data based on the same microarray platform as used for developing the classifier. However, the MG- and SG-classifiers had similar performance when conducting classifier validation in independent data based on a different microarray platform. The latter was also true when only validating sets of MG- and SG-features in independent datasets, both between and within similar and different platforms.

Keywords

microarray classification metagenes breast cancer

Background

Microarray gene expression analysis has in several studies been applied to elucidate the relation between clinical outcome and gene expression patterns in breast cancer and has demonstrated improvement of recurrence prediction.^1–14 In some studies, genes in such profiles might be fully or partially missing in the test data used for validation due to the choice of microarray platform or the presence of missing values associated with a given probe. Furthermore, an obtained gene list can have none or few genes in common with other gene lists addressing the same clinical outcome,^15,16 due to usage of different microarray platforms, different methods for measuring mRNA expression levels, variation in patient sampling,¹⁵ lab variation/measurement noise,¹⁷ and differences in data processing such as different normalization methods.¹⁸ Furthermore, a wide array of feature selection methods is available for gene selection, which also affects the constitution of such final gene lists.¹⁵ These feature selection methods encompass filter approaches; selection of top features from a ranked list of genes; wrapper methods where model selection algorithms are wrapped in the search process of feature subsets (ie, the Gini index in random forest);¹⁹ and embedded methods where the feature selection is an integrated part of the classification method, such as iteratively eliminating redundant features with minimal information regarding classification performance.²⁰ More recent approaches to gene selection include recursive feature elimination based on support vector machines (SVM-RFE). This approach uses the coefficient of the weight vector to compute a feature ranking score, from which features with the smallest ranking scores are built into the model, for example, a leave-out-N number of genes approach.²¹ The advanced version combines SVM-RFE with a minimum-redundancy and maximum-relevancy filter, where relevance of each feature is determined by the mutual information among genes and class labels, and the redundancy is given by the mutual information among the genes.²²

In addition to the above mentioned factors, the choice of classification method also impacts the final model and gene lists. Furthermore, the validations of such gene lists in independent data are very heterogeneous, with the majority testing significant differences in survival, which barely reflect the actual classification accuracy, while few studies conducts validations in terms of classification accuracies.

To overcome the above obstacles, individual genes could be considered part of a larger network, that is, their expression being controlled by the expression level of other genes or that a group of genes belong to a specific pathway performing a well-defined task. These genes may be controlled by the same transcription factor or located in the same chromosomal region. Such grouping has been collected in public databases such as the Kyoto Encyclopedia of Genes and Genomes (KEGG)²³ the Molecular Signature Database (MsigDB),²⁴ and the Gene Ontology database (GO).²⁵ In relation to breast cancer, for example, cell cycle upregulation or deregulation of other pathways are associated with metastasis^2,3,26,27 Furthermore, it has been shown that metastasis progression²⁸ and tumor grading²⁹ in breast cancer are associated with accumulated mutations in several genes, leading to amplification or inactivation of genes, and even large genomic losses or gains in specific chromosomal regions affecting gene expression levels.

Our previous studies showed that the expression levels of such specific entities, called metagenes (MGs), are significantly associated with metastatic outcome in breast cancer across eight different datasets.^30,31 Several studies have defined metagene/gene modules derived from microarray data using various methods such as penalized matrix decomposition which clusters similar genes but without similar expression profiles³² hierarchical clustering,³³ correlation,³⁴ or combining a priori protein-protein interactions with microarray gene expression data defining interaction networks as features.^35,36 Few studies have attempted to use such predefined gene sets for prediction models. One such study used a compendia of microarray cancer genes for defining metagene/gene modules by performing hierarchical clustering of these genes expressions and seeding genes within the clusters into gene sets annotated in the public databases.³³ A second study defined metagene/gene modules as sets of significantly correlated or anticorrelated genes combined with prior information about the genes.³⁴ One of the strengths of using gene sets as features is that this circumvents the necessity of sharing all genes between studies. Furthermore, grouping the genes together also reduces the dimensionality of the datasets and thus functions as feature reduction. Therefore, profiles consisting of MGs might be used for developing predictive classifiers that can be validated in independent data.

This study systematically assesses and compares the performance of MG- and SG-(single gene) feature sets and MG- and SG-classifiers extracted from the same samples in predicting metastasis outcome among lymph node–negative breast cancer patients who have not been treated with adjuvant therapy.

These comparisons were first made by model building and classification within the same dataset using 10-fold cross validation. Furthermore, the comparisons were also done across datasets in two ways: (1) application of the entire classifier on the test data and (2) only the features from the classifier are transferred to the test data for model building and testing and evaluated by leave-one-out cross validation (LOOCV). In each case, we also examined two possible scenarios. In the first, the validations were conducted between studies using the same microarray platform (Affymetrix classifier/feature set validated on an independent Affymetrix dataset), while the second, encompassed validations across studies with different platforms (Agilent-developed classifier/feature set validated on an independent Affymetrix dataset).

Results

Features and models

The 71 metagenes used in this study wer determined as gene sets covering similar biological pathways having common transcription factor binding sites or genes located in the same chromosomal region (Supplementary Table 1), which in previous studies have proven to be associated with breast cancer metastasis across eight different datasets using a rank-based method (Fig. 1).^30,31 An overview of the eight datasets is shown in Table 1. The final MGs consist, on average, of 21 genes, with the smallest MGs consisting of only 2 genes and the largest, of 65 genes. This rank-based method was also applied to each gene across the same eight datasets, which led to identification of 283 rank-significant SGs (Supplementary Table 2) shared by the four datasets to be used later. Amongst the 283 SGs, 119 genes were also present on the gene lists underlying the MGs (data not shown). These 71 MGs and 283 SGs were used for selecting the optimal MG- and SG-feature sets and development of their corresponding MG- and SG-classifiers.

Table 1.

Overview of datasets.

Dataset	Chip	Probes (K)	Patients	Outcome	Treatment	Define MG	Define SG	Train	Test	Ref.
Amsterdam	Agilent/Rosetta	25	295, N⁺, N^−-	DM	None, et, ct	√	√			[14]
Amsterdam (AG1) (subset of the above)	Agilent/Rosetta	25	151, N^−-	DM	None	√	√	√		[14]
Rotterdam (AF1)	Affymetrix HG-133A	22	286, N^−-	DM	None	√	√	√		[3]
HUMAC	Spotted oligonucleotides	29	60, N^−-	ME	None	√	√			[7]
Huang	Affymetrix 95av2	12	52, N⁺	RE	Ct	√	√			[13]
Sotiriou 2003	Spotted cDNA	7.6	99, N⁺/N^−-	RE	Et, ct	√	√			[1]
Sotiriou 2006	Affymetrix HG-133A	22	179, N⁺/N^−-	DM	Et	√	√			[12]
Uppsala	Affymetrix HG-133A+B	44	236, N⁺/N^−-	DF	None, ct, et	√	√			[52]
Stockholm	Affymetrix HG-133A+B	44	159, N⁺/N^−-	RE	None, ct, et	√	√			[11]
TRANSBIG (AF2)	Affymetrix HG-133A	22	147, N^−-	DM	None			√^a	√	[35]
Mainz (AF3)	Affymetrix HG-133A	22	200, N^−-	DM	None			√^b	√	[53]

Notes: The table shows name of dataset, microarray chip, number of probes, patients, outcome, patient treatment, datasets used to define features and for training and testing and the references.

designate used as training only when validating AF3;

designate used for training only when validating AF2.

Abbreviations: K, thousands; N⁺ and N⁻, node-positive and -negative patients; DM, distant metastasis; Me, metastasis; re, relapse; DF, death from breast cancer; et, endocrine therapy; ct, chemo therapy; none, no adjuvant therapy.

Table 2.

The number of metagene and single genes features in the 24 models.

Dataset	AG1		AF1		AF2		AF3
Features method	#MG	#SG	#MG	#SG	#MG	#SG	#MG	#SG
RF	4	21	15	21	14	14	21	26
R-SVM	18	20	57	25	5	22	10	71
S-SVM	29	17	67	35	9	19	64	122

Figure 1.

Metagene and single gene selection procedure.

Two (AG1 and AF1) of the eight datasets used to define the features were, therefore, solely used for training purposes (Table 1). The AF1 corresponds to the Affymetrix-based Rotterdam dataset and AG1 is a subset of the Agilent-based Amsterdam dataset. Both dataset containing only node-negative samples. The AF2 and AF3 datasets are both based on the Affymetrix platform and function as test sets for classifiers and feature sets developed and selected by the AG1 and AF1 datasets (Table 1). Furthermore, AF2 was used for training to validate on AF3 and vice versa (Table 1). The following three classification methods were used for model building: random forest (RF) and support vector machines with a radial-based kernel (R-SVM) or a sigmoid-based kernel (S-SVM). These were optimized to achieve the best mean of sensitivity and specificity, referred to as balanced accuracy (bAcc).

In order to build the models, the SGs or MGs within each training dataset were ranked according to their random forest importance values. For a given feature, this value reports the standardized drop in prediction accuracy when the class labels are permuted.³⁷ This rank was then used for model building by subsequently adding one feature at a time in a top-down forward wrapper approach starting with the top two features. To avoid creating bias during gene selection and training of the final classifier and on classification performance, 10 times repeated 10-fold cross validation bAcc was used as optimization measure, as this optimization metric has previously been shown to give an excellent bias-variance balance.³⁸ The above described procedure led to a total of 24 models, with 12 composed of MG and SG features, respectively (Table 2). In the metagene models, each metagene contributes one input value calculated as the median expression level of the genes underlying that particular metagene. We, therefore, considered a metagene model composed of four metagenes to have four inputs or four features. These MG- and SG-models varied in complexity, consisting of 4 to 122 features (Table 2). Comparison of the number of features in each model displays a slightly higher complexity of SG-models with MG-models ranging from 4 to 67 features having, on average, 26 features per model (Table 2), while SG-models varied from 14 to 122 features with an average of 34 features per model (Table 2). This suggests that each SG-feature is less informative compared to the MG-features and thus a larger number of SG-features is a requirement for reaching optimal performance.

Internal performance of MG-and SG-models

To reduce variability and complexity and keeping validation parameters as constant as possible, the performance of MG- and SG-models were evaluated within the same dataset from which they were initially developed. This internal model performance was evaluated using 10 times repeated 10-fold cross validation (Fig. 2). This validation scheme partitions the training data into 10 nearly equally sized folds. Subsequently, 10 iterations of training and validation are performed. During each of these iterations, a different fold of the training data is left out for validation, and the remaining folds are used for learning. The mean accuracy of all 10 folds validated is thus the 10-fold cross validated accuracy of the model. By repeating this process 10 times, a more robust and unbiased estimation of the generalization performance is obtained.³⁹ It should, therefore, also be noted that the individual classification performances are artificially elevated due to information leakage caused by using the entire dataset for ranking. However, as metagenes are a simple linear combination of single genes, we assume that the comparisons between the MG- and SG-model performances are similarly affected by this leakage.

Figure 2.

The internal dataset validation procedure.

As indicated by the internal classification performances, the models predicted outcome with high accuracy (Fig. 3). Comparison across the three classification methods within the majority of each training dataset showed that MG-models perform slightly better than SG-models (Fig. 3). However, these differences were only minor and nonsignificant, suggesting that these results tend to converge to a common optimization level independent of feature type.

Figure 3.

Internal classification performance.

Feature set transferability

The feature sets within the MG- and SG-models were then transferred to independent datasets to examine if these features could be used to build a model for predicting outcome within the test data as illustrated in Figure 4A and B. The transferability was assessed by leave-one-out cross validation (LOOCV). Briefly, in LOOCV, one sample is left out at a time and used for testing, while the remaining samples are used for training a classifier, which is used to classify the left out sample. This process is repeated until all samples have been left out for testing.

Figure 4.

Between study classifier or feature set validation. (A) Between different platform validations. The best classifiers developed from the training set (AG1) are either directly (transfer classifier) applied and validated in the independent test data (AF2 or AF3) or the features only from the best classifier are used within the test data for model building and testing by leave-one-out cross validation (LOOCV). In each case comparison of MG- (blue) and SG-classifier (red) or feature set performance is conducted using the same training data, classification method and test data. (B) Between similar platform validations. The best classifiers developed from the training set (AF1, AF2, or AF3) are either directly (Transfer classifier) applied and validated in the independent test data (AF2 or AF3) or the features only from the best classifier are used within the test data for model building and testing by leave-one-out cross validation (LOOCV). In each case, comparison of MG- and SG-classifier or feature set performance is conducted using the same training data, classification method, and test data.

The results show that transfer of the feature sets are able to classify samples with a mean LOOCV bAcc ranging from 54.0% to 72.0%, and where all the MG- and SG-gene lists have a mean LOOCV bAcc of 65.7% and 63.4%, respectively (Fig. 5). This suggests that the majority of these gene lists perform significantly better than random and that the selected MG- and SG-feature sets, being optimal in one dataset, display transferability across studies and can train and build a predictive model in independent samples.

Figure 5.

Exported feature set classification performance.

Comparison of classifier performance based on MG- and SG-feature sets in independent datasets

Comparison of the classification performance between the transferred MG- and SG-feature sets between different platforms (AG1-feature sets validated in AF2 or AF3) (Fig. 4A) showed MG features significantly outperformed SG-features using R-SVM (P = 3.7 × 10^−-9). This is also true when comparing across the three classification methods (P = 0.02) (Fig. 5). However no significant difference in performance was found when using RF (P = 0.30) or S-SVM (P = 0.39). This could suggest that the overall effect is solely due to the significance associated with the R-SVM method (Fig. 5). Comparison of the similar-platform performances (AF1-feature sets validated on AF2 or AF3, AF2-feature sets validated on AF3, and AF3-feature sets validated on AF2) (Fig. 4B), revealed that MG- and SG-features performed equally well for all three classification methods used (Fig. 5).

Classifier transferability and performance

We next used the entire classifier developed in the training sets, based on the features and rules associated with the classifier, to classify the independent samples in the entire test data and, therefore, to determine if the classifier can be exported and used in the test data (Fig. 4A and B). These results showed that the classifiers are less transferable than the feature sets, which is reflected by the weak mean bAcc ranging from 52.5% to 61.5%, with MG- and SG-classifiers having an overall mean bAcc of 56.0% and 58.2%, respectively (Fig. 6).

Figure 6.

Exported classifier classification performance.

Comparison of classifier performance

Comparison of MG- and SG-classifiers external validation performance, when the classifier was developed on the Agilent dataset (AG1) and validated on the Affymetrix datasets (AF2 and AF3), showed that MG- and SG-classifiers performed equally well, with the exception of using S-SVM, which favored the MG-classifiers (Fig. 6) (P = 0.006). Comparison of classifier performance across the three methods revealed that MG-classifiers performed slightly better than SG-profiles (Fig. 6). However, this difference was not statistically significant (P = 0.42).

Comparison of MG- and SG-classifiers developed from an Affymetrix dataset and validated on another independent Affymetrix dataset showed that SG-classifiers performed significantly better than MG-classifiers for each of the three classification methods (RF, P = 0.02; R-SVM, P = 0.015; S-SVM, P = 0.007) and across the three classification methods (P < 0.001).

Discussion

A strong prediction model should be robust, reproducible, and ideally exportable to allow validation in independent datasets. In this study, predictive models based on metagenes or single genes being significantly associated with breast cancer metastasis outcome were developed. The metagene features are composed of gene lists sharing a biological consensus, while the single genes are features representing the expression of one gene. The study examined the transferability of MG- and SG-feature sets and classifiers and compared their classification performance on independent datasets. The genes present in the MG-feature sets were always present in the validation, as we had assured that the single genes and metagenes were shared by all four datasets. However, such sharing of gene lists between classifiers derived from different studies is rarely the case. The reason that different studies derive gene expression based classifiers having few or no genes in common is probably due to the selection bias caused by the microarray platform used for measuring gene expression, patient sampling, and the way the data were analyzed. In this respect, a study by Ein-Dor and coworkers showed that the same dataset could derive classifiers having comparable performances but differing substantially with respect to which genes are contained in these classifiers, thus also leading to a selection bias.¹⁵ In our study, the problem with missing common genes was circumvented by using features shared by all four datasets used in the study.

We found that both MG- and SG-feature sets could be transferred to independent datasets and were able to build a predictive model achieving good results for classifying node-negative breast cancer metastasis outcome.

The finding that the SG-feature sets are exportable agrees well with few predictive gene lists being exportable to independent data. One such gene list is the 70-gene Mammaprint prognosis gene list embedded in the 70-gene correlation classifier originally developed to predict metastasis within five years (defined as poor prognosis), or no metastasis within five years (good prognosis).² Transfer of this 70-gene Mammaprint list for model training and testing in independent datasets have proved successful using the original correlation classification method^7,40 or by SVM,⁷ while another study only showed random performance when validating this gene list in independent data.³⁵ Due to missing genes, other studies have only validated a subset of the 70 genes successfully by correlation,⁸ VFI-classification,⁸ and centroid classification.⁴¹ A second frequently validated gene list is the 76-gene list underlying the 76-gene classifier based on a regression determined risk score trained to predict distant metastasis within five years among lymph node–negative breast cancer patients.³ Transfer of the 76-gene list for model building and testing in fully independent datasets have been proven with limited success in one study by using SVM achieving balanced accuracies of 37% to 64%,³⁵ while a subset of 46 of the 76 genes performed well in predicting metastasis from low-malignant breast cancer, achieving balanced accuracy of 75%.²⁷ A third list is a “wound signature” containing 512 genes, which is able to predict increased risk of metastasis in three types of cancers, including breast cancer,⁴² The entire wound signature gene list has been transferred to independent data for model building and testing, obtaining results in the range of our transferred SG-gene lists using decision trees,⁴³ but also using a subset of 252 genes has provided similar performance when predicting metastasis outcome among low- malignant breast cancer patients using SVM.²⁷ A fourth study derived a 70-gene Cox-ranked gene list, which was developed to achieve optimal significant hazard ratios with respect to survival analysis in a cohort of patients with early breast cancer.⁵ Subsets of this list were transferred to two independent datasets for validation by model training and testing using nearest centroid classification obtaining balanced accuracies of 72% and 59%.⁵

Validation of the classifiers with some of the gene lists mentioned above using the original classification method and cutoffs for class/outcome prediction has been conducted for the 70-gene profile achieving balanced accuracies of 63%,⁴⁴ while validation of the 76-gene profile has achieved balanced accuracies of 53% to 65%,⁴⁵ 59%,⁴⁴ and 62%.⁴⁶ These reported balanced classification accuracies are similar or fairly lower than those obtained by the MG- and SG-classifiers in our study. Although, these MG- and SG-classifier performances are far from optimal to be used in the clinic, they do perform better for long-term outcome prediction than clinicopathological risk criteria illustrated by the findings from three studies. The study by Daemen and coworkers showed that the St. Gallen, NIH, and the Nottingham prognostic index assessors predicted metastasis outcome in a group of 147 breast cancer patients with balanced accuracies of 51%, 52%, and 57%.⁴⁷ Another study by Sun and coworkers demonstrated that the St. Gallen assessor predicted 5-year relapse free survival in 97 node-negative breast cancer patients with only 51% balanced accuracy.⁴⁸ Furthermore, the study by Schmidt and coworkers showed that the St. Gallen assessor predicted 5-year and 10-year distant metastasis free survival with a balanced accuracy of 57% and 54%, respectively, and that the Adjuvant! Online algorithm predicted the same endpoints with 56% bAcc in 410 node-negative patients.⁴⁹

Interestingly, although the 76-gene profile was developed to predict outcome among lymph node–negative patients, as in our study, the majority of MG- and SG-classifiers obtained in our study actually performed slightly better than those reported by the above studies. However, the reported low performance is likely caused by the classifiers being specific to the dataset from which they were developed, and the results, therefore, cannot be generalized. Despite the fact that the primary tumors from both the training and test sets all are lymph node–negative, they might still not be very representative of each other, due to, for example, biological variation, follow-up time of the studies (affecting outcome-coding), and cross-study/lab variation, thus impairing classifier external classification performance. The lack of classifier transferability has been shown in other studies, for example, in the classification of normal versus tumor tissue⁵⁰ and in the prediction of pathologically complete response to breast cancer chemotherapy treatment.⁵¹ This stresses the importance of homogeneity between the samples used for building classifiers and those used for validation.

We evaluated and compared the classification performance between MG- and SG-models within the same datasets. We found that MG- and SG-models had equal performance during internal model building and performance testing, with a trend of MG-models performing slightly better than SG-models. However, the model sizes could imply that SG- models need to contain more features than the MG-models to obtain a similar performance, suggesting that the individual MG-features are more informative than each SG-feature.

The study found no significant difference in performance only between exported MG- and SG-feature sets when used for training and testing on independent datasets. In this setup, the models based on transferred features are trained and tested in the same independent dataset, rendering the measurements underlying the intratraining and testing iterations comparable. One explanation for this similar performance could be that there is a great deal of overlap between the genes constituting the metagenes and those underlying the list of 283 single genes, as the 119 genes are shared. A second explanation could concern the way the metagenes were defined and scored. In our study, the MGs cover lists of genes having a consensus. The first advantage of being defined as such is that the metagenes are conserved and robust across microarray platforms. A second advantage is that the metagenes are narrowed down to the gene set enrichment analysis (GSEA) leading edge genes, thereby picking the best genes within the predefined lists. However, the limitations of the metagenes are that they only consider interactions with members within the defined metagene, but do not take interactions with genes beyond the members of the defined pathways and gene sets. Furthermore, although the human genome has been sequenced, there are still many genes with unknown functions, and, thus, these have still not been annotated to a specified functional gene set. Such undiscovered networks might be discovered by single gene classifiers composed of mixed biological predictors, which might be more easily discovered when using the same microarray technology, reducing the variation caused by the shift in microarray platforms.

In this study, the scoring of the metagenes was based on the median expression values among the GSEA leading edge within each predefined metagene. The fact that only a weak difference in performance between MG- and SG-models was detected suggests that the definition of metagenes and/or their expression calculation might not be able to detect such a significant difference in performance, at least not for metastasis outcome among lymph node–negative breast cancer patients. Other studies have scored gene module/metagene activity in a different way compared with our median expression score by either using the arithmetic mean,⁵² the sum of discrete values within each module/set,³³ the expression values of a median gene in a module/gene set (defined as the gene within the gene set having the smallest sum of distances to other genes within the given gene set/module),⁵³ average rank score (calculated as the average rank of the relative expression levels in a pathway normalized by the total number of genes),⁵⁴ or by probabilistic inference using log-likelihood ratios.⁵⁵

Another reason could be that exported MG- and SG-feature sets used for training and testing, and that we only used random forest and support vector machines as classification methods. The above confinement implied, as some of our unpublished results suggest, that classifiers based on random forest and support vector machines have a better classification performance when validated in completely independent datasets compared with other classification methods such as logistic regression or neural networks. In this respect, an option is to validate if the MG- and SG-feature set performances also are similar when using other classification methods.

Interestingly, other studies have found a similarly slight performance difference between models composed of gene modules or individual genes. These studies have also addressed breast cancer metastasis outcome, but using a different module definition and scoring. The gene modules in one study were defined as a subset of genes from gene compendia with correlated expression across arrays and showed that classifiers consisting of gene set modules had a slightly better classification performance, compared with classifiers of individual genes.³³ A second similar study, comparing the performance of classifiers consisting of gene sets defined by MsigDB, with classifiers consisting of individual genes, found that the two groups of classifiers had similar performance, but that gene set classifiers were more stable.⁵⁶ A third study by Blazadonakis and coworkers⁵⁷ used the 70-gene Mammaprint gene list² and a previously determined 59-gene gene list⁵⁸ to extract the presence of significant gene ontology biological processes (GOBPs). The underlying genes in each of these GOBPS, beyond those present in the gene lists, were used to construct of pool of genes for constructing new gene-lists. Interestingly, comparison of classification performance between the original gene-list and the GOBP-derived gene lists in model training and testing within the validation datasets showed that using the GOBP-derived gene-list performed slightly better than the gene lists from which they were derived.⁵⁷ However, compared with our study, the validations in these three studies were confined to using a single classification method (Bayes classifier),³³ or centroid classification.^56,57 Also, the predicted outcome differed compared with our study, being either defined as a “good” or “bad” outcome relative to 5-year time to metastasis^33,56 or 5-year breast cancer survival.⁵⁷

The results from applying the classifiers developed in the training sets directly upon the test sets revealed that SG-classifiers trained on an Affymetrix dataset and validated on an independent Affymetrix dataset performed significantly better than MG-classifiers. This suggests that the single gene expression values are better for defining classification functions to classify independent data. Therefore, although the genes underlying each MG have been limited to the leading edge, signals from highly predictive genes might be diluted within the metagenes by the median expression scoring across the metagenes, and therefore lose predictive power compared with sets of single genes, which has been picked by feature selection, making each of them highly predictive. This performance difference could also be due to classifier functions/rules based on single gene expression measurements being more transferable than leading edge median expression measurements.

In contrast, no significant difference in performance was found between the MG- and SG-classifiers when the classifier was trained on an Agilent dataset and tested on an Affymetrix dataset. This suggests that when switching platforms, measurements underlying the classifier functions and rules differ significantly between the training and test set, that is, the Agilent and Affymetrix measurements being defined as log ratios and log intensities, respectively. The finding that the MG- and SG-classifiers are equally impaired suggests that Agilent-defined rules are not applicable to an Affymetrix dataset either when using median expression measures or individual gene expression measures, which agrees well with a previous finding showing poor correlations between corresponding measurements conducted on the Agilent and Affymetrix platforms.⁵⁹ Interestingly, a previous study has shown that Agilent log ratios show a bigger variability compared with Affymetrix log intensities, even after correcting the Agilent variance using log ratios.⁶⁰ The higher Agilent variability suggests that it would be less feasible to conduct Agilent to Agilent validations compared with Affymetrix to Affymetrix validations.

Conclusions

In this study we compared the performance of metagene-or single gene-based feature sets and classifiers. As the function of the genes within breast cancer predictive profiles are frequently conserved, but not the individual genes, as we expected, gene sets having a biological consensus would both have predictive power and potentially also better validation performance than classical single gene lists when validated in new samples. Surprisingly, the metagene- and single gene-based features had equal performance. When comparing classifier performance in independent datasets, we found only a significant difference between MG- and SG-classifier performances when validation was conducted on datasets measured upon the same microarray platform from which the classifiers were developed. In this situation, SG-classifiers significantly outperformed MG-classifiers.

Methods

Datasets used in this study

This study used a total often different datasets. Eight datasets were used for defining the metagene and single gene features. These samples samples from the studies,^1,11–14 and samples from the Gene Expression Omnibus (GEO-) series GSE2034,³ GSE4796,⁷ GSE3494.⁶¹ In the further study, the GSE2034 (abbreviated AFI) and a subset of 151 node-negative samples from the Amsterdam dataset by van de Vijver¹⁴ (abbreviated AGI) were used for internal validation within the same dataset by 10 times repeated 10-fold cross validation. Furthermore, these two datasets were also used for defining gene sets used to build and train a classifier within the independent test datasets and also for building classifiers to be validated in our independent datasets. The following samples from two datasets were used as independent test datasets: 147 samples from GSE7390⁴⁰ (abbreviated AF2) and all samples from GSE11121⁶² (abbreviated AF3).

Dataset Processing

The normalizations performed in the studies were retained because the authors found these methods optimal for the eight datasets and because initial ranking of metagenes and single genes was performed separately in each data set. Furthermore, the normalizations of the four datasets, AG1, AF1, AF2, and AF3, were retained for the reasons mentioned above. However, all four datasets were standardized, having a mean of zero and a standard deviation of one. Calculations and classification were also conducted using the R-environment. For random forest and support vector machines, we used the randomForest and e1071 packages, respectively.

Single gene and metagene features

To determine which single genes should be used to build single gene-based gene expression profiles, we focussed on the eight publicly available datasets used in our two previous studies.^30,31 The determination of single genes was done by applying the microarray meta-analysis described in our previous study³⁰ upon the same individual gene expression values of each individual probe/gene used to derive the metagenes in the eight datasets. This method ranks each individual gene in each dataset according to its signal-to-noise ratio (SNR). The SNR finds the features that will discriminate between two classes by calculating a score that gives the highest score to those features whose expression levels differ most on average in the two groups while favoring those with small deviations in scores in the respective classes. The SNR for a feature j is calculated as:

S N R_{j} = \frac{\bar{X_{A, j}} - \bar{X_{B, j}}}{s_{A, j} + s_{B, j}}

(1)

In this formula, X_A,j, X_B,j are the mean gene expression of class A (metastasis group) and class B (nonmetastasis group) and s_A,j, s_B,j are their associated standard deviations for feature j. In this setting, features that obtain the most positive values are correlated with the metastasis class, while the most negative values are most correlated with the nonmetastasis class.

Following gene ranking within each dataset, the meta-analysis calculates the genes’ mean rank across datasets and determines if this mean rank is significantly high or low, according to a significance cutoff at FDR ≤ 0.05. Using gene symbols, 283 genes (Supplementary Table 2) were found significant and shared by the AG1, AF1, AF2, and AF3 datasets and used in the further analysis. The MGs are 71 selected gene sets covering a specific biological pathway, chromosomal region, or a gene set sharing a DNA transcription factor binding motif. In a previous meta-analysis across eight breast cancer microarray datasets using individual gene enrichment score ranks calculated by GSEA. The 71 metagenes were shown to be associated with breast cancer metastasis^30,31 These are listed in Supplementary Table 1. The gene sets covering biological pathways are defined by KEGG (http://www.genome.ad.jp/KEGG), GenMapp (http://www.genmapp.org), and Biocarta (http://www.biocarta.com), while the transcription factor motifs are collected from TransFac (http://www.gene-regulation.com) and sets of genes regulated by the same microRNA from the mirBase (http://microrna.sanger.ac.uk). The gene sets belonging to these were defined using bioinformatic prediction as described in our previous study.³⁰ However, as only a fraction of the genes within each gene set display differential expression between the classes, we limited the final gene sets to those that constituted the leading edge in a GSEA analysis. These leading edge genes are considered to be the core of genes driving the enrichment signal.⁶³ These leading edge genes thus form each of the final metagenes, and the score of each metagene is defined as the median expression of these leading edge genes.

Classifier Building

SGs or MGs within each training dataset were ranked according to their random forest importance value. For each feature, this value reports the standardized drop in prediction accuracy when the class labels are imputed.³⁷ For each feature, this rank was used for model building by subsequently adding one feature at a time in a top-down forward-selection wrapper-approach starting with the top two features. To avoid creating bias during gene selection and training of the final classifier or on classification performance, we used 10 times repeated 10-fold cross validation accuracies as a performance measure as this metric has previously been shown to give an excellent bias-variance balance.³⁸ In this study, the models were developed to achieve the best mean sensitivity and specificity thus forcing the overall accuracy to give a balanced sensitivity and specificity. Three different classification methods were used for model building which included Random Forest (RF)³⁷ and SVM with a radial-based kernal (R-SVM) and a sigmoid-based kernel (S-SVM).⁶⁴ As all the classification methods have hyperparameters, we determined the optimal combination of these parameters using a grid search built into the 10-fold cross validation procedure. For this purpose the tune command from the e1071 R-package was applied. In random forest, we optimized the number of trees in the forest (ntree) from settings of 2000, 3000, 4000, and 5000 trees and the number of subselected predictors for node-splitting (mtry) with settings of 1, 0.5 times the number of features), 1 times the square root of features, 2 times the number of features, and the total number of features. In all support vector machines, the slack variable penalizing cost parameter (C) was optimized using settings of 0.01, 0.1, 1, and 10. The γ-parameter controlling the spreading of samples in feature space was optimized with the settings 0.001, 0.01, 0.1, times the the square root of features

Classification performance assessment

We compared the bAcc of SG- and MG-profiles developed from the four datasets (AG1, AF1, AF2, and AF3) at three levels. At each level, we report the classification accuracy as the mean of sensitivity and specificity, which is referred to as the balanced accuracy (bAcc). Within the same dataset (AG1 or AF1) this was done by 10 times repeated 10-fold cross validation classification accuracies. For each combination of either MG or SG and classification method, this led to generation of 100 different models. Therefore, the internal performance for either MG-or SG-models was defined as the mean performance of these 100 models. When transferring features (defined by AG1, AF1, AF2, or AF3) for classifier building and testing in independent data, leave-one-out cross validation bAcc was used as a performance metric. Although the 10 times repeated 10-fold cross validation is known to be a robust performance metric, we wanted to classify each patient in the validation datasets one by one to mimic the clinical situation and, therefore, the LOOCV bAcc was chosen as a performance metric.

When transferring the best trained classifier (defined by AG1, AF1, AF2, or AF3) from the training sets to classify the independent samples, the classification corresponding to one-fold validation upon all test samples is reported. During transfer of features and classifiers, we examined between different platform and between similar platform classification performances. The between platform (Agilent validated on Affymetrix) covers AG1 validations on AF2 or AF3. The similar platform covers validation of using the same microarray platform as the external validation set (Affymetrix validated on Affymetrix), that is, AF1 on AF2 or AF3, but also AF2 validated on AF3 and vice versa.

Endpoint/outcome

Several studies that have developed classifiers predicting a dichotomous endpoint, have all optimized classifier performance using a particular longitudinal cutoff,^2,3,8,22 thus excluding patients not experiencing an event within the longitudinal cutoff and excluding patients experiencing events after this particular time point. These two circumstances could enhance the performance of these classifiers with respect to classifying early metastasis events, but they are prone to perform poorly when classifying late metastasis events.^40,46 However, a study by Thomassen and coworkers used full follow-up time for classifier development, addressing if a patient would ever develop a metastasis for the length of the entire study⁷ and thus have the strength of giving equal weight to both early and late metastasis events. Furthermore, time-to-event analysis sometimes can be misleading when considering classification, and transformation of time-to-event into an binary outcome can blur prediction of the classes.⁶⁵ In addition, genes being significantly correlated with survival time are not always optimal for classification.⁶⁶

The outcome differs in the eight datasets used for defining the single genes and metagenes, that is, local and regional recurrences are included in some studies. However, non-metastatic relapses constitute a minority in clinical cohorts. Therefore, the outcome is defined as metastasis or no metastasis after time of diagnosis.

Comparison of external validation performance

To test the significance of the performance difference between MG- and SG-features/classifiers, we used a repeated down-sampling approach consisting of the following five steps: (1) The MG- and SG-features/classifiers classification results upon the entire test data were initially converted into a balanced test-result by down-sampling. Down-sampling obtains a class-balanced dataset from an imbalanced dataset by removing a subset of randomly selected samples from the majority class, where the number of samples removed equals the sample size of the minor class. In this study the majority class is the non-metastasis class. (2) The number of samples correctly classified by MGs but incorrectly by SGs and vice versa is counted; (3) the significance of the difference in these counts in determined using a χ² test;⁶⁷ (4) The P value of this test is stored and the steps 2 to 4 are repeated 1000 times; and (5) the median P value from the 1000 tests is reported as the significance between the MGs and SGs.

List of abbreviations used

AF1, Affymetrix dataset 1 (Rotterdam dataset); AF2, Affymetrix dataset 2 (Transbig dataset); AF3, Affymetrix dataset 3 (Mainz dataset); AG1, Agilent dataset 1 (node-negative Amsterdam samples); bAcc, balanced accuracy; GEO, Gene Expression Omnibus; GO, Gene Ontology; GOBP, Gene Ontology Biological Process; GSEA, gene set enrichment analysis; KEGG, Kyoto Encyclopedia of Genes and Genomes; LOOCV, leave-one-out cross validation; MG, metagenes; MsigDB, Molecular Signature Database; mRNA, messenger RNA; RF, random forest; R-SVM, support vector machine with a radialbasis kernel; SG, single gene; S-SVM, support vector machine with a sigmoid kernel.

Author Contributions

Designed the study: MB, MT, QT, TK. Performed scripting, calculations, and data analysis: MB. Developed methods for statistical analysis: MB, QT. Provided the metagene datasets: MT. Wrote the manuscript: MB. Proofread the manuscript: MT, QT, TK.

Funding

Author(s) disclose no funding sources.

Competing Interests

Author(s) disclose no potential conflicts of interest.

Disclosures and Ethics

As a requirement of publication author(s) have provided to the publisher signed confirmation of compliance with legal and ethical obligations including but not limited to the following: authorship and contributorship, conflicts of interest, privacy and confidentiality and (where applicable) protection of human and animal research subjects. The authors have read and confirmed their agreement with the ICMJE authorship and conflict of interest criteria. The authors have also confirmed that this article is unique and not under consideration or published in any other publication, and that they have permission.

Footnotes

Acknowledgements

This work was partially funded by the Ministry of Public Health,the Danish Cancer Society,the Clinical Institute at University of Southern Denmark,and the Human Microarray Center associated with the department of Clinical Genetics at Odense University Hospital. Furthermore,we acknowledge authors and people who made the microarray datasets used in this study publicly available.

References

Sotiriou

, Neo

S.Y.

, McShane

L.M.

. Breast cancer classification and prognosis based on gene expression profiles from a population-based study. Proc Natl Acad Sci USA. 2003; 100: 10393–8.

van ‘t Veer

L.J.

, Dai

, van de Vijver

M.J.

. Gene expression profiling predicts clinical outcome of breast cancer. Nature. 2002; 415: 530–6.

Wang

, Klijn

J.G.

, Zhang

. Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer. Lancet. 2005; 365: 671–9.

Finak

, Bertos

, Pepin

. Stromal gene expression predicts clinical outcome in breast cancer. Nat Med. 2008; 14: 518–27.

Naderi

, Teschendorff

A.E.

, Barbosa-Morais

N.L.

. A gene-expression signature to predict survival in breast cancer across independent data sets. Oncogene. 2007; 26: 1507–16.

Tan

, Thomassen

, Kruse

T.A.

Feature selection for predicting tumor metastases in microarray experiments using paired design.

Cancer Inform. 2007; 3: 213–8.

Thomassen

, Tan

, Eiriksdottir

, Bak

, Cold

, Kruse

T.A.

Prediction of metastasis from low-malignant breast cancer by gene expression profiling.

Int J Cancer. 2007; 120: 1070–5.

Karlsson

, Delle

, Danielsson

. Gene expression variation to predict 10-year survival in lymph-node-negative breast cancer. BMC Cancer. 2008; 8: 254.

Starmans

M.H.

, Krishnapuram

, Steck

. Robust prognostic value of a knowledge-based proliferation signature across large patient microarray studies spanning different cancer types. Br J Cancer. 2008; 99: 1884–90.

10.

Damasco

, Lembo

, Somma

M.P.

, Gatti

, Di

C.F.

, Provero

A signature inferred from Drosophila mitotic genes predicts survival of breast cancer patients.

PLoS One. 2011; 6: e14737.

11.

Calza

, Hall

, Auer

. Intrinsic molecular signature of breast cancer in a population-based cohort of 412 patients. Breast Cancer Res. 2006; 8: R34.

12.

Sotiriou

, Wirapati

, Loi

. Gene expression profiling in breast cancer: understanding the molecular basis of histologic grade to improve prognosis. J Natl Cancer Inst. 2006; 98: 262–72.

13.

Huang

, Cheng

S.H.

, Dressman

. Gene expression predictors of breast cancer outcomes. Lancet. 2003; 361: 1590–6.

14.

van de Vijver

M.J.

, He

Y.D.

, Van't Veer

L.J.

. A gene-expression signature as a predictor of survival in breast cancer. N Engl J Med. 2002; 347: 1999–2009.

15.

Ein-Dor

, Kela

, Getz

, Givol

, Domany

Outcome signature genes in breast cancer: is there a unique set?

Bioinformatics. 2005; 21: 171–8.

16.

J.X.

, Sieuwerts

A.M.

, Zhang

. Pathway analysis of gene signatures predicting metastasis of node-negative primary breast cancer. BMC Cancer. 2007; 7: 182.

17.

, Stolovitzky

, Klein

Quantitative noise analysis for gene expression microarray experiments.

Proc Natl Acad Sci USA. 2002; 99: 14031–6.

18.

Sontrop

H.M.

, Moerland

P.D.

, van den Ham

, Reinders

M.J.

, Verhaegh

W.F.

A comprehensive sensitivity analysis of microarray breast cancer classification under feature variability.

BMC Bioinformatics. 2009; 10: 389.

19.

Diaz-Uriarte

, Alvarez de

A.S.

Gene selection and classification of microarray data using random forest.

BMC Bioinformatics. 2006; 7: 3.

20.

Saeys

, Inza

, Larranaga

A review of feature selection techniques in bioinformatics.

Bioinformatics. 2007; 23: 2507–17.

21.

Guyon

, Weston

, Barnhill

, Vapnik

Gene selection for cancer classification using support vector machines.

Machine Learning. 2002; 46: 389–422.

22.

Mundra

P.A.

, Rajapakse

J.C.

SVM-RFE with MRMR filter for gene selection.

IEEE Trans Nanobioscience. 2010; 9: 31–7.

23.

Ogata

, Goto

, Sato

, Fujibuchi

, Bono

, Kanehisa

KEGG: Kyoto Encyclopedia of Genes and Genomes.

Nucleic Acids Res. 1999; 27: 29–34.

24.

Subramanian

, Tamayo

, Mootha

V.K.

. Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proc Natl Acad Sci USA. 2005; 102: 15545–50.

25.

Ashburner

, Ball

C.A.

, Blake

J.A.

. Gene ontology: tool for the unification of biology. The Gene Ontology Consortium. Nat Genet. 2000; 25: 25–9.

26.

Pawitan

, Bjohle

, Amler

. Gene expression profiling spares early breast cancer patients from adjuvant therapy: derived and validated in two population-based cohorts. Breast Cancer Res. 2005; 7: R953–64.

27.

Thomassen

, Tan

, Eiriksdottir

, Bak

, Cold

, Kruse

T.A.

Comparison of gene sets for expression profiling: prediction of metastasis from low-malignant breast cancer.

Clin Cancer Res. 2007; 13: 5355–60.

28.

Zhang

, Martens

J.W.

, Yu

J.X.

. Copy number alterations that predict metastatic capability of human breast cancer. Cancer Res. 2009; 69: 3795–801.

29.

Buness

, Kuner

, Ruschhaupt

, Poustka

, Sultmann

, Tresch

Identification of aberrant chromosomal regions from gene expression microarray studies applied to human breast cancer.

Bioinformatics. 2007; 23: 2273–80.

30.

Thomassen

, Tan

, Kruse

T.A.

Gene expression meta-analysis identifies metastatic pathways and transcription factors in breast cancer.

BMC Cancer. 2008; 8: 394.

31.

Thomassen

, Tan

, Kruse

T.A.

Gene expression meta-analysis identifies chromosomal regions and candidate genes involved in breast cancer metastasis.

Breast Cancer Res Treat. 2009; 113: 239–49.

32.

Zhang

, Zheng

C.H.

, Liu

J.X.

, Wang

H.Q.

Discovering the transcriptional modules using microarray data by penalized matrix decomposition.

Comput Biol Med. 2011; 41: 1041–50.

33.

van Vliet

M.H.

, Klijn

C.N.

, Wessels

L.F.

, Reinders

M.J.

Module-based outcome prediction using breast cancer compendia.

PLoS One. 2007; 2: e1047.

34.

Teschendorff

A.E.

, Miremadi

, Pinder

S.E.

, Ellis

I.O.

, Caldas

An immune response gene expression module identifies a good prognosis subtype in estrogen receptor negative breast cancer.

Genome Biol. 2007; 8: R157.

35.

Garcia

, Millat-Carus

, Bertucci

, Finetti

, Birnbaum

, Bidaut

Interactome-Transcriptome integration for predicting distant metastasis in breast cancer.

Bioinformatics. 2012; 28(5) 672–8.

36.

Chuang

H.Y.

, Lee

, Liu

Y.T.

, Lee

, Ideker

Network-based classification of breast cancer metastasis.

Mol Syst Biol. 2007; 3: 140.

37.

Breiman

Random forests.

Machine Learning. 2001; 45: 5–32.

38.

Hastie

, Tibshirani

, Friedman

The Elements of Statistical Learning: Data Mining Inference and Prediction. 2nd ed. New York, NY: Springer; 2009.

39.

Kohavi

A study of cross-validation and boot-strap for accuracy estimation and model selection.

Proceedings of The Fourteenth International Joint Conference on Artificial Intelligence. 1995; 2: 1137–43.

40.

Buyse

, Loi

, Van't Veer

. Validation and clinical utility of a 70-gene prognostic signature for women with node-negative breast cancer. J Natl Cancer Inst. 2006; 98: 1183–92.

41.

Blazadonakis

M.E.

, Zervakis

M.E.

, Kafetzopoulos

Integration of gene signatures using biological knowledge.

Artif Intell Med. 2011; 53: 57–71.

42.

Chang

H.Y.

, Sneddon

J.B.

, Alizadeh

A.A.

. Gene expression signature of fibroblast serum response predicts human cancer progression: similarities between tumors and wounds. PLoS Biol. 2004; 2: E7.

43.

Chang

H.Y.

, Nuyten

D.S.

, Sneddon

J.B.

. Robustness, scalability, and integration of a wound-response gene expression signature in predicting breast cancer survival. Proc Natl Acad Sci USA. 2005; 102: 3738–43.

44.

Haibe-Kains

, Desmedt

, Piette

. Comparison of prognostic gene expression signatures for breast cancer. BMC Genomics. 2008; 9: 394.

45.

Haibe-Kains

, Desmedt

, Sotiriou

, Bontempi

A comparative study of survival models for breast cancer prognostication based on microarray data: does a single gene beat them all?

Bioinformatics. 2008; 24: 2200–8.

46.

Desmedt

, Piette

, Loi

. Strong time dependence of the 76-gene prognostic signature for node-negative breast cancer patients in the TRANSBIG multicenter independent validation series. Clin Cancer Res. 2007; 13: 3207–14.

47.

Daemen

, Gevaert

, De

M.B.

Integration of clinical and microarray data with kernel methods.

Conf Proc IEEE Eng Med Biol Soc. 2007; 2007: 5411–5.

48.

Sun

, Goodison

, Li

, Liu

, Farmerie

Improved breast cancer prognosis through the combination of clinical and genetic markers.

Bioinformatics. 2007; 23: 30–7.

49.

Schmidt

, Victor

, Bratzel

. Long-term outcome prediction by clinicopathological risk classification algorithms in node-negative breast cancer—comparison between Adjuvant!, St. Gallen, and a novel risk algorithm used in the prospective randomized Node-Negative-Breast Cancer-3 (NNBC-3) trial. Ann Oncol. 2009; 20: 258–64.

50.

Warnat

, Eils

, Brors

Cross-platform analysis of cancer microarray data improves gene expression based classification of phenotypes.

BMC Bioinformatics. 2005; 6: 265.

51.

Stec

, Wang

, Coombes

. Comparison of the predictive accuracy of DNA array-based multigene classifiers across cDNA arrays and Affymetrix GeneChips. J Mol Diagn. 2005; 7: 357–67.

52.

Guo

, Zhang

, Li

. Towards precise classification of cancers based on robust gene functional expression profiles. BMC Bioinformatics. 2005; 6: 58.

53.

, Shen

, Song

. Module-based prediction approach for robust inter-study predictions in microarray data. Bioinformatics. 2010; 26: 2586–93.

54.

Yang

, Cheng

, Zhang

Average rank-based score to measure deregulation of molecular pathway gene sets.

PLoS One. 2011; 6: e27579.

55.

, Yoon

B.J.

, Dougherty

E.R.

Accurate and reliable cancer classification based on probabilistic inference of pathway activity.

PLoS One. 2009; 4: e8161.

56.

Abraham

, Kowalczyk

, Loi

, Haviv

, Zobel

Prediction of breast cancer prognosis using gene set statistics provides signature stability and biological context.

BMC Bioinformatics. 2010; 11: 277.

57.

Blazadonakis

M.E.

, Zervakis

M.E.

, Kafetzopoulos

Integration of gene signatures using biological knowledge.

Artif Intell Med. 2011; 53: 57–71.

58.

Blazadonakis

M.E.

, Zervakis

The linear neuron as marker selector and clinical predictor in cancer gene analysis.

Comput Methods Programs Biomed. 2008; 91: 22–35.

59.

Kuo

W.P.

, Jenssen

T.K.

, Butte

A.J.

, Ohno-Machado

, Kohane

I.S.

Analysis of matched mRNA measurements from two different microarray technologies.

Bioinformatics. 2002; 18: 405–12.

60.

Pedotti

, ‘t Hoen

P.A.

, Vreugdenhil

. Can subtle changes in gene expression be consistently detected with different microarray platforms? BMC Genomics. 2008; 9: 124.

61.

Miller

L.D.

, Smeds

, George

. An expression signature for p53 status in human breast cancer predicts mutation status, transcriptional effects, and patient survival. Proc Natl Acad Sci USA. 2005; 102: 13550–5.

62.

Schmidt

, Bohm

, von

T.C.

. The humoral immune system has a key prognostic impact in node-negative breast cancer. Cancer Res. 2008; 68: 5405–13.

63.

Subramanian

, Kuehn

, Gould

, Tamayo

, Mesirov

J.P.

GSEA-P: a desktop application for Gene Set Enrichment Analysis.

Bioinformatics. 2007; 23: 3251–3.

64.

Vapnik

The Nature of Statistical Learning Theory. New York, NY: Springer; 1995.

65.

Dupuy

, Simon

R.M.

Critical review of published microarray studies for cancer outcome and guidelines on statistical analysis and reporting.

J Natl Cancer Inst. 2007; 99: 147–57.

66.

Chiu

S.H.

, Chen

C.C.

, Lin

T.H.

Using support vector regression to model the correlation between the clinical metastases time and gene expression profile for breast cancer.

Artif Intell Med. 2008; 44: 221–31.

67.

Demsar

Statistical comparisons of classifiers over multiple data sets.

Journal of Machine Learning Research. 2006; 7: 1–30.

Prediction of Breast Cancer Metastasis by Gene Expression Profiles: A Comparison of Metagenes and Single Genes

Abstract

Background

Methods

Results

Conclusion

Keywords

Background

Results

Features and models

Internal performance of MG-and SG-models

Feature set transferability

Comparison of classifier performance based on MG- and SG-feature sets in independent datasets

Classifier transferability and performance

Comparison of classifier performance

Discussion

Conclusions

Methods

Datasets used in this study

Dataset Processing

Single gene and metagene features

Classifier Building

Classification performance assessment

Endpoint/outcome

Comparison of external validation performance

List of abbreviations used

Author Contributions

Funding

Competing Interests

Disclosures and Ethics

Footnotes

Acknowledgements

References