The goal of feature selection is to identify informative genes from thousands of available genes. The informative genes are those that have high discriminative powers, and have low correlations between each pair of them 16 . Selecting informative genes helps not only overcome the curse of dimensionality problem and thus improve the prediction accuracy, but also reveal meaningful biological explanations of the dataset. Theoretically, any wrapper or filter feature selection method, such as t-test, mutual information measurement, etc, can be used. However, one drawback of these approaches is that the feature selection criterion is designed regardless of the classifier design. In 17 , it has been observed that almost all feature selection methods have some assumptions of the distribution of the data, and these assumptions usually affect the performance of the classifiers. Therefore, it is important to design a feature selection method that is suitable for the classification method that will be used.

Bressan and Vitrià 17 showed that there is a close link between nonparametric discriminant analysis (NDA) 18 and instance-based classifiers. In that work, a modified NDA was applied to improve the performance of NN for face recognition. Since CBR-based methods also belong to instance-based classifiers, we believe that the idea of NDA also helps to improve, at least not downgrade the performance of CBR. Our NMM criterion is defined based on the notion of NDA. Instead of directly using the furthest within-class distance as in the original NDA method, in our method training samples in each class ω _k are firstly grouped into many clusters { } so that the samples in each cluster have similar patterns. The objective of NMM is to maximize the between-class distance of samples while minimize the within-cluster distance. For one sample x _i ∈ ω _k , we define its nearest between-class neighbor as

Similarly, its furthest within-cluster neighbor is defined as

where C [x] indicates the cluster that x belongs to.

Then the nonparametric margin of x _i is

where is the nonparametric nearest between-class distance for x _i , and is the furthest within-cluster distance. Obviously, the larger Θ _i is, the more likely that x _i is correctly classified. Therefore the learning objective of NMM is to select a subset of genes from Φ to maximize the nonparametric margin for all the samples, i.e. to maximize

where δ _i is the sample x _i in the space of selected features, which is represented as δ _i = [h ₁(x _i ), ⋯, h _T (x _i )] ^T .

Not surprisingly, we find that if each class contains only one cluster, the NMM criterion is equal to the optimization objective of NDA (see proof 1). Since it has been proved that there are close relationships between NDA and instance-based classifiers such as NN 17 , we believe that our margin maximum criterion also benefits the design of a robust CBR classifier. Moreover we replaced the furthest within-class distance with furthest within-cluster distance, which makes our approach more robust to outliers, considering that the outliers that usually exist in microarray data might make the furthest within-class distance extremely large. Another major difference between our method and NDA is that, NDA performs feature reduction by finding a weighted combination of all the features, while NMM aims at selecting a subset of features. This property is important since the selected features can be used to reveal some biological significance.

In our method, we normalize the original data and then perform feature pre-selection and sample clustering to define within-cluster neighbors. We use the same normalization method as in 19 , which includes base 10 log-transformation as well as normalization to mean 0 and variance 1. For the data that contains negative values, we do not perform log-transformation.

In microarray data, the gene dimension is extremely large compared to the small number of samples. Many of these genes are not differentially expressed across the samples of different classes and thus do not contain very useful information. It is likely that too many non-informative genes in the data will undermine the clustering results. In order to improve the clustering performance, we implement gene pre-selection before clustering. Another benefit of removing some non-informative genes is that it can drastically ease the computational burden in subsequent processing procedures.

Approaches that can remove non-informative genes have been studied in many literatures, for instance t-test 20 , mutual information (MI) maximization 16 , etc. Instead of these parametric methods, we use a nonparametric scoring algorithm presented in 13 . For binary classification which involves two classes ω ₀ and ω ₁, the score of a feature ϕ _m is

where ⟦A⟧ equals 1 if A is true, otherwise 0. |ω| is the number of samples in ω.

The genes whose scores are below a threshold θ _p will be removed, and the remaining genes are used for further processing. Compared with the parametric methods such as t-test and MI maximization, this method is less sensitive to outliers, since it does not rely on any statistical values (mean, standard deviation, etc.) of the data, which can be highly affected by outliers.

This nonparametric method can be easily generalized to multiclass problems by considering all the possible binary cases. For a K class problem, the score of a feature ϕ _m is

After gene pre-selection, we group samples in each class into some clusters. Although there are many choices of clustering approaches, hierarchical clustering 21 is the most commonly used one for microarray analysis. The preference of hierarchical clustering in microarray analysis is due to its good performance 2 and, moreover, it does not require a pre-specification of the number of clusters.

We use the most common type of hierarchical clustering. At the initial level, each sample forms its own cluster. At each subsequent level, the two 'nearest' clusters are combined to form one bigger cluster. We use method = 'furthest' which means the distance between two clusters is the maximum of all the distances between any sample in one cluster and any sample in the other cluster. The 'furthest' metric is used since it is not highly sensitive to outliers compared with the other metrics such as 'nearest' and 'average'. We empirically set a threshold θ _h for clustering, which means that for each class, the clustering procedure terminates when the distance between any two clusters is larger than θ _h .

Here the NMM criterion is optimized in an additive approach, which operates iteratively. At each iteration, one feature is selected. Assuming that until the (t - 1)-th iteration the margin is J _t-1, at iteration t the feature h _t will be selected to maximize

During the optimization procedure, however, when one feature is selected, for each sample its nearest between-class neighbor and furthest within-cluster neighbor might change. In another word, the optimization of J _t might change J _t-1, and for each sample, many other samples might become its nearest between-class neighbor or furthest within-cluster neighbor in subsequent processing. Therefore we should maintain the distance between any two samples in each iteration, which is computationally expensive. In order to reduce computational complexity, we maximize the following formula instead of directly optimizing Equation (4),

Proof 2 shows that Equation (11) is a low bound of Equation (4), which implies that we can maximize Equation (4) by optimizing Equation (11).

In this section we carry out experiments on four publicly available real data sets that have been widely studied. Brief descriptions of these data sets are as follows. Please refer to the original papers for more details of each data set.

Similar to that on simulated data, here we also compared ANMM4CBR with SVM, kNN and LogitBoost. BW and MRMR were used to select features for SVM and kNN classification. Since the standard SVM is tailored for binary classification, in multiclass data sets we used the one-versus-all (OVA) 26 approach, which firstly solves many binary problems and then combines the results to solve the multiclass problem. Given a k class problem, OVA trains k binary classifiers, each focuses on classifying one class against the others. A new sample will take the class label of the classifier with the largest real valued output from all k classifiers. For LogitBoost, we used the same approach of 6 , in which multiclass problems were solved by combining OVA results in a Bayes framework.

Table 1 gives the classification results of the six methods on the four microarray data sets. The results demonstrate that these algorithms show different performance on different data sets. On Leukemia data, all methods achieve comparable results, with ANMM4CBR and MRMR+SVM perform slightly better. On Colon data, ANMM4CBR performs better than the other methods by a large margin. We can see that with different number of selected features, ANMM4CBR consistently achieves the highest accuracy. On SRBCT data, the results are different with different numbers of features. When the feature number is small, SVM and LogitBoost perform better than ANMM4CBR; when the feature number is large, ANMM4CBR performs better. Table 1 also shows that, results of ANMM4CBR on GCM are not encouraging. SVM performs better than the other algorithms on GCM data.

We now take a closer look at the results in Table 1. We can see that ANMM4CBR performs much better than all the other algorithms on the Colon data, while only achieves comparative results on the Leukemia data. This is because Leukemia is a simple data on which many algorithms have reported impressive results. Therefore it is not surprising that all six algorithms in our experiment can have similar good results. In contrast, it was reported in 27 that the Colon data might have a sample contamination problem, and therefore the much better performance of ANMM4CBR on Colon data demonstrated its robustness to noise in the data sets.

Although when the feature number is 40 and 50, ANMM4CBR performs the best on SRBCT, on the two multiclass data sets ANMM4CBR cannot achieve comparative results with SVM and LogitBoost. It is shown in Table 1 that SVM and LogitBoost perform better than ANMM4CBR, and ANMM4CBR performs better than kNN. However, we argue that this does not indicate that ANMM4CBR cannot get good results on multiclass problems. Note that the same as kNN, ANMM4CBR can be directly used to solve a multiclass problem. Therefore in ANMM4CBR method the number of iterations is equal to the number of selected features. But in SVM and LogitBoost algorithms, we used OVA method to make the final prediction, which needs to solve k (class number) binary problems. When each binary classifier selects s features, the total number of selected features will be O(s × c). This means that with the same iteration number, SVM and LogitBoost have to use more features than ANMM4CBR and kNN.

Here we made another experiment on GCM. We compared ANMM4CBR with MRMR+SVM, which showed the best performance on GCM data in Table 1. In each comparison of this experiment, the number of features selected by ANMM4CBR was equal to the total number of genes that are selected for all the binary classifiers. Since we performed experiment for 100 times and in each time the total gene number may be different, we firstly carried out SVM experiment and then calculated the total number of genes. The results are shown in Table 2, which demonstrate that ANMM4CBR outperforms SVM by a large margin when they choose the same number of genes.

Since ANMM4CBR is a CBR-based method, we would like to compare it with other CBR methods that have been applied to microarray classification problems. Because both source code and data sets used in 11 are not available, we did not compare our method with the gene-CBR method in 11 . We compared ANMM4CBR with the mixture of experts for case-based reasoning (MOE4CBR) method 12 , which builds CBR classifiers based on the idea of mixture of experts. We applied our ANMM4CBR method to the same microarray data with the same experimental set as that in 12 , i.e., using the training and testing data suggested in 3 on the Leukemia data, and using leave-one-out cross-validation on the Lung data and average the results obtained from 20 trials. The Lung data contains 39 lung cancer samples with 18,117 gene expression levels. This data set is classified into two categories, recurrence (23 samples) and nonrecurrence (16 samples). The Lung data was not used in previous experiments because there are missing values. The same as that in 12 , here missing values were imputed using the weighted k-nearest neighbor method 28 .

In 12 , the classification accuracies on Leukemia and Lung data are 74% and 70% respectively. 712 out of 7,129 genes were selected for Leukemia data classification and 1,811 out of 18,117 genes were selected for Lung data classification. When the same number of genes are selected, the classification results of ANMM4CBR are 91% on Leukemia and 75% on Lung. Moreover, on the Leukemia data, the best result obtained by ANMM4CBR is 94% when only 23 genes are selected. This shows that ANMM4CBR outperforms MOE4CBR, especially on the Leukemia data set.