Cost-sensitive feature selection for multi-label classification: multi-criteria decision-making approach Open Access

In the big data and artificial intelligence (AI) era, feature selection has become an essential step toward reducing computational complexity, enhancing model performance, and improving interpretability. Although traditional machine-learning algorithms typically assume a single label for each instance, real-world scenarios are often more complex, with a single instance potentially associated with multiple labels. Consequently, multi-label feature selection has become a prominent research topic, and numerous methods have been developed to address these challenges [1]. Feature-selection methods are generally classified into three categories. Filter methods assess feature quality based on predefined criteria or statistical measures. In contrast, wrapper methods make use of the learning algorithm. Embedded methods, on the other hand, integrate feature selection into the learning process. Filter-based methods are particularly preferred for high-dimensional datasets, owing to their simplicity and computational efficiency [1]. In particular, information-theory-based measures have been extensively utilized in feature selection because of their ability to elucidate the relationships between features and labels.

Many existing feature selection methods often assume that the data are readily available and free of charge. Obtaining data often incurs significant time, money, and computational resource costs [2]. For instance, a predictive model for disease risk may rely on features, such as blood test results, imaging scans, and genetic information. Although genetic tests offer valuable insights, they are typically expensive and not routinely performed. By integrating cost considerations into feature selection, healthcare providers can prioritize affordable and frequently available tests, such as blood tests, while balancing predictive accuracy with budget constraints. The goal is to optimize model performance while minimizing data acquisition costs [3]. Although high-cost tests may be irreplaceable in high-risk applications, low-cost tests can serve as valuable preliminary screenings in resource-constrained environments. This approach ensures that the models are practical and effective, thereby enhancing patient care while managing healthcare expenses. In general, two main costs are associated with learning from data: test costs and misclassification costs. Test costs refer to the expenses incurred in obtaining data, whereas misclassification costs represent the costs of incorrect predictions made by the model. This study explicitly emphasizes the feature cost, a subset of test costs that accounts for expenses related to obtaining specific features of the data. The primary challenge in cost-sensitive feature selection is achieving an effective balance between the feature cost and predictive accuracy. Selecting low-cost features can reduce the overall costs but may compromise the predictive performance of the model. In a multi-label scenario, this is further complicated by the need to simultaneously maximize feature relevance for multiple labels while minimizing feature costs. This creates two conflicting objectives: optimizing for one often detracts from the other.

However, MCDM has become a leading approach for tackling complex problems that involve multiple, often conflicting criteria in a structured way. These methods empower decision makers to consider a range of factors and derive comprehensive solutions. Classical MCDM methods like elimination and choice expressing reality (ELECTRE), technique for order of preference by similarity to ideal solution (TOPSIS), complex proportional assessment (COPRAS), Viekriterijumsko Kompromisno Rangiranje (VIKOR), weighted aggregated sum product assessment (WASPAS), and multiplicative multi-objective optimization by ratio analysis with full multiplicative form (MULTIMOORA) have been extensively used to address these challenges. However, to overcome the limitations of traditional approaches, researchers have introduced new methods, such as evaluation based on distance from the average solution (EDAS), combinative distance-based assessment (CODAS), combined compromise solution (CoCoSo), and WISP [4]. In addition, to better manage the inherent uncertainty in decision-making processes, most of these methods have been adapted to operate within fuzzy environments. This adaptation enabled the incorporation of imprecise or ambiguous information, resulting in more robust and realistic solutions for various fields.

Therefore, the application of an MCDM approach to address the cost-sensitive multi-label feature-selection problem is logical and promising. Unlike single-label classification, multi-label classification requires consideration that each feature may have a different level of contribution to the prediction of each label. A feature that is inexpensive and valuable for one label may be irrelevant to another, adding to the complexity of feature selection. Although considerable work has been conducted on cost-sensitive feature selection for single-label classification, multi-label scenarios remain relatively unexplored. To address this gap, we propose a novel methodology that fuses feature importance with feature cost within an MCDM framework. This fusion aims to optimize feature selection for multi-label classification tasks by balancing predictive performance with cost efficiency. The key contributions of this study are as follows:

• (1)

  We introduce a novel approach that frames an MCDM problem for cost-sensitive multi-label feature selection by leveraging mutual information. Our method uniquely fuses feature importance along multiple labels with cost factors, thereby enabling a more comprehensive evaluation of features in multi-label scenarios.

• (2)

  We converted the data to Fermatean fuzzy sets, which are powerful tools for handling uncertainty and imprecision in decision-making processes. We then employed the Fermatean fuzzy WISP method to rank features, considering both their relevance to the labels and their associated costs.

• (3)

  Unlike existing methods that integrate cost through penalties and select features via a greedy search [5, 6], the proposed approach adopts an MCDM-based strategy for feature ranking. This method achieves globally optimal outcomes by balancing the trade-offs between conflicting objectives, marking a significant advancement over existing techniques.

• (4)

  We conducted extensive experiments on ten benchmark datasets to evaluate the performance of the proposed method. The results demonstrated the effectiveness of our approach in selecting relevant features while minimizing costs, outperforming existing methods in terms of predictive accuracy and cost efficiency.

The structure of the paper is as follows: Section 2 reviews related work. Section 3 introduces the key notation and fundamental concepts. Section 4 details the proposed methodology for cost-sensitive multi-label feature selection. Section 5 outlines the experimental setup and reports the results obtained from experiments on ten benchmark datasets. Lastly, Section 6 concludes the paper and proposes directions for future research.

In this section, we review some of the key studies related to multi-label feature selection, cost-sensitive feature selection, and MCDM.

Multi-label feature selection has evolved significantly over the years, with various approaches addressing the complexities of real-world scenarios. Early approaches predominantly utilized problem-transformation techniques [1], which converted multi-label problems into multiple single-label problems to facilitate feature selection. However, these approaches often fail to identify intricate nonlinear relationships between features and labels. The field has subsequently witnessed a surge in information-theory-based feature selection methods. Early work on single-label feature selection [7, 8] was later extended and adapted to address the more complex challenges of multi-label feature selection [9]. One approach utilized both mutual and interaction information to guide feature selection [10], while a subsequent method introduced a scalable criterion to handle large label sets more efficiently by leveraging simple dependency functions [11]. Building on these developments, a multi-label feature selection algorithm incorporating two conditional mutual information terms [12] was proposed. Additionally, a framework for information-theoretic methods was developed, proposing a novel approach based on three lower-order information-theoretic terms [13]. However, despite these advancements, existing methods do not account for the costs associated with data in multi-label feature selection.

Several researchers have incorporated feature-associated costs into single-label feature selection. One framework introduced cost constraints into existing single-label feature selection algorithms [3], while another method employed simulated annealing to account for test costs during the feature selection process [14]. While significant progress has been made in cost-sensitive feature selection for single-label scenarios, comparatively less attention has been given to multi-label data. One approach transformed logical labels into label distributions and ranked features using neighborhood mutual information with cost considerations [5]. More recently, another method [6] utilized normalized conditional mutual information with a cost penalty to evaluate feature relevance, followed by a greedy sequential search to derive a cost-constrained feature subset. However, existing multi-label cost-sensitive feature selection methods often overlook the overall feature space structure, leading to suboptimal solutions. This gap in the literature highlights the need for more comprehensive and globally aware cost-sensitive feature selection methods for multi-label data.

Since the introduction of fuzzy sets by Zadeh [15], fuzzy-based MCDM methods have found widespread application in various fields, including supply chain, health and safety, energy, and waste management. This field has experienced significant advancements with the introduction of more sophisticated fuzzy set theories. Atanassov [16] generalized fuzzy sets to intuitionistic fuzzy sets, followed by developing Pythagorean [17] and Fermatean fuzzy sets [18], each offering increased flexibility in representing complex information. Fuzzy-based MCDM tools are particularly adept at handling conflicting criteria, which makes them potentially valuable for feature selection tasks. The WISP method recently developed [4], integrates the weighted sum and product approaches. In their comparative study [19], the authors evaluated WISP and other MCDM methods for real-world decision-making problems. Since then, several extensions of the WISP have been proposed. For instance, the WISP method has been adapted to various set theories, such as fuzzy sets, by one group of researchers [20], and to grey sets for sustainable supplier selection by another [21]. It was later extended to rough sets [22] and intuitionistic fuzzy sets [23], and further expanded to interval-valued Pythagorean fuzzy sets [24]. Another extension applied it to interval-valued Fermatean fuzzy sets for telescopic forklift selection [25]. Early attempts to address this gap include the work of one group [26], MCDM methods to evaluate feature selection techniques for text classification. Another proposed an MCDM-based approach for feature selection in text classification, utilizing ridge regression to develop a decision matrix and the COPRAS method to rank the features [27]. Despite these adaptations, its application to feature selection remains largely unexplored.

This section presents the essential notation and core concepts that will be used throughout the rest of the study. The basic definitions are presented in  Supplementary File.

Consider $D$ as a multi-label dataset consisting of $N$ instances and $L$ labels. Subsequently, each instance $Ii$ is characterized by a feature vector and label vector, expressed as $Ii=(Xi,Li)$ for $i=1,…,N$⁠. Here, the feature vector $Xi$ is $(xi1,…,xiM)$ and the label vector $Li$ is $(li1,…,liL)$⁠. The objective of multi-label classification is to learn a function that can predict the label vector $Lk$ for any given feature vector $Xk$⁠.

Mutual information is a key concept in information theory that describes the amount of information shared by two random variables. It is rooted in Shannon’s entropy, a measure introduced by Shannon in his seminal work on information theory [28], which introduced the concept of entropy to quantify uncertainty in random variables.

Recently, an innovative MCDM approach was introduced that combines the weighted sum and weighted product methods [4]. This newly developed method builds upon the foundations of established techniques such as MULTIMOORA, WASPAS, and CoCoSo, and offers a more comprehensive solution for complex decision-making scenarios. In this study, we extend their method to FFSs and apply it to solving the cost-sensitive multi-label feature selection problem. The detailed steps of this method are provided in the  Supplementary File.

This section describes the design of the proposed cost-sensitive multi-label feature selection method, which consists of three distinct stages.

• (1)

  Feature Relevance Assessment: We started by calculating the mutual information between each feature and label. This calculation measures feature relevance and provides a foundation for subsequent decision-making.

• (2)

  Decision Matrix Construction and Fuzzy Transformation: We construct a decision matrix by integrating the cost information with the previously determined feature relevance. In this matrix, features serve as alternatives, whereas labels and their associated costs serve as criteria. To better manage the uncertainty and imprecision inherent in the decision-making process, we transform this decision matrix into Fermatean fuzzy sets. This transformation enhanced the flexibility of the proposed approach.

• (3)

  Feature Ranking: We employed the Fermatean fuzzy WISP method to rank features based on their relevance to labels and associated costs. This final stage synthesizes information from the previous steps to produce a ranked list of features. The Fermatean fuzzy model was chosen over other generalized fuzzy models due to its enhanced capacity to capture a wider range of uncertainties and offer a more refined representation of both membership and non-membership degrees.

The subsequent steps will elucidate each stage of this procedure in detail:

• Step 1. Let $X:i$⁠, $i=1,…,M$⁠, represent the feature vector and $L:k$⁠, where $k=1,…,L$ represents the column-wise label vectors. Let $Ci$ denote the cost of $i$-th feature. We considered the features as alternatives and labels, along with cost, as criteria. The decision matrix is obtained by calculating the mutual information as follows:

• Step 2. Next, this decision matrix is transformed into FFSs as follows [29]:

where

with

Furthermore, the hesitation degree $πij$⁠, is computed as $πij=1−μij3−νij33$ for all $i=1,…,M$⁠, and $j=1,…,L,L+1$⁠. Thus, the problem of cost-sensitive multi-label feature selection is transformed into an MCDM problem. Here, the criteria with respect to the labels are the maximization criteria, whereas the cost is the minimization criterion. The objective is to maximize the relevance of the feature with respect to the labels and minimize the cost associated with the feature.

• Step 3. The sum and product of each alternative for the maximization and minimization criteria were calculated as follows:

where $wj$ is the weight corresponding to the criterion $j$⁠, $wj∈[0,1]$⁠, and $∑j=1Lwj=1$⁠. For the criteria with respect to the labels, the information entropy values were considered as the weights, whereas the importance of the cost was considered as the weight for the cost criterion.

• Step 4. The Fermatean fuzzy values were then converted to crisp values using the score function as follows:

• Step 5. The utility measures are calculated in the following manner:

• Step 6. The utility measures are recalculated as follows:

• Step 7. The final utility score $ui$ is obtained as follows:

• Step 8. The overall ranking of the features was determined by sorting the alternatives based on their utility scores. The feature with the highest utility score was considered the best.

The pseudo code of this proposed method is provided in the  Supplementary File.

In this section, we demonstrate the effectiveness of the proposed method through a comprehensive set of experiments that utilize various evaluation metrics across multiple datasets.

In the experiment, we utilize ten benchmark multi-label datasets collected from the MULAN [¹] and UCO [²] libraries. The characteristics of each dataset are listed in Table 1. Here, “label cardinality” denotes the mean number of labels assigned to each instance, while “label density” represents the normalized label cardinality [30]. Because of the absence of inherent cost information in the datasets, we adopted a common approach from the existing literature [3] to simulate the cost factors. Specifically, we assigned each feature a random value between 0 and 1 to represent its associated cost.

To evaluate the performance of the algorithms, we employed three example-based metrics: ranking loss, coverage, and average precision, accompanied by two metrics based on labels: Macro-F1 and Micro-F1 [1]. The lower scores of ranking loss and coverage indicate better performance (denoted by $↓$⁠), while greater scores of average precision, Macro-F1, and Micro-F1 signify superior performance (denoted by $↑$⁠). The experiments were conducted using an ML-kNN classifier [31] with k = 10 and a smoothing parameter of 1. To ensure the reliability of the results, we employed a fivefold cross-validation approach.

We compared the proposed approach against two existing cost-sensitive multi-label feature selection techniques: cost-sensitive feature selection on multi-label data (CFSM) [5] and cost-constrained feature selection with a mutual information approach (CFSMIA) [6]. Furthermore, it was compared to seven conventional multi-label feature selection algorithms: D2F [10], IGMF [32], LRDG [33], ML-COPRAS [27], MLACO [34], SCLS [11], and STFS [13]. These methods were chosen based on their relevance to the proposed approach, despite the existence of numerous alternatives.

We compared the proposed method with two existing cost-sensitive multi-label feature selection methods and seven conventional multi-label feature selection techniques. Each algorithm is evaluated under various cost constraints to assess its performance. First, the algorithms ranked the features, and each metric was applied as the total feature cost was progressively limited. The metric values across different cost constraints are presented in Figure 1, where the $x$-axis represents the cost values, and the $y$-axis indicates the corresponding metric values at each cost level. The averages of these values are compiled in Table 2 to provide a clear quantitative summary; the best values are highlighted in bold. We also conducted a win-tie-loss analysis to compare the proposed method against each benchmark method. The last row of the table includes the average metric values for each comparison method and their average rankings. The results for the other four metrics are provided in a separate  Supplementary File.

With respect to average precision, as illustrated in Figure 1, the proposed method consistently outperformed the comparison methods across most cost levels. While some algorithms show non-monotonic trends, with fluctuations in average precision as costs increase, the proposed method exhibits a more stable and reliable performance throughout the cost spectrum. This increasing trend reflects the ability of the algorithm to rank features by considering their relevance to labels and the associated costs. The proposed method also achieved a superior performance for the other four metrics evaluated. Notably, it performs well across all cost constraints on the social dataset. In contrast, most algorithms encountered challenges on the Genbase and Yeast datasets, likely because of the high relative difference between the number of features and instances in these datasets. Despite this, the proposed method continued to outperform the other algorithms, demonstrating its robustness across diverse datasets.

Based on the average values presented in Table 2, the proposed method outperforms all other methods in every dataset except for one (Education), where it ranks lower. It also shares the top position with ML-COPRAS in the Genbase dataset. Additionally, in terms of ranking loss and coverage, the proposed method delivered the best results across nine datasets, with the exception of the Recreation dataset. Macro-F1 attained the highest scores in all datasets, whereas Micro-F1 yielded seven datasets. Overall, these results highlight the effectiveness and adaptability of the proposed method for cost-sensitive multi-label feature selection.

In this section, we employ two nonparametric tests, the Friedman [35] and Bonferroni-Dunn tests [36] to systematically test the significance of the proposed method. Table 3 lists the Friedman test results for various evaluation metrics, along with the critical value. The null hypothesis of equal performance is rejected because the $FF$ values exceed the critical value. Subsequently, we utilized the Bonferroni-Dunn post hoc test to analyze the relative performance of the comparison methods. The performance of the two algorithms differs significantly if the average rank differs by one critical difference (CD). Figure 2 presents CD diagrams with respect to various evaluation metrics. Although the performance of the proposed method is comparable to that of some methods, it demonstrates a significantly superior average classification performance compared to the other algorithms.

To intuitively evaluate the stability of the proposed method across different datasets, we conducted a stability analysis using radar charts. Performance scores were normalized to the range $[0.1,0.5]$⁠. A more regular polygon shape formed by the connecting lines indicates greater stability of the corresponding algorithm across different datasets. Figure 3 illustrates the stability analysis results of the proposed method for ten benchmark datasets and five evaluation metrics. The results indicated that the proposed method was more stable than the other methods.

In this study, a novel approach to cost-sensitive multi-label feature selection was proposed that effectively integrates feature importance with feature costs within an MCDM framework. The method begins by constructing a decision matrix utilizing mutual information along with cost values, which are then transformed into Fermatean fuzzy numbers. Subsequently, the Fermatean fuzzy WISP method is employed to rank the features. Unlike existing approaches that rely on the penalty-based integration of costs and employ greedy search strategies, the proposed MCDM-based technique offers a more comprehensive and flexible solution for balancing feature importance and costs. This strategic advantage was demonstrated through extensive experiments conducted on ten benchmark datasets, where the proposed method consistently outperformed state-of-the-art feature-selection algorithms across five evaluation metrics. The ability to optimize feature selection while accounting for costs represents a significant advancement, particularly in real-world applications, where data acquisition can incur substantial time, financial, and computational expenses. Future research will focus on extending the proposed method to handle missing values in datasets, a challenge frequently encountered in many practical applications. In addition, this approach can be adapted for parallel processing, potentially enhancing its efficiency and scalability. By addressing these limitations, the versatility and applicability of a cost-sensitive multi-label feature selection approach can be significantly enhanced, making it more robust and practical for a wide range of real-world scenarios.

This work was supported by the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (2021R1A2C1014338 and RS-2024-00344752). This research was also supported by the Department of Integrative Biotechnology, Sungkyunkwan University (SKKU) and the BK21 FOUR Project.

The supplementary material for this article can be found online.