A hybrid deep learning model for analyzing the sentiments of products

With the widespread use of the internet, businesses can now reach a global audience, and customers can access products from anywhere in the world. The traditional boundaries of commerce have been transcended, rendering brick-and-mortar stores no longer the sole avenue for purchasing products [1]. The advent and popularity of digital storefronts and e-commerce sites have revolutionized the retail landscape, enabling customers to conveniently explore and procure a diverse range of products from virtually anywhere in the world [2]. The internet is flooded with an overwhelming abundance of reviews and comments, often influencing consumer decisions and shaping brand reputations. This has significantly transformed the way consumers engage with businesses, placing more emphasis on digital interactions and feedback.

Customer reviews have become an essential factor in the decision-making process for both consumers and businesses. For individuals, these reviews offer first-hand insights into the experiences of others, providing a more authentic and detailed understanding of a product or service beyond traditional advertising or marketing efforts [3]. This peer feedback helps consumers make informed choices, reducing uncertainty and risk when selecting similar products. For businesses, customer reviews serve as a direct line of communication with their audience, offering real-time feedback on customer satisfaction, product performance, and areas that may require improvement. Positive reviews can bolster a company's reputation, while negative feedback can highlight critical issues that need attention.

In today's digital landscape, where content is generated at an overwhelming pace, especially on social media platforms, manually sifting through reviews and opinions is impractical. This is where sentiment analysis (SA) comes into play [4]. By leveraging machine learning (ML) and natural language processing (NLP) techniques, sentiment analysis automates the process of evaluating the emotional tone behind customer feedback. It enables companies to efficiently monitor consumer attitudes, identify trends, and respond to emerging issues in a timely manner. As the volume of online feedback continues to grow, sentiment analysis has become an essential tool for businesses aiming to maintain a competitive edge in an increasingly customer-centric market.

Sentiment analysis can be performed at multiple levels, including document, sentence, and aspect levels. At the document level, the analysis focuses on determining the general sentiment of the whole text, categorizing it as positive, negative, or neutral [4]. Sentence-based sentiment analysis, on the other hand, focuses on individual sentences, assigning a sentiment to each one separately, which is useful when mixed sentiments are present in a text. ABSA is more fine-grained [5]. It focuses on particular features or aspects mentioned in the text and analyzes the sentiment associated with each of them. For example, in a product review, aspect-based analysis might identify sentiment about the “battery life” and “camera” separately. This method provides more detailed insights and is especially valuable in complex reviews or feedback. ABSA is the most modern and advanced technique because it captures sentiment at a granular level, offering richer and more actionable insights for businesses.

Current approaches to ABSA often depend on lexicon-based methods or ML models. The lexicon-based methods are tailored to specific domains and can be difficult to construct, while ML models require well-structured input for effective training [6]. Another limitation of older approaches is their inability to accurately extract features, resulting in incorrect identification of aspects and opinions [7]. This research introduces an improved ABSA method using a hybrid model, “DeBERTa-IDCNN-BiLSTM,” designed to enhance accuracy by addressing the limitations of existing techniques. The proposed model delivers the following key contributions, enhancing the overall effectiveness of product recommendations:

1. Examine and determine the key factors that impact the performance of feature extraction and sentiment analysis models

2. A novel hybrid model combining Transformer, deep learning, and recurrent neural network (RNN) approaches is developed to leverage the strengths of both Transformer and CNN models for sentiment analysis

3. The proposed approach has been rigorously tested on four widely recognized datasets. Experimental results show that the model surpasses recent baseline approaches, confirming its superior accuracy and reliability.

The rest of the paper is structured as follows. Section 2 reviews past approaches to ABSA. Section 3 presents a comprehensive description of the DeBERTa-IDCNN-BiLSTM-CRF model. Section 4 presents the results, while Section 5 provides the conclusion.

This section reviews several studies carried out in the field of sentiment analysis. It highlights significant progress achieved through the use of machine learning and deep learning methods.

Sangeetha et al., [8] propose a hybrid model that leverages an optimization-based algorithm and the RNN-LSTM model to classify sentiments into different categories. The model was tested using product reviews from Amazon. It showed improved accuracy and strong performance in sentiment classification. Another study introduced a hybrid model called DistilRoBiLSTMF [9]. This model aims to extract rich contextual information from complex sentences. Its primary goal is to detect and classify sentiments accurately. The results demonstrated the model's ability to handle challenging sentence structures and improve sentiment identification. A different study introduced a model for ABSA that identifies the sentiment polarity for each specific aspect [10]. This model incorporates contextual location information, position weighting mechanisms, and bidirectional GRU layers to enhance accuracy in sentiment detection.

In addition, Garcia & Berton proposed a hybrid approach to SA, combining lexicon-based and learning-based techniques [11]. The proposed approach is validated on Twitter data. Gupta et al. proposed a hybrid deep-learning architecture for sentiment categorization [12]. Deep convolutional networks have proven highly effective at extracting local features, leading to strong performance in analyzing lengthy text sequences. Additionally, Kim and Jeong highlighted that the growing popularity of social media and e-commerce platforms has driven significant interest in sentiment analysis among large companies. [13]. The findings revealed that, compared to other algorithms, the Random Forest model achieved the highest accuracy. Another study addresses the ABSA challenge of words with dual meanings [14]. The study explored how contextual factors influence word polarity, ultimately affecting both the overall product ratings and evaluations of specific features.

Other innovative approaches have been introduced as well. Albadani et al. [15] use a transformer to generate word embeddings based on a graph neural network. It learns to choose edge types and capture intricate relationships. This approach helps in obtaining node representations for accurate sentiment classification. Nawaz [16] introduced a new framework, ASIF, for sentiment classification in reviews. The framework includes feature extraction and classification from both text and images. Five datasets were used to assess ASIF's performance. Another research work proposes the Two-State GRU (TS-GRU) model, which incorporates a feature attention mechanism to enhance sentiment analysis [17]. It focuses on identifying and classifying sentiment polarity through sequential modeling.

Based on the above discussion, it can be concluded that the primary challenges with current methodologies lie in their inability to effectively process complex user comments and their decreased accuracy when dealing with short, unstructured, or multi-aspect reviews. Given these limitations, there is a need for further research to develop more robust sentiment analysis systems. To address these challenges, this research proposes a novel framework, combining feature extraction with sentiment analysis using DeBERTa-IDCNN-BiLSTM and CRF, designed to improve sentiment analysis performance and address these issues.

This section provides a detailed description of the proposed model. The goal of the proposed approach is to enhance the extraction of aspects and identify the sentiments expressed in user text regarding various aspects. The proposed model is composed of data collection, data preprocessing, feature extraction using DeBERTa and IDCNN, and classification using attention based BiLSTM and CRF. Figure 1 shows the workflow of the proposed model.

Data serves as the essential foundation for effectively conducting sentiment analysis. The experiments utilize four diverse review datasets from multiple domains, as indicated in Table 1. The first dataset, DS-I, contains reviews of five different electronics products. The collection of product reviews is a valuable asset for tasks involving sentiment analysis and opinion mining. This dataset comprises a substantial quantity of customer evaluations pertaining to their product purchases on an e-commerce platform, making it well-suited for training and assessing sentiment analysis models. The second dataset is from SemEval (DS-II). SemEval is widely acknowledged and utilized for the assessment of semantic analysis tasks. The third dataset, DS-III, contains movie reviews, chosen to validate the proposed model's results across domains. The last dataset “MAMS” [30], containing 35,902 instances, has also been included for a more comprehensive evaluation. This dataset focuses on MAMS reviews, specifically providing descriptions related to food and services.

A number of essential actions are conducted during the preprocessing stage of a review dataset in order to make sure the data is clear and organized in an ideal way for model understanding [4]. First, all text is converted to lowercase to ensure consistency. Tokenization follows, dividing the text into individual words or tokens. Punctuation and special characters are removed to reduce noise in the data. Stopwords, such as common conjunctions, are eliminated to improve model performance. This process helps simplify the data for better analysis and processing.

Despite implementing these techniques, the textual data may still contain superfluous statements and extraneous phrasings, such as telephone numbers, monetary figures, dates, timestamps, and the like, which can impede the performance of the ABSA model. To address this issue, a technique known as “redundancy deletion” is employed. During this process, the Natural Language Toolkit's (NLTK) regular expression (regexp) tokenizer is utilized to identify and eliminate unwarranted statements and sentences. By getting rid of unnecessary and duplicate data, the ABSA model can focus on the important parts of the text, resulting in a more precise sentiment analysis.

BERT introduced bidirectional contextual embeddings, transforming NLP by jointly considering left and right contexts. Later models refined this approach: RoBERTa improved training efficiency through dynamic masking and larger datasets; XLNet combined autoregressive and bidirectional modeling using permutation-based objectives; and ALBERT reduced parameters with factorized embeddings and cross-layer sharing for efficiency. DeBERTa (Decoding-enhanced BERT with disentangled attention) advanced further by separating content and position encoding and incorporating relative position bias in attention [21]. These innovations allow DeBERTa to capture fine-grained semantic relations and consistently outperform BERT, RoBERTa, XLNet, and ALBERT. Accordingly, this study employed DeBERTa for token sequence embeddings. Each input sequence is first tokenized and embedded as follows:

Where [CLS] is a special classification token, and $ti$​represents the $ith$ token in the sequence. The input embeddings for DeBERTa are composed of content embeddings $Ec$ ​and position embeddings ​$Ec$⁠, which are not directly added as in BERT but kept disentangled. Each token embedding is represented as:

where f represents the function combining content and position information. DeBERTa's disentangled self-attention computes attention scores using both content to content and position to content interactions.

Here:

1. $Qc,KCT$ are content-based query and key vectors

2. $QcKPT$ are position-based query and key vectors

3. $d$ is the dimension of the key vectors

An attention mask is used to indicate the importance of each token in the sequence: Attention Mast = ${m0,m1,m2,...,mn}$⁠. Where $mi=1$ indicates that the token is to be attended to and $mi=0$ indicates padding or masked tokens. After passing through all transformer layers (e.g 12 layer in DeBERTa-Base), the output hidden states are:

Where $hi$ is the final hidden representation for the token $ti$⁠, and s is the hidden size. These hidden representations, specifically the embeddings of the individual tokens, are then used as input for downstream tasks such as sentiment analysis or classification, depending on the goal of the model. The final embedding for each token $ti$ ​captures its context, meaning, and positional relevance within the user review, which is crucial for understanding sentiment or other textual features.

An alternative method to traditional feature extraction approaches is the Iterated Dilated Convolutional Neural Network (ID-CNN), which is particularly effective for sequential data, such as text [22]. ID-CNN builds on the concept of dilated convolutions, where convolutional layers utilize dilated filters to capture long-range dependencies between elements in the sequence while maintaining computational efficiency. This method is especially advantageous in text processing as it enables the network to gather contextual information over a wider range without significantly increasing the model's complexity. ID-CNN uses dilated filters, where the filter skips certain positions in the input, allowing it to cover a broader range of tokens without increasing the filter size.

The combination of DeBERTa embeddings and ID-CNN leverages DeBERTa's contextual understanding with the long-range dependency-capturing capabilities of ID-CNN, resulting in more effective feature extraction. The feature map $R$ derived from the DeBERTa embeddings can be expressed as:

Here + is the concatenation operator and $rn$ is the DeBERTa embedding for the $n−th$ token.In the next step, the ID-CNN processes the DeBERTa embeddings by applying dilated convolutions. A filter $w∈R1*H$ is applied over a window of $l$ embeddings, with a dilation rate $d$⁠, to handle both long and local range dependencies. This dilated convolution operation results in the creation of new features from the DeBERTa embeddings. For a window of embeddings $r(i:i+l−1)$ the output feature $ci$ computed as:

Here w is the convolutional filter. $r(i:i+l−1)$ represents the DeBERTa embeddings for the tokens within the window. $b$ is the bias and $f$ is the nonlinear activation function.

To reduce the dimensionality of the feature map and capture the most salient features, max-over-time pooling is applied. This pooling process extracts the highest value for each feature dimension, generating a fixed-length output that encapsulates the most relevant information from the input. The pooled representation focuses on key features detected by the ID-CNN. Max-over-time pooling is particularly useful for tasks like text classification and sentiment analysis, where it helps distill the most critical features from the input data.

The output features generated by the ID-CNN from the DeBERTa embeddings represent a highly refined and informative set of representations that encapsulate both local and global dependencies in the input sequence. By leveraging the contextual richness of DeBERTa embeddings and the long-range capturing abilities of dilated convolutions, the ID-CNN extracts the most relevant features from the input text. After applying max-over-time pooling, the model produces a fixed-length feature vector that highlights the most salient patterns and relationships across the entire sequence. The combination of DeBERTa and ID-CNN ensures that the output features not only capture meaningful syntactic and semantic information from the text but also retain a strong representation of both local word interactions and broader contextual insights, ultimately enhancing the model's performance across a variety of natural language processing tasks.

In this section the BiLSTM-CRF has been presented which is the core of this research. This architecture combines the strengths of Bidirectional LSTMs for capturing sequential information and Conditional Random Fields for sentiment classification.

Once the features have been extracted by the ID-CNN, the output is passed to the BiLSTM layer for further processing. The BiLSTM layer is particularly adept at modeling the sequential nature of text, as it processes the input in both forward and backward directions. Given the sequence of features X $={x1+x2+x3+…+rT}$ where $T$ represents the sequence length, the forward and backward passes of the BiLSTM generate hidden states for each time step $t$⁠. The forward hidden state $htfw=LSTMfw(x1,ht−1fw)$ captures the past context, while the backward hidden state $htbw=LSTMbw(x1,ht+1bw)$ captures the future context. These hidden states are concatenated to form the final hidden state for each word $ht=[htfw;htbw]$⁠, allowing the model to incorporate context from both directions.

After processing the entire sequence, a pooling operation is typically applied to aggregate the hidden states, producing a fixed-length feature vector. This representation is denoted as $HSen$ and act as the input to the following classification layer. The final sentiment classification is performed by passing this pooled representation through a fully connected layer, where the classification scores $S$ are computed as:

Here, $W$ is the weight matrix, and $b$ is the bias term. These scores are then converted into probabilities for each sentiment class using the SoftMax function:

The predicted sentiment is obtained by selecting the class with the maximum probability. During training, the model minimizes the cross-entropy loss to optimize its predictions. In addition to this basic architecture, attention mechanisms can be introduced to enhance the model's performance. Attention allows the BiLSTM to focus on the most relevant parts of the feature sequence, which is particularly useful when certain words or phrases carry more weight in determining sentiment. The attention mechanism computes a set of attention scores $αi$​for each word in the sequence using a scoring function:

These attention scores indicate the importance of each hidden state in the final sentiment prediction. The context vector $c$⁠, representing the most important information from the input sequence, is computed as:

The context vector, which encapsulates the most relevant information from the input sequence, is concatenated with the hidden states and passed through a fully connected layer to predict the sentiment polarity of the text. By combining both the sequential context and the most important features highlighted by the attention mechanism, the model can effectively classify the sentiment of the input. The inclusion of attention enhances the interpretability of the model by indicating which words or phrases in the text had the greatest influence on the polarity classification, allowing the model to focus on the key elements that contribute to the positive, negative, or neutral sentiment. This not only improves the accuracy of sentiment polarity classification but also provides valuable insights into the underlying reasons for the model's predictions.

This research used K-fold cross-validation in experiments. For this specific experiment, k was set to 5 for the evaluation. K-Fold cross-validation is an essential method in sentiment analysis, as it addresses the limitations of a single train-test split. Dividing the data into k subsets, it allows for multiple rounds of training and testing, resulting in a more comprehensive assessment of the model's performance. This approach aids in assessing the model's generalization ability, preventing overfitting, and producing dependable and robust sentiment analysis outcomes.

The experimental results of the proposed approach are presented in Table 2. The results reveal that the model attained impressive performance metrics, with 93.02% accuracy, 91.48% recall, and 92.06% precision on the Hu and Liu dataset (DS-I). Similarly, on the SemEval dataset (DS-II), the proposed model attained an accuracy of 90.21%, a recall of 89.19% and a precision of 89.64%, while on the DS-III, it achieved an accuracy of 88.76%, a recall of 87.02%, and a precision of 87.98%. The last dataset, DS-VI, achieved an accuracy of 92.86%, a recall of 91.72%, and a precision of 92.08%.

The performance of the model on datasets DS-I, DS-II, DS-III, and DS-IV is depicted through confusion matrices in Figure 2. These matrices present the counts of correctly and incorrectly classified instances, offering an in-depth analysis of the model's ability to differentiate between various classes within each dataset.

The reliability of the proposed model was assessed through a comparative study using both deep learning models (BiLSTM, BiGRU, CNN) and traditional machine learning classifiers (SVM, Lexicons, Rules Mining). Deep learning models were chosen for their strength in capturing context, while traditional models offer simplicity and effectiveness on smaller datasets. This comparison highlights the proposed model's superiority and offers insights into best practices for ABSA. BERT + ARM [4], POS-Lexicon [5], BERT-BASE [23], BERT-ADA [24], CNN-BiLSTM [25], SVM [26], ABSA-PER [27], TF-IDF-RF [28].

Figure 3a illustrates a comparison of the accuracy achieved by the proposed method against current state-of-the-art techniques, while Figure 3b shows a comparison of their respective F1 scores.

The comparative analysis highlights the model's superior capability in capturing nuanced sentiment patterns, though it may increase computational complexity. These findings emphasize the importance of combining state-of-the-art neural architectures with innovative training techniques to achieve high-performance sentiment classification in diverse scenarios. While the proposed model demands higher computational resources due to the complexity of transformer-based DeBERTa and BiLSTM components, it significantly outperforms simpler models in terms of accuracy and feature extraction capabilities. This trade-off between computational cost and performance suggests that our model is well-suited for applications where accuracy is prioritized and sufficient computational resources are available. Paired t-tests were applied on the accuracy and F1 scores of the proposed model against all baseline methods across the four datasets. The results indicated statistically significant improvements (p < 0.05) in every case, and 95% confidence intervals further confirmed the robustness of the proposed approach. Notably, the lower bounds of the confidence intervals for the proposed model consistently exceeded the upper bounds of several baselines, demonstrating that the performance gains are both consistent and meaningful.

An ablation study was conducted to systematically assess the contribution of individual components within the proposed model. Variants were created by altering or removing specific modules such as IDCNN, DeBERTa, the attention mechanism, and the CRF layer, while keeping the remaining architecture unchanged. This comparison provides insights into how each component influences aspect-level feature extraction and sentiment classification. The complete pipelines of all variants and their averaged performance across the four datasets are reported in Table 3. The ablation results clearly show that the proposed model outperforms all reduced variants, achieving the highest accuracy, recall, and precision across datasets. Each component—DeBERTa, IDCNN, attention mechanism, and CRF—contributes to performance, but their integration delivers the most robust aspect-level sentiment classification.

Online product reviews have a significant influence on consumer purchasing decisions and the sales of products or services. These reviews not only aid in improving the quality of products and services but also provide valuable insights to new users looking for recommendations. To enhance the effectiveness of such reviews, this study proposed a DeBERTa-IDCNN-BiLSTM-CRF for conducting ABSA on product reviews. The proposed model is evaluated on four benchmark datasets, and the results show that the proposed model outperforms baseline baselines in multidomain benchmark datasets. The model can be integrated into e-commerce platforms to summarize feedback and enhance recommendations, and into customer service systems for real-time monitoring of complaints and improved responsiveness. Its architecture supports scalability and real-time processing, making it practical for large-scale deployment. Future work may extend this study by exploring multilingual reviews and additional contextual factors, such as accommodation, spoken language, branding, and facilities, to capture broader variations in customer sentiment.