Spam detection using sentence-BERT with attention-enhanced GRU and SVM Open Access

Emails have become a fundamental medium for communication, yet spam remains a widespread problem that affects both email and text messaging. Fraudulent actors often use deceptive tactics, such as familiar area codes or impersonation, to extract personal or financial information, a practice now termed “smishing” [1]. Recent data reveal a dramatic increase in spam activity, with reported messages increasing from 1.27 million in September 2021 to 10.89 billion in August 2022, and financial losses reaching USD 10 billion in 2021 [1]. These trends highlight the urgent need for advanced detection strategies to protect digital communications.

Various methods have been explored to combat SMS spam. Traditional approaches, such as blacklisting and heuristic filters, offer basic protection but are often inadequate against sophisticated attacks. Recent efforts emphasize artificial intelligence, using machine learning to analyze and classify messages more effectively. Success in AI-based detection is highly dependent on text representation, feature extraction, and classification algorithms. Previous studies have also demonstrated the benefits of hybrid feature selection [2] and content-based filtering [3] in enhancing spam detection performance.

This article presents a significant advancement in spam detection, utilizing advanced text analysis methods. Our approach leverages Sentence-BERT embeddings, an attention-enhanced Gated Recurrent Unit (GRU) network, and a Support Vector Machine (SVM) for classification. By capturing both contextual and sequential dependencies in the text, our method processes these features through the GRU for enhanced performance. Furthermore, we use the T5 model for data augmentation to improve the balance and robustness of the dataset. Our key contributions include.

1. Sentence-BERT Embeddings: We generate rich contextual embeddings using Sentence-BERT, capturing nuanced semantic relationships within the text.

2. Attention-Enhanced GRU: By integrating an attention mechanism into a GRU network, our model better identifies important sequential patterns that distinguish spam from ham.

3. SVM Classification: A Support Vector Machine is used for the final classification, capitalizing on the expressive features extracted by the GRU–attention architecture.

4. Data Augmentation with T5: We apply the T5 model to augment the dataset with paraphrased spam messages, enhancing balance and robustness in representing diverse spam expressions.

Numerous studies have addressed phishing and smishing detection using various machine learning and deep learning methods. In Ref. [4], the “SmiDCA” model extracts 39 features from smishing messages, achieving accuracies of 96.40% for English and 90.33% for non-English messages, and maintaining an accuracy of 96%, 16% after dimensionality reduction. Likewise [5], presents a comprehensive model targeting smartphone vulnerabilities, combining an SMS content analyzer (Naive Bayes), a URL filter, a source code analyzer, and an APK download detector, achieving 96.29% accuracy.

Joo et al. [6] integrate message monitor, content analyzer, classifier, and knowledge base for spam detection. Deep learning approaches in Refs. [7, 8] use CNN and LSTM models, achieving accuracy above 98%. A Hidden Markov Model with word weighting, as proposed in Ref. [9], captures distributional patterns for effective spam filtering. More recent architectures, including Transformers [10] and Bi-GRU-CNN hybrids [11], leverage contextual and sequential modeling to achieve 99% accuracy.

Traditional models still show strong performance. For example, combining Multinomial Naive Bayes with Linear SVC [12] achieves 98.38% accuracy after preprocessing, while a GRU-based classifier [13] reaches 99%. These results highlight the continued effectiveness of both conventional and neural methods.

In parallel, robust feature selection and content-based filtering have gained prominence. Tarek and Abd-El-Hafeez [2] introduces a hybrid feature selection method with automatic threshold detection, while [14] leverages frequent and associated itemsets for improved classification. Mahmoud and Abd-El-Hafeez [15] extends this approach by optimizing the selection of correlated frequent items.

Content-based filtering has also been applied in security contexts. Studies [3, 16] show its effectiveness in detecting inappropriate multimedia content, with a focus on precision, scalability, and real-time processing; principles that can also benefit spam detection. Other work includes extracting images and URLs for web sanitization [17] and using interactive knowledge graphs for topic extraction [18], improving both semantic transparency and classifier interpretability.

Recent studies reflect a growing emphasis on hybrid spam detection systems combining deep semantic embeddings and classical classifiers. Sharma et al. [19] reviewed deep learning applications in sentiment and spam classification, highlighting the effectiveness of architectures such as GRU-SVM and BERT-SVM. In a related direction, Lopez-Joya et al. [20] analyzed the internal structure of social bots and proposed a Bi-GRU model enhanced with self-attention and interpretability layers. Batiuk and Dosyn [21] combined Sentence-BERT embeddings with XGBoost for textual classification in social networks, reinforcing the robustness of semantic feature extraction. Additionally, Sayeed and Dutta [22] applied BERT and GRU architectures for detecting malicious URLs in brushing scam campaigns, showing the adaptability of these models to cybersecurity threats.

Building on these prior advances, our work proposes a unified spam detection framework that combines recent innovations in contextual embeddings, sequential modeling, and classification strategy. Our proposed framework brings together three complementary components: (1) Sentence-BERT embeddings to capture rich and contextual semantic information, (2) a GRU enhanced with self-attention to better model sequential patterns and highlight important features, and (3) an SVM classifier that replaces softmax for more refined decision boundaries. This combination, further supported by T5-based data augmentation, forms a unified architecture that offers strong performance and improved generalization. To the best of our knowledge, this specific integration has not yet been explored in the context of spam detection.

Sentence-BERT (SBERT) [23] extends BERT [24] by using siamese and triplet architectures to produce fixed-size, semantically meaningful sentence embeddings; typically of 384 dimensions. Unlike BERT, which outputs token-level embeddings requiring aggregation, SBERT is fine-tuned to generate sentence vectors directly, optimized for tasks like semantic textual similarity (STS), clustering, and retrieval.

With around 110 million parameters in its base version, SBERT maps similar sentences close together in the embedding space, enabling efficient comparison. It combines semantic precision with computational efficiency and has shown strong performance across various NLP tasks [25, 26].

XLNet [27] integrates autoregressive and autoencoding objectives via Permutation Language Modeling (PLM), which maximizes the expected log-likelihood over all word orderings. This removes the need for [MASK] tokens and allows bidirectional context without data corruption.

Based on a transformer with 340 million parameters and 768-dimensional embeddings, XLNet effectively captures long-range dependencies. It excels in tasks like text classification, translation, and sentiment analysis [28–30], and its permutation-based training enhances contextual understanding.

The Gated Recurrent Unit (GRU) [31] is a recurrent neural network (RNN) variant designed to capture long-range dependencies while mitigating the vanishing gradient problem. Unlike traditional RNNs, GRUs employ gating mechanisms; specifically, an update gate and a reset gate; that regulate the flow of information, as illustrated in Figure 1.

At each time step t, the reset gate r_t determines how much of the previous activation h_t−1 to consider:

Using r_t, the candidate activation $h̃t$ is computed as:

The update gate z_t then controls how much of $h̃t$ is used versus the previous activation:

Finally, the new activation h_t is obtained by blending h_t−1 and $h̃t$⁠:

GRUs offer efficient training due to their simplified structure, achieving performance comparable to LSTMs without separate memory cells. They are widely adopted in applications like natural language processing [33], speech recognition [34], and time series analysis [35].

The attention mechanism [36] plays a crucial role in deep learning models, particularly in NLP and machine translation, by enabling selective focus on different segments of an input sequence.

It operates by computing attention scores based on the similarity between Query and Key vectors, which are normalized through a softmax function and used to weight the Value vectors. The weighted sum forms the output, emphasizing the most relevant parts of the input [36].

This mechanism enhances the model's ability to manage long-range dependencies and intricate relationships, significantly improving tasks like sequence-to-sequence translation [37].

The spam detection approach outlined in this study comprises a systematic four-step process, as illustrated in Figure 2.

1. Beginning with an initial preprocessing and data augmentation step (Step 1 in Figure 2), text messages are augmented using the T5 model and undergo thorough cleaning to enhance dataset balance and integrity.

2. In the subsequent stage (Step 2 in Figure 2), contextualized representations or embeddings are generated for the augmented data using Sentence-BERT. This crucial task captures deep semantic nuances and contextual dependencies effectively.

3. Advancing to Step 3 in Figure 2, the obtained enriched embeddings serve as input for a Gated Recurrent Unit (GRU) network with an attention mechanism. The GRU network processes these embeddings to capture sequential relationships within the text data, further enhancing the model's understanding of dependencies between words.

4. The final stage (Step 4 in Figure 2) involves the classification of the processed embeddings using a Support Vector Machine (SVM). This step determines whether a message is spam or ham based on the features extracted by the preceding stages.

For a comprehensive understanding of each stage in the proposed approach, detailed descriptions are provided in the subsequent subsections.

To mitigate class imbalance, we applied the T5 model [38] to generate paraphrased versions of spam messages. Known for text-to-text transformations, T5 rephrased original texts into semantically equivalent variants. This enriched the spam class, improved balance, and increased the model's robustness by exposing it to diverse spam patterns.

To encode messages, we used Sentence-BERT (SBERT) [23], which generates contextualized embeddings that capture semantic relationships. Specifically, we employed the sentence-transformers/bert-base-nli-mean-tokens model, fine-tuned for semantic similarity and well-suited for spam detection.

Let T be a text message and GenerateSentenceBertEmbeddings the embedding function. The resulting vector is:

This yields a 384-dimensional embedding that encodes semantic nuances of T, facilitating accurate classification.

To prepare embeddings for downstream processing, we applied z-score normalization:

where μ and σ are the mean and standard deviation of the embedding vector. This standardization ensures zero mean and unit variance, improving model performance.

The embeddings generated by Sentence-BERT and normalized serve as inputs to a Gated Recurrent Unit (GRU) network, which is enhanced with an attention mechanism. The self-attention mechanism is integrated into the GRU to improve the model's focus on relevant parts of the embedding sequence.

Below is a simplified schematic of the self-attention mechanism integrated within the GRU, using the embeddings generated by Sentence-BERT as input sequences:

The final classification is handled by a Support Vector Machine (SVM) rather than a softmax or neural classifier. This choice leverages the SVM's strength in binary classification with high-dimensional embeddings like those from Sentence-BERT. As a margin-based method, SVMs offer strong generalization with minimal tuning and are less prone to overfitting. Decoupling feature extraction (via attention-enhanced GRU) from classification improves modularity and interpretability. SVMs also provide theoretical robustness and resilience to noise; key advantages in spam detection, where misclassification can be costly.

We utilized ResearchGridSV for hyperparameter tuning to determine the optimal configuration for our models. Table 1 summarizes the selected hyperparameters and their respective values for each component of the proposed model.

To determine these values, we explored a range of configurations using grid search on the validation splits. For example, we tested GRU units in {64, 128, 256}, dropout rates in {0.2, 0.5}, and SVM C values in {0.01, 0.1, 1.0}. Across these settings, the model's performance remained consistently high, with only minor variations in F1-score and accuracy. This stability suggests that the proposed architecture is robust to reasonable changes in hyperparameter values.

We used the publicly available SMS Spam Collection Dataset [39], which contains 5,572 English SMS messages (747 spam, 4,825 ham). This benchmark, compiled by Almeida et al. [39], is widely used for spam detection.

An initial cleaning step removed 403 exact duplicates, resulting in 5,169 messages (747 spam, 4,422 ham). Due to class imbalance (13.4% spam), we augmented the spam class using the T5 transformer model for paraphrasing, increasing the dataset to 8,560 messages (3,735 spam, 4,825 ham). A second cleaning removed 347 duplicate or near-duplicate entries introduced during augmentation, yielding a final set of 8,213 unique messages (3,697 spam, 4,516 ham). This balanced distribution (approximately 45% spam) helps reduce bias during training.

All preprocessing was completed before data splitting. We used stratified 5-fold cross-validation to preserve class ratios in each fold. A summary of the dataset statistics before and after augmentation is provided in Table 2.

We evaluated model performance using five standard classification metrics: accuracy, precision, recall, F1-score, and the ROC curve with AUC, alongside the confusion matrix for detailed class-level insights.

1. Accuracy: Proportion of correct predictions.

1. Precision: Proportion of correctly predicted positives.

1. Recall (Sensitivity): Proportion of actual positives correctly identified.

1. F1-Score: Harmonic mean of precision and recall.

1. Confusion Matrix: Cross-tabulates predicted vs. actual labels, highlighting classification errors.

2. ROC Curve & AUC: The ROC curve plots recall vs. false positive rate. The AUC quantifies class separability; higher values indicate better discrimination.

We evaluated the generalization ability of our spam detection model using 5-fold stratified cross-validation, which splits the dataset into five equal parts while preserving class distribution. In each fold, four parts are used for training and one for testing, ensuring every sample is used for both purposes. This strategy maximizes data utilization, exposes the model to varied data distributions, reduces overfitting, and yields stable performance metrics by averaging results across folds.

Beyond the core evaluation, we conducted targeted experiments:

1. Individual Configuration Evaluation: We tested GRU alone, GRU with attention, and GRU with attention combined with SVM. LightGBM and XGBoost were also assessed as alternative classifiers with GRU + attention. Hyperparameters for each classifier were tuned via grid search.

2. Embedding Comparison: We compared XLNet and SBERT embeddings across configurations, highlighting SBERT's contribution to performance gains.

3. Holistic Model Evaluation: All configurations and embeddings were assessed using accuracy, precision, recall, and F1-score, providing a comprehensive performance overview.

This evaluation framework ensures consistent assessment across all configurations and embedding strategies.

The results from our experiments, summarized in Table 3, provide detailed information on accuracy, precision, recall, and F1-score for each configuration and embedding.

Based on the results presented in Table 3, several insights can be drawn regarding the performance of classifiers with SBERT and XLNet embeddings:

1. The GRU model, coupled with XLNet embeddings, achieves an impressive accuracy of 99.09%. Despite this high accuracy, there is a slight trade-off between precision (99.73%) and recall (98.24%). This indicates that while the model is highly accurate in identifying spam, it may miss a few instances. On the other hand, when paired with SBERT embeddings, the GRU model demonstrates a higher accuracy of 99.57%. The precision (99.86%) and recall (99.19%) are well-balanced, suggesting a more effective spam detection model with SBERT.

2. The GRU with attention model, when using XLNet embeddings, shows excellent performance with an accuracy of 99.57%. The precision is perfect at 100%, and the recall is also high at 99.05%, making it a robust choice for spam detection. Similarly, with SBERT embeddings, the GRU with attention model maintains high performance with an accuracy of 99.63%, a perfect precision of 100%, and a recall of 99.19%.

3. The combination of GRU with attention and LightGBM, in combination with XLNet embeddings, exhibits a high precision of 99.86% but at the cost of a lower recall of 97.97%. This implies that while it effectively identifies spam, it may produce some false negatives. When paired with SBERT embeddings, GRU with attention and LightGBM performs better with an accuracy of 99.39%, a balanced precision of 99.86%, and an improved recall of 98.78%.

4. Similarly to the combination of GRU with attention and LightGBM, the combination of GRU with attention and XGBoost with XLNet embeddings maintains a strong precision (100%) but shows a lower recall (97.97%). This balanced performance may still miss some spam messages. With SBERT embeddings, GRU with attention and XGBoost shows strong performance with an accuracy of 99.63%, and balanced precision (100%) and recall (99.19%).

5. The combination of GRU with attention and SVM with XLNet embeddings shows exceptional performance, achieving the highest accuracy of 99.76% and a balanced precision-recall trade-off (99.86% and 99.59%, respectively).

6. The proposed method, leveraging SBERT embeddings, outperforms all other experiments, achieving an outstanding accuracy of 99.88% and a harmonious precision-recall balance (100% and 99.73%, respectively). This indicates the superior capability of the proposed method in accurately detecting spam with minimal false negatives.

Also, the Figures 3–5 provide a clear overview of the achieved results.

From Figures 3–5, several key observations can be made about the performance of different configurations using SBERT and XLNet embeddings:

1. SBERT embeddings consistently outperform XLNet in precision, recall, and accuracy, confirming their effectiveness for spam detection.

2. GRU with attention delivers strong, reliable results with both embedding types, demonstrating its versatility in sequential modeling.

3. Replacing the softmax layer in GRU with attention by SVM, LightGBM, or XGBoost further improves performance, offering a more effective approach to spam classification.

These findings underscore the efficacy of the proposed method, particularly when leveraging SBERT embeddings, showcasing its potential for advanced spam detection applications.

Further, we plotted the Receiver Operating Characteristic (ROC) curve and calculated the Area Under the Curve (AUC), as shown in Figure 6. The AUC value for our model is 0.99.

The near-perfect AUC score of 0.99 highlights the robustness and reliability of our model in accurately identifying spam messages. This exceptional AUC value demonstrates the model's strong discriminative ability, confirming its effectiveness in distinguishing between spam and non-spam messages.

Furthermore, to gain deeper insight into the classification results, confusion matrices for the GRU with attention model using different classifiers (SVM, LightGBM, XGBoost) are presented in Figure 7. This figure compares the results for SBERT and XLNet embeddings.

The analysis of these matrices reveals noteworthy patterns. Across all methods, there was a high success rate in correctly identifying the 902 non-spam messages. However, a significant challenge was encountered in the classification of spam messages, with all configurations exhibiting a considerable number of false positives. For instance, the GRU with attention and LightGBM configuration misclassified 15 spam messages as non-spam. Notably, our proposed approach with Sentence-BERT embeddings achieved perfect identification of all true positives but encountered difficulty in correctly classifying two spam messages. This highlights the robustness of our method.

Beyond classification performance, inference speed is crucial for real-time spam detection. While training demands significant computational resources, deployment primarily depends on how quickly the system processes new messages.

On a machine with an Intel i7 processor, 32 GB RAM, and an NVIDIA GeForce GTX 950M GPU, the proposed method using SBERT embeddings processed each message in about 31 ms, compared to 45 ms for XLNet embeddings. These values fall within the typical range reported for transformer-based models (10–50 ms, depending on hardware and configuration) [40, 41].

Overall, the SBERT variant offers the best trade-off between accuracy and speed, making it well-suited for high-throughput, real-time spam detection.

To better position the effectiveness of our method within the context of existing research, we conducted a comparative evaluation with several recent state-of-the-art approaches tested on similar spam detection datasets, documented in the related works section and described in Refs. [4, 7–9, 11–13]. Detailed descriptions of these methods are provided in the Related Works section. This comparison serves to highlight the performance improvements achieved by our proposed framework in terms of classification accuracy.

Table 4 presents a summary of recent methods selected from the literature, including deep learning and transformer-based models. The comparative results clearly demonstrate that our model outperforms these approaches, achieving the highest reported accuracy of 99.88%. This performance gain can be attributed to the synergy between contextualized embeddings (SBERT), attention-enhanced sequential modeling (GRU with attention) and a robust classifier (SVM), all reinforced through data augmentation.

To rigorously assess the performance and reliability of our spam detection configurations, we employed both the bootstrap method and the Friedman test, offering complementary insights into model stability and the significance of observed differences.

We used a non-parametric bootstrap approach to estimate 95% confidence intervals (CIs) for each configuration's accuracy. This involved generating 500 bootstrap samples by resampling the test data with replacement and computing accuracy for each. The resulting distribution was used to derive empirical CIs.

Table 5 presents the CIs for both embedding strategies:

From a statistical perspective, the overlap between the SBERT and XLNet intervals is minimal for the top-performing configurations, indicating a consistent advantage for SBERT embeddings. The narrow CI for C5 (GRU + Attention + SVM) reflects low variance across resamples, while the broader range for C3 (GRU + Attention + LightGBM) suggests higher sensitivity to data variation.

We applied the Friedman test to assess whether performance differences among configurations (C1–C5) are statistically significant. The test yielded χ² = 19.36 with a p-value of 0.00067 for XLNet, and χ² = 23.80 with a p-value of 0.0069 for SBERT.

In both cases, results are statistically significant (p < 0.05), confirming that model configuration meaningfully affects spam detection performance.

The bootstrap analysis indicates that C5 is the most stable configuration, and the Friedman test confirms that performance differences among models are statistically significant. These findings highlight the strong performance of the GRU + Attention + SVM architecture, particularly with SBERT embeddings.

Our approach relies on pretrained models such as SBERT and XLNet, which may be demanding for devices with limited resources. Future work will explore lighter alternatives like DistilBERT or MiniLM, and optimization methods such as quantization and pruning to speed up inference.

The current evaluation is limited to English SMS data. Testing in other languages and domains (e.g., email, chat, social media) will help assess how well the method adapts to different contexts.

We also observed a few missed spam cases and have not yet studied resistance to adversarial or evolving spam patterns. Expanding the evaluation and exploring online or continual learning could further improve adaptability.

This study presented a spam detection framework using Sentence-BERT embeddings. Combined with an attention-enhanced GRU for sequential modeling and a SVM classifier in place of softmax. The approach reached 99.88% accuracy with a 95% CI of [99.70%, 100.00%], demonstrating the effectiveness of combining semantic encoding, temporal modeling, and optimized classification. Inference speed was also competitive, supporting deployment in real-time filtering systems.

Beyond technical performance, content-based spam filtering also raises important ethical and societal questions. These systems can greatly reduce exposure to harmful content such as scams and phishing, but they must be designed to protect legitimate communication, respect user privacy, and be transparent in how they work. This calls for models that are easy to understand and accountable, helping to build trust and ensure fair decisions.

Future work will explore:

1. Advanced Embeddings: Testing lighter or more adaptable embeddings to improve efficiency and generalization.

2. Dynamic Ensembles: Developing adaptive ensemble methods to handle evolving spam patterns.

3. Imbalance Mitigation: Applying sampling or loss-based strategies to address class imbalance in practical deployments.