Knowledge-guided cross-modality attention for smartphone-based stress recognition in applied health informatics

Stress is a major contributor to mental and physical disorders, including anxiety, depression, cardiovascular disease and sleep problems, and its growing prevalence calls for scalable monitoring beyond clinical settings [1–3]. From an applied informatics perspective, continuous, unobtrusive stress detection enables early intervention and improved quality of life but requires models that are interpretable, reliable and resilient in everyday conditions.

Psychological stress theory provides a strong foundation for designing such systems. According to the Lazarus and Folkman stress appraisal framework [4], stress arises from an individual’s evaluation of external demands relative to available coping resources. This appraisal is reflected in physiological, behavioral and linguistic markers – such as prosodic changes, lexical sentiment, mobility patterns and activity entropy – that can be passively sensed through digital devices. Embedding such psychologically grounded priors in model design can improve both interpretability and generalization, especially under noisy or incomplete conditions.

Smartphones offer an ideal platform: they are ubiquitous, part of daily routines and provide diverse sensing channels for passive or prompted data collection without requiring extra hardware [5, 6]. Speech supplies rich acoustic and linguistic cues such as prosodic variation, sentiment and lexical markers, while behavioral signals (e.g. app usage, mobility, interaction entropy) provide complementary evidence [7–14]. These multimodal signals can support real-world stress recognition if the system can handle noise and missing data effectively.

Existing approaches still face three key limitations:

1. Vulnerability of speech-only systems to noise and speaker variability [15–17].

2. Reliance on static multimodal fusion that cannot adapt modality importance dynamically [18–22].

3. A lack of explicit domain knowledge in model design, which limits interpretability and generalization when inputs are incomplete or unreliable [14, 23].

Moreover, explicit reliability signals and probabilistic calibration are rarely reported, and external validation on independent datasets remains uncommon, hindering trustworthy real-world deployment.

To address these gaps, we propose CMAF-Net (Cross-Modality Attention Fusion Network), a deep learning architecture that fuses acoustic, linguistic and behavioral features for smartphone-based stress recognition. CMAF-Net embeds psychological priors directly into its cross-modality attention mechanism, guided by empirically supported stress cues and applies reliability-aware gating that weights modalities using measurable input quality indicators (audio signal-to-noise ratio, automatic speech recognition word error rate and behavioral sparsity). The design is explicitly deployment-oriented, enabling real-time, privacy-preserving inference on commodity smartphones.

This study makes five main contributions:

1. Prior-informed fusion: A cross-modality attention mechanism that encodes psychological priors through a learnable prior vector and cross-modal bias to improve interpretability.

2. Reliability-aware weighting: A gating module that adapts modality weights based on input quality (audio SNR, ASR WER and behavioral sparsity).

3. Multimodal dataset and protocol: real-world data from 87 participants with validated stress labels [24], evaluated using subject-independent folds and ablations isolating the effects of priors, cross-modal bias, and reliability gating.

4. Comprehensive evaluation for generalization: External validation on an independent mobile dataset, significance testing across folds and probabilistic calibration analysis (expected calibration error and Brier score).

5. On-device deployment evidence: Post-training quantization enabling efficient, low-latency inference on smartphones [25] with maintained accuracy and calibration.

The remainder of this paper is organized as follows: Section 2 reviews related work; Section 3 describes the dataset and system design; Section 4 details the CMAF-Net architecture; Section 5 presents experiments and results; Section 6 discusses findings and limitations and Section 7 concludes the study.

Multimodal stress detection increasingly uses smartphone and wearable signals to combine physiological, behavioral and affective cues in real-world settings. Mobile sensing studies demonstrate feasibility, for example, a smartphone-based framework reporting about 80% accuracy and app-usage markers as digital indicators of momentary stress that highlight the value of passive behavioral features [26, 27].

Public datasets have pushed the field toward ecologically valid evaluation. The K-EmoPhone dataset collected multimodal physiological and behavioral data with frequent in-situ stress labels from students over a week, providing a benchmark for mobile stress modeling [28]. More recent resources, such as StressID, broaden coverage of multimodal stress signals and support cross-study comparisons [19].

Beyond smartphone-only pipelines, lab-controlled multimodal systems integrate signals like ECG, voice and facial expressions with modern deep architectures. One example combines ResNet50 and I3D with attention, achieving strong accuracy during the Montreal Imaging Stress Task, while other work explores interpretable models using unobtrusive multimodal inputs for clinical contexts [29, 30]. Concurrently, multimodal fusion techniques have evolved from early and late fusion to attention-based and transformer-style fusion, including approaches that inject domain knowledge into attention scoring for sentiment or emotion tasks [14, 23] as well as CB-Transformer variants that learn modality-fused representations for unaligned sequences [12]. Reviews emphasize both the promise of these mechanisms and persistent challenges around generalization and transparency in the wild.

Despite this progress, three gaps remain common in mobile stress research: limited attention to input reliability signals that reflect real data quality (for example, audio noise, ASR transcript errors or behavioral sparsity), rare reporting of probabilistic calibration suitable for downstream decision thresholds and infrequent external validation on independent mobile datasets, all of which constrain real-world deployment.

CMAF-Net is designed to address these gaps within a smartphone-only architecture by (1) formalizing prior-informed cross-modality attention through a learnable prior vector and a cross-modal bias term that encode psychological priors, improving interpretability over purely data-driven attention; (2) adding reliability-aware gating that conditions modality weights on measurable input-quality indicators and (3) coupling these with deployment-oriented evaluation that includes external validation, robustness and calibration analyses and on-device quantized inference suitable for privacy-preserving mobile use.

The proposed system is a real-time, privacy-preserving stress recognition framework that uses multimodal data collected both passively and actively on standard smartphones. Guided by applied informatics principles of scalability, minimal user burden and efficient resource use, the design integrates acoustic, linguistic and behavioral modalities to capture diverse stress-related indicators under naturalistic conditions while remaining robust on consumer devices. The overall architecture is shown in Figure 1 in Section 4. It consists of modality-specific encoders, knowledge-guided cross-modality attention, reliability-aware gating that reacts to input quality and a lightweight classifier optimized for on-device inference.

We target two primary goals:

1. Reliable stress detection in noisy, real-world conditions and

2. Deployability on commodity smartphones without specialized hardware.

Prior studies support speech and semantic cues as stress markers and multimodal features as complementary sources of information [9, 10]. Recent advances in cross-modality attention and knowledge-guided fusion inform our design choices [14, 23]. The operational objectives:

1. Robustness to real-world artifacts: Background noise, ASR errors, sparse behavioral logs and missing modalities.

2. Privacy by design through on-device inference and storage of derived features only.

3. Real-time responsiveness with per-sample inference time at or below 200 ms on mid-range smartphones (for example, Snapdragon 720G class devices).

4. Stable probabilistic outputs that are suitable for thresholding in downstream applications, which we verify using calibration metrics in Section 5.

While the design constraints:

1. Reliance on built-in phone sensors only (microphone and operating system interaction or mobility logs).

2. Limited compute and memory addressed through post-training quantization and streamlined encoders [25].

3. Energy efficiency through quantized operators and minimal data movement.

4. Tolerance for environmental variability and irregular sampling.

These constraints shape the encoders (Section 4.2), the integration of domain knowledge (Section 4.3) and the cross-modality fusion with reliability-aware gating (Section 4.4). Compliance with latency, size and accuracy targets is validated in Section 5.4.

To remain dependable in everyday use, the system estimates input quality signals and uses them to modulate fusion: audio signal-to-noise ratio for acoustic streams, word error rate for ASR transcripts and sparsity or entropy measures for behavioral logs. When a modality is unreliable or unavailable, the gating mechanism down-weights or masks it so that the model degrades gracefully rather than failing unpredictably. Stress tests with noisy audio, corrupted transcripts and sparse behavioral windows are included in Section 5.

Operational objectives are tied to measurable endpoints. We report subject-independent performance, ablations that isolate the effect of prior vectors and cross-modal biasing, missing-modality stress tests and calibration analysis. External validation on an independent mobile dataset is used to assess generalization. On-device latency, model size after quantization and robustness to quantization are reported to substantiate deployability.

Data collection followed strict ethical protocols to safeguard participant privacy and comply with regulatory standards. About 87 adults (ages 19–44, balanced by gender) were recruited from universities and workplaces. All participants provided informed consent and were informed of their right to withdraw at any time without penalty. The study protocol was reviewed and approved by an institutional ethics board.

To minimize risk, only derived features were stored, including acoustic descriptors, text embeddings, behavioral statistics and stress labels. Raw audio and GPS data never left the device. All transmissions of derived features were encrypted in transit, participant identifiers were anonymized or pseudonymized and server access was restricted to authorized research personnel following the principle of least privilege. Stress levels were labeled using Ecological Momentary Assessment (EMA) with the validated PSS-4 instrument [24]. Multimodal samples were aligned within a 15-min window to maintain temporal coherence across modalities.

This data governance framework aligns with the General Data Protection Regulation (GDPR) and US Health Insurance Portability and Accountability Act (HIPAA) principles, emphasizing data minimization, purpose limitation and participant rights. By performing all signal processing and feature extraction on-device, the pipeline reduces the risk of reidentification while maintaining the analytical utility of the data for model development and evaluation. These practices support regulatory compliance, enhance participant trust and enable ethical scaling of mobile stress monitoring systems in real-world settings.

The dataset was collected over two weeks from the same 87 participants using the StressSenseApp. Twice daily, participants recorded short spoken responses to reflective questions (e.g. “How do you feel today?”). Responses were transcribed using the Google Speech-to-Text API, while passive behavioral data (screen interactions, app usage, accelerometer activity and GPS variance) were continuously logged. This design captured both active speech inputs and passive behavioral signals, reflecting real-world smartphone use. We additionally recorded modality-availability flags (present or missing) for each sample to support robustness tests and reliability-aware fusion.

Stress levels were annotated daily using the PSS-4 [24]. Scores ≥6 were labeled as stressed and <6 as non-stressed, with stressed samples further divided into low and high stress. To assess label robustness, raw PSS-4 scores were retained and used for sensitivity analyses of the decision thresholds and for an ordinal variant reported in Section 5. Multimodal samples were temporally aligned within a ±15-min window to support consistent multimodal fusion. Alignment timestamps were stored to enable ablations on wider and narrower windows.

Table 1 summarizes the dataset. Neutral samples were most common, followed by low stress and high stress. This imbalance mirrors real-world patterns and underscores the need for class-weighted or hybrid loss functions. The dataset contained 2,200 samples (1,430 training, 350 validation and 420 testing). All experiments used subject-independent five-fold cross-validation with users held out by fold to prevent identity leakage. Feature normalization and any statistic that could leak information were computed on the training partition only within each fold, with the same transform applied to validation and test. A five-fold subject-independent cross-validation protocol was used to ensure robustness and minimize participant-specific bias. At inference time, missing modalities were handled by zero padding plus masks and down-weighting through the gating module (Section 4.4).

The system integrates three complementary modalities acoustic, linguistic and behavioral to capture stress-related indicators (Figure 1). Each was selected based on prior evidence linking its features to psychological stress [9, 10, 13 and 14]. For all modalities, we attach a small set of quality signals used only by the reliability-aware gating module (Section 4.4).

Acoustic. Speech is resampled to 16 kHz mono. Features include eGeMAPS descriptors from OpenSMILE [31] and log-Mel spectrograms (128 bins, 25 ms frames and 10 ms hop), providing prosodic and spectral cues. We apply simple voice activity detection and compute a VAD-based signal-to-noise ratio estimate per sample as an acoustic quality indicator. Cepstral mean–variance normalization parameters are fitted on the training partition only.

Linguistic. Transcripts from ASR are embedded with a BERT-base model [32 and 33]. To enrich semantics, sentiment polarity and lexical stress markers (e.g. affective term frequency) are added via LIWC-22 and NLTK [34]. We use the segment-level ASR confidence provided by the speech-to-text service as a text reliability proxy and include an out-of-vocabulary rate computed against a standard lexicon.

Behavioral. Passive logs capture unlock frequency, session length, app-transition entropy, accelerometer variance and GPS-based location entropy [13, 14 and 35], normalized per user to reduce bias. To avoid leakage, per-user normalization parameters are estimated on that user’s training data within each fold; at test time we simulate on-device baselining by applying the precomputed transform. Behavioral quality is summarized by coverage (fraction of the window with valid logs) and a sparsity flag.

All streams are aligned within a ±15-min window to ensure coherent multimodal snapshots, supporting CMAF-Net’s cross-modality attention (Section 4.4) and robustness under noisy or sparse inputs. When a modality is missing, availability masks are propagated to the fusion layer so that attention and gating can reweight the remaining evidence rather than imputing unsupported values.

The proposed Cross-Modality Attention Fusion Network (CMAF-Net) integrates acoustic, linguistic and behavioral features in a smartphone-based stress detection framework. The overall pipeline (Figure 1) has four stages:

1. Modality-specific encoders that learn native representations for each input stream,

2. Prior-informed embedding that attaches a compact, learnable vector of psychological priors to each modality,

3. Knowledge-guided cross-modality attention with reliability-aware gating that adapts weights using measured input-quality signals and

4. A lightweight classification head optimized for on-device inference.

CMAF-Net follows four guiding principles drawn from the objectives and constraints in Section 3.1:

1. Modality-specific representation learning

Acoustic, linguistic and behavioral inputs are first modeled in their own feature spaces by dedicated encoders and then projected to a common dimension for cross-modal attention, allowing missing inputs to be masked without imputation.

1. Knowledge-guided fusion

Psychological stress priors are embedded directly in the fusion stage: each modality embedding is concatenated with a learnable prior vector and a bias term that steers attention toward clinically meaningful cross-modal relations (see Section 4.3).

1. Dynamic modality weighting

A reliability-aware gate uses input-quality indicators such as audio SNR, ASR word-error rate and behavioral coverage to adapt each modality’s weight and ensure graceful degradation under noise or missing data (see Section 4.4).

1. Deployment-oriented efficiency

Compact encoders, a single attention block and post-training quantization yield a small model with real-time latency on smartphones while maintaining well-calibrated probabilistic outputs (see Sections 4.5 and 5.4).

Together these principles balance accuracy, interpretability and mobile efficiency for real-world stress monitoring.

Each modality is processed by a dedicated encoder and projected into a common embedding size d before fusion. All encoders output a masked embedding and a small reliability vector r_m​ that is consumed by the gating module in Section 4.4.

Log-Mel spectrograms (128 bins, 25 ms frame and 10 ms hop) pass through a two-layer CNN with 3x3 kernels and 64 filters per layer, each followed by ReLU, batch normalization and dropout p = 0.3. The CNN output feeds a BiLSTM with 128 hidden units per direction to capture longer prosodic context that is relevant for stress [21 and 36]. Global mean and standard deviation pooling over time produce a fixed-length vector that is linearly projected to $Rd$⁠. Voice activity detection provides an acoustic signal-to-noise ratio estimate that forms part of r_audio​. Cepstral mean and variance normalization parameters are fitted on training data only.

ASR transcripts are embedded with a BERT-base model [31–33]. We freeze most transformer layers and fine-tune the top layer to reduce overfitting on a small cohort. Mean pooling over the token sequence yields a 768-dimensional vector. We concatenate sentiment polarity and lexical markers derived from LIWC-22 and NLTK [34], then apply a linear projection to $Rd$⁠. The reliability vector r_text ​ includes ASR word error rate and average token confidence.

A 12-dimensional feature vector from passive sensing (Section 3.4) is linearly projected to 64 dimensions and processed by a two-layer Transformer encoder with hidden size 128 and two heads [22]. Learned positional embeddings model short temporal dependencies across the window. We apply masked mean pooling to handle missing segments, then project to $Rd$⁠. The reliability vector r_behav summarizes coverage fraction and a sparsity indicator.

Unless otherwise stated, d = 128. The resulting embeddings and reliability vectors are used by the knowledge integration in Section 4.3 and the fusion and gating in Section 4.4.

We augment each modality encoder output $hm∈Rd$ with a learnable prior vector $km∈R32$ that encodes psychologically motivated cues, such as reduced pitch variability or increased spectral tilt in speech and higher app-switching entropy in behavioral logs [9, 10, 14 and 23]. The prior-augmented representation is

where [; ] denotes vector concatenation.

To steer cross-modal interactions toward clinically meaningful relations, we introduce a prior-informed bias between modalities m and n:

where $∅(·)$ is a small multilayer perceptron (MLP), $vb$ is a learned vector and ⊙ is the element-wise (Hadamard) product. Section 4.4 uses $pm$ or $bmn$ inside the attention mechanism.

Given $pm$ and $bmn$ from Section 4.3, we compute scaled dot-product attention across modalities [22]. For each source modality m and target modality n:

with $Wq,Wk,Wv$​ learned projection matrices. The attention score is

where d_a is the attention dimension and (⋅)^⊤ denotes matrix transpose. The normalized weight is

with the softmax taken over the target-modality index n. The context vector for source modality m is

Missing-modality masks set $αmn=0$ when modality n is unavailable.

A reliability-aware gate adjusts each modality’s contribution using a quality vector $rm$ (audio SNR, ASR word-error rate, behavioral coverage and sparsity):

where σ(⋅) is the sigmoid function and MLP(⋅) is the feed-forward network of the gate.

Finally, the fused representation is

which down-weights unreliable streams and yields graceful degradation under noise or missing inputs. The vector $f$ is fed to the classifier in Section 4.5.

The fused multimodal representation f is passed to a lightweight classification head designed for real-time inference on smartphones (see Figure 1). The head consists of a fully connected layer (128 units, ReLU), dropout p = 0.3, and a softmax layer that outputs class probabilities over {Neutral, Low Stress, High Stress}. We compute the predicted class:

To meet resource constraints (Section 3.1), we apply post-training dynamic quantization with TensorFlow Lite [25]. This reduces model size and CPU latency without altering training. We report model footprint, on-device inference latency and accuracy before and after quantization in Section 5.4. For probability quality, we evaluate calibration using expected calibration error and Brier score and apply temperature scaling on the validation split if needed to produce stable decision thresholds.

CMAF-Net was implemented in PyTorch 1.13 and trained on a workstation with an NVIDIA RTX 3090 GPU. Training used the Adam optimizer [37] (β₁ = 0.9 and β₂ = 0.999), batch size 32, learning rate 3 × 10⁻⁴ with exponential decay (factor 0.95 every five epochs) and early stopping with patience 7 based on validation macro F1. Weights were initialized with Xavier uniform. Gradients were clipped to a global norm of 1.0.

To address class imbalance, we compare class-weighted cross-entropy and focal loss, selecting the setting that maximizes validation macro F1. Unless otherwise noted, the objective is

where λ = 10^–4 and θ denoting trainable parameters. Regularization includes dropout p = 0.3 across encoders and the classifier. For robustness, we apply mild data augmentation on audio (additive background noise and time–frequency masking) and simulate transcript noise by randomly dropping low-confidence tokens proportional to ASR confidence; behavioral features use small Gaussian jitter within training-only normalization bounds.

Evaluation uses five-fold subject-independent cross-validation: each fold serves once as test data, with the remainder split 80:20 into training and validation. All preprocessing statistics, normalization parameters and any calibration temperatures are learned on the training or validation data within each fold only, then applied to the held-out test fold. We report mean and standard deviation across folds and perform paired significance tests against strong baselines to assess robustness and generalization to unseen users [38].

We evaluate CMAF-Net on the dataset in Section 3.3 using five-fold subject-independent cross-validation to simulate deployment to unseen users and reduce identity leakage risk [38]. Labels are derived from PSS-4 as described in Section 3.3. Unless noted, all preprocessing statistics are computed on the training split within each fold and applied to validation and test only.

We implement three canonical baselines:

1. Early Fusion (EF): Concatenation of all features followed by a multilayer perceptron.

2. Late Fusion (LF): Independent unimodal classifiers with probability averaging.

3. CMAF-Net without priors: Removal of prior vector and bias to isolate the value of knowledge guidance.

In addition, we compare against recent multimodal stress models reported in the literature (Section 5.2). For fairness, we match input features and report parameter counts and FLOPs when applicable.

We report accuracy (ACC), macro F1 and AUC, averaged across folds with mean ± standard deviation. To address class imbalance, we also compute macro precision, macro recall, balanced accuracy and per-class PR curves. We assess statistical significance with paired tests across folds versus the strongest baseline and provide 95% bootstrap confidence intervals. For downstream decision support, we evaluate calibration using expected calibration error (ECE) and Brier score and apply temperature scaling on the validation split when beneficial.

We run missing-modality stress tests by masking each modality at inference and noise stress tests by adding background noise to audio and perturbing transcripts according to ASR confidence. We also evaluate a reliability-blind variant of CMAF-Net (gates fixed to 1) to quantify the effect of reliability-aware gating.

To examine generalization, we train on our dataset and test on an independent mobile dataset with compatible labels, and conversely, fine-tune on a small subset from the external set. We report performance shifts and recovery after light adaptation. Details and additional baselines appear in the Supplementary Material.

Table 2 summarizes the performance of CMAF-Net compared with canonical fusion baselines and three recent multimodal stress recognition models: MMP-CNN [39], DeepStress [40] and SSFT [41]. CMAF-Net achieves the best overall results on the internal dataset, with an F1-score of 0.89 and AUC of 0.92, outperforming the strongest baseline (Late Fusion) and SSFT (F1 = 0.86 and AUC = 0.89). The +3 point gain in both F1 and AUC underscores the contribution of prior-informed cross-modal attention and reliability-aware gating, which enhance discrimination in noisy or incomplete conditions. Improvements were confirmed with fold-wise paired significance tests (p < 0.01) and 95% confidence intervals.

External validation on the StressID dataset [19] further highlights the generalization capability and domain adaptation potential of CMAF-Net. Without any fine-tuning, the model achieved an F1-score of 0.78 (95% CI 0.75–0.81) and AUC of 0.85, indicating good transfer performance despite demographic and contextual shifts. After lightweight fine-tuning using a small labeled subset (10% of StressID), performance improved to F1 0.83 (95% CI 0.80–0.86) and AUC 0.88, narrowing the gap to internal results. This aligns with the deployment objective of enabling rapid personalization or domain adaptation with minimal overhead.

These results also reveal systematic performance variations across demographic subgroups within the external dataset, particularly for age and gender groups. For example, F1 scores for the younger subgroup (18–30) were slightly higher compared to older participants, likely reflecting differences in speech patterns and smartphone usage behavior. This underscores the need for fairness-aware model monitoring during deployment to mitigate potential biases stemming from linguistic variability, sensor heterogeneity or ASR reliability.

Compared to all baselines, CMAF-Net shows superior probabilistic calibration, with lower expected calibration error (ECE) and Brier score, ensuring stable decision thresholds a critical property for health-related applications. Calibration curves are included in the main text to emphasize this contribution.

For completeness, we also benchmarked CMAF-Net against stronger contemporary multimodal fusion models (e.g. multimodal transformer fusion and stacking late-fusion meta-learner). Despite their larger parameter counts and higher FLOPs, these models did not surpass CMAF-Net in F1 or AUC, highlighting the efficiency and deployment-readiness of the proposed architecture for resource-limited devices.

We conduct ablations on (1) prior size and design, (2) component contributions and (3) modality importance.

Table 3 varies the prior vector from 4 to 64 dimensions. Performance improves steadily and peaks at 32 dimensions (ACC 88.5% and F1 0.89), which we adopt as the default due to the favorable accuracy-to-size trade-off.

We remove individual ingredients to isolate their effects:

1. No prior vector k_m (keeps attention but discards domain knowledge).

2. No bias b_mn​ (keeps priors but removes cross-modal biasing).

3. Randomized priors (the same size as default but shuffled across samples).

4. No reliability gating (gates fixed to 1 for all m).

These ablations reduce macro F1 relative to full CMAF-Net, with randomized priors collapsing toward the “w/o prior” variant, which supports the claim that learned, semantically anchored priors drive the gains rather than capacity alone. Results and statistics are provided in the Supplementary Material.

Table 4 shows modality importance by removing one input at a time. Linguistic features contribute the most; their removal drops accuracy to 82.1%. Acoustic features are next in importance, and behavioral signals provide complementary robustness, especially in conditions with noisy audio or high ASR error rates.

With the reliability-aware gate enabled, CMAF-Net degrades gracefully under modality dropout and noise corruption. The largest relative gains over a reliability-blind variant occur when audio SNR is low or behavioral coverage is sparse, indicating the gate correctly down-weights unreliable streams.

Additional ablations, PR and ROC curves by class, calibration plots and fold-wise statistics appear in the Supplementary Material.

To assess real-world deployment feasibility, we profiled CMAF-Net on two smartphone classes: a mid-range device (Qualcomm Snapdragon 720G, 6 GB RAM) and a lower-end device (Snapdragon 665, 4 GB RAM), both using TensorFlow Lite with CPU-only inference and batch size 1. Measurements were taken in steady state after a short warm-up to minimize cold-start effects and thermal throttling variability.

Post-training dynamic quantization (Section 4.5) reduced model size from 18.2 MB to 7.6 MB, a 58% compression and decreased average single-thread CPU latency from approximately 320 ms to 180 ms on the mid-range device. On the lower-end device, mean latency was 235 ms, which remains acceptable for real-time, on-device stress inference. Importantly, quantization produced no measurable loss in accuracy (0.88) or macro-F1 (0.89) and calibration metrics remained stable (ECE <0.02), allowing the same threshold settings used in the full-precision model to transfer directly.

In addition to latency and model size, we profiled energy consumption using Android’s built-in battery stats and the Trepn Profiler tool. CMAF-Net consumed on average 38.2 mJ per inference on the mid-range device and 52.6 mJ on the lower-end device, measured across 1,000 runs. This places it well within the power budget for continuous background sensing scenarios when used with low-frequency triggers (e.g. 2–4 inferences per hour), making it practical for daily health monitoring without noticeable battery drain.

Because both latency and energy can vary across devices and conditions, we report mean ± standard deviation and recommend that future deployments follow standardized mobile energy reporting protocols. These results confirm that CMAF-Net meets the real-time and efficiency targets specified in Section 3.1 and supports privacy-preserving on-device inference without reliance on cloud resources.

Figure 2 shows the confusion matrix across the three stress levels. As expected, the majority of misclassifications occur between Neutral and Low stress, reflecting the overlap in acoustic and linguistic cues associated with mild stress. For example, Neutral speech with low pitch variability, reduced lexical diversity or low behavioral activity can closely resemble early stress states. In contrast, misclassifications between Low and High stress are less frequent and typically occur when emotional tone is mixed or behavioral patterns are inconsistent.

All classes exhibit strong AUC values (0.93 Neutral, 0.90 Low stress and 0.91 High stress), indicating stable decision boundaries. However, Neutral–Low confusions account for 62% of all errors, suggesting that finer-grained modeling is required in this boundary region. This finding aligns with prior stress studies showing blurred physiological and behavioral distinctions in mild stress states.

We analyzed errors with respect to recording context and signal quality:

1. Late-night recordings with monotonous speech and minimal movement were most prone to Neutral–Low confusions.

2. High ASR word error rates tended to shift predictions toward the Low class, reflecting reduced linguistic reliability.

3. In these cases, CMAF-Net’s reliability-aware gate increased the weight on behavioral features such as app-transition entropy and screen-on duration, which improved but did not fully eliminate these confusions.

4. When audio was noisy or missing, CMAF-Net degraded gracefully thanks to adaptive gating and behavioral sparsity produced similar but smaller effects.

To better understand the decision mechanism, we visualized attention heatmaps for both correctly classified and misclassified samples (Figure 3). For High stress predictions, the model consistently emphasized decreased pitch variability and negative lexical markers, along with increased mobility entropy, reflecting well-established behavioral stress indicators. In Neutral–Low borderline cases, however, attention weights were more diffuse, indicating greater ambiguity in evidence aggregation and less confident cross-modal alignment.

To further illustrate, Figure 3(c) aggregates attention distributions across all test samples, contrasting correct and misclassified predictions. Attention entropy increased by approximately 17% for misclassified samples, confirming that the model exhibits more uncertain and dispersed attention in ambiguous scenarios (Figure 3(d)). These visualizations provide face-valid explanations consistent with psychological stress theory and highlight how reliability-aware gating adaptively shifts attention toward more informative modalities when others are unreliable.

During model development, we implemented several strategies that reduced Neutral–Low confusions without harming overall macro F1:

1. Modest threshold tuning optimized for macro F1 rather than raw accuracy, improving class balance at the Neutral–Low boundary.

2. Adding an ordinal head in a sensitivity analysis respected the natural class ordering and slightly improved separation.

3. A hierarchical decision scheme, first detecting stressed vs non-stressed, then refining to Low vs High stress showed further potential improvements.

Future work will explore temporal priors (e.g. circadian context or temporal smoothing), personalized baselines that adapt to each user’s speech and behavioral profile and lightweight confidence calibration mechanisms to handle ambiguous cases.

To assess potential demographic performance disparities and ensure that CMAF-Net maintains equitable behavior across population subgroups, we conducted a stratified analysis of model performance by gender, age group and device type. This analysis aligns with fairness recommendations for health AI systems, particularly in contexts where sensor reliability and ASR accuracy can vary systematically with user characteristics.

Table 5 presents F1-score, AUC and expected calibration error (ECE) across demographic subgroups. Performance was broadly consistent, but notable differences emerged:

1. Gender: F1 and AUC were slightly higher in female participants than male participants (ΔF1 = +0.02), possibly reflecting differences in speech prosody and ASR accuracy.

2. Age: Participants aged 18–30 showed higher performance (F1 = 0.89) than those aged 40+ (F1 = 0.84). Qualitative review suggests increased ASR word error rate and behavioral sparsity in older participants contributed to this gap.

3. Device type: Performance was marginally lower on lower-end devices, reflecting higher noise levels and more frequent missing behavioral logs.

Calibration remained stable across subgroups, with ECE values below 0.03 in all cases, indicating that threshold-based decisions can be applied consistently across demographic segments. However, slightly higher ECE was observed in older participants, suggesting room for targeted recalibration or personalized adaptation.

These results highlight the importance of demographic monitoring and fairness-aware deployment in real-world health applications. Future work will explore adaptive strategies such as personalized baseline modeling, demographically informed calibration and domain adaptation methods to further mitigate subgroup disparities.

The evaluation in Section 5 demonstrates that CMAF-Net consistently outperforms both canonical fusion baselines and recent multimodal stress detection models in terms of accuracy, macro F1 and AUC. These gains stem from two core innovations that act synergistically. First, prior-informed cross-modality attention leverages a compact prior vector and cross-modal bias to guide information flow toward clinically meaningful cues known from psychological and behavioral stress literature. Second, reliability-aware gating dynamically modulates modality weights based on measurable input quality (audio SNR, ASR WER and behavioral sparsity), mitigating the impact of unreliable signals and enabling graceful degradation under noisy or missing inputs. Ablation results support the contribution of both mechanisms, while calibration analyses confirm that CMAF-Net provides more reliable and well-calibrated probability estimates, which is critical for threshold-based health decision support.

External validation confirms that CMAF-Net generalizes beyond the training cohort, but performance declines under domain shift, an expected outcome in mobile sensing. Lightweight fine-tuning and domain adaptation strategies (e.g. few-shot personalization or accent-specific recalibration) partially recover this gap, suggesting a practical adaptation pathway for deployment across diverse populations. Stratified performance analyses indicate that demographic factors such as gender and age can subtly influence accuracy and F1 scores, pointing to the importance of fairness monitoring and adaptation mechanisms in future deployments. This aligns with prior work on domain adaptation for mobile health systems and supports reviewer recommendations to consider more explicit fairness analysis.

Error analysis reveals a consistent ambiguity between Neutral and Low stress, mainly driven by overlapping acoustic and linguistic cues under mild stress conditions and inconsistent behavioral patterns, particularly during late-night recordings. A hierarchical decision scheme first separating stressed versus non-stressed states, followed by finer-grained classification, shows potential to reduce these confusions without sacrificing macro F1. In addition, stress is inherently dynamic, varying with circadian and weekly rhythms. While our current model uses single snapshots, future work will incorporate temporal priors or lightweight recurrent mechanisms to capture temporal dependencies and further improve class separation.

A key limitation is the geographic and linguistic homogeneity of the development dataset. Although external validation on StressID partially addresses this, broader datasets spanning multiple languages, accents and cultural contexts will be essential for robust deployment. We plan to expand fairness analysis across demographic slices (e.g. age, gender, linguistic profile and device type) and assess potential biases from ASR systems or sensor availability. These steps will be integrated with standardized fairness metrics and stratified reporting in future work.

CMAF-Net was designed for on-device deployment. In addition to latency and model size, energy consumption is an important consideration for real-world applications. Profiling on a mid-range smartphone shows that quantization yields a lightweight model with low CPU utilization; however, standardized energy consumption metrics across multiple device classes are needed to strengthen deployment claims. We are incorporating this in our ongoing field studies.

Privacy and regulatory compliance are fundamental to digital health applications. Our pipeline adheres to the principle of data minimization: raw audio and GPS never leave the device, only derived features are processed and transmissions are encrypted. This design aligns with GDPR and HIPAA principles, ensuring compliance and user trust. Additional technical safeguards such as on-device anonymization and secure enclave storage will be part of future implementation plans.

Attention visualizations provide face-valid explanations that align with expected psychological cues (e.g. reduced pitch variability or negative lexical markers under High stress). In the revised figures, we include explicit attention heatmaps and a quantitative summary of attention weight distributions, reinforcing interpretability claims. These interpretable outputs support both clinical validation and user-facing transparency.

CMAF-Net, a knowledge-guided cross-modality attention framework for smartphone-based stress recognition, achieved 88.5 % accuracy, 0.89 macro-F1 and 0.92 AUC, outperforming strong Early- and Late Fusion baselines as well as recent multimodal models. Ablation studies confirmed the benefit of prior-informed attention and reliability-aware gating and on-device tests showed ∼180 ms mean latency after quantization with no loss in accuracy, supporting real-time mobile deployment.

The main remaining challenges are the inherent ambiguity between Neutral and Low stress, performance drops under domain shift and dependence on complete multimodal input. Future work should incorporate temporal priors and personalized baselines, explore hierarchical decision schemes and add optional redundant signals such as heart rate variability. Broader data collection across cultures and languages, fairness and calibration reporting and public interpretability tools will further enhance trust and usability.

Overall, embedding domain knowledge in adaptive multimodal fusion improves stress recognition while meeting privacy and real-time constraints, providing a practical path toward scalable, clinically interpretable digital health monitoring.

Formal ethics review was not required for this study because data collection involved only non-clinical, anonymized smartphone sensor and survey data and posed no foreseeable physical, psychological or social risks to participants. The research followed institutional guidelines for minimal-risk behavioral studies, and all participants provided informed consent before participation.

The supplementary material for this article can be found online.