A multimodal deep learning framework for real-time defect recognition in industrial components using visual, acoustic and vibration signals Open Access

Industrial quality assurance has undergone significant transformation over the past 2 decades, evolving from manual inspection techniques toward highly automated and intelligent systems. In the context of Industry 4.0, where smart factories rely on real-time data streams and autonomous decision-making, defect detection has become a cornerstone of operational reliability and cost efficiency. Traditional defect inspection methods, primarily reliant on human vision or basic threshold-based signal processing, have proven inadequate in scenarios requiring high speed, precision, and adaptability. The complexity of modern industrial environments demands more sophisticated systems that can detect diverse types of defects, ranging from superficial scratches and micro-cracks to internal structural inconsistencies, often occurring simultaneously.

As manufacturing systems increasingly incorporate automation, sensors, and networked computing infrastructure, the opportunity to utilize multimodal data for defect recognition becomes highly feasible. The use of visual imaging, acoustic emission sensing, and vibration monitoring presents a comprehensive means of capturing both surface-level and latent anomalies in industrial components. Yet, integrating these modalities in an effective and computationally efficient manner remains a technical challenge. Visual signals are rich in spatial information but lack penetration depth. Acoustic signals can detect transient internal events, such as crack propagation or material fatigue, but are susceptible to ambient noise. Vibration signals, often captured through accelerometers, provide crucial information about mechanical instability but are typically difficult to interpret in isolation. The integration of these heterogeneous data streams into a unified, intelligent decision-making framework is the central motivation behind this study.

While our framework is evaluated in a simulated environment, future work should validate its robustness in actual industrial settings with factors such as variable lighting, ambient noise, and sensor drift.

Despite notable advances in deep learning and machine perception, existing defect detection systems are often siloed by modality, leading to limited generalization, high false positive rates, and reduced robustness under varying operational conditions. Visual inspection systems based on convolutional neural networks (CNNs) have demonstrated success in identifying surface anomalies but perform poorly with hidden or evolving faults. Conversely, acoustic-based models using recurrent neural networks (RNNs) or temporal convolutional networks (TCNs) can capture time-series signatures of structural faults but may fail to distinguish between background machinery noise and defect-related anomalies. Vibration analysis, while sensitive to internal changes in system dynamics, requires specialized preprocessing and suffers from signal attenuation in complex mechanical assemblies.

Moreover, current implementations rarely operate in real time, and when they do, they are constrained by high computational loads or reduced accuracy. The lack of real-time, interpretable, multimodal solutions limits industrial adoption, particularly in high-throughput environments such as automotive manufacturing, aerospace assembly lines, and semiconductor fabrication. There is a critical need for a unified framework capable of jointly processing visual, acoustic, and vibration signals to provide accurate, low-latency defect detection in simulated environments, with potential extension to real-world scenarios despite challenges like environmental variability.

The overarching objective of this research is to develop a real-time, multimodal deep learning framework for defect recognition in industrial components by fusing visual, acoustic, and vibration signals. The specific goals of the study are as follows.

1. To design a novel neural network architecture that effectively combines convolutional, sequential, and transformer-based modules to process spatial and temporal features from three distinct modalities.

2. To construct and annotate a multimodal dataset of defect samples acquired under controlled industrial conditions, enabling reproducible experiments and benchmarking.

3. To evaluate the model's performance against unimodal baselines and classical machine learning classifiers in terms of accuracy, inference time, and robustness under varying noise and lighting conditions.

4. To incorporate explainability modules such as Gradient-weighted Class Activation Mapping (Grad-CAM) and SHAP values to enhance transparency and support industrial validation.

This research is situated at the confluence of artificial intelligence, signal processing, and manufacturing engineering, and addresses several pressing concerns in modern industry.

1. Economic Impact: Early and accurate detection of component defects reduces production downtime, minimizes waste, and lowers the cost of warranty claims.

2. Operational Safety: Failures due to undetected defects can result in catastrophic system breakdowns, especially in critical infrastructures like transportation and energy sectors.

3. Technological Advancement: By enabling real-time analysis on edge devices, this work advances the capabilities of on-site quality control systems without reliance on centralized cloud computing.

4. Research Contribution: The proposed fusion strategy and network architecture introduce a scalable template that can be adapted to other multimodal industrial monitoring problems, such as fault diagnosis in rotating machinery or anomaly detection in pipelines.

Several research and practical challenges are tackled in this work.

1. Multimodal Alignment: Synchronizing heterogeneous sensor data captured at different frequencies and sampling rates poses a challenge for end-to-end model training.

2. Noise Robustness: Real-world industrial environments introduce acoustic and vibrational noise that can obscure meaningful signals, necessitating robust preprocessing and denoising strategies.

3. Computational Efficiency: Achieving real-time performance requires architectural optimization and possibly model compression without significant loss of accuracy.

4. Generalization: Models must be capable of detecting both known and unknown defect types across various industrial components and settings.

The core contributions of this work are summarized as follows.

1. A unified, multimodal deep learning architecture that leverages CNNs, Bi-directional GRUs, and transformer encoders to extract and fuse features from images, audio spectrograms, and accelerometer data.

2. A curated multimodal dataset comprising annotated recordings of defective and defect-free components across multiple real-world industrial scenarios.

3. A real-time inference framework optimized for deployment on edge devices using TensorRT and ONNX for reduced latency.

4. Explainable AI modules integrated into the defect classification pipeline, allowing insight into modality-specific contributions and aiding human interpretability.

By addressing the limitations of existing unimodal systems and proposing a scalable, interpretable, and real-time solution, this paper contributes a substantial advance to the field of pattern recognition and artificial intelligence in industrial contexts.

Defect detection in industrial systems has traditionally relied on unimodal approaches, each leveraging a single sensor modality. Among the most prevalent are visual inspection systems, which employ high-resolution imaging and convolutional neural networks (CNNs) to identify surface-level faults such as cracks, scratches, or deformations. These systems have seen widespread deployment due to the simplicity of image acquisition and the success of pretrained CNNs such as ResNet, VGG, and EfficientNet in extracting hierarchical spatial features (Ma et al., 2018). However, visual systems suffer when defects are internal or when lighting and texture variability degrade image quality.

Parallel to visual techniques, acoustic signal-based methods have been employed, particularly in rotating machinery and additive manufacturing (Singh and Ahmad, 2025). Methods such as time-domain feature extraction combined with recurrent neural networks (RNNs) or long short-term memory (LSTM) networks allow for capturing transient fault signatures that may not be visible. However, these systems are prone to noise contamination and require carefully controlled acoustic environments for optimal performance.

Vibration analysis, typically using accelerometers, offers another unimodal route for detecting structural anomalies (Bhuiyan and Uddin, 2023). Frequency-domain transformations like Fast Fourier Transform (FFT) or Wavelet Packet Decomposition (WPD) help reveal hidden patterns related to imbalance, misalignment, or bearing faults. Deep learning models such as autoencoders and temporal convolutional networks (TCNs) have recently been used to detect deviations in these signals (Lu et al., 2022), yet these methods alone lack the specificity needed in complex manufacturing scenarios where multiple failure modes coexist.

The limitations of unimodal methods have catalyzed the development of multimodal fusion approaches, which integrate data from multiple sensors to enhance detection accuracy and robustness. Multimodal defect detection systems offer a richer, complementary representation of faults by combining spatial, acoustic, and dynamic features. Recent research by Singh and Ahmad (2025) demonstrated the effectiveness of combining vibration and acoustic data for fault detection in manufacturing, significantly outperforming single-modality baselines. Similarly, Ma et al. (2018) introduced a deep coupling autoencoder model to fuse temperature, vibration, and sound data, achieving robust classification across diverse defect types.

More advanced architectures now utilize cross-modal attention and joint representation learning to align features across modalities. For example, Cao and Shi (2025) applied residual neural networks combined with joint embedding techniques for rotating machinery diagnostics, yielding accurate health status identification even under noisy conditions. In rail surface diagnostics, Wang et al. (2025) demonstrated the utility of fusing vibration signals with vision data, resulting in improved detection of micro-level wear and internal fractures.

The integration of multimodal inputs into deep learning frameworks is typically realized at one of three levels: early fusion, intermediate fusion, or late fusion (Liu et al., 2024a, b). Early fusion involves combining raw signals or features from all modalities at the input level, allowing the network to learn cross-modal correlations from the outset. Intermediate fusion, often implemented using parallel subnetworks, allows each modality to learn its unique representation before merging them in deeper layers. Late fusion, in contrast, combines the output probabilities or decisions from individual networks, which may lose cross-modal interactions.

Recent works favor intermediate fusion using specialized deep learning blocks. Kullu and Cinar (2022) proposed a dual-path neural architecture, where vibration and image data streams were processed separately and then merged via attention-based fusion. Their system demonstrated enhanced robustness in predicting component fatigue in dynamically varying environments. Similarly, Zhang et al. (2025) designed a transformer-based fusion model that used modality-specific encoders followed by a joint attention mechanism, applied successfully in fault localization for belt conveyors.

Despite promising accuracy gains, many multimodal systems are computationally intensive and unsuitable for real-time or edge device deployment. Industrial settings often impose latency constraints, demanding inference times under 100 ms. Lightweight architectures like MobileNet or model pruning strategies have been proposed to address this (Zhou et al., 2022), but few studies implement full-stack pipelines capable of handling real-time synchronization, buffering, and multimodal fusion in deployment environments.

Notably, Bhuiyan and Uddin (2023) emphasized the need for compact, deployable architectures when using vibration and acoustic sensor data. Their deep transfer learning models reduced training time but still required GPU acceleration. More recent attempts to optimize such systems include quantized models and ONNX runtime frameworks to meet real-time performance requirements without sacrificing accuracy (Wang et al., 2024).

Another emerging concern in industrial AI is model interpretability. Black-box decisions are often insufficient for high-stakes manufacturing decisions. Tools like Grad-CAM, SHAP, and LIME have been adapted to multimodal frameworks to highlight the contribution of each modality toward a decision. Zhao et al. (2023) used modality-wise attention maps to determine whether sound or image data had a greater influence on a given fault classification, improving operator trust.

In practice, explainability enables model debugging, system validation, and compliance with regulatory or safety protocols. Few models, however, offer integrated interpretability modules alongside their inference pipeline, leaving a gap in usability for manufacturing stakeholders.

While numerous studies have validated the effectiveness of vibration–acoustic or image–vibration fusion for defect detection, the integration of all three modalities—visual, acoustic, and vibration—remains underexplored. Only a handful of papers, such as those by Liu et al. (2024a, b) and Zhang et al. (2023), attempt full sensor fusion, often with incomplete datasets or lacking interpretability features.

Furthermore, many approaches do not include synchronized data acquisition frameworks or ignore real-time constraints, limiting industrial deployment potential. There is also a lack of publicly available benchmark datasets that feature all three sensor types recorded simultaneously during defect development processes. These limitations constrain reproducibility and comparative evaluation of new methods.

Recent advancements in large language models (LLMs), such as GPT-4 and LLaMA, have extended beyond natural language processing to multimodal signal processing, where they integrate text, images, audio, and time-series data through unified transformer architectures. For instance, models like CLIP and Flamingo have demonstrated success in cross-modal tasks by leveraging vast pre-trained embeddings to align visual and acoustic signals with descriptive prompts, enabling zero-shot defect detection in industrial settings (Radford et al., 2021). In vibration analysis, LLMs have been adapted for anomaly detection by converting time-series data into tokenized sequences, allowing for semantic reasoning over multimodal inputs (Huang et al., 2024). These approaches excel in interpretability and generalization, as they can incorporate domain knowledge via prompting without extensive retraining.

However, LLMs pose challenges for real-time industrial applications due to their high computational demands and latency, often requiring significant GPU resources that exceed edge device capabilities. In this work, we prioritize lightweight, specialized neural modules over LLMs to ensure low-latency inference, though future extensions could explore hybrid LLM-fine-tuning for enhanced zero-shot adaptability in defect recognition.

This study proposes a multimodal deep learning framework for real-time defect recognition in industrial components, leveraging three synchronized data sources: visual imagery, acoustic emission signals, and vibration time-series data. The core idea is to capture spatial, temporal, and dynamic fault indicators using tailored neural network components that process each modality independently before performing intermediate-level fusion via cross-attention.

Let the dataset be defined as:

where.

1. $Ii∈RH×W×C$⁠: image of the component,

2. $Ai(t)∈RT×1$⁠: acoustic waveform,

3. $Vi(t)∈RT×d$⁠: vibration signal with $d$ axes,

4. $yi∈{0,1}$⁠: binary defect label (or $yi∈{1,…,K}$ for multi-class).

Each modality undergoes specialized preprocessing, feature extraction, and transformation before concatenation into a shared representation space.

Visual images are first normalized and resized to $224×224$ resolution. To extract rich spatial features, a ResNet-50 backbone is employed, whose convolutional blocks map the input image into a compact feature representation $fv∈RDv$​.

where $ϕv(·)$ denotes the image encoding function.

Acoustic data $A(t)$ is transformed using Short-Time Fourier Transform (STFT) into a log-magnitude spectrogram:

A 2D CNN is used to extract time–frequency features:

We also experimented with 1D wavelet scattering for robustness to nonstationarity, as it provides multi-scale representations better suited for transient signals like sudden crack emissions. However, wavelet scattering increased inference time by 40% due to higher computational complexity. To balance efficiency and handling of non-stationary signals, we selected STFT with a window size of 2048 samples and 50% overlap, supplemented by adaptive thresholding in the spectrogram to suppress transient noise. For further improvements while maintaining efficiency, techniques like constant-Q transform (CQT) could be explored in future iterations, offering logarithmic frequency resolution at comparable cost to STFT.

The vibration signals $V(t)$ are multi-axial and sampled at high frequency. We extract statistical and frequency-domain features, including:

1. RMS, Peak, Crest Factor,

2. Spectral Kurtosis: $SK=μ4σ4$​,

3. Teager-Kaiser Energy Operator (TKEO): $Ψ[x(t)]=x2(t)−x(t+1)x(t−1)$

These features were selected based on their proven efficacy in vibration-based fault diagnosis: RMS and Peak capture amplitude variations indicative of imbalances; Crest Factor highlights impulsive events like cracks; Spectral Kurtosis detects non-Gaussianity in frequency distributions for transient faults; TKEO emphasizes energy bursts from material fatigue. This compact set (5 features per axis) reduces dimensionality from raw time-series, improving transformer efficiency while retaining discriminative power, as validated by feature importance rankings in preliminary PCA analysis (explaining >85% variance). Alternatives like full FFT spectra were avoided due to higher computational load.

Time-series are then encoded using a Transformer-based encoder:

The input sequence is embedded with positional encodings and passed through multi-head self-attention:

where MHSA is the multi-head self-attention block and FFN is the position-wise feed-forward layer.

Given the disparate sampling rates (visual at 30 FPS, acoustic at 16 kHz, vibration at 5 kHz), synchronization is achieved via timestamp-based alignment. Each signal is timestamped using a unified clock from the data acquisition system. For interpolation, we apply cubic spline interpolation to upsample lower-rate signals (e.g. visual frames) to match the highest rate, ensuring smooth transitions without introducing artifacts. Cross-correlation is computed on signal energy envelopes (smoothed RMS over 100 ms windows) using the normalized cross-correlation formula: $(τ)=∑(x(t)−x̅)(y(t+τ)−y̅)∑(x(t)−x̅)2∑(y(t+τ)−y̅)2$⁠, where $τ$ is the lag maximizing alignment (limited to ±50 ms). This corrects for minor drifts, with average alignment error <5 ms.

After extracting modality-specific embeddings $fv$​, $fa$​, and $fvib$​, we perform feature fusion using a cross-attention transformer module.

Define fused embedding as:

This cross-attention mechanism enables inter-modality relationships to emerge. Specifically, we compute the query from one modality and keys/values from another:

where $WQ,WK,WV$​ are learnable projection matrices.

This mechanism encourages cross-modality conditioning, e.g. visual features querying temporal vibration anomalies.

The fused feature vector $F∈RDF$​ is passed through a multi-layer perceptron (MLP) with dropout and batch normalization:

For binary classification, $yˆ∈[0,1]$ is the probability of defect presence; for multi-class, a softmax over $K$ classes is used.

We utilize focal loss to handle class imbalance:

where $γ=2$ and $αt∈{0.25,0.75}$ are tuned hyperparameters.

In addition, a triplet loss is included to encourage discriminative feature space geometry:

where $fa$⁠, $fp$⁠, and $fn$ are anchor, positive, and negative feature embeddings, and $δ=1$ is a margin.

Total loss:

where $λ$ is a regularization coefficient.

To support deployment in real-time industrial environments, we design the pipeline with.

1. Streaming Sensor Interface: Captures synchronized RGB frames, acoustic samples, and vibration data.

2. Signal Buffering and Framing: Implements sliding windows of size $T=2048$⁠, hop size = 512.

3. ONNX Runtime + TensorRT Acceleration: Model is converted from PyTorch to ONNX and optimized using TensorRT with mixed precision (FP16).

4. Latency Profiling: Ensures end-to-end inference time $<80$ millisecond on NVIDIA Jetson Xavier.

5. Temporal Sliding Windows for Long-Term Faults: To mitigate the risk of missing evolving defects with fixed-time windows, we incorporate a sliding window mechanism with a hop size of 512 samples, aggregating predictions over multiple overlapping frames using a majority vote ensemble. This allows detection of long-term fault development without increasing latency beyond 10 ms per inference cycle.

6. Data Management for High-Volume Signals: To handle the large volume of vibration data (e.g. ∼600 KB per minute at 5 kHz sampling), we implement online compression using techniques like downsampling to 1 kHz for non-critical periods and lossless encoding (e.g. FLAC for time-series), reducing storage by up to 70% while preserving key frequency components. During deployment, a FIFO buffer discards processed windows, ensuring memory efficiency on edge devices.

To enhance interpretability.

1. Grad-CAM is applied to the visual stream to localize defect regions.

2. SHAP values are computed for vibration and acoustic feature importance.

3. Attention heatmaps are extracted from the cross-modal attention layers.

These visualization methods help operators understand the basis of predictions and refine data collection or operational protocols.

To verify the contribution of each modality, we design ablation variants.

1. Visual-only baseline: CNN + MLP

2. Acoustic-only: STFT + CNN + MLP

3. Vibration-only: Transformer + MLP

4. Visual + Acoustic, Visual + Vibration, and All Modalities

Each model is trained under identical conditions for comparative analysis.

Table 1 shows the hyperparameter configuration used in the model training process, including the learning rate, batch size, epochs, optimizer, and specific settings for layers such as dropout and transformer heads.

To validate the performance, robustness, and practical applicability of the proposed Multimodal Deep Learning Framework for Defect Recognition, we conducted a series of controlled experiments across diverse settings. These experiments are designed to address the following questions.

1. Accuracy: How well does the model perform in detecting defects, both visible and latent?

2. Efficiency: Can the model deliver predictions in real time on edge hardware?

3. Robustness: Does performance hold under noisy, adversarial, or degraded input conditions?

4. Interpretability: Can we reliably trace back decisions to specific input signals?

All experiments are performed using a synchronized multimodal dataset collected in a simulated manufacturing line environment using industrial-grade sensors.

Experiments were conducted on a server equipped with an NVIDIA RTX 3090 GPU (24 GB VRAM), Intel Core i9-10900K CPU, and 64 GB RAM, using PyTorch 1.12.1 for model implementation. Training involved a batch size of 32, AdamW optimizer with a learning rate of 1e−4 and weight decay of 1e−2, over 60 epochs with early stopping based on validation F1 score. Data was preprocessed offline, with images augmented via random rotations (up to 15°), flips, and brightness adjustments (±20%); acoustic signals via time-stretching (0.8–1.2x) and additive Gaussian noise (SNR 20–40 dB); and vibration signals via jitter and scaling. Evaluation used 5-fold cross-validation to ensure robustness, with metrics computed on the held-out test set.

The dataset used in this study comprises 6,000 samples, each containing.

1. A high-resolution RGB image (⁠$224×224$⁠)

2. A 2-s acoustic signal sampled at 16 kHz

3. A 3-axis accelerometer vibration signal sampled at 5 kHz

Each sample is manually labeled by experts into one of four classes.

1. Normal component

2. Surface defect (e.g. cracks, corrosion)

3. Internal defect (e.g. subsurface delamination)

4. Mixed/compound defect

The dataset is split into 70% training, 15% validation, and 15% test sets. We ensured temporal independence and no leakage across modalities.

Table 2 summarizes the dataset class distribution.

We employed the following standard metrics.

1. Accuracy (Acc): Overall percentage of correct classifications

2. Precision (P): $TPTP+FP$​

3. Recall (R): $TPTP+FN$​

4. F1 Score: $2·P·RP+R$​

5. Area Under Curve (AUC) of ROC

6. Latency: Mean inference time per sample (ms)

We compared the proposed model to several baselines.

1. CNN-only (Image)

2. LSTM-only (Acoustic)

3. Transformer-only (Vibration)

4. Late Fusion Ensemble

5. Multimodal (Ours)

Table 3 shows the model performance across all classes.

Figure 1 illustrates the confusion matrix of the proposed method.

Table 4 presents the results of the ablation study, showing the contribution of each modality (image, acoustic, and vibration) to the overall performance. It demonstrates that using all modalities together yields the highest accuracy (94.7%) and F1 score (0.93), highlighting the effectiveness of combining visual, acoustic, and vibration signals in defect detection. The performance significantly improves when incorporating multiple modalities, as opposed to using individual modalities.

To assess the impact of each modality, we conducted ablation experiments:

Figure 2 shows the ROC curves for each model variant.

The proposed model is deployed on an NVIDIA Jetson Xavier NX, with average inference times shown in Table 5.

As evident, the model can run inference comfortably within <100 ms, meeting real-time thresholds.

We employed Grad-CAM to visualize image regions responsible for classification. Figure 3 shows the Grad-CAM activation map on defective images, where the highlighted regions indicate the areas of the image most responsible for the model's classification decision. This visualization technique helps to provide a clear understanding of how the model interprets various image features related to defects.

We also used SHAP for vibration data feature importance. Figure 4 illustrates the SHAP (Shapley Additive Explanations) importance scores for vibration channels, revealing the contribution of each feature in the vibration signal towards the model’s prediction of defects. The higher the SHAP score, the more critical the corresponding vibration feature is to the defect detection process. Interpreting these scores, higher SHAP values for spectral kurtosis indicate its role in detecting abrupt vibrations from internal cracks, while RMS features contribute more to steady-state anomalies like imbalances.

In acoustic data, attention heatmaps showed strong localization of defect sounds around $t=0.4−0.6s$⁠. Figure 5 shows the attention weights in the cross-modality model, particularly highlighting the strong localization of defect sounds in the acoustic data around $t=0.4$ to $t=0.6$ seconds. The attention mechanism emphasizes these key time frames as highly influential for the detection of defects based on sound.

These interpretability outputs support model trustworthiness in production settings.

Table 6 shows the significance testing results, including the p-values for various model comparisons. The p-values indicate statistically significant differences between our proposed model and the image-only, acoustic-only, vibration-only, and late fusion models. Specifically, our approach significantly outperforms each of these individual modalities, with p-values of 0.002, 0.001, 0.003, and 0.012, respectively, all of which are below the 0.05 threshold for statistical significance.

To validate statistical significance, we conducted paired t-tests between the proposed model and baselines.

All differences are statistically significant at $α=0.05$⁠.

Beyond sensor dropout (where random modality occlusion led to a 3.8% accuracy drop), we evaluated robustness under simulated real-world adversities: (1) Additive Gaussian noise to acoustic/vibration signals (SNR 10–30 dB), resulting in 91.2% accuracy (3.5% drop); (2) Lighting variations in images (brightness ±50%, contrast ±30%), yielding 92.5% accuracy; (3) Sensor drift emulation via gradual offset addition (±10% over sequences), maintaining 90.8% accuracy. These tests, conducted on a 10% subset of the test set, demonstrate stability, with F1 scores above 0.89 across conditions. Future field trials will further validate against uncontrolled factors.

The empirical findings of this study affirm the significant advantage of using a multimodal deep learning architecture for real-time defect recognition in industrial components. As demonstrated in Section 4, the proposed system outperformed all unimodal and late-fusion baselines, achieving a test accuracy of 94.7% and an F1 score of 0.93. These metrics indicate both high sensitivity and precision in classifying surface and internal defects. Notably, performance gains were especially pronounced in the “Mixed Defect” class, where individual modalities tend to struggle due to overlapping signal features.

The success of the multimodal strategy lies in its ability to integrate complementary information from visual, acoustic, and vibrational domains. Visual input contributes rich spatial context for surface anomalies; acoustic signals offer temporal insights into transient emissions associated with material deformation or cracking; vibration signals encode dynamic mechanical properties like resonance shifts and imbalance. The fusion of these representations through a cross-attention transformer allows the model to dynamically weigh the importance of each modality, adapting to defect type and context.

Ablation results, as shown in Table 4 of Section 4, reinforce the hypothesis that no single modality suffices across all defect classes. For instance, vibration-only models achieved reasonably good performance in detecting internal faults due to sensitivity to substructural shifts but underperformed on surface defects. Conversely, image-only models did well with visual scratches and weld faults but could not detect subtle fatigue propagation. This performance disparity validates the premise of heterogeneous signal synergy: certain defects manifest more clearly in specific domains, and deep networks can be trained to exploit these distinctions.

The robustness of the model under degraded input conditions was also evident during experiments with sensor dropout. When one modality was randomly occluded, performance dropped only marginally (average of 3.8%), suggesting redundancy and fault tolerance. This is particularly relevant in industrial settings where sensor malfunction or environmental noise is inevitable.

The choice of intermediate fusion via cross-attention plays a pivotal role in the observed performance gains. Unlike early fusion, which often suffers from representational incompatibility, or late fusion, which fails to capture cross-modal dependencies, intermediate fusion allows for modality-specific abstraction followed by contextual alignment. This design enables the network to.

1. Extract specialized features per domain

2. Evaluate interactions such as visual anomalies co-occurring with specific acoustic bursts

3. Preserve interpretability by localizing contributions from each modality

Cross-attention layers also lend themselves to interpretable diagnostics, as shown in Figure 5, revealing which modalities influenced the decision and when, adding transparency crucial for high-stakes applications like aerospace and heavy manufacturing.

Another noteworthy aspect is the real-time deployment capability of the system. With a total inference latency of 67 ms, the proposed pipeline comfortably meets real-time constraints common in assembly lines and continuous monitoring setups. This low latency is a result of model optimizations including.

1. Channel pruning in convolutional layers

2. Use of ONNX format for inference

3. Mixed-precision (FP16) execution via TensorRT

Moreover, the modular architecture supports edge-based deployment on devices like NVIDIA Jetson Xavier, reducing reliance on cloud infrastructures and enhancing system resilience in remote or latency-sensitive environments.

The integration of Grad-CAM and SHAP for interpretability ensures that the model’s decision-making process is transparent to quality control engineers. Grad-CAM visualizations localize defects in images with high accuracy, while SHAP importance plots highlight which vibration signal components contribute most to decisions. This dual-level explainability satisfies not only academic curiosity but also practical industry needs where traceability and validation are critical for regulatory compliance.

In addition to visual attribution, temporal interpretability through attention heatmaps (Figure 5) further enhances the diagnostic value of the system. Operators can observe which temporal segments of the acoustic or vibration signal triggered defect classification, allowing targeted inspection and process feedback.

While the model demonstrates strong performance and deployability, several practical challenges arise.

One of the technical complexities lies in synchronizing sensors with disparate sampling frequencies. Visual data (typically 30 FPS) differs markedly from acoustic (16 kHz) and vibration (5 kHz) streams. Detailed synchronization methods are described in Section 3.2.4, which effectively reduced drift-induced misalignments. This strategy effectively reduced drift-induced misalignments.

Collecting a large-scale, well-annotated multimodal dataset remains a bottleneck. Defect occurrences in high-quality manufacturing environments are rare, making supervised learning data-hungry. Collecting a large-scale, well-annotated multimodal dataset remains a bottleneck. Defect occurrences in high-quality manufacturing environments are rare, making supervised learning data-hungry. The original dataset exhibited imbalance, with mixed defects comprising only 15% of samples compared to 35% for normal components. To alleviate this, we used a hybrid data augmentation approach: Geometric transformations for images; Time warping and noise injection for signals; Synthetic minority oversampling (SMOTE) for rare classes, which generated synthetic samples by interpolating between minority instances and their k-nearest neighbors (k = 5).

1. Geometric transformations for images

2. Time warping and noise injection for signals

3. Synthetic minority oversampling (SMOTE) for rare classes

Despite this, future work may benefit from semi-supervised learning or self-supervised contrastive methods to reduce annotation overhead.

Sensor variability across installations (e.g. different accelerometer gains or camera resolutions) may affect generalization. In our study, we mitigated this through.

1. Per-modality standardization

2. Adaptive instance normalization layers

3. Modality-wise batch normalization

However, transfer learning for unseen environments remains an open challenge for broader deployment.

Compared to the best-performing state-of-the-art systems in similar domains, our approach yields.

1. +10.2% accuracy gain over CNN-only vision-based defect detectors (Ma et al., 2018)

2. +9.1% gain over acoustic-only classifiers using BiLSTM (Singh and Ahmad, 2025)

3. +6.4% gain over transformer-based fusion without attention mechanisms (Bhuiyan and Uddin, 2023)

Moreover, the interpretability and deployability features in our system set it apart from other works focused solely on accuracy, affirming its suitability for industrial integration.

Despite its strengths, the system has limitations.

1. While sliding windows address short-term limitations, very long-term fault trends may require integration with time-series forecasting models like LSTMs in future work.

2. Model generalization outside the training environment requires further study.

3. Sensor failure detection and recovery were not fully explored, which is critical for robust long-term deployment.

This study presented a novel multimodal deep learning framework for real-time defect recognition in industrial components, utilizing a synergistic integration of visual, acoustic, and vibration signals. By addressing the critical limitations of unimodal and late-fusion systems, the proposed model offers an accurate, interpretable, and deployable solution for intelligent manufacturing environments.

The core of our framework lies in its intermediate-level feature fusion architecture, which enables context-aware interactions between spatial and temporal signals via a cross-attention transformer mechanism. This design allows the model to adaptively prioritize modality-specific cues based on defect type and signal quality. The integration of modality-aligned neural encoders (CNN for vision, STFT-CNN for acoustics, and Transformer for vibration) contributes to a robust feature representation that generalizes across surface-level and internal faults.

Experimental results on a carefully constructed dataset of 6,000 synchronized samples demonstrated the clear superiority of the proposed system over conventional baselines. The model achieved a classification accuracy of 94.7%, an F1 score of 0.93, and operated within a 67 ms latency, fulfilling real-time inference requirements for edge devices. Furthermore, ablation studies confirmed that each sensor modality contributes unique and complementary information to defect classification, while statistical significance tests validated the reliability of observed improvements.

Importantly, the framework incorporates explainability modules such as Grad-CAM for image regions, SHAP for signal attribution, and attention heatmaps for modality interaction. These interpretability features are essential for practical industrial deployment, aiding engineers in validating model decisions and enabling traceability in critical systems.

From a systems engineering perspective, the model was optimized for real-world feasibility through deployment on NVIDIA Jetson Xavier NX, model compression via ONNX + TensorRT, and timestamp-based signal synchronization strategies. This ensures the method is not only theoretically sound but also scalable and field-ready.

In conclusion, this work advances the frontier of industrial artificial intelligence by bridging the gap between high-performance defect detection and real-time, explainable, and multimodal deployment. Future work can build upon this foundation by exploring semi-supervised training, domain adaptation for unseen factories, and adaptive sensor health monitoring to further enhance system resilience and generalizability.