Pavement Damage Detection System Using Big Data Analysis of Multiple Sensor

The researches to detect the road damage has been developed in several aspects because road damage can affect driving safety seriously. In the past, cars embedded with several sensors were used to investigate road maintenance. However, the inspection vehicles are expensive, and it is not able to monitor the road in real time (Gueta and Sato 2017). Researchers have found that the sensors on these mobile devices can solve many problems that traditional sensors due to the development of mobile phones. Although the signal is not accurate enough than other sensors, lots of data and signal processing technologies can cover it. Also, mobile phones are equipped with various sensors so that we can collect various types of data as well. Mohan et al. (Mohan, Padmanabhan et al. 2008) proposed a detection system to use a mobile phone to monitor road cracks and road traffic conditions. In this research, accelerometers, microphone sensors, and GPS sensors were used to collect data and amplitude of acceleration was used as a threshold to detect road cracks. Aksamit P et al. (2011) designed a mobile phone application to detect road damage by setting the threshold of sensor signal power. The road information was collected using the built-in accelerometer and GPS module of the mobile phone to detect the road surface damage. This method achieved to use mobile phones to detect damages in real-time.

Previous researchers (Eriksson, Girod et al. 2008, Carrera, Guerin et al. 2013, Theodora S. Brisimi, Setareh Ariafar et al. 2016, Alqudah and Sababha 2017, Singh, Bansal et al. 2017) have already raised their method and more or less realised it. The most researches (Carrera, Guerin et al. 2013, Theodora S. Brisimi, Setareh Ariafar et al. 2016, Singh, Bansal et al. 2017) have been used accelerometer and GPS systems in mobile phones to detect and localise pavement damage. Alqudah and Sababha (2017). Eriksson, Girod et al. (2008) designed and developed a device consisting of an accelerometer and GPS unit, which is called the porthole portal. Sensors built-in mobile phones are not as universal in the time of their research (prior to 2007) so that sensors were installed together in their device.

The purpose of this paper is to use acceleration sensor data and audio data collected by mobile phones and other devices to detect the cracks of the pavement. In this paper, we used a the wavelet transform (WT) and short-time fast Fourier transform (STFT) for analysing the data to obtain the precise frequency of the signal of the damage. Those two methods were compared to find a better solution for the detection of cracks. At the same time, audio files were used to find cracks with the WT and STFT method. The crack image captured from the video is as a reference to verify the accuracy of the processing results.

The driving tests performed on Sutongli road in Suzhou. A car equipped with several sensors was used to detect the crack on the pavement. In order to identify the damages on the pavement quantitatively, some devices are used for translating the damages characteristics into data. In this test, the vibration of the vehicle was the main data collected from mobile phone and sound recorder, because the vibration will affect the driving condition directly when the vehicle crosses damage. This vibration is the main factor to make people uncomfortable. The signal of the vibration is also more straightforward than the other data collected from other sensors. The main idea of this method is that the accelerometer and mobile phones are installed on the car to measure the vibration intensity when the vehicle crosses damage. The sound sensor and camera were used in this test at the same time, and the location of the crack was obtained from the GPS information.

The test route is Sutongli road, which is located in the southeast of Xi’an Jiaotong Liverpool University. This is a north-south highway, starting from the intersection of Sutongli road and Wusong road. The final location is the intersection near the aquaculture zone of Tongli agricultural science and technology demonstration zone, with a total length of 3.8 kilometres. The roadmap is shown in Fig. 1.

There are several main test apparatuses: vehicle, accelerometer, mobile phones, camera, log book, sound sensor and laptop. Due to small passenger cars are more sensitive to pavement condition (Xu, Yang et al. 2018), reliable experimental data can be obtained by using a small SUV, Jeep Compass 2007.

The equipment mainly includes three types of equipment for the data collection, which are equipment collecting acceleration data, sound data, and road images. For the collection of acceleration, an accelerometer, MPS-SEN04, was used to detect the acceleration of a vehicle more accurately. Three mobile phones installed with “physics toolbox”, including Huawei mate10, Huawei Honor Play, One Plus 5T, were used to collect acceleration data from mobile phones. The sampling rate of the mobile phone sensor is 50Hz, and the acceleration sensor is 100Hz. At the same time, Newman recorder collects the sound data from the pavement, and Yi 4K sport camera films the pavement. Table 1 shows the details of equipment which can show the sample range and accuracy of sensors in used mobile phones and professional accelerometers provided by the manufacturer. Three types of mobile devices were used to minimize the data collection errors, and an accelerometer was used as a comparison reference.

In these tests, the vibration of the vehicle and images of pavement were collected from the sensors. In order to simulate the similar condition of normal driving, mobile phones were placed on the seat as shown in Fig. 2(a). When the vehicle passes over the crack, the vibration of the vehicle can be measured by mobile phones due to the shaking of the vehicle. Simultaneously, GPS data can be recorded from mobile phones so that the locations at each time can be collected. An accelerometer was installed under the car chassis near the wheel to measure the acceleration accurately as shown in Fig 2(b). The sound recorder was installed inside the car near the wheel position. The specific sounds representing cracks can be collected when a car passes over cracks. The camera was installed at the back of the car as shown in Fig 2(c), and it can record the video of the pavement condition.

The driver made three artificial car shakings to make triggers for all sensors to react at the same time before the test so that it is beneficial to further data analysis, although all data were recorded with the real time. The speed of the vehicle was maintained at more than 30 km/h in order to make it similar to the normal driving condition.

The raw data includes the amplitudes of acceleration and sound with the time, but it is not able to assess cracks from amplitude variations. A reason is that the signals of mobile phones are not bright enough to detect the small crack. Even if accurate sensors can detect the signals of cracks, it is not able to define that the signal represents the crack of pavement. Because signals can be peaking up with driving conditions like a break, acceleration of the vehicle, changing the traffic lane, etc. Therefore, the amplitude of the signal is not able to make a clear assessment of pavement condition. In this paper, in order to analyse the data in detail, the data represented by the time-amplitude domain is converted into a time-frequency-amplitude domain. Based on this transformation, the specific wavebands can be found that the value of frequency representing the damage of pavement is located within specified range. Finally, it can define the frequency of the crack from the analysis of lots of acceleration and sound data

For the signal analysis, the frequency is obtained by transformation of the time-amplitude domain signal. Fourier transform is a widely used method for signal processing. In the Fourier transform, the base functions are the complex oscillations exp(iωt), where t is the time, and ω is the single frequency parameter that determines the basis function in the family(Seraj, Meratnia et al. 2017). The Fourier transform can divide any period of the signal into N sinusoidal signals with different frequencies and amplitudes, and the amplitude and frequency characteristics of the signal can be directly observed. The function of the FFT is:

The time information can be removed from the Fourier Transform so that Gabor improves the Fourier Transform by introducing the concepts of “windows”(Cohn 1995). Short Time Fourier Transform (STFT) overcomes this drawback of Fourier Transform by dividing the signal into a short-time period, and a window means a data set divided by a fixed time interval. The signal is multiplied by a non-zero window function in STFT, and the window function moves along the axis of time. The Fourier transform can be performed in every divided window. Finally, the obtained spectrums are expanded into a two-dimensional image so that frequency variation can be represented with the time. The STFT is defined as Equation (2) (Han, Renaudin et al. 2015), where, w[n–T] is the window function and x[n] is the signal to be transformed.

Although the STFT solved the time-frequency resolution, the window function dominantly affects the results of time-frequency variations. If the window size is not appropriately evaluated, unexpected results can be obtained from STFT. In order to increase the accuracy of the analysis, it is necessary to evaluate the window size appropriately and automatically. The wavelet transform (WT) makes the automatic adjustment of window possible so that it can be expected to receive accurate analysis results. The difference with STFT is that the signal is multiplied by a wavelet function in WT analysis. The signal is transformed separately at different segments of the time-domain signal. The wavelet transform is defined by the following equation:

Where, u and s are the shift and scale parameters, respectively and u is the wavelet’s position along the signal so that WT(s,u) can get the similarity between the signals at time t. The change of u will affect the position of the timeline in the centre of the corresponding time-frequency window. With the increase of s, the central frequency of the time-frequency window decreases and the central time increases. By definition, the wavelet base function ψ_so is defined by Equation (4):

Morlet wavelet has accurate separation performance in the time domain and frequency domain (Su, Wang et al. 2010) and hence Morlet wavelet is used as the generating function ψ(t) for time-frequency analysis in this paper.

From the driving test, acceleration can be obtained from mobile phones. The amplitude of acceleration is influenced by situations such as the location of the cell phone and the contact status, etc. Because the mobile phones collect acceleration within the vehicle, an amplitude of acceleration is not able to be a threshold to assess the damage of pavement when the vehicle passes over a small crack. In this paper, STFT and WT were used to convert the time-acceleration domain to time-frequency domain in order to analyse the data of the acceleration sensor signal and the sound sensor signal. Although the amplitude of signal representing the small crack is not larger enough, it can be detected by the analysis of frequency. The results of STFT and WT were compared to define the better solution.

In the STFT analysis, window size is one governing factor of the results and hence sensitivity analysis was performed to determine the appropriate window value. The window size should be narrow enough to ensure that the signal representing the crack is included within a window (Djurović and Rubežić 2007). But if the window size is too narrow, it is not able to obtain the results with high-frequency resolution. In this paper, the window size was determined before starting the detailed analysis.

The acceleration is also peaking up when the vehicle passes over the special pavement conditions. Fig. 3(a) shows the joint between at the bridge. The signals of accelerometer and mobile phones peak up clearly so that these data can be selected to find the most appropriate window size. Fig. 3(b) and (c) shows the raw data of acceleration collected from accelerometer and mobile phone. The acceleration sensor of the mobile phone has much lower sensitivity than the accelerometer. The frequencies of data collection are 100 Hz in accelerometer and 50 HZ in mobile phones, respectively. we assumed that the window size of STFT is 0.25 second, one second, two seconds, and four seconds to find the most appropriate window size. The acceleration sensor usually provides the signal in three-axis. Because this paper focused on the frequency pattern recognition, the data in vertical direction is used as input data in the analysis. In the accelerometer, it is found that two peaks in a second which is the signal from crushing between joint and front and rear wheels. However, the mobile phone placed on the car seat is not able to collect two peaks.

In order to define the window effect in STFT, STFT analysis was performed with different window sizes: 0.25 second, one second, two seconds, and four seconds. Fig. 4 shows the STFT results of accelerometer data. Fig. 4(a) shows the STFT results in case of 0.25 second and it can be seen that the amplitude of the frequency between 10 Hz and 50 Hz increases twice in 0.25 second interval. Because the front and rear wheels collected both the joint and the two collided vibrations. But it is difficult to find the frequency representing crack in narrow window size. Since both signals increased at 0.25 second window size are overlapped due to larger window size in a second window size, it is not able to analyse the individual signals. But it can be seen that the frequency band increases regularly compared to the result of 0.25 seconds window size (See Fig. 4(b)). If the window size increases up to two seconds, the discrimination of frequency band representing the joint can be increased in between 10 Hz and 20 Hz (See Fig. 4(c)). When the window size increases up to four seconds, no visible features can be found from Fig. 4(e). If the window size increases too much, it is difficult to find a signal indicating cracks, as it is overlapped with other signals as well as signals indicating normal road. In order to find the frequency band, the window size is decided as two seconds.

Fig. 5 shows the STFT analysis to identify the window size in acceleration data of mobile phone. In 0.25 seconds window size, it can be seen that the amplitude of the frequency between 10 Hz and 25 Hz increases clearly but it is difficult to estimate the frequency representing the joint because the frequency band is increased in overall range. The frequency ranges in between 10 Hz and 25 Hz is increased only once contrary to the result of the accelerometer (see Fig. 5(a)). In a second window size, the frequency band representing the joint is gradually revealed (see Fig. 5(b)). As the window size increases up to two seconds, the frequency band representing the joint becomes more clear (see Fig. 5(c)). However, if the window size increases to four seconds, it becomes difficult to identify cracks due to the influence of other signals other than the signal representing the joint (see Fig. 5(d)).

As shown in Fig.4 and Fig.5, if the window is decided in between 0.25 seconds and a second, the frequency band increases in overall range so that it is not able to determine the frequency representing the joint. The signal becomes less noticeable when the window is four seconds because of the overlapping of signals. Conversely, the smaller the window, the more easily the signal is disturbed by the noise signal. From Fig.4 and Fig.5, it can be found that two seconds window size can be used to find the crack location in STFT analysis.

In order to verify the practicability of the method, three groups of signals were randomly selected and tested. Fig. 6-8 is the experimental result of using the acceleration sensor and mobile phone data, and Fig. 6-8 (b,d,f,h,j) is the result of using the acceleration sensor and the Fig. 6-8 (c,e,g,i,k) is using mobile phone data. Moreover, here window size is 2s using in STFT, and because the sampling frequency of mobile phone sensor and the external acceleration sensor is different, the frequency of the signal in the analysis is different. The acceleration sensor is 100HZ/2 = 50HZ, and the mobile phone sensor is 50HZ/2 = 25HZ.

In Fig. 6(a), two signals corresponding to cracks are visible at 6.5 s. After the signal is transformed by STFT, the signal is shown in the frequency domain in Fig. 6 (b).

For obvious cracks in the road, sensors and phones can distinguish between frequency and time. From the Fig.6 (d,h), The external acceleration sensor collected three obvious frequency signals in the large crack section, roughly 12HZ, 20HZ and 37HZ. It can be found from figure 6 (e,i) that due to the low sensitivity of the mobile phone sensor, only one obvious frequency was collected, which was 10HZ.

The small cracks do not have a signal with large amplitude, signal vibration is more small. The processing of STFT is no longer enough to obtain the obvious frequency. In this case, it can be seen from the comparison in Fig. 7 (f,g,j,k) that WT can still process the signal frequency, indicating that the WT method is more universal.

Due to the high sensitivity of the sensors mounted on the wheel frame, it is possible to directly collect the feedback from the road. At the same time, however, there are more noise signals. From Fig. 6-8, it is found that the signal at the crack position is very obvious and even there is some noise interference, the crack signal can be extracted well, we can get the obvious frequency of the signal. At the same time, we can remove the non-crack regions and the extract the crack signal according to the frequency distribution of the signal.

From Fig. 6 to Fig. 8, it can be found that both the STFT and the wavelet transform method can accurately find the crack position. Due to the sensitivity and installation position of the accelerometer, the signal strength is enough. The signal intensity corresponding to the large crack shown in Fig. 6 is high. From the frequency domain diagram in Fig 6(d) and Fig 6(h), the vibration spectrum has three distinct peaks, which indicates the frequency characteristics of the vibration signals. And the frequency corresponding to the damaged road surface can be found around 12, 20 and 37 HZ. The wavelet transform can obtain a more accurate corresponding frequency of the damaged road surface. However, for the small crack shown in Fig.7, the STFT method cannot obtain visible information, and the result obtained by the wavelet transform is still apparent. Moreover, both methods can distinguish the target signal and the noise signal. In the specific operation process, it is easy to find out the potential by double filtering the signal frequency and signal power. From Fig. 6-8, comparing different spectrograms, it is found that the wavelet transform performs better for frequency extraction.

When the car passes through different road surfaces and damaged positions, the contact between the tire and the ground changes, which further causes the vibration of the car, thereby emitting sounds of different frequencies. However, these sounds are hard to be entirely distinguished by human. However, depending on the STFT, the frequencies corresponding to these sounds can be distinguished. Through sample learning, the frequency corresponding to the damaged road surface can be determined.

Fig. 9 shows that a small crack is detected using an audio file. Moreover, it can be found that it has increased amplitude and increased frequency as it passes through the damaged position. It can be seen from the original data that the collected audio files have more noise, Even if there is a significant increase in the sound in Fig.9 (a), this information cannot be used to judge the damage of pavement. After STFT processing, by filtering the amplitude and frequency, the frequency corresponding to the damaged of the road surface can be judged to be about 200 to 600 Hz through statistical regularity of multiple samples.

This paper attempts to use STFT and wavelet transform to detect crack of pavement from acceleration signals and audio signals. The three-axis accelerometer built into the mobile phone is analyzed. And two kinds of time-frequency analysis methods is proposed: short-time fast Fourier transform and wavelet transform. Acceleration data collected by an acceleration sensor mounted on a car wheel frame and three mobile phones are used, and a voice recorder is used to collect the audio signal of the road surface feedback and the camera collect the image of the pavement. For visible vibration signals, STFT and WT perform well, and the frequency of the corresponding signal of crack is extracted clearly. However, the signal from small cracks is not so clear. The amplitude is small, and the frequency is more complicated. Therefore, the STFT and WT are compared in a different situation. The results of STFT show the limitations of the algorithm. The fixed window value cannot adapt to the change of data. For small crack, STFT cannot completely extract the signal of damage. The analysis based on WT method performs better, due to a good fit to the window size of the signal changes, the WT method can get the clear frequency of the signal Whether the signal is strong or weak. For future research, we will use machine learning methods to match the data after extracting some eigenvalues from the wavelet transform.