Automatic Pavement Crack Detection Based on Image Recognition

The road pavement is damaged by various physical influences, which results in a lot of congestion in downtown traffic. If the crack starts to be generated, it is extended to be larger damage by the continuous physical bumping from the car wheels. Therefore, it is very important to measure and to repair cracks in the pavement at an early stage. In the past, the cracks of pavement were inspected manually. However, the manual inspection is not only time-labour consuming, inefficient works, and making lots mistakes but also it can lead to security incidents. Most studies aim to find out and to repair the cracks quickly and effectively from the pavement. Cubero-fernandez, etc. (2017) evaluated the type of cracks in the pavement. Normally, the crack is mainly classified into three categories: transverse cracks, longitudinal cracks, and alligator cracks (Johnson 2000).

The purpose of this paper is to evaluate the damage of the road surface by extracting cracks from the image and classifying cracks. The improved pre-treatment method before segmentation is proposed to enhance the accuracy of extracting images of the cracks and an SVM (Support Vector Machine) classifier based on LBP (Local Binary Patterns) feature is used to the category the crack type.

Many inspection methods have been used in the research of pavement crack detection, especially using image processing (Wang, Gopalakrishnan et al. 2016). Various analytical techniques using high-speed cameras and high-resolution cameras have been widely used for maintenance in order to detect cracks on the packaging surface. In the early 1970s, the French National Road Management Agency (LCPC) conducted a study on the surface damage detection system for pavement (Autret et al., 1974) and developed the first pavement detection vehicle (GERPHO). Fukuhara (1987) developed an automatic measurement system for pavement damage (APDS) that applies new hardware equipment and proposes the maximum variance threshold segmentation algorithm. This study was also carried out to classify and identify road pavement cracks. In the early 1990s, Surface Crack Assessment System (PCES), developed by Earth Technologies in the United States, introduced digital image processing technology (Wang, 2000). Then, Koutsopoulos et al. (1993) develop an automated packaging analyzer (ARAN).

The studies to develop the algorithms for road pavement crack recognition have been recently started (Cubero-Fernandez et al., 2017). Kirschke and Velinsky (1992) proposed a method to identify pavement cracks using an algorithm based on histogram. When a threshold value calculated by a histogram method is applied to segment a road image represented by grayscale, an image higher than a threshold value is removed. Thus, after applying the threshold, the image reveals more clearly the cracks on the image of pavement. Li and Liu (2008) proposed a method of estimating new thresholds based on Adjacent difference histogram (NDHM). The proposed algorithm is based on two main aspects: a) the value of the standard deviation of the image is closely related to the pavement cracks; b) the threshold can obviously segment the crack image from the original image. Based on these two condition, they proposed the objective function. Kapela (Kapela, Śniatała et al. 2015) proposed the cracks recognition system based on the image processing technique called Histograms of Oriented Gradients.

In this paper, the automatic crack detection system in pavement was proposed through the video and GPS (Global Position System) data obtained in the driving test. The obtained video data is converted to each frame for the image processing. Thereafter, the grayscale processing, filtering and histogram equalisation are used to enhance the image to distinguish the crack image from the image of common pavement. It is also possible to remove the road line from the original image by filtering the pixels having high grayscale which is closed to the white colour. The crack image can be segmented from the pre-treated image by the threshold which is calculated by the OTSU or Niblack methods. However, even theses image processing has been successfully finished, the analyse images are still affected by the several conditions like shadows and image noise. Therefore, in order to acquire more clear processed images of the cracks, the potential crack regions were clarified by using a standard deviation method of grayscale values in this paper. This method automatically finds potential cracks regions in the pre-treated images. If a clear crack is detected though the proposed algorithm, the crack will be extracted for the classification stage. Then, the crack location is automatically pinpointed on the map based on GPS information. Finally, algorithm classifies each type of crack through the SVM method, which is a machine learning technique.

This paper is organised as follows; the related work is briefly introduced in section 2. In Section 3, the method to enhance the image is recommended. In Section 4, a new method to find the possible crack region and threshold segmentation have been proposed, and at this section, the results have been presented. In Section 5, the method to classify the crack based on SVM classifier is introduced. Section 6 shows the proving experiment.

When the road test is conducted to collect the image of pavement, lots of noises can be collected together with the image of cracks. If such noises are not removed, subsequent image processing and analysis cannot be performed properly so that the pre-treatment of image is very important. In order to improve the quality of the image, the proper filtering to reduce noise needs to be applied in the pre-treatment process. In this paper, we propose the grayscale converting method, filtering method, image evaluation through histogram, and road line filtering pre-treatment techniques for the pre-treatment of image.

The image of the pavement crack is represented by RGB (Red, Green, and Blue) scale including brightness and colour information. RGB model is composed of three components which are red, green and blue. Each component of the image is represented by 8 bits, and each pixel takes a value from 256 values depending on the degree of colour. There are many factors to be considered, if the image represented by RGB scale is directly analysed. Therefore, in this paper, the image represented by RGB scale is converted into the grayscale image containing only brightness information. In the converted grayscale images, the range of grayscale value of a pixel is changed from 0 to 255, which can be represented by one byte. The larger grayscale value of the pixel represents the whiter and brighter. A pixel having a gray scale value of ‘0’ indicates pure black and a pixel having a gray scale value of ‘255’ indicates pure white, respectively. Figure 1a shows the original image of a crack represented by RGB scales. Each pixel of the original image is composed of each values of red, green, and blue as shown in Fig 1b.

Figure 1a shows the original image of the crack. Figure 1b shows the marked area of the original image in Figure 1a which includes the RGB values of each pixel.

In this paper, the image is represented by the grayscale which is converted from the RGB scale into the YUV space representing the brightness, Chrominance, and Chroma. Rackley and West (1999) used the two-dimensional matrix function to represent brightness information from RGB scales. This function allows the values of RGB scales to be represented in YUV space (Corry, 1998). In particular, the brightness (Y) can be calculated from the GRB value, which is shown in equation (1):

According to the Eq. (1), all pixels of original image (see Fig. 2a) can be converted to the grayscale image as shown in Fig. 2b. Therefore, various parameters can be reduced in the subsequent image processing, and it is possible to analyse the image data more easily based on the images pre-treatment.

The grayscale values of images are concentrated in a narrow interval. The histogram equalization can be used to adjust the distribution of grayscale value to enhance the local contrast, so that the crack and the background are more distinct. Histogram equalization can enhance the contrast very well, making the details of the figure clearer and achieving image enhancement. Histogram equalization is a method of enhancing the image enhancing image contrast by stretching the pixel intensity distribution range. In this paper, histogram equalization process is performed using a correction method based on cumulative distribution function. (Kim 1997, Stark 2000, Stark 2002). Equation (2) shows the algorithm for histogram equalization (Ulku and Camurcu 2014).

Where, f(x, y) is the output image of the histogram equalization, R is the range of the output image (in this paper, R is 255), X₀ is the minimum grayscale value in the input image, X_k is maximum grayscale value in the input image, n_i is the total number of the grayscale, N is the total number of pixels in the image.

Figure 3 is a schematic diagram about the process of grayscale equalization. Figure 3(c) is a gray histogram of input image (see Fig. 3(a)) which shows the distribution of the number of pixels according to the grayscale. Fig. 3(c) shows that the gray values of most pixels are concentrated between 70 and 145 in which represents the pavement pixels. Because of this overcrowded distribution, images of cracks and normal pavement are not clearly distinguished so that it can disturb the crack extraction from the original image. Through the histogram equalization, newly processed image can be obtained as shown in Fig. 3(b). Fig. 3(d) shows that the histogram distribution area is wider after the histogram equalization, which means it makes the contrast of the image stronger. As shown in Fig. 3(b), the crack is clearly distinguished from the background image, then, in the following stage, the crack will be easier to extract.

Most road pavement images include road lines, which can be seen as a noise in the image processing. Therefore, it is essential to remove road lines using appropriate filtering from the road pavement images. In this paper, Eq. (3) was adopted as the filtering method to remove the road line. As results from lots of driving tests in this paper, the gray scale value of the road line after the histogram equalization was always higher than ‘150’ and the gray scale value of most road pavement and cracks was lower than ‘150’. Therefore, the algorithm of this study includes filtering to remove pixels that grayscale value has higher than ‘150’. Fig. 4 shows the results after the filtering applying the filtering of Eq. (3).

Where, f(x, y) is the imported images after Histogram equalization and M is the total pixels in an image.

The mean filter is mainly used for noise reduction. The concept of the mean filter is to evaluate the average grayscale value of the pixels in the filtering window instead of the pixel values in the image so that the mean filtering is capable of filtering out the image noise that are randomly changed within local area of image. Mean filtering can remove the shape change of pavement in the image as well so that mean filtering can quickly eliminate most of the noise without damaging the crack image.

The surface of pavement is composed of very small asphalt particles. Since each particle does not have the same colour, there is large difference among the grayscale values of pixels representing the asphalt particles even if it is a neighbouring pixel in an image. In other words, the grayscale values of pixels can change randomly within a local area of a pavement image. However, low grayscale values that represent dark colours appear along the crack line in the image of cracks. Thus, applying the averaging the values of each pixel in local area can make the grayscale values of the pixels which has random behaviour change to more stable area. A average value of filter mask in local area including several pixels can be replaced to the original pixels, which is a method to provide smoother the image (Rakshit, Ghosh et al. 2007, Gupta 2011). The details of mean filtering are shown in Eq. (4):

Where, S(x, y) is the input image and the filter mask has m by n pixels. In Fig. 5a and b shows the results from the mean filtering by using mask size of 3 by 3, 5 by 5, respectively.

In order to find out only a specific shape from an image, it is necessary to segment the pixels with appropriate standard. In this paper, the threshold segmentation is proposed to extract the crack image from pre-treated images. OTSU and Niblack methods are introduced in this paper to calculate threshold values for the crack extraction and the results of those methods are compared. The OTSU method calculates the threshold value for the whole image but the Niblack method divides the image into several windows to calculate the thresholds for each divided window separately. The difference in between two methods is whether the estimation of the threshold value is based on the global area or the local area. In this paper, the potential crack region is estimated from the image processed by the threshold value so that the crack can be more clearly revealed. The potential crack region was estimated from the standard deviation of the grayscale values of adjacent pixels. Therefore, it is possible to extract the crack more clearly than the conventional method and the shape of the extracted crack can be defined clearly.

OTSU (1979)) proposed the global threshold of the pixel grayscale value to separate the image into two categories of grayscale. It tried to define an interclass variance to express the difference of grayscale value between the two categories, and when the interclass variance get maximum, the responding grayscale value can be a threshold. The specific implementation is as follows:

In this method, the threshold is determined by the maximum interclass variance methods (Otsu 1979). The pixels of input image are represented from 0 to 255 of grayscale values and the probability of the grayscale level ‘i’ is calculated from Eq. (5).

Where, n_i is the number of pixels at level ‘i’ and N is the total number of pixels (N = n₀ + n₁ + n₂ + ⋯ + n₂₅₅).

If the threshold is assumed certain value as T to segment the images to two categories, C₀ and C₁, C₀ contains pixels representing grayscale values from 0 to T and C₁ includes pixels representing grayscale values from T + 1 to 255. Therefore, the probability (ω₀) can be calculated by Eq. (6):

The mean grayscale (μ₀) of the target region, C₀, can be represented by Eq. (7).

The probability (ω₁) and mean grayscale (μ₁) of the target region, C₁, are calculated by Eq. (8) and (9), respectively.

The mathematical expectation between the two classes;

Then, the interclass variance σ² is as following;

Fig. 6 shows the result of evaluating the threshold value by applying the interclass variance method. Most of the pixels are located in the region which represents higher grayscale than the threshold value (C₁). However, all the pixels representing the crack are located in the region which represents lower grayscale than the threshold value (C₀). It can be seen that the image segmentation method based on the threshold can segment into two different categories in an image. In Fig. 6, the threshold value (T) is 109 when the interclass variance is reached to the maximum so that this value can be used as the threshold to segment the image. The result of the image applying the threshold is shown in figure 7.

Niblack methods is introduced in this paper to calculate threshold values for the crack extraction Niblack method divides the image into several windows to calculate the thresholds for each divided window separately. If the background image does not have the same brightness overall in an image, darker image such as a shadow can be also assessed as a crack in the Otsu segmentation method which is calculating the threshold value in a whole image. However, in the Niblack’s segmentation method, threshold values can be calculated for each local windows divided from the image. The threshold value of Niblack’s method is calculated as the average and standard deviation of the pixels in each window. Therefore, even if a window contains a shadow or dark image without large deviation of grayscale, the lower threshold value is calculated in that window and then shadow can be filtered. Eq. (12) shows the threshold value calculation.

Where m is mean value, k is adjustment factor (normally k is -0.2), and s is standard deviation (Vijayan, Reshma et al. 2017). Fig. 8 shows the results of Niblack segmentation method. The noise effect from shadow decreases obviously compared with the results of Otsu segmentation method.

Both the OTSU and Niblack algorithms distinguished cracks through thresholds, but a lot of noise still exists. Therefore, further improve the algorithm is necessary and the potential crack region is estimated in this paper.

In the Niblack method, the variance can be evaluated by the standard deviation which can be possible to define the potential crack area. Fig. 9(a) shows the average of the images and Fig. 9(b) shows the standard deviation calculated from the divided windows. The image representing standard deviation is calculated by applying the equation from 5 to 11. In this procedure, the potential crack region can be evaluated to extract the area having a lower grayscale than the threshold. The following Equations can extract the potential crack region and Fig. 9(c) shows the result of the potential crack region.

Where, P(i, j) is the image after the mean filtering, M(i, j) is the potential crack image, and the F(i, j) is the generated image.

A processed image with the potential crack region is shown in Figure 10a. Although the potential crack region includes some noise in addition to the crack region, it can be seen that it includes the crack region precisely. Fig. 10b shows the extracted image through the potential crack region and most noise disappears and only cracks and a few noise are shown in Fig. 10b.

Various types of cracks exist in a complex road condition. Therefore, it needs to be checked the proposed algorithm in this paper is applicable to detect various cracks. Fig. 11a shows the original image when cracks occur in multiple directions, and the results for the Otsu and Niblack methods are shown in Fig. 11b and c, respectively. Fig. 11d shows the result of applying the potential crack region proposed in this paper. Figure 11e shows that the road line is dominant a large area of an image. The results for Otsu and Niblack segmentation method are shown in Fig. 11f and g, respectively. Fig. 11h shows the result of applying the potential cracking region proposed in this paper. Regardless of the case, it can be seen that the shape of the crack can be clearly distinguished when the potential crack region is applied. It is important to evaluate the potential crack region because clear crack extraction is required in automatic crack detection.

Through comparison of these typical experiments, it is found that the Niblack with pre-treatment algorithm proposed in this paper can suppress the image noise and reduce its interference to the crack region extraction, making the extraction more accurate.

Then, the minimum exterior rectangle is used for extracting the crack image. This method tries to mark the boundary point to extract the crack, and the extracted image is used for classification. And at the same time, the image is marked with GPS information. Figure 12 will show the process of extraction.

After the segmentation of the crack image, a concrete image of a crack is got. And the extent of pavement can be obtained by classifying the crack. In this chapter, this paper uses SVM (Support Vector Machine) classifier based on LBP (Local Binary Pattern) image features as the crack classification algorithm. At the same time.

LBP operators have been widely used in texture classification, image and video retrieval and face recognition (Ojala, Pietikäinen et al. 2000, Ojala, Pietikainen et al. 2002). The local binary mode is a texture description operator in the gray range. The original LBP operator defines a 3×3 window (texture unit), and then the gray value of the center pixel is regarded as a threshold value, and then the center pixel is compared with the adjacent 8 pixels. If the surrounding pixel value is greater than the value of the center point, the pixel position is marked as 1, otherwise 0. Thus, an 8-bit binary number can be generated at 8 points in the 3×3 neighborhood, and then an integer is obtained by assigning different weight values according to the position, and the integer is the LBP value of the window. The equation is as follows:

Where g_c is the pixel value of the center pixel of the window, is the gray value of the eight connectivity pixels, and s(x) is a banalization function, which is defined as follows:

After we got the LBP coding image of an original image, the texture feature will be described by the histogram. Here, we use an image to introduce the process of LBP operator. The binary number of 3×3 patterns is 8, the total combination of LBP operators is 256(2⁸). After the feature description, the texture feature points, the edge and spot can be included in a histogram which is formed by LBP cedes.

In this paper, in order to highlight the details, the image was divided into 9 sub-blocks, and the histogram of each sunblock was connected to describe the crack image. Figure 14 shows the process in order to eliminate the different size of sub-block, the histogram divided by an adjustment factor, which is equal to 10000 × width of subblock × hight of the subblock.

Support Vector Machine (SVM) is a machine learning algorithm based on statistical learning theory, which is widely used due to its excellent classification performance. The theory of SVM is looking for a hyperplane to meet the requirement of categorization (Luo, Wu et al. 2013). In the method, the support vector tries to find the maximum interval between classes and ensure the accuracy.

The feature of crack is described by LBP feature so that the crack recognition is a nonlinear classification model. In this classifier, the input data is in a high dimensional feature space, and the data cannot be separated. The kernel function which is a nonlinear mapping function is used for efficient computation of inner products to map a single vector to a vector of higher dimensionality. Common kernel functions include linear, polynomial, RBF and sigmoid. Here RBF kernel function is applied in this paper.

To verify the effectiveness of SVM classification algorithm based on LBP features in pavement crack classification. In this experiment, the training sample set includes sixteen images of three different crack type, which were randomly collected from road. And the testing sample set includes thirty-four images of different crack type. The experiments were carried out in Matlab 2017b environment. The classification results are in table 1.

Table 1 shows that the method using sub-block in LBP operator in this paper can get higher accuracy of recognition compared with the method only using LBP. And we can notice that the recognition of average accuracy, transverse crack, longitudinal crack alligator cracks and none crack using sub-block in LBP operator perform better than the method only using LBP.

To verify the universality of the algorithm, the verification experiment is conducted in Su tong li road, Suzhou, China. The result of experiment is in the table 2.

The GPS information of the crack was found in this method; here the location will be marked in a three-dimensional space. The schematic of cracks location is in Fig. 15. In this figure, x-axis is longitude; y-axis is latitude and z-axis is altitude. And the red point is transverse cracks, blue point is longitudinal cracks, and the green point is alligator cracks.

In this paper, we have developed an automatic method for crack extraction and crack classification. This method combined with image processing, Niblack threshold method with pre-treatment, and SVM classifier based on LBP feature.

First, the crack of the image is enhanced after the graying processing, histogram equalization, filtering the road line and mean filtering. After that, two classical segmentation methods, Niblack and OTSU method are applied to extracted the crack. Because the above method works well in removing the image noise and shadows, a pre-treatment is proposed to find the potential crack region. Then, the crack part was extracted to do the classification. In the classification stage, the SVM classifier and LBP operator is used to categorize the crack, and here the sub-block of the image is used to highlight the details of the image. Though comparison, we find that the LBP using sub-block based SVM classifier has a better reorganization. Through the verification experiment, the results show that the proposed method has a good performance in crack detection and identification.