Image-Based Multiview Change Detection in Concrete Structures

Visual inspection is a common technique to examine and assess the current state of buildings. This technique is cumbersome and time-consuming as it normally involved inspectors travelling to sites to assess the structures’ conditions based on their visual appearance. Hence, this procedure cannot be conducted frequently due to high labour cost, prone to human error, and site inaccessibility. Failure to detect problems can lead to disastrous effects.

Many buildings in Thailand contain damages, such as cracks and surface paint on the structures that require frequent monitoring and inspection to assess if the structures need repair. The structures require closed monitoring to determine if the change in damages has a significant impact on the structure. This research project aims at improving techniques in change detection to determine if the structures and damages have changed in appearance over time. This project explored the use of a camera for data collection, which is also used to acquire images of problematic areas for monitoring. The acquired images are used in the proposed change detection system, which exploits the technique called Structure from Motion to help to remove geometrical errors, a common problem in change detection systems.

Figure 1 shows example pictures of cracks many building in Thailand obtained from a DSLR camera. These building has been deteriorating due to human-work and long lifetime. Although, these structures will be repaired but still has areas where damages are present, therefore, frequent inspection and monitoring is required. In this paper, we propose techniques to improve a change detection system. We obtained reference images from a DSLR camera in a problematic area and collected new images in the same view at a different time to build a 3D model from new images via the Structure from Motion technique. Then, the reference images will be registered onto the 3D model, in which the camera positions of reference images were used to create synthetic images. The synthetic images were used to compare with the real images of reference images to create change masks. This process is called geometrical registration, which is essential in removing unwanted noise due to viewpoint difference in a change detection process. The contribution of this paper is, firstly, we show that the 3D model from SfM can be used to monitor problem areas, and secondly, we propose a new technique in geometrical registration to improve change detection systems.

The rest of the paper is organized as follows, Section 2 is Literature Review, which summarizes previous work in visual inspection and 3D modelling, and a change detection system. The methodology is explained in Section 3, and Section 4 presents experiments, results and discussion. The paper ends with a conclusion in Section 5.

For inspection, photographs can be used as a mean to record for defects from inspection sessions. Photographs provide valuable information, such as the texture, colour and 3D cues about the surface of the assessed structures. For example, water ingress can cause changes to the colour of the tunnel surfaces. Other methods, such as the LiDAR system, cannot provide such texture rich information. Therefore, photographs are extremely useful tools in detecting changes in structural components. For crack detection, Miyamoto et al., (2007) compute the difference in intensity between each pixel and the average intensity of each row in an image matrix. A pixel that differs considerably from the average is deemed to belong to a crack on a concrete structure. Fujita et al., (2006) apply a line filter using the Hessian matrix and a threshold is applied to extract the crack regions on a concrete structure. Zhu and Brilakis, (2010) propose an algorithm for detecting concrete columns based on texture using artificial neural networks. Liu et al., (2002) applies a Support Vector Machine classifier to classify if crack features appear in an image patch, which is pre-processed to extract potential crack features based on intensity. Abdelqader et al., (2006) apply a Principal Component Principles (PCA) algorithm, which can be used to reduce the dimensions of feature vectors based on eigenvalues, to extract cracks from concrete bridge decks. The images are first pre-processed by line filters in three directions, vertical, horizontal and oblique; then further processed by the PCA algorithm and classified based on the nearest neighbour algorithm. Song et al., (2005) create a stitched image of construction sites from a series of images taken by a robotic camera. The image is used as documentation to record the progress of the construction. It can be seen that images have been applied in the inspection work of structural systems to aid with the monitoring and damage assessment of the structure.

Change detection is a technique that can be used to detect changes in anomalies such as cracks on structures’ surface. A change detection system normally consists of pre-processing steps (i.e. geometrical and photometric adjustments) and a change detection step. The pre-processing steps remove unwanted noise, which is not a real change, from images before the change detection step is applied. Lim et al., (2005) propose a system for monitoring the changes in cracks from multi-temporal images. The system is based on a 2D projective transformation that can accurately extract the size of the cracks, which are then monitored in the images as cracks propagated. Delaunoy et al., (2008) applied an SFM system to synthesize new views by using a geometrical adjustment for the change detection in a coral reef. In this work, it is concluded that an SFM system provides an accurate method for synthesizing new views for change detection. Guo et al., (2009) proposed change detection is the main component in the image interpretation step for pipe inspection. Guo et al., (2009) presented algorithms for crack detection generally involve a preprocessing step and a crack identification step. The preprocessing step applies image processing techniques to extract potential crack features, such as edges and indicates a change in the monitored structural components. Chen and Hutchinson, (2010) propose a framework for concrete surface crack monitoring and quantification. The method is based on optical flow, which is used to track the movement of cracks. The regions, where the cracks become larger, are labelled as having changed. Chaiyasarn, (2011) proposed a system for multi-view change detection using images to detect changes of crack on a concrete beam with the use of the SfM for geometrical adjustment. Saur and Krüger, (2016) present a change detection system using images obtained from a UAV to detect changes between image pairs from video frames. It can be seen that pre-processing steps are important for change detection systems, in which this project is aimed to tackle.

In this paper, a change detection system is proposed as shown in Figure 2. The dataset used in the experiments were obtained from Chaiyasarn, (2011), which is an image dataset containing a series of crack images of a beam in a 3-point bending test. This dataset was created to simulate the change in cracks in a laboratory setting to verify a change detection system. The system starts with image acquisition (explained in section 3.1), where a set of images are collected in periodic time. Let the first set S₀ taken at time t₀ be images of concerned areas for monitoring purpose, this set is called a queried set, and let subsequent sets be S₁,...,S_n which are taken at time t₁,...,t_n, the subsequent sets are reference images. The images from S₁,...,S_n were used to create 3D models via the Image-based 3D modelling module (explained in Section 3.2). The images from S₀ were registered into 3D models via the Camera registration module. In the Image synthesis step, a 3D model is rendered using the camera parameters from a queried image I_q, which is a real image from the set S₀ to create a synthesized image I_s (explained Section in 3.3). This process is called geometrical adjustment, which is an essential pre-processing process in change detection to remove noise due to the effect from a different camera viewpoint. In the Change detection module, a change mask is created from I_s and I_q to see if there are changes between images (explained in Section 3.4).

Firstly, images are collected to create a reference set S₀ at time t₀ for monitoring purpose. Then, we collected images to create other subsequent sets S₁,...,S_n at time t₁,...,t_n > t₀. The subsequent sets were used in creating 3D models using a DSLR camera. As shown in Figure 3(a), the Point of Interest (POI) strategy was used to obtain images to ensure full coverage of the concrete beam and to obtain a highly-detailed 3D model of the concrete beam under loading at the different time. Figure 3(a) shows the POI strategy, in which images were taken around an interested object, and Figure 3(b) show example images of a concrete beam.

The image datasets S₁,...,S_n at time t₁,...,t_n > t₀ from section 3.1 is used to create 3D models of a concrete beam using the Agisoft Photoscan software package, which is based on Structure from Motion and interested readers can refer to Snavely et al., (2006) for more detail of the theory and technology. Figure 4(a) shows a sparse point cloud model of a concrete beam and Figure 4(b) shows a dense cloud model with camera calibrations for all images of dataset t₁. To obtain a watertight model, a mesh can be created to provide a more realistic 3D model as shown in Figure 5(a). The mesh is composed of a collection of triangular mesh, in which texture from 2D images is projected onto. The textured model is shown in Figure 5(b).

The images from the datasets S₁,...,S_n were taken when cracks started to appear on the concrete beams. The images from these datasets were used to create 3D models so that synthesized images can be created to compare with the reference dataset S₀. The dataset of images S₀ are registered onto S₁,...,S_n by matching similarity between images and then the images were initialized through the triangulation process. The 3D model and the camera parameters were then re-optimized to update the cameras as exemplified in Figure 6.

A textured 3D model and camera parameters from section 3.2 are used to synthesize an image from an interested area. This is called a geometrical adjustment process, where a real image is compared with a synthesized image as if they are viewed from an identical viewpoint and location. A queried image is a real image, which is from S₀ will be called I_q with the camera parameters as a rotation matrix R_q and a translation vector t_q and an intrinsic matrix K_q. I_q is registered onto any of the subsequent set, e.g. S₁, whose 3D model is P_r, then a synthesized image I_s can be created using the following equation,

Figure 7(a) shows an example of a real queried image I_q and synthesized image I_s, the two images must be adjusted photometrically. We matched lighting between the two images using histogram matching. Also, we applied a mask to blackout the background on I_q and I_s to remove unwanted noise. Figure 7(b) shows a queried image with lighting adjusted and the background removed. As shown in Figure 7(b) and (c) I_q and I_s appear to have identical viewpoints and similar lighting.

We applied a similarity measure to compare images. The Mean Squared Error (MSE) method is applied, this metric indicates the difference between two images, 0 means perfect similarity and 255 means the two images are dissimilar. MSE can be described as

where I(i,j) is a pixel value at i,j of image 1 and K(i,j) is a pixel value of image 2.

To create a change mask, we compared pixel values between I_s and I_q by subtracting grey scale values between the two images using the following equation,

Then the change mask B(x) is generated according to the following decision rules using the following equation,

Where τ is a specified threshold. The threshold is usually chosen empirically in order to produce different change masks.

In our experiment, we collected images for a reference set S₀ for monitoring purpose and set S₁,...,S_n for creating 3D models. The camera was used to take pictures to ensure that the overlap between consecutive images is at least 50% so that 3D models can be created. We collected 6 images for the set S₀, and we collected 19, 33, 29, and 26 images for S₁, S₂, S₃, and S₄ of size 3456 x 2304, respectively, from a concrete beam.

The 3D model of each set S₁, S₂, S₃, and S₄ has the details as shown in Table 1. Figure 4 and 5 show the example results (S₁) of our work, the sparse model, the dense model and the textured model. As can be seen from the figure, the dense and textured model provide good 3D visualization for the concrete beam.

For camera registration, we registered images from the set S₀ onto 3D models from the sets S₁, S₂, S₃, and S₄ as explained in Section 3.2. Table 2 shows the example results of registration errors in the 3D model from the set S₁. As can be seen from the table, the errors from image S₀ is similar to the errors from set S₁.

In this experiment, we applied the change detection technique for queried and synthetic images in 4 different datasets, S₁, S₂, S₃, and S₄, in which queried images are original queried image I_q from S₀ and synthetic images are synthesized image I_s from S₁ with the same camera parameters as I_q. The concrete beam was loaded to simulate crack propagation such that the difference between the reference set S₀ and set S₁, S₂, S₃, or S₄ is apparent enough to observe changes between images. Table 3 shows the summary results from example images. It can be seen from the table that all image pairs have similar MSE, and the MSE values increase from S₁ to S₄ as the cracks propagated. Nevertheless, the values of MSE are small in all pairs of images. This is due to the fact that the texture of our 3D models is fine enough to provide sufficient detail in the synthetic images.

Table 4 shows the results of the change masks between all image pairs. The threshold for the change mask is set as 55, which was chosen empirically. The change pixels are in white and the non-change pixels are in black. As observed from the results, the changed pixels are visible and accurate in all image pairs, which means that our method can detect changes of cracks between images. The last column of Table 4 shows the result of change masks overlaid on top of original query images I_q to show the change in cracks in the original images for monitoring purpose and for specifying damages occurring on the surface of a structure.

This paper proposed techniques to monitor problematic areas using change detection techniques. The paper proposed techniques that deal with geometrical errors for a change detection system. It can be seen that the 3D model-based image can be used to monitor problematic areas periodically. The images acquired later can be used to update the 3D model such that the 3D model will become more detailed and allow synthetic images to be more realistic. The system can be combined with damage detection system so that problem areas can be identified automatically, although this is beyond the scope of this paper. In order to obtain more accurate 3D models, more images should be collected. Nevertheless, this research project has demonstrated that we can use 3D model-based image registration to monitor problematic areas.

This research project demonstrates the method to create synthetic images from a 3D model to remove geometrical errors in change detection systems. To improve our results, we require a better 3D model. The current watertight 3D model used in our work may not be sufficient to provide a more realistic synthetic image. Since the watertight model available is created from mesh, in which the texture in each mesh has been smoothened, therefore there will be some loss in the details of synthetic images due to the smoothing process. To improve the 3D model, more close-up images may be required. Also, the proposed change detection algorithm is simple, it may not be suitable for a complex surface. Nevertheless, our method works well with a simpler surface, such as a surface of a concrete beam.

The pipeline of a change detection system for concrete structures is presented in this study. The system acquires images and creates a 3D model. More image data can be obtained to allow a more detailed 3D model to be created, which make synthetic images to be more realistic. The change masks can be used to monitor the problematic areas for periodical inspection and monitoring.

The research proposed method for accurate geometrical adjustment in a change detection system. The adjustment method was achieved by creating a synthetic image from a 3D model. The results of our synthetic images provide a result which can help to remove geometrical errors, although further improvement is required to make a 3D model better to allow a more detail synthetic image for a change detection system in complex structures.

The authors would like to thank the Faculty of Engineering, Thammasat University, Thailand for providing scholarship and support for the research project.