Photogrammetry and Augmented Reality for Underground Infrastructure Sensing, Mapping and Assessment

According to the UN World Urbanization Prospects (UN, 2018), about 55% of the world’s population lives in urban areas, with countries like India, China, and Nigeria adding hundreds of millions more people in coming decades. The growth of urban population calls for better city management and planning, emergency response, and expansion of city infrastructure. Smart city initiatives promise to aid in delivering sustainable city growth by introducing a highly integrated network of sensors that will provide information for rational decision-making and efficient resource allocation (Barlow, 2018; Tierney, 2017).

A major source of inefficiency in city planning and growth where smart city techniques can have a positive impact is in providing organized, open knowledge of subsurface infrastructure location and conditions. Subsurface infrastructure encompasses utilities (water, stormwater, wastewater, gas, electricity, telecommunications, steam, etc.), geotechnical formations, and the built underground (including tunnels, subways, garages and subsurface buildings). There are both technical and political challenges regarding the locating, mapping, and assessment of underground infrastructure and the exchange of information among stakeholders (Petroski, 2017). Such deficiency causes unexpected delays in construction, prevents adequate maintenance and periodic inspection, and reduces provided service quality. Cities like New York have been under continuous development for several decades, with aging infrastructure in unknown location and conditions, which occasionally lead to catastrophic events (NY Times, 2018).

In the US, some initiatives addressing this challenge are present in cities like Chicago (CNET, 2017). There, a common integrated framework for subsurface infrastructure visualization and registration was introduced. However, this framework only includes data provided by new construction.

Traditional methods for collecting geometric information collate as-built drawings, historical records, and dead reckoning with information gathered by above-ground geophysical instruments, such as ground penetrating radars (GPRs), magnetometers, and acoustic sensors (Huston, 2017, 2017a).

In this context, three-dimensional (3-D) information about the location and condition of subsurface urban infrastructure can facilitate assessment, construction, emergency response, management, and planning. It has direct impact on the implementation of smart city approaches for sustainable development of urban areas.

Current commercial geophysical sensing systems such as GPR use either wheel encoders or GPS to localize acquired data. However, wheel encoders are limited to linear, planar data acquisition, while GPS is unavailable in tunnels and has degraded accuracy in the urban canyons of densely populated areas where the location and condition of subsurface utilities is critical (Groves, 2013). Traditional dead reckoning techniques for positioning, such as inertial measurement units (IMUs), suffer from drift over time due to accumulated errors in the signal integration. An alternative to overcome or mitigate inertial drift is to use photogrammetric techniques (Durrant-Whyte, 2006, 2006a; Dissanayake, 2011). These techniques have been extensively studied in the robotics community and are applied in inertial-odometry frameworks (Bloesch, 2017) and/or in simultaneous localization and mapping (SLAM). However, such techniques require specialized domain-knowledge, especially computer vision and real-time rendering, making it difficult for researchers and engineers in other domains to readily exploit them. The growing interest in augmented reality (AR) is changing the technical landscape.

In the past, the gaming industry pushed the development of graphic processing units (GPUs), introduced by Intel in 1999 (Moller-Akenine, 2008), that are now ubiquitous in personal computers and heavily applied in machine learning. A similar trend has appeared in the present with the development of AR applications that are pushing the boundaries of mobile computing and networking and drawing the attention of the largest software and hardware companies. Efforts are thus being made to provide easier to use application programming interfaces (APIs) and development frameworks for the creation of AR applications. Such efforts wrap sophisticated algorithms and facilitate the use of photogrammetry in other systems and knowledge domains.

Notwithstanding, the development of modern game engines, such as Unity (Unity, 2019), allows software developers to implement applications once and deploy over multiple platforms, reducing attrition in cross-platform systems integration as well as handling challenges in AR projection. Furthermore, AR frameworks enable faster feedback between the user and the environment, leading to efficient resource allocation (e.g. contrast using a traditional printed map versus visualizing directions using GPS in a smartphone or digital screen). In the sense of emergent intelligent systems (Minsk, 1988), faster feedback through improved visualization has the potential to increase resource allocation (Pollalis, 2012), e. g. by saving time in subsurface utility locating.

This paper presents the results of endeavours towards integrating geophysical instruments with photogrammetry-enabled smartphones, 3-D subsurface map assembly from collected data sets, and AR visualization. The following projects are considered in this paper:

1. GPR systems integrated with smartphones for improved positioning and registration.

2. AR visualization of buried objects.

These efforts may improve 3-D locating by integrating above-and-below-ground information, leveraging systems that are already AR-friendly for improved localization, mapping, and visualization of sensed data.

The paper is organized as follows: In Section 2 an overview of photogrammetric localization and mapping and AR frameworks is given; Section 3 describes the Google’s Tango AR framework; Section 4 presents a geophysical sensing system integrated with Tango, as well as data collected with this new system; Section 5 introduces the AR representation of subsurface features using Microsoft’s HoloLens; Section 6 covers technical challenges and possible future directions; Section 7 gives the concluding remarks.

Accurate position registration is a fundamental step for collecting and collating subsurface geometric information. In terms of geophysical test instruments, position registration takes the form of localization of the instruments relative to a coordinate system fixed on the earth. While there continues to be improvements in capabilities, GPS/GNSS positioning instruments are often inadequate for localization due to the required precision and multipath degradation of signals in urban canyons (Groves, 2013). The use of IMUs is attractive to complement GPS/GNSS positioning since IMU sensors are ubiquitous in modern smartphones and do not depend on remote communication (Barrios, 2015). However, they have limited accuracy and suffer from position drift, requiring an alternative or complementing pose estimation device.

Photogrammetric methods are an alternative and complementary set of position localization methods that combine optical digital photographic sensing with computationally-intensive image processing to render the position and pose of an instrument and may be used to reduce IMU drifting. There is extensive literature in SLAM algorithms that leverage on photogrammetric techniques. However, open-source, cutting-edge photogrammetry frameworks usually are implemented in C/C++ and require the use of Robot Operating System (ROS) which is popular in the robotics community but not common in other areas of research. While C/C++ are used for computationally efficient photogrammetry algorithm implementation, their library ecosystem and requirement of compilation and linking render them less welcoming to new users than other newer programming languages. Furthermore, open-source libraries have only been tested on specific operating systems and specific library versions, complicating cross-platform development and reducing portability.

In recent years, several AR application development frameworks such as Tango, ARCore (for Android devices) or ARKit (for iOS devices), have enabled the computation of complex, photogrammetric-based, pose estimation algorithms in smartphones without the need of expert knowledge in computer vision.

AR frameworks implement photogrammetric algorithms, as well as data fusion algorithms to merge pose estimated from camera imagery with device pose estimation from inertial measurement unit (IMUs). Furthermore, AR frameworks may also provide the platform for building a photogrammetric map of above-ground survey area prior to geophysical testing, a feature fundamental to mitigate IMU drift. Photogrammetric mapping aids in the identification of the area by matching stored and online visual features, called a loop closure, reducing IMU drift and allowing persistence of AR 3-D objects by means of virtual anchoring. The above-ground maps may be in the form of dense point clouds, point descriptors or surface maps. The present state-of-the-art is that the photogrammetric registration instrument both constructs and uses the above ground landmark map, i.e. SLAM (Dissanayake, 2011).

Furthermore, cross-platform game-engines, such as Unity, readily perform both the optical measurements and calculations for AR projection by leveraging the AR frameworks. The disadvantage is that the implemented algorithms are proprietary and may not suit specific applications. However, the savings in development time and implementation cost make this approach attractive for system prototyping.

In the first project presented here, the Google’s Tango AR framework, now supplanted by ARCore, was used. A Tango-enabled smartphone, i.e. with adequate calibration and compatibility with the Tango software development kit (SDK), is here denoted a Tango device. The Tango AR framework leverages mainly on visual-inertial odometry (VIO) for accurate localization, providing also an offline mapping capability, i.e. it requires pre-mapping of the survey area for improved pose estimation. The Tango device can generate dense point cloud 3-D area models for data collating and area representation in Unity 3D, with application, for instance, in virtual preservation of historical heritage.

An AR-based technique combines present-time photogrammetric data with pre-collected photogrammetric maps of the site to render position and pose estimates relative to the site map coordinate system. Geographic coordinates may be obtained by anchoring the photogrammetric map origin to the appropriate geographic coordinates and transformation of local cartesian coordinates into the desired geographic coordinate system.

Attaching photogrammetric position registration to a geophysical instrument, such as a GPR, enables the position-registered free 3-D movement of the instrument during data collection. This technique, as opposed to conventional grid and measuring wheel registration, has the potential to dramatically reduce test times, permit testing over larger areas, and allow for more flexible test paths. Processing the geophysical data and merging with known information about the site leads to earth-registered 3-D maps of the subsurface infrastructure.

An early prototype system for the integration of GPR system and photogrammetry techniques using the Tango device was proposed in (Pereira, 2018). In that work, the Tango device would trigger a radar scan based on its displacement, e.g. every 25.4 mm. However, the commercial GPR system data processing pipeline was not engineered expecting this type of triggering, causing the loss of a few of the GPR A-scans (i.e. received signals) if not well calibrated. Matching of registered pose and collected data was also challenging, as the system does not provide a timestamp for each A-scan. Use of the Geophysical Survey Systems (GSSI) SDK is of future research interest and may allow for real-time data exchange and 3-D GPR image formation.

The system presented here, shown in Figure 1, uses a Lenovo Phab II smartphone as the Tango device. In contrast to the previous work, the system now registers position based on the GPR cart wheel encoder pulse count, e.g. every 50 encoder pulses. An Arduino counts the pulses, and a second Arduino communicates with Tango to require pose registration every 50 pulses. This enables more consistent data collection, and pose and data matching. The application registers data locally, i.e. on device, but may also be posted to a server using an HTTP request.

A smartphone application, developed using Unity 3D, registers the timestamp and the device poses provided by the Tango device. A (GSSI) GPR system composed of an SIR-30 control unit and a 400 MHz antenna is used for GPR data collection.

Prior to the survey, the Tango device creates a photogrammetric 3-D map of the area. In the Tango framework, the photogrammetric map consists of the most relevant visual features of the recorded imagery, avoiding the use of dense point-cloud representation to improve photogrammetric algorithm performance during online pose estimation.

Two surveys assessed the capability of free-form scanning with the system. Since the GPR system is ground-coupled, the vertical coordinate of device pose is set to a constant value at ground level. The depths of collected data are estimated using a homogeneous dielectric constant of 4. Local pose is collated on satellite imagery of Google Maps based on visual documentation of the survey, i.e. pictures of the starting position. Subtracting a mean value removes the ground reflection from the acquired GPR data.

In the first survey, only a region of the site was mapped, as illustrated in Figure 2. Apart from the initial pose, no other form of area marking or documentation is required during survey. In traditional GPR survey it would be necessary to make measurements and draw markings on the ground and take intensive annotations for post-survey processing and data interpretation. Those time-consuming activities are not required in the AR-integrated system. Furthermore, traditionally at each linear data collection, the system is brought to a halt to save data. In the system presented here data is continuously acquired, in a free-form path, pointing to potential automation of GPR survey and inviting the development of more sophisticated GPR data processing, using for instance, cognitive radar techniques.

A top-view of the local device path appears in Figure 3. In the mapped area (crosswalk area), pose registration is in good agreement with satellite imagery distances. However, in the area not mapped drift started to build, but mitigated when loop closures occurred, as indicated by the arrow. This illustrates the advantage of performing pre-mapping or, ideally, SLAM, in conjunction with geophysical sensing.

The resulting 3-D subsurface image appears in Figure 4. The street is removed but other features of the above-ground area are shown for enhanced visualization. Post-survey matching of the registered poses and collected scans assuming a constant number of scans per registered pose forms the image. Since more scans are collected than poses registered, linear interpolation is performed between subsequent poses. An alternative is to window-average collected waveforms. Of note is the presence of several hyperbolas in the initial scans shown in Figure 4, corresponding to rebars that are present only in the crosswalk area. The occurrence of loop closure correcting against drift is evidenced by sudden large displacements. In future work post processing may be able to correct those deviations (e.g. forcing alignment with neighbouring points).

The second survey fully mapped the area shown in Figure 5. To demonstrate and validate pose estimation consistency, the first pass of the scan is made back-and-forth along the same line. As shown in Figure 6, the path overlaps and returns to the starting position. The distance covered is in good agreement with satellite imagery distance estimates. The resulting subsurface scan is shown. The arrow in Figure 7 points to the located buried utility present in the area.

Using AR techniques, it is possible to render images through an AR headset, with the 3-D maps as overlays on top of streets for personnel on-site.

A laboratory sandbox setup demonstrates the visualization of buried objects through AR goggles, Figure 8. A metal cylinder was buried in a laboratory sandbox. A multistatic GPR consisting of three receive and one transmit antennas scanned for subsurface features, Figure 9. The GPR system collects one A-scan every centimetre in the along-track direction, such that the along-track position of buried features may be determined by reading the collected B-scans.

This system returns three B-scans, one for each receive antenna, that form the basis of a post-processing step in Unity to introduce a 3-D model of the buried feature. The Unity application ports to a HoloLens. The game development platform Unity and HoloLens are already integrated such that the engine performs the AR image projection. Furthermore, the HoloLens uses SLAM to track device pose. Figure 10 shows the view through the HoloLens goggles, the subsurface pipe imaged with the goggles, and the manual location of the pipe in the sand.

Some of technical challenges with advancing AR-facilitated subsurface infrastructure locating, mapping and imaging are:

1. Develop effective methods of persistent anchoring of AR instruments to above-ground landmarks with subsurface infrastructure features – Current commercial systems photogrammetric mapping is device-specific, e.g. a map created with Tango cannot be used by HoloLens. Better cross-platform integration of 3-D maps and anchors is desired.

2. Mapping buried in-filled trenches that distort acoustic and electromagnetic waves – The configuration of urban subsurface infrastructure is often congested and complicated by geotechnical structures. In filled trenches with utility pipes at the bottom are a common complication that arises from cut and cover construction. Differing electromagnetic and acoustic refractive and attenuative properties between the trench and infill distort and guide waves, making it difficult to ascertain the location and condition of the pipes buried at the bottom of trenches, especially for narrow trenches.

3. Integrating the AR systems with open source geospatial data formats – Open source data collection for urban data is a movement that continues to gain momentum. It is favourable for the collected mapping data to be interfaced with these open source formats in order to increase usage.

4. Creating AR formats that present buried utilities on flat screens that are realistic – The projection of subsurface pipes with an AR system is not very convincing on a flat screen but appears realistic with a 3-D mixed reality goggle, such as a HoloLens, Figures 11 and 12. Compensating for this distortion is necessary to AR experience in flat screen devices, such as smartphones and pads. The insertion of virtual trenches and similar artificial optical cues is potentially a viable method, as shown in Figure 12.

The digitization of information flows in civil infrastructure applications is proceeding at a rapid pace. AR facilitated subsurface infrastructure location, monitoring and management is at an early stage of development. Proof of concept feasibility of the primary processes and identification of some of the possibilities have been presented.

This work has been supported by US National Science Foundation grants 1647095 and 1640687, the University of Vermont SPARK Fund, VT EPSCoR and US Ignite.