Learning-based vision perception for industrial robots
Introduction
Industrial robots have to face new and complex environments, as more and more industrial production systems need them. Traditional robots based on model or human configuration cannot deploy to new environments to perform different tasks as quickly as people wants. Fortunately, vision perception can help industrial robots overcome these problems, because most of the information related to the environment and the tasks to be performed can be input into the robot system in the form of visual information. Meanwhile, the rapid development of machine learning technology, such as deep learning, has been proved to be able to greatly improve the visual perception ability. Therefore, the learning-based vision perception can further enhance the ability of industrial robots, which enables their fast deployment and high autonomy.
However, there are still many challenges in the application of these learning-based visual perception technologies in actual industrial environments. This special collection on “Learned-based vision perception for industrial robots” aims at bridging the gap between industrial robots and learning-based vision perception. The topics of this collection mainly include different machine learning technologies for industrial robots and recently developed computer vision technologies for industrial robots, including object detection, segmentation, tracking and multimodal perception.
The papers
Finally, the special collection accepted 12 papers for including in “Learning-based vision perception for industrial robots.” These papers can be roughly categorized into three groups, i.e. visual perception based on deep learning for robots, visual information processing for robot systems and surveys for robot-related application. This section gives an overview of these accepted papers.
The first paper (Liu et al., 2020) is to propose a novel anomaly detection to address the problem of very rare abnormal patterns when robots detect abnormal samples in the industrial environment. This method designs a double encoder-decoder generative adversarial networks to detect anomalies and uses the double encoder-decoder approach to map high-dimensional input images to a low-dimensional space. Anomaly detection is achieved by calculating the distance between two low-dimensional vectors obtained from two encoders. By comparison with traditional anomaly detection models, the proposed method has better accuracy and F1 score.
In the second paper (Jampour et al., 2021), Jampour et al. introduce an autonomous vision-based shelf-reader robot, called Pars, and demonstrate its effectiveness in accomplishing shelf-reading tasks. Motion blur on images captured while it moves is efficiently handled and the region of interest for retrieving a book’s information is detected. Moreover, six parallel cameras enable Pars to check books and decide moving paths. Its performance is evaluated in a library with 120,000 books, and results show that it can discover problems such as missing and misplaced books.
Chan et al. propose to track targets based on standard hedging and feature fusion for robot (Chan et al., 2021). The proposed method achieves better accuracy of visual target tracking even in the complex background, by efficiently learning the discriminative information between targets and similar objects, then using standard hedging algorithms to dynamically balance the weights between different feature optimization components. Furthermore, spatial regularization and extended Kernelized correlation filter are integrated in the proposed method for robust tracking.
Tian et al. also propose a novelty detection method by self-supervised learning and channel attention mechanism (Tian et al., 2021). The conventional pixel-level reconstruction cannot effectively extract the global semantic information of the image. To tackle this problem, an auxiliary task of reconstructing rotated image is set to enable the network to learn global semantic feature information. Then, the channel attention mechanism is introduced to perform adaptive feature refinement on the intermediate feature map to optimize the correspondingly passed feature map.
The Siamese network trackers based on template matching have gained great response in the community because of their excellent performance. Nevertheless, the feature extractor of these trackers is still AlexNet rather than other deep neural networks, and these trackers’ templates are not updated. To solve this problem, Zhou et al. present a dynamic template update strategy for the Siamese trackers, through which the tracking performance potential of the deep network is unlocked (Zhou et al., 2021).
The next group of papers focuses on visual information processing for robot systems. The purpose in Kulecki et al. (2021) is to implement and evaluate two representative grasping methods for the robotic arm equipped with a two-fingered gripper. The robot detects objects in the environment and uses grasping methods to determine the reference pose of the gripper. The whole pipeline, including perception, grasp planning and motion execution on the mobile robot system, is implemented and evaluated in various scenarios.
Although deep neural networks have been successfully implemented in computers with powerful computation, it is rarely deployed in STM32 microcontrollers. Guan et al. make a good attempt in Guan et al. (2021) to implement intelligent control for quad-rotor aircrafts with a STM32 microcontroller using deep neural networks. A 32-bit micro-controller STM32F103C8T6 is adopted as the main control chip for the control system of the quad-rotor aircraft, and a deep neural network is deployed. This method provides a good application example of edge Artificial Intelligence (AI), it also provides a design reference for implementation of AI algorithms on unmanned aerial vehicle or terminal robots.
Garcia et al. (2021) tackle the controller acquisition problem of unknown sensorimotor models in non-holonomic driftless systems, which can simplify and speed up the process of setting up industrial mobile robots with feedback controllers. An offline exploratory method consisting of two stages is proposed. The first stage focuses on completing the kinematics model of the system, whereas the second stage explores the sensorimotor space in a predetermined pattern.
The mass electronics sector is one of the most critical sources of waste, in terms of volume and content with dangerous effects on the environment. To improve the outcome of recycling PCB waste, Doroftei et al. (2021) design and develop a robotic system for automated dismantling of PCB waste. The robot system uses data provided by an artificial vision system to guide a custom tool attached to the last link of a six degrees of freedom manipulator robot. The custom tool includes a programmable screwdriver combined with an innovative rotary dismantling element.
To enable a robot to accurately build a 3D map in industrial environments, even when GPS signals are invalid, Lin et al. propose a robust and precise SLAM system which optimally integrates the GPS data and a Light Detection and Ranging (LiDAR) odometry (Lin et al., 2021). To effectively verify reliability of the GPS data, Verifying GPS data with Lidar data (VGL) algorithm is proposed and the algorithm uses the points from LiDAR. EG-LOAM algorithm is implemented to eliminate the accumulative errors by means of a reliable GPS data. On the KITTI data set and the customized outdoor data set, the proposed system is able to generate a high-precision 3D map in both GPS-denied areas and areas covered by GPS.
A novel graph structure is proposed to represent point features, line features and plane features, where the features correspond to the graph nodes and their geometric relation corresponds to the graph edges (Guo et al., 2021). Furthermore, a hybrid descriptor for points and lines (HDPL) is obtained using graph convolutional neural networks. The experimental results show that HDPL has rotational invariance, scale invariance, viewpoint invariance and noise immunity compared against other methods.
In the last paper, Shaik and Rufus (2021) review the shape sensing techniques used using large area flexible electronics (LAFE), where shape perception of humanoid robots using tactile data is mainly focused. In this review, the authors investigate papers published in the past 15 years and emphasize contact-based shape sensors. Moreover, fiber-optics-based shape sensing methodology is discussed for comparison purpose. Authors find that LAFE-based shape sensors of humanoid robots with neural networks and machine learning algorithms can provide good results with best resolution in 3D-shape reconstruction. The most suitable approach for large object shape sensing using LAFE is also suggested.
