A compliant teleoperation control method based on real-time dynamic impact force feedback and operator-sensing

In recent years, telerobotics has gradually been applied to complex and delicate operating environments, such as ultrasound diagnosis and precise industrial assembly. These application scenarios are characterised by complex environments, high requirements for real-time communication and fine contact forces (Bruno and Oussama, 2016). The advantages of teleoperation technology over conventional robotics are clear: it doesn’t need the traditional trajectory planning work and allows inexperienced robot operators to complete the task in their professional fields, e.g. the complex space task scenarios, the ultrasonic diagnosis scenarios (Kang et al., 2020; Tsumura et al., 2021), the remote industrial assembly scenarios (Li et al., 1999; Kumar et al., 2020) and the disarming of explosives scenarios (Conway et al., 1987; Gao and Gao, 2009).

To avoid the dangerous situations caused by teleoperating errors, reseachers have developed teleoperation control methods a lot. The most widely used method is direct and bilateral control method, which provides realtime slave robots states to the operators and requires the operators to send control commands in a timely manner (Bruno and Oussama, 2016). In another way, depending on the usage requirements and application scenarios of the task, the teleoperation devices are designed to meet the needs of bilateral motion control and force feedback, which makes their dissimilar mechanisms with the slave robots (Pan et al., 2017; Peer et al., 2005). For these reasons, the motion mapping methods is based on their tips(mostly their end-effectors) instead of the traditional joints. Meanwhile, compared to force/position control method based on position-position architecture, the one that based on position-force architectures can avoid excessive operator inertia from the robot, which is a more desirable bilateral control architecture complex scenarios (Bruno and Oussama, 2016). With low communication latency and small loads, maintaining system stability and transparency in scenarios with good communication environments, operators are able to carry out tasks successfully.

A large number of existing studies have analyzed teleoperation systems from control theories perspective (Yang et al., 2021; Li et al., 2022). Among the passive and stability analyses for a wide variety of teleoperation systems, Chen and Huang (2003) conducted stability analysis in a virtual reality force-aware proximity sensing teleoperation robot system Guo et al. (2013) investigated the stability of teleoperation robot systems by means of forward time observers European scholars such as Gil et al. (2004) applied stability analysis to haptic function teleoperation devices in more specific system stability studies, Tugal et al. (2017) proposed a; Zames-Falb-based multiplier method for the analysis of monotonically bounded environments Chen et al. (2019) proposed an RBF-based neural network-based adaptive robust control design method for nonlinear bilaterally operated robots to help solve stability problems in complex environments; in studies concerning transparency, the wave variable coding method proposed and adopted by Niemeyer et al. for high-frequency communication and lower time delay has enabled the solution of the time delay problem in high-frequency communication (Stramigioli et al., 2002; Niemeyer and Slotine, 2004). Feizi et al. (2022) enhance the force-tracking and the activation of the stabilizer performance by a real-time frequency-based delay compensation approach Yang et al. (2023) developed a fixed-time synchronous control for a class of uncertain flexible telerobotic systems, which provides an example of synchronous control in uncertain systems. Besides, in shared control field, Michel et al. (2021) proposed a method to combine the shared control teleoperation method with adaptive impedance control, making teleoperation method to be used in assembly field (Gomes, 2011).

Although the above researches have greatly ensured the stability of the teleoperation system, the teleoperation system still needs more researches on the compliant motion control of the slave robot to ensure the accuracy and safety. Usually, when robots motion is controlled by the algorithm programmed off-line, trajectory with planned velocity can avoid robots motion jitter and achieve the compliant motion effects (Guo et al., 2019; Niemeyer et al., 2016). However, considering that it is impossible to plan the velocity of the trajectory while slave robots is under direct control, jitter from the joints caused by the dissimilar mechanisms and unplanned trajectories will break the stability of the slave robots to a great extent. Scenarios where positional accuracy is required, such positional curves with jitter cannot simply be smoothed out by simply using a filtering algorithm, which would destroy the accurate position mapping relationship between the slave robot and the teleoperation device. Additionally, with conventional bilateral force feedback algorithms, such jitter is often difficult to perceive by the operator. These problems can seriously affect the flexibility and accuracy of the teleoperation robot and impede precise teleoperating.

This paper explicitly analyses the motion mapping process of the teleoperation robot and the principle of impact jitter under a dissimilar mechanisms teleoperation architecture, thus obtaining the cause of jitter due to impact on the robot. Then a new feedback force control method is proposed, which is based on the relationship between impact jitter and feedback forces. The control method largely allows the operator to actively sense and avoid impact jitter in real-time and teleoperate the robot in a compliant manner. Finally, the effectiveness of this approach was demonstrated in an experimental test with a teleoperation system based on a Force Dimension omega.7 teleoperation device, a Rokae xMate3 pro robot and an industrial computer with a real-time operating system(RTOS).

According to the Handbook of Robotics (Bruno and Oussama, 2016), the used teleoperation device and the slave robot are usually in dissimilar mechanisms. As shown in Figure 1, the key motion points of the teleoperation device are used as tip points (usually the end-effector points of the device) to establish the positional mapping relationship in order to achieve the direct control with the slave robot. The motion information of the teleoperation device is used as input to the system, and the positional information of the tip point relative to the base is obtained from its positive kinematics, while the motion of the slave robot is solved by the position information and inverse kinematics. The position information of them are transformed by the mapping relationship.

The displacement of the two should remain linear, $pstB$ can be solved for as follows

while k_P is the scaling factor. The above equation determines the relationship between the positions of the points.

As shown in Figure 2, for the mapping points in the orientation at moment t, the initial orientation of the mapping point of the teleoperation device is noted as R_Om, the transformation of the motion is represented by the rotation matrix as R_m and the new orientation after its motion is R_Omt. The initial orientation of the mapping point of the robot is R_Os, the transformation of the motion is represented by the rotation matrix as R_s and the new orientation after its motion is R_Ost.

In the rotation mapping relationship, R_m and R_s should be equal, then we can solve R_Ost as follows:

Generally speaking, scaling of orientation is not required in teleoperation mapping, so the above equation determines the orientation relationship of the points. The complete mapping transformation matrix can be constructed by combining equation (2) with equation (1): $TOst=ROstBpst01$⁠. The displacement on the robot joints can be further calculated to complete the solution of the robot motion poses based on the inverse kinematics. This posture mapping relationship can be applied to the case of dissimilar teleoperation devices and slave robots, ensuring the accuracy of the position between the tips of the teleoperation devices and the slave robots.

Figure 3 shows a schematic mapping of the motion profile of the teleoperation device to the motion profile of the robot. With a strict mapping relationship, the incoherent and irregular motion characteristics entered by the operator will also be retained. Figure 3 also reveals the correspondence between the proposed teleoperation compliant motion curve and the slave robot joints motion curve. Ideally, even if the operator input a compliant motion curve into the system, the obtained slave robot joints velocity and acceleration curves by the inverse solution of the pose-mapping algorithm are still discontinuous.

In addition, focusing on the motion of slave robot joints, if the velocity curve is discontinuous or the absolute acceleration value is too large, a rigid shock to the robot will occur at the point of velocity discontinuity; if the acceleration curve is discontinuous or the absolute acceleration derivative value is too large, a flexible shock to the robot will occur at the point of acceleration discontinuity. From the above analysis, the unpredictable joint motion caused by the dissimilar kinematic mechanisms between devices will trigger rigid and flexible shocks on the joint motors, thus causing the slave robots in motion to exhibit jitters and undesirable state.

Bilateral force feedback is an important mechanism for operator sensing robot’s operational state (Michel et al., 2021; Selvaggio et al., 2021). For the case of dissimilar mechanisms of teleoperation devices and slave robots, a force feedback approach with force-position architecture should be constructed. On the one hand, for the position and velocity information provided by the teleoperation device, the robot mapping point and the external contact force magnitude are controlled using a proportional-derivative (PD) controller for conductivity; on the other hand, by installing force sensors on the robot mapping point, the force magnitude on the teleoperation device will be consistent with the value of the force sensors, thus building a bilateral force feedback path between the teleoperation device and the robot, with the relationship as follows:

Where d stands for desired, desired displacement p_sd and desired velocity $ṗsd$ are both obtained in the teleoperation device; F_m stands for the feedback force on the master teleoperation device and F_sensor stands for the contact force collected by the force-sensor(a force sensor is usually connected at the end of the slave robot).

The above equation shows that although the bilateral force feedback mechanism enables the operator to feel the force state at the robot mapping point in the case of Cartesian spatial motion with small external forces, but the difference between the slave displacement p_s and p_sd desired displacement is small(the difference between $ṗs$ and $ṗsd$ is also small), and the feedback force F_m obtained by the system cycle will be weak for the operator to sense the shock jitter phenomenon. In addition, on robots without a rear drive design, force feedback mechanisms based solely on the machine’s human sensor design do not provide effective feedback on impact jitter phenomenon.

From the above analysis, although the teleoperating relationship is defined by the positional mapping between the teleoperation device and the robot, but the mapping relationship also brings the internal jitter phenomenon to the slave robot. The bilateral force feedback mechanisms commonly used in teleoperation systems are unable to provide effective feedback on this jitter phenomenon, which means that the system requires a new approach to help the operator sense and achieve a compliant control of the slave robot.

As shown in Figure 4, for the mentioned teleoperation robot jitter problem previously, an improved method that feedbacks the jitter impact in the form of force is provided, and the direction of the force will be opposite to the direction of the velocity input by the operator; the operator perceives it in real time and then circumvents the discontinuous points of velocity and acceleration of the robot motion.

As in equation (4), based on an time interval Δt at moment t, the integral expression for the operator input work ΔW_m in per unit of time is obtained by integrating the feedback force F_m(the force on the teleoperation device) with the displacement p (displacement of the mapped motion point of the teleoperation device) as the integration variable; the integration expression for the robot input work ΔW_m per unit of time is obtained with the robot. The angle of rotation q on the joint is used as the integration variable for the moment τ on the robot joint to obtain the expression for the robot output work ΔW_s per unit time.

In order to achieve the effect that operator senses the slave robot impact feedback when using the teleoperation device, the ΔW_m should be scaled up and adjusted to ΔW_{m_back}. For different configurations of the robot, the degrees of freedom is assumed to be n here. Based on the causes of rigid and flexible shocks and their effects on the jitter phenomenon of different robot configurations, the rigid shock scaling factor a and the flexible shock scaling factor b are set and the rigid shock linear amplification function $kR(q̈)$ and the flexible shock linear amplification function $kF(q̈)$ are constructed, where $q̈$ is a vector of n × 1 formed by the joint angular acceleration of each joint of the robot and $q⃛$ is a vector of n × 1 formed by the derivative of the joint angular acceleration of each joint of the robot.

Considering the general case, the slave robot has different configurations and shows inconsistent jitter effects when different joints are subjected to the same impact, so the n × 1 weighting coefficient vector for each joint is set, besides rigid and flexible shocks are expressed as k_{R_joint}. These two vectors are generally given by using experimental experience. Combining equation (5) to establish the mapping between ΔW_s and ΔW_{m_back}.

Taking Δt to be a sufficiently small value, the slave robot output work ΔW_s and the operator input work ΔW_{m_back} are expressed respectively.

where θ_{m_back} represents the angle formed by vector [F_{m_back}(t + Δt) + F_{m_back}(t)] and vector [p(t + Δt) − p(t)]. Since the compensating force F_{m_back}(t) of the teleoperation device is always in the opposite direction to the velocity $ṗ(t)$ of every movement during the motion, when Δt is sufficiently small, the vector [F_{m_back}(t + Δt) + F_{m_back}(t)] will be approximately opposite in direction to the vector $ṗ(t+Δt)$⁠, which is cos θ_{m_back} ≈ − 1.

Similarly, ΔW_s is scalarized as,

where θ_s represents the angle formed by the vector [τ(t + Δt) + τ(t)] and the vector [q(t + Δt) − q(t)]. Since the torque τ(t) of the robot is always in the same direction as the joint velocity $q̇(t)$ during motion, when Δt is sufficiently small, the vector [τ(t + Δt) + τ(t)] will be the same as the vector [q(t + Δt) − q(t)], the velocity vector $ṗ(t+Δt)−q(t)$] at the moment t + Δt) in approximately the same direction, which is cos θ_{m_back} ≈ 1.

Decompose the vectors required for further calculations in a Cartesian space coordinate system.

Substituting the scalarized ΔW_s into the mapping relationship between ΔW_s and ΔW_{m_comp}, we can find |F_{m_back}(t + Δt) + F_{m_back}(t)|.

From the relationship between the vectors F_{m_back}(t), F_{m_back}(t + Δt) and [F_{m_back}(t + Δt) + F_{m_back}(t)] Using the parallelogram rule as well as the cosine theorem to solve for the vector F_{m_back}(t + Δt).

Let α be the angle formed by the vector F_{m_back}(t) and the vector F_{m_back}(t + Δt), which are opposite to the velocity vectors $ṗ(t)$ and $ṗ(t+Δt)$⁠, respectively, of the known operator-operated teleoperation device in the opposite direction, so the inverse cosine theorem can be used to solve for the velocity vectors $ṗ(t)$ and $ṗ(t+Δt)α$ as follows.

As shown in Figure 5, according to the parallelogram rule, the vector [F_{m_back}(t + Δt) + F_{m_back}(t)] is on the angle bisector of the vector F_{m_back}t, F_{m_back}t + Δt formed by the angle bisector, so by the cosine theorem, for the angle α/2 formed by the vector F_{m_back}t and the vector [F_{m_back}(t + Δt) + F_{m_back}(t)], the with.

Separating |F_{m_back}(t + Δt)| from the above equation.

The vector F_{m_back}(t + Δt) has the same direction as $ṗ(t+Δt)$⁠. Let the directional cosines of $cosαṗ(t+Δt),cosβṗ(t+Δt)$⁠, $cosγṗ(t+Δt),Fm_backt+Δt$⁠, which is shown in equation (17).

After the above discussion and calculation, this section completes the separation and solution of F_{m_back}(t + Δt). The impact feedback force F_{m_back} is obtained for each moment by iteratively calculating.

The impact feedback force is combined with the force sensor values of the original bilateral force feedback to optimize the original bilateral force feedback mechanism and obtain a bilateral force feedback method incorporating dynamic impact feedback, as in equation (18).

And now we obtain the specific impact feedback force, which enables the operator to sense the jitter states and make avoidance in real-time.

In order to verify the practical effectiveness of the compliant teleoperation control method based on dynamic impact feedback force and real-time sensing proposed in this paper, the following experimental method is used in this section.

As shown in Figure 6, the hardware used in the experiments includes the omega.7 teleoperation device, the xMate seven-degree-of-freedom robot, the KWR75 E six-dimensional force sensor from Kunwei and an IPC. The xMate 3pro robot from Rokae is a collaborative robot with a positioning accuracy of 0.03 mm and a more open underlying controller; Nexcom’s IPC is equipped with a 7Gen. IntelⓇCoreTM i7 with the RTOS real-time operating system installed, the 1 kHz high-frequency communication between the robot and the teleoperated devices can approximately reduce interference caused by communication delays on the experiment; the KWR75 E six-dimensional force sensor from Kunwei has the accuracy less in 0.01 N;

To address the effectiveness of the previously mentioned direct control methods for position mapping, the system should verify the effectiveness of the follow up between the teleoperation device and the slave robot. In the experiment verification of the teleoperation, the real-time performance and accuracy of the motion should be included. Considering the random and irregular nature of the human motion curve, the replication of a given motion curve is difficult for the operator to achieve. These characters should be remained in the experiments of this paper. As shown in Figure 6, the robot and the teleoperation device run a reciprocal trajectory shaped like an “O”. Operator will avoid the jitter state by sensing the value of feedback impact force.

To further observe the accuracy and real-time performance of the motion, the scale factor of the mapping was set to k_P = 1. The motion trajectories of the teleoperation device was plotted in blue (left), and the slave robot in Cartesian space in red (right), respectively. The results are shown in Figure 7.

The motion trajectory is decomposed in Cartesian space to make a plot of its position-time relationship, where the motion profile of the robot is in red and the motion profile of the teleoperation device input is in blue, as shown in Figure 8.

From the combined analysis of Figures 7 and 8, the ideal position accuracy error can be obtained under the position mapping condition. We collect four points, which displayed on Figure 8, to analysis the time-delay(X stands for t time/ms and Y stands for the displacement on the X-axis direction). Moreover, we calculated the time-delay in every millisecond and the results is shown in Figure 9, and the time-delay is around 30 ms. Considering the processing effect of the band-pass filtering algorithm with velocity protection, the real-time performance is wonderful; if k_P < 1 is taken, the position accuracy can still be further improved.

Given the randomness and uncertainty of motion in the actual operating environment, it is impossible for the operator to enter the same motion position information in both experiments. In another way, the main principle of our given approach eliminate jittering by incorporating real-time impact force sensing and avoidance by the operator. The input motion information may vary considerably. For this reason, the experiments were set up to allow the operator to input a motion curve shaped like an “O” as soon as possible, using a teleoperation device, record the motion data and then analyze the data.

Specifically, to determine the feedback from a rigid shock, two of the seven joints of the robot were selected as having the most representative acceleration values for the shock jitter phenomenon, and their relationship to the feedback force was compared in the time domain, using the red curve to represent the robot’s motion data and the blue curve to represent the feedback force data from the teleoperation device. The specific data and curves are shown in Figures 10 and 11.

Similarly, for the method of determining feedback from flexible shocks, the two joints with the most representative acceleration values for the flexible shock jitter phenomenon were selected from the seven joints of the robot and compared in the time domain in relation to the feedback forces. Using the red curve to represent the robot’s motion data and the blue curve to represent the feedback force data on the teleoperation device, the specific curve relationships are shown in Figures 12 and 13.

From the motion data in Figure 10, Figure 11. and Table 1, the joint acceleration controlled by the original algorithm(max 12.9 units on 2nd joint, over 29.3 units on 4th joint) is higher than the compliant algorithm(max 4.8 units on 2nd joint, max 10.9 units on 4th joint), which means that it takes more rigid shocks to the slave robot due to the jitter(most of them are caused by the kinematics mapping reasons). Meanwhile, due to the weak feedback effect of the original control method, the operator couldn’t sense the jitter, which makes the slave robot suffers a lot from the changing velocity impact. From the perspective of flexible impact displaced on Figure 12, Figure 13. and Table 1, the joint acceleration derivative controlled by the original algorithm(max 1446.3 units on 2nd joint, max over 2502.3 units on 4th joint) is higher than the compliant algorithm(max 1076.2 units on 2nd joint, max 1857.1 units on 4th joint), which means that it takes more flexible shocks to the slave robot due to the jitter. Meanwhile, the frequency of the flexible shocks is quite different between the two algorithms. The lower shocks frequency and lower average shock units show that the compliant algorithm benefits a lot from the operator sensing and actively avoidance.

The proposed compliant teleoperating method provides an effective feedback way for operating sensing impact, and makes the operator adjusting the teleoperate trajectory and avoid the unstable and jitter states, thus, reducing the rigid/flexible impact of the slave robot.

The way of direct teleoperation robotics would cause jitter on slave robot, even if the operator input a compliant motion curve into the system. This is because the motion information is obtained by the robot inverse solution(kinematically dissimilar mechanisms of the slave robots and the teleoperation devices determines the difference of inverse solution), that the input trajectory is determined by the operator in real-time system without any trajectory planning works. Obviously, the slave robot joint motion information with velocity discontinuity will bring rigid impact, while the acceleration discontinuity bringing flexible impact. The principle of teleoperation robotics jitter is clearly analyzed.

According to the principle of slave robot jitter, a new force feedback method is designed to enhance the feedback of slave robot jitters. The feedback force is based on acceleration and acceleration derivative, using the Work theory to design a proper value of the feedback force. Specifically, cosine theory is used to calculate the accurate direction of the feedback force, completing the whole feedback processing. The way of the designing is a promotion of the original bilateral control, not only feedback the jitter state but also maintaining good operational performance.

Due to the promoting force feedback method, the operator enables to perceive the jitter state through the feedback force during teleoperating process. Sensing the larger force, which is contrary to the motion direction, the velocity of the operator motion is going to be slow down and avoid the jitter state of the slave robot, making the teleoperating robot more stable. It is generally a good way to remind the operators to adjust their input trajectory. However, operator could also overcome the feedback force with larger operating force to pass the jitter point if necessary.

According to the experiments results on figures and Table I, the compliance and effectiveness of the promoting teleoperating method are proved. The motion data shows that jitter from rigid impact and flexible impact has obviously reduction. Based on the benefits of the operator-sensing and jitter states avoidance, operator is able to operate the robot in a more stable, more precise and more compliant way.

The proposed method is useful in the delicate application scenarios such as ultrasound diagnosis and precise industrial assembly, increasing the security and stability of the slave robot. However, there are also some compliant control difficulties in some scenarios that needs shared control strategy instead of direct control strategy, which can’t be deal with by the proposed method. The study to improve the method to fit the shared control is also needed and developed in the future works.

A bilateral force feedback method with dynamic jitter sensing is proposed for the jitter problem during the motion of the teleoperation in the direct control method. However, the direct control method of teleoperation cannot plan the slave robot trajectory as the traditional robot control method does, and the robot will receive impact jitters such as rigid impact and flexible impact, which seriously affects the accuracy and stability of slave robot motion. In this paper, we propose a compliant teleoperation control method based on dynamic impact force feedback and operator-sensing. By improving the conventional bilateral force feedback method, the impact that the robot may experience during motion is calculated as a feedback force, allowing the operator to sense and avoid the impact in real-time. The effectiveness of the method has been verified in experimental validation, ensuring the compliant motion of the slave robot in various scenarios.

Funding: This work was supported by the National Natural Science Foundation of China No. 52205017 and Zhejiang Lab Open Research Project (NO. K2022NB0AB08).