Robot-Assisted System with Ensenso 3D Camera for Safe Handling of Nuclear Waste

With the help of the software, the Enseno 3D camera takes over the perception and evaluation of the depth information for the operator, whose cognitive load is considerably reduced as a result. The assistance system combines the haptic features of the object to be gripped with a special gripping algorithm.

"The scene cloud is used by our system to automatically generate several stable gripping positions. Since the point clouds captured by the 3D camera are high-resolution and dense, it is possible to generate very precise gripping positions for each object in the scene. Based on this, our "hypothesis ranking algorithm" determines the next object to pick up, based on the robot's current position," explains Dr Naresh Marturi, Senior Research Scientist at the National Centre for Nuclear Robotics.

(The principle is similar to that of the skill game Mikado, where one stick must be taken away at a time without moving any other sticks).

The determined path guidance enables the robot to navigate smoothly and evenly along a desired path to the target gripping position. Like a navigation system, the system supports the operator in guiding the robot arm to the safe grasp, if necessary, also past other unknown and dangerous objects. The system calculates a safe corridor for this and helps the operator not to leave the corridor through haptic feedback.

Camera
"For all our autonomous grasp planning, remote control and visual object tracking tasks, we use Ensenso N35 model 3D cameras with blue LEDs (465nm) mounted on the end effector of the robots along with other tools," says Dr Naresh Marturi. Most of the systems from the Extreme Robotic Lab have so far been equipped with a single 3D camera. "However, recently to speed-up the process of 3D model building we have upgraded our systems to use additional three scene mounted Ensenso 3D cameras along with the one on-board the robot."

The Ensenso N series is predestined for this task. It was specially designed for use in harsh environmental conditions. Thanks to its compact design, the N series is equally suitable for the space-saving stationary or mobile use on a robot arm for the 3D detection of moving and static objects. Even in difficult lighting conditions, the integrated projector projects a high-contrast texture onto the object to be imaged by means of a pattern mask with a random dot pattern, thus supplementing the structures that are not or only weakly present on its surface. The aluminum housing of the N30 models ensures optimal heat dissipation of the electronic components and thus stable light output even under extreme ambient conditions. This ensures the consistently high quality and robustness of the 3D data. Even in difficult lighting conditions, the integrated projector projects a high-contrast texture onto the object to be imaged by means of a pattern mask with a random dot pattern, thus supplementing the structures that are not or only weakly present on its surface.

Cameras of the Ensenso N camera family are easy to set up and operate via the Ensenso SDK. It offers GPU-based image processing for even faster 3D data processing and enables the output of a single 3D point cloud of all cameras used in multi-camera operation, which is required in this case, as well as the live composition of the 3D point clouds from multiple viewing directions. For the assistance system, the researchers have developed their own software in C++ to process the 3D point clouds captured by the cameras.

"Our software uses the Ensenso SDK (multi-threaded) and its calibration routines to overlay texture on the high-resolution point clouds and then transform these textured point clouds into a world coordinate system," explains Dr Naresh Marturi. “Ensenso SDK is fairly easy to integrate with our C++ software. It offers a variety of straightforward functions and methods to capture and handle point clouds as well as camera images. Moreover, with CUDA support, the SDK routines enable us to register multiple high-resolution point clouds to generate high-quality scene clouds in global frame. This is very much important for us, especially to generate precise grasp hypothesis.”

Main advantages of the system

• Operators no need to worry about scene depth or how to reach the object or where to grasp it. The system can figure out everything in the background and helps the operator to move exactly to the place where the robot can best grasp the object.
• With haptic feedback, operators can feel the robot in their hand even when the robot is not present in front of them.
• By combining haptics and grasp planning, operators can move the objects in a remote scene very easily and very quickly having very low cognitive load.This saves time and money, avoids errors and increases safety.

Outlook
Researchers at the Extreme Robotic Lab in Birmingham are currently developing an extension of the method to allow the use of a multi-fingered hand instead of a parallel jaw gripper. This should increase flexibility and reliability when gripping complex objects. In future, the operator will also be able to feel the forces to which the fingers of the remote-controlled robot are exposed when gripping an object. Fully autonomous gripping methods are also being developed, in which the robot arm is controlled by an AI and guided by an automatic vision system. The team is also working on visualization tools to improve human-robot collaboration to control remote robots via a "shared control" system.

This is a promising approach for the safety and health of all of us: the handling of hazardous objects such as nuclear waste is ultimately a matter of concern to us all. By reliably capturing the relevant object information, Ensenso 3D cameras are making an important contribution to this globally prevalent task of increasing urgency.