Augmented reality (AR) is not just about fun and games. “It also has a variety of uses in industrial automation,” notes Eran Nadel, director of innovation projects and rapid development manager for manufacturing engineering at Siemens Digital Industries Software. He should know because he and his colleagues are among the select group of engineers who have been developing AR technology for use in manufacturing. These engineers have been superimposing virtual elements constructed from CAD and other data onto views of real objects depicted on mobile devices, desktop computers, smart glasses, and headsets. It is this superimposition of the virtual upon the real that differentiates AR from virtual reality (VR), which immerses users in a completely virtual scenario constructed solely from digital elements.
Nadel reports that the focus at Siemens has been mainly on using AR to support various kinds of human-centered activities. An example is a maintenance application in which Siemens’ AssistAR product guides technicians through routine maintenance on a gearbox. The process begins after the technician uses a LiDAR infrared camera to generate the real-world image of the gearbox and establish its orientation in space. Then, AssistAR provides step-by-step instructions by highlighting elements, such as screws and faceplates, on a conventional computer screen in the correct order in the workflow. It also animates the elements to show how they are removed, replaced, or reassembled.
In its simplest form, AR is used to identify a component, machine, or area of a factory, and give the user some contextual information about it. “This information can be anything from documents, pictures, and videos to IIoT [Industrial Internet of Things] data,” says Peter Richmond, portfolio manager for XR and 3D visualization at Aveva. More advanced applications include 3D, real-time models, step-by-step holographic animations of a task, and remote multi-user collaboration.
Four industrial uses
Industry is deploying AR in at least four ways. The first is training new workers or even guiding more experienced workers through various workflows, like performing routine maintenance on the gearbox. “AR helps users to achieve faster upskilling and creates more confident workers by standardizing procedure execution and providing in-the-field training,” says Vineesh Kapoor, director of product management at Emerson Automation Solutions.
The second way industry is using AR is asset identification. “Augmented reality can help operators and technicians find assets quickly and easily—a difficult task when they must identify one device out of hundreds or thousands,” notes Kapoor. With a mobile device using augmented reality, a field worker can be guided directly to an asset that needs attention.
A third industrial use of augmented reality is knowledge transfer, particularly in the field. Demand for this kind of AR application has grown partly because of the travel restrictions imposed during the COVID-19 pandemic and partly because of the shortage of skilled experts. “Senior personnel often can’t travel to a site to help with solving a problem,” says Kapoor. With AR technology, however, these experts can see exactly what local engineers and field technicians see and collaborate with them from afar.
This ability to assist in remote communications and streamline collaboration also shows promise for process design and equipment layout—the fourth way industry is deploying AR. Users can place virtual equipment in the real world to check whether it will fit in the allotted space before actually spending the money and time to install it in real life. Users can also immerse themselves in a proposed manufacturing cell to see how robots will interact with other machinery, the workers tending the cell, and the surroundings.
For a long time now, manufacturing engineers have been able to construct such cells and populate them with robots and CNC machines built with CAD/CAM software like SolidWorks and Delmia from Dassault Systèmes. Indeed, the company’s Delmia software already contains prebuilt models of more than 1,400 robots. “We calculate the kinematic motion, so it’s a logical progression to put it all together in a timed sequence,” says Mike Buchli, senior SolidWorks product and portfolio manager. “It’s not that much different than a video game.”
These simulations have an important limitation, however. “Even though you have been able to put the robots in your computer model and move them around, you were not really immersed in that cell,” explains Buchli. Now, with AR technology like his company’s 3D Experience platform, users can get the sense of standing next to a 6 ft robot as it reaches across the cell to load a machine.
Making a mesh of things
Implementing augmented reality is usually harder to do than implementing virtual reality because an AR system needs a mechanism for determining where elements are in the real world so that it can overlay the virtual elements with precision. Because the computer-game industry has already solved many of these problems, many AR developers serving the manufacturing sector have been borrowing technology from the game industry.
For example, some use a headset fitted with a video camera and a mixture of other sensors, such as infrared sensors and radar. “Besides gathering a regular video image, it’s also measuring depth and establishing perspective,” says Arnav Mukherjee, development director for viewer and experimental technology at Dassault Systèmes. Meanwhile, the software superimposes a virtual grid on the image of the real environment. “We can plop a machine on top of that mesh, and it’s going to be scaled appropriately in the view.”
Because game engines already generate such meshes and superimpose virtual objects on them, some AR developers are relying on them for this purpose. Dassault Systèmes isn’t among those developers, however. SolidWorks can generate these meshes and place CAD models on them. “Our users expect us, as suppliers of engineering software, to take care of all the headaches of creating images inside the headset,” explains Mukherjee.
According to Nadel, Siemens is also streamlining the generation of AR graphics. The company has integrated its AssistAR software with its product lifecycle management (PLM) software to automate the extraction of both the CAD data and the assembly, inspection, or maintenance sequence from the PLM software. As a consequence, the CAD data and work sequences are delivered as an electronic work instructions package that AssistAR can read to generate AR visuals on PCs and screens typically found on production lines. Siemens is making these visuals available to integrators and users through an AR converter called JT2Unity.
“This integration of CAD data through PLM is unique,” notes Nadal. “In other cases, data conversion and extra process authoring are required.”
Preparing your CAD data
These extra steps to prepare the CAD models for use in AR visuals is necessary for a couple of reasons. First, “CAD models come in various forms from various vendors,” says Aveva’s Richmond. “So, a typical first step is to translate the CAD files into a form suitable for a 3D modeling tool such as Blender,” an open-source toolkit for creating animation and 3D models.
Another reason for data preparation is that models built from CAD and other data are usually too big for most AR devices to handle in real time. “Whether geometry, textures, sheets, or information drawn from bills of material, most of this data is still created by tools that do not have real-time consumption at the top of mind,” explains Nicolas Fonta, senior product manager of augmented, virtual, and mix reality at Autodesk Inc. “CAD data, for example, is often represented in a parametric way that is not suitable for most real-time graphics. When meshed, the amount of detail and resolution of the data is difficult for a phone, tablet, or AR glasses to ingest and render in real time.”
The data, therefore, need to be reduced somehow. “You can take away the screws and other details that are not critical for the AR experience,” offers Mukherjee at Dassault. You can also cut out and use only the relevant portion of a machine or production line.
Until a few years ago, most preparation of CAD data was done manually using tools like Autodesk 3ds Max, a process that required a measure of specialized knowledge, according to Fonta. “More and more, however, partially automated solutions are emerging,” he says. “The best have been offshoots of the tools used to create video games and their 3D elements.” Also emerging are services for polygon reduction, level-of-details generation, texture remapping and compression, and streaming.
Even so, Fonta advises designers to create smaller and simpler CAD models from the outset, especially when they know that their models will be likely to be used in AR visuals. This means developing CAD models to have the simplest surfaces and fewest polygons possible. “It’s best to model efficiently from the start, rather than try to decrease the polygon count later,” says Fonta. Such reductions after the fact run the risk of erasing important details.
Preparation is also required for layering other, non-graphics data, like metadata or building information modeling (BIM) data, on top of graphics information. “BIM data, for example, can be shown on top of a visual representation of a building,” says Fonta. “Or BOM information can be shown on top of visual representation of a consumer product, or IoT data layered on top of physical industrial machinery.”
Augmenting your realities
Once the CAD and other data are properly prepared, they can be imported into AR authoring software for the creation of the final application. The latest generation of this software has 2D and 3D graphical editing tools, scripting tools, and visual editors designed to streamline the generation of applications customized for the hardware on which it will be deployed. Aveva’s Richmond says these tools can help automate parts of the process.
Still, generating AR applications requires a measure of skill and familiarity with the necessary tools and techniques. “Like any skill, though, it would be possible for automation engineers at user companies to learn how to use these applications,” notes Richmond. To help them do that, most AR-software developers have refined their visual editing tools to reduce the skill barrier and make the tools more accessible to what Richmond calls the “citizen developer.”
“For more advanced cases,” adds Kapoor at Emerson, “we recommend working with integrators and suppliers to create and maintain libraries in the cloud.” Storing libraries there not only streamlines access to files, but keeps them up to date with the latest AR software.
The investment of time and money in AR visuals doesn’t end with their creation. “Like any software, AR solutions are not set-and-forget applications,” explains Kapoor. “Maintaining your AR visuals is a necessity to ensure that they continue to pay returns on your investment.”
Leaders relevant to this article: