Applied AI for Assembly and Training

A sweet spot for applying artificial intelligence (AI) in industry lies in connecting AI and vision systems for quality control. A recent entrant into this field is focused on democratizing access to vision-based AI and expanding its use beyond quality inspection and into training and process control.

According to Rapta, its AI platform uses live video to capture an assembly process as it’s performed by experts. Rapta’s AI platform uses these videos to standardize work instructions across an organization to teach and guide anyone in the assembly process. 

The Rapta platform learns by replicating how the human eye and brain work together to understand nearly any manufacturing assembly task. 

Matthew Thornton, chief marketing officer at Rapta, explained, “We decided to mimic humans rather than LLMs [AI large language models like ChatGPT] in developing this. Humans are so adept at physical dexterity; that’s why we train our AI like a human would learn—across zones 1, 2 and 3 of vision, where zone 1 is for focusing on a small area, zone 2 encompasses a wider area of view and zone 3 includes peripheral vision. We teach it what to focus on and when to mitigate noise like a human does through positive and negative reinforcement.”

The proprietary AI used by Rapta reportedly builds on technologies pioneered by Google DeepMind and Facebook and combines convolutional neural networks (which use three-dimensional data for image classification and object recognition), proprietary deep learning models and deployment techniques to provide the company’s turnkey system.

Aaron Brown, CEO of Rapta, said the company’s technology can be used to learn any manufacturing process—from airbag and tractor assembly to electronic panel building. 

The platform

The Rapta platform includes the following components:

• AI platform software
• AI computer (Brown noted that Nvidia GPUs are key to Rapta’s platform) 
• Basler industrial power over Ethernet cameras 
• Gantry 
• LED lighting 
• HMI touchscreen 

Teaching the system an assembly process is done using Rapta’s visual setup tool, where a knowledgeable worker performs the assembly for the system’s cameras. Rapta’s deep learning AI analyzes the recorded video to learn the correct sequence of steps based on what it was shown, with no coding required.

Once deployed, the Rapta platform shifts from being trained to being an assembly supervisor. The platform delivers continuous training and quality assurance in real time, ensuring the assembly work is completed correctly. 

“It's all about getting a product built properly with 100% quality without having to directly watch everyone on the factory floor,” said Brown. “By having the Rapta tool automate the supervising task, you no longer have to have a supervisor walking around to, basically, babysit adults doing their job. The AI validates the process so that the worker knows they've got the right things happening in the right sequence every single day.”

Beyond assembly applications, the Rapta platform can also integrate with manufacturing and enterprise systems such as MES and ERP to help with real-time forecasting and ordering. Because all Rapta data streams are analyzed and correlated, the data can be integrated with other systems using APIs (application programming interfaces) to provide a more comprehensive view of a company’s total operation. 

Focused attention

Because people are often the bottleneck to scaling a manufacturing business, particularly due to the ongoing shortage of available skilled workers, Rapta’s technology was designed to help with initial and ongoing worker training by continuously showing performance data to the operator at the workcell.

“You start by teaching Rapta your company’s tribal knowledge about how to do steps correctly and incorrectly,” said Brown. “It’s all done visually with no coding or data science experience required. It can even adjust for lighting in facility. Once the user input is provided to the Rapta AI, it can build the model in about 5-10 minutes and is ready for deployment.”

Brown added that “we kept the models small and focused to avoid issues with AI—like you hear about with LLMs—where they can get confused with too many parameters. But more importantly, we wanted to make it easy to train the AI for accessibility.”

The Rapta platform also features a timer on the dashboard to indicate how long specific assembly tasks should take as performed when the AI was taught. That timer can be adjusted, Thornton said, for application by both expert assemblers and newbies to help the worker understand their assembly effectiveness.

OCR demo

Automation World was recently given a demonstration of the Rapta platform across two applications—OCR (optical character recognition) and in situ training.

In the OCR application, the Rapta platform was used to inspect a panel build to validate that all the wire labels were installed in the right sequence and on the right components. 

The system can read everything in the panel—including wires and labels. “If it finds a problem, it shows the worker in real time where the problem is in the panel,” said Brown. “This gives the worker a chance to remediate the problem before continuing the job. What we really want to do is avoid having to call over a supervisor [to review the work]. Once the identified problem has been corrected, the system will scan it and give the OK to proceed. That's the powerful thing about applied AI from our perspective—it's all about helping the human operator be successful.”

By virtualizing the supervisory function to ensure 100% quality, Brown said the Rapta OCR feature—initially designed as a feature for the electrical panel building industry—can be used effectively in any industry. 

“Every customer uses OCR for different reasons because of the myriad problems that can occur in assembly,” said Brown. “For example, the wrong type of pump could be installed in the wrong spot or serial numbers get flipped on a rifle scope, causing it to leave the assembly area with the wrong serial number. That’s why industry needs the ability to automatically read and validate text.”

In-situ demo 

The second demonstration of the Rapta system focused on in situ control of the manufacturing process. This allows users to quickly see what they’ve done to determine if parts were installed correctly. 

“The system will warn the user if components are installed, say, upside down or if components are spaced too far apart from an adjacent component on the din rail,” said Brown.

A good case example for this application can be seen in the common assembly process of bolt torquing. When a worker is required to torque, say, eight bolts on an assembly, problems can arise if the bolts aren’t torqued in the proper sequence. Problems with improper torquing include pieces coming loose and compressed gaskets.

Brown noted that, in addition to visual inspection of the bolt torquing process, the Rapta platform can process inputs from torque guns to validate that the right event happened to the right bolt with the right amount of torque applied.

In situ application of the Rapta platform can also extend into supply chain verification to identify part anomalies. 

“If a part you receive is out of tolerance, even an expert assembler is going to have assembly problems,” said Thornton. “And if that out-of-tolerance part in a new batch from a supplier isn’t noticed, that can create significant assembly problems and production slowdowns while you try to figure out what’s creating the problem.”

The platform can also highlight design flaws if a manufacturing process proves to not be repeatable.