Major advances in industrial robotics technology are not just coming soon; some of them are already here. And one of the biggest drivers behind these advances is OpenCV—Open Source Computer Vision—which offers free software libraries for computer vision development.
I first learned of OpenCV during a meeting with Jeff Bier, founder of the Embedded Vision Alliance (some of my conversation with Jeff is captured in my post “Computer Vision: An Opportunity or Threat?”). It was Jeff’s description of the types of content presented at the Embedded Vision Alliance Summit that led me to learn more about OpenCV and what it could mean for the future of industrial robotics through a presentation by OpenCV’s president and CEO Gary Bradski (who is also a senior scientist at Willow Garage).
The routines offered via OpenCV are focused on real time image processing and 2D and 3D computer vision. The routine library offers versions for Linux, Windows, Mac, Android and iOS and can be programmed with C++, C, C#, Java, Matlab, and Python.
While watching the presentation by Bradski, delivered at a recent Embedded Vision Alliance Summit (you can view the presentation as well after registering on the EVA site at no charge), I learned that OpenCV algorithm modules are particularly useful in industrial robotic applications because they address relevant issues such as image processing, calibration, object recognition/machine learning, segmentation, optical flow tracking and more.
Because OpenCV library materials can be used for so many different types of applications, the site currently has some 160,000 downloads of materials each month and has so far delivered more than 6 million downloads.
Noting that the paradigm of industrial robotic application is welding, Bradski points out that in such applications robots are essentially blind, dumb and repetitive, albeit highly accurate. Because these welding robots typically lack vision, the products to be welded (usually cars) have to be staged and moved into the robotic welding area with incredible accuracy, which requires a lot of planning and orchestration.
“Programmed with ladder logic, these (welding) applications are a throwback to 1980s robotics technology,” says Bradski, which means it is mostly used in applications where there is a high margin, and the products are rigid and have a long shelf life (which is why industrial robotics have such a foothold in the automotive industry). “Products like cell phones are typically assembled by hand because it takes so long to fixture a robotic line for cell phone assembly. By the time you had the line set up, you'd be on to the next version of the cell phone.”
Robotics 2.0, in Bradski’s parlance, deals with robots being able to more easily handle greater levels of variance. To make this jump to Robotics 2.0, robots must be able to see (ideally in 3D), should be controlled by PCs rather than PLCs, and will no longer conduct only rote moves; instead they’ll operate adaptively.
“In order to understand and grasp objects, robots need to recognize objects and object categories as well as know the pose of objects in six degrees of freedom relative to robot coordinate frames,” says Bradski. Both pose and recognition can be obtained in 2D and 3D.
To fully grasp that Bradski’s vision is not one that exists in pure academic research but is rooted in real world application, it’s helpful to know that Bradski organized the vision team for Stanley, Stanford University’s autonomously controlled vehicle that won the DARPA Grand Challenge. Stanley used sensors and lasers to gain 2D and 3D perception for planning and control. According to Bradksi, the laser on Stanley was used to teach the vision system, i.e., the laser would find the “good road” and then use the vision system to find more of it. What the vision system captured was segmented into a red, green and blue pixel distribution. This segmented road, composed of bad trajectories and good trajectories, was then integrated into a rolling world model and passed to the vehicle’s planning and control system.
The vision system technology used on the Stanley project has ben adapted for use by the Google Robot Car and in Google Maps’ Street View.
In related autonomous vehicle news, Nissan has just announced that it plans to sell an autonomous driving vehicle by 2020 (see video at bottom of this article). If you think this announcement is a bit far-fetched, consider that Nissan also committed to delivering a zero-emissions vehicle by 2010, which it did with the Nissan Leaf.
Following his work on Stanley, Bradksi went on to found Industrial Perception Inc. (IPI) in March 2012. The initial thought was to develop robotic vision systems for manufacturing, but logistics proved to be more of a sweet spot for vision-enabled robotics, according to Bradski. “There’s a lot of boxes in the world,” he says, “so we focus on navigation —understanding surfaces, tracking motion, calibration and metrology, pick and place, and truck and container loading and unloading. We do a lot of model building (3D model capture) to train robots” rather than training them by hand/teach pendant.
Bradski says IPI is currently working on a robot that can unload boxes at a rate of one box every four seconds (a logistics industry goal). The human rate for this is 6 seconds. “We are approaching the human rate now,” he says of the ability of the current robot he is testing.
Rest assured that the robotic advances learned through Bradski’s experience in logistics applications will certainly find their way to the manufacturing floor sooner rather than later.
The next Embedded Vision Alliance Summit will be held October 2 in Littleton, Mass. Read more about the Summit and how to register for it.
The video below highlights Nissan’s plan to deliver autonomous driving by 2020.
Leaders relevant to this article: