You know that you made the right decision to invest in a remote monitoring service when it catches a problem hours before your staff would have even noticed it. Such was the case at a Centria plant that custom coats coils of metal in Ambridge, Pa. A developing embossing problem was beginning to cause the metal to tear. Had it continued, the problem would have led to significant losses in product.
The vigilance of an off-site engineering crew averted these losses. From their offices, engineers with the InSite monitoring service offered by Milwaukee-based vendor Rockwell Automation Inc. identified the nascent problem and initiated the appropriate corrective action.
This incident is just one example of why a growing number of manufacturing facilities have found it profitable to hire vendors to monitor and maintain their machinery remotely. Although machinery builders and automation providers have been eyeing the enabling technology for quite some time now, communications were often cumbersome, thereby limiting the kinds of services they could offer. Now that relatively recent Internet Protocol (IP)-based Ethernet applications for remote diagnostics and maintenance have removed that limitation, more vendors are rolling out remote services.
Starting big
Builders of very large and expensive capital assets, such as power-generation equipment, have been delivering remote diagnostics for a while, according to Steve Carlson, Rockwell’s product manager for its remote diagnostics business. The high price tags on such equipment made it easier to justify the custom, proprietary connections that had been necessary to provide the services in the past.
Communications standards and falling price-to-performance ratios, however, have put the same ability within the budgets of smaller projects. “With the proliferation of Ethernet and IP-based communications on the factory floor, a lot of builders of smaller machinery are beginning to follow in the footsteps of the larger capital-asset builders,” says Carlson. “In many cases, though, they are offering more than just diagnostics services.”
Rockwell Automation got into this business because it had already developed the business processes and technology infrastructure necessary for supporting its own automation products in large, complex integrated-manufacturing environments. As machinery builders prepare to deliver suites of managed services remotely, they must build similar infrastructures for monitoring potentially hundreds of machines and managing and documenting incidents on each. “So, we’re partnering with OEMs (original equipment manufacturers) to leverage our infrastructure and processes to deliver those services to their customers,” explains Carlson.
At Centria, Rockwell Automation’s InSite service came bundled with a controls upgrade that the coater undertook on a coating line when it added a more modern second line. Since then, Rockwell engineers have been monitoring the controllers on both lines over a high-speed broadband connection, keeping the drives tuned, and looking for signs of trouble.
Whenever control parameters deviate beyond predetermined limits, the engineers notify Centria’s production people and begin troubleshooting the problem by checking the real-time and historical operating data logged by the control system. “It’s like having someone continuously standing over your shoulder, constantly pointing out potential problems and then letting you know how to correct them,” says Ron Mahan, plant engineer at Centria.
He reports that the service has helped his company not only to boost the productivity of its coating process but also to reduce downtime, scrap rates and damage to equipment. Before the control upgrade and monitoring service, several breakdowns a week had been costing the company at least $3,000 to $5,000 per hour in downtime. In the first year since the implementation, Centria needed a service technician to come on-site only twice.
Overcoming insecurities
Although remote diagnostic services have been around for a long time, many manufacturers have been reluctant to use them. One reason is that they are afraid to open their production systems to the outside. These fears are justified because it can take less than two minutes to penetrate an unprotected supervisory control and data acquisition (SCADA) system running the Microsoft Windows operating system, according to Mike Rothwell, business unit manager for Americas Automation Systems at supplier Phoenix Contact Inc., of Middletown, Pa.
“Opening up a system to remote access via the Internet, or even to limited access within a plant-wide local area network (LAN), can introduce the risk of exposing confidential information to unauthorized users,” he says. “Worse yet, it can give someone—a hacker, a disgruntled employee—the opportunity to sabotage a machine or an entire plant.”
To protect users against such intruders, Innominate Security Technologies AG, a Phoenix Contact company based in Berlin, Germany, has designed its mGuard security products to be deployed at the machine. Its “device-attached security” architecture allows the machine, rather than the service provider, to initiate any contact for relaying service requests and supporting information. Because outgoing Internet connections are easier and safer to administer, Rothwell believes that this strategy will lower the cost of delivery and permit providers to put more tools such as video streaming in their toolboxes.
These benefits accrue from solving another problem—connectivity. Until recently, modem connections have been the favored means of delivering remote maintenance and diagnostic services. They are direct and avoid the complexity of managing incoming connections through the firewall from service providers for all of the machinery and processes in the plant. Although modem connections are somewhat secure, especially for stand-alone machines, they still pose a security risk for networks.
Moreover, they are notoriously slow. “A modem connection does not have enough bandwidth,” explains Thorsten Hoes, director of central engineering at Ferromatik Milacron Maschinenbau GmbH, a builder of plastic injection-molding machines in Malterdingen, Germany. In order to offer maintenance services to its customers over the Internet, the builder sought a way to exploit the high-speed communications made possible by the IP-based Ethernet interfaces that it and most other machine builders are putting into their controllers today.
The approach that it found was Innominate’s mGuard. This solution solves the scalability and bandwidth problems that had hampered earlier Ethernet-based interfaces for remote maintenance. Its device-attached security architecture creates virtual private networks (VPNs) and, when necessary, permit adding tunnel technologies via HTTP (HyperText Transfer Protocol) and proxy servers. The technology not only secures the connection, but also lets Ferromatik’s technicians and engineers view their customers’ user interfaces on their personal computers (PCs). “The response time is less than a half second,” says Hoes. “It’s like you’re in front of the machine.”
Molding facilities can get various kinds of online and telephone support with this teleservice. Ferromatik’s application engineers can help them troubleshoot the molding process, or its machine specialists can solve problems that might develop in the machine over time. “If a customer has some errors in the plastic parts that he is producing, our application guys can look at the current operating parameters, edit them, and get feedback from the customer,” explains Hoes.
On the other hand, when something goes wrong with the machine itself, specialists can check the logbooks in the machine and figure out what went wrong. Depending on the problem, a specialist can either tell the customer what to do over the phone or dispatch a technician to the machine. “Not every problem can be solved with the teleservice,” notes Hoes. “If, for example, a hydraulic component is drawing more current than expected, we can inform the customer that someone from our service department is on his way to find out why.”
Since the technicians have an idea of what the problems might be ahead of time, they can be sure to bring the right parts and avoid the hassle of having to come back later. “Our customers are under pressure to meet their production schedules,” says Hoes. “So, if a machine goes down, we have to react very fast. Our service people have to fix the problem on their first visit.” Since launching the teleservice in September 2007, Ferromatik has about 30 injection-molding facilities, big and small, using it. There are many more machines connected to the service, however, because some molders have more than one machine. These facilities connect all of their machines to one VPN tunnel device. “When a customer needs support, the designated person gives us a call and connects us to the right machine,” says Hoes.
Enabling secure communications
Whether the machine or the provider initiates a service call, some measure of security is crucial. An experienced information-technology (IT) engineer should establish a third-party access policy, even on a direct phone line or DSL (digital subscriber line) connection, in order to prevent unauthorized access. “Without a policy in place to lock it down, even a phone connection is vulnerable,” warns Martins Jansons, network consultant with the Industrial Communications Group of supplier Siemens Energy and Automation Inc., in Norcross, Ga.
“If you’re using the Internet, you also need technology to make sure that no one is eavesdropping,” he adds. For this reason, he and other experts recommend user-authentication procedures, content encryption, VPNs or a combination of these technologies. VPNs establish security over and above authentication and encryption by separating the traffic of various user groups or by restricting access through custom routing mechanisms.
As has been the case with many advances in automation, the enabling technology underlying remote diagnostics and maintenance has been the continual increases in processing speed and memory of the controllers. “This allows for the embedding of functionality that was not possible before,” explains Ted Thayer, automation systems product manager at vendor Bosch Rexroth Corp., in Hoffman Estates, Ill.
“For example, a programmable logic controller (PLC) that I used back in 2002 had a processor that ran at 20 megahertz (MHz) and had about 128 KB (kilobytes) of total memory,” he offers. “The lowest-end PLC that we sell now has a processor that clips along at 166 MHz with 32 MB (megabytes) of memory, which allows for using software tools such as a Web server and provides for extensive Ethernet functionality.”
Protocol for pulses
As crucial as Ethernet and other communications technologies are, they are not the only ones that make remote diagnostic and maintenance services possible. Standards also have been playing an important supporting role. Perhaps the most prevalent one is the simple network-manager protocol (SNMP) promulgated by the Internet Engineering Task Force (IETF), an open standards organization that is part of the Internet Society, in Reston, Va.
The protocol allows management systems to monitor networked devices for problems by presenting management data, such as diagnostics information, as variables that application software can query. “It allows diagnostics information to be more transportable over existing infrastructures, both intra-networks and inter-networks,” notes Jansons at Siemens. “Most devices, including a lot of PLCs, support it now. So, not only can they use diagnostics in their native control languages, but they also can leverage this information by sending it via Ethernet.”
This capability has great potential for the support services that machine builders and automation vendors are offering users of their equipment. “More than 80 percent of machine faults can be effectively diagnosed remotely,” estimates Kerry Sparks, senior field marketing specialist at Cleveland-based Eaton Corp. A technician at a remote location can either correct the problem by modifying the software on the PLC or human-machine interface (HMI), or alert local technicians to take the necessary corrective action.
“With good predictive modeling, estimates are that 30 percent to 50 percent of machine downtime can be prevented when the right people are notified of impending problems in a timely fashion through paging, e-mail or text messaging,” concludes Sparks. In these ways, checking a machine’s pulse from afar permits an early diagnosis that can cure problems before they have a chance to hamper a machine’s productivity and cause unscheduled downtime.
Subscribe to Automation World's RSS Feeds for Feature Articles
Leaders relevant to this article: