How IT System Uptime Yields Huge Cost Savings

A large pharmaceutical manufacturer in the European Union develops new designs, tests raw materials and produces, inspects, packages and ships its products worldwide from a single plant location. The company’s vision is to be the global leader in innovation and manufacturing of its products. In keeping with that vision, they recently solved a production line bottleneck using Cogent DataHub software.

Due to a recent change to security enforcement in Microsoft Windows, minor disruptions in the manufacturer’s control system that used to take a few seconds to resolve were shutting down entire production lines for up to five hours. 

“It was a lot of cost,” said Stephen Doody, plant automation team leader with the pharmaceutical manufacturer. “Not only were we losing manufacturing time, but the knock-on effect meant that half of our control staff was tied up for another five hours to get the line back running.”

DCOM security issues

This problem is typical of the ongoing challenge in the automation industry to maintain a stable, working system within a changing hardware and software environment. 

In this case, the stable system is a vision inspection system installed more than 15 years ago at significant cost. The change was an initiative by Microsoft to raise security standards for data networking in Windows software.

At the plant, IT systems are deeply integrated with the manufacturing process. The vision inspection systems, for example, have a direct, real-time connection to the main controller using OPC DA, an industrial protocol based on Windows COM technology.

Networking OPC DA requires DCOM, whose security settings are complicated to configure. To run the system efficiently, the plant automation team had minimized DCOM security. However, with the mandatory application of the Windows security patch from Microsoft, only the two highest levels of DCOM authorization are permitted. Configuring and enabling these higher settings created problems. 

The plant also had other security requirements that made recovery from any OPC disruption extremely difficult and time-consuming, causing multi-hour production delays.

A new approach: Tunneling

What Doody needed was a different way to connect the OPC clients on the PCs running the vision control systems with the OPC server on the main controller. A web search on DCOM issues brought him to the concept of OPC tunneling and Cogent DataHub software.

“While investigating tunneling I found the Cogent DataHub website, read the case studies and support testimonials, and learned how the application was easy to integrate into systems like ours,” Doody said. “It looked like once [it was] implemented, I could then hand it off to our production support group and they could take it over without me having to supervise.”

For a trial, Doody configured a tunnel connection on one production line between the main controller PC and four vision system PCs. Originally, security for both the vision system PCs and the main controller PC were configured on the plant LAN. All logins were done through the Active Directory domain controller and had to be identical.

With the tunnel, the DataHub instance on the main controller connects to the OPC server using the normal login credentials. The DataHub instance on each vision system PC makes a tunneling connection to the DataHub instance on the main controller and receives the data. Each of those DataHub instances is configured as an OPC DA server, allowing the vision system to connect as an OPC client.

Now, because the OPC client on each vision control system connects to a local DataHub instance, Doody has been able to remove those PCs from the plant LAN. He no longer needs to enforce user logins with the domain controller. Each user logs in independently of the main control login. This means that any irregularity or dropped connection no longer requires re-synchronizing security logins across multiple machines.

After a week of testing the first system, Doody felt confident to implement the solution on the remaining three lines. Now all connections are using DataHub tunneling and the benefits are clear.

“Before, if an OPC connection dropped, it could take anywhere from one to five hours to get the PCs that were off the domain back on and signed up and authenticated with the user account, initialize the network cards, connect to the plant LAN and line PC, connect to OPC, calibrate the application and more,” said Doody.

“Now I’m getting no reports of any machine down time,” he continued. “Our application is even faster connecting to the local instance of the OPC server. Even before the DCOM hardening stuff, we would get intermittent dropouts between our clients and servers. The only error message we’d get would be: ‘Check OPC client installed.’ No detail whatsoever and very little error logging.

“DataHub tunneling makes it so much easier to debug why the connection is failing. So far, we’ve had one instance where OPC failed to connect on our application. Looking at the DataHub Data Browser we could see that it was because the tunneling application wasn’t running on our server side,” he said. “Before, we would have been shrugging our shoulders and going through numerous reboots of the PC waiting and hoping that it would re-establish this connection. And this [connection failure instance] didn’t require a call out to an engineer to investigate the problem. One of our process support guys was able to diagnose and fix the problem. That’s another big win for us.”

Doody added that the “whole solution was pretty much a no-brainer for management and easy to sell to the financial controllers because it was saving a heap of money, rather than upgrading the lines. Engineering was brought in as well, because it was easy to implement and maintain. And then the proof of concept, getting test licenses easily and reading the how-to guides to implement it on our systems—it was all fantastic and seamless for our work.”

Xavier Mesrobian is vice president of sales and marketing at Skkynet Cloud Systems Inc.