Manufacturing is on the rise in the U.S. For the past two or three decades, it has been in decline, with plant shuttering caused by company cutbacks, as well as consolidation of companies, plants and operations. There have been private equity buyouts as well. To increase profits and return for investors, operating and maintenance budgets and staffing have been systematically cut.
One area of maintenance that has routinely been ignored is the plant distributed control system (DCS) or programmable logic controller (PLC) system. The neglect of these systems is commonplace, and it has often proven to be an expensive oversight. There are a variety of reasons that this happens. In some cases, it is simply because those running the plant are unaware that it is necessary. In other cases, owners or managers find it difficult to quantify the ROI they might get from the upgrade when they really don’t understand what that ROI might be. In most cases, the assumption is made that the new system will provide the same functionality as the old one, but in a more reliable fashion.
Not enough attention is given to the benefits of state-of-the-art systems with their powerful analytics and functionality. It is difficult to quantify the increases in quality or productivity that might result when a 20-year-old system—or one even older—is replaced with today’s technology. For a plant to run effectively, it is important that the patches, bug fixes and security changes be put in place when they are provided by the manufacturer. Systems cannot run effectively if left unattended.
A simple analogy is automobile maintenance. Even the best cars, with the best maintenance records, still need oil changes, and new tires and brakes periodically. And they should be returned to the dealer for recall repairs. We commonly tell clients that a control system’s life is about 15 years. This often gets shocked looks or comments. Clients often consider “upgrades” unnecessary. When you buy a car, you don’t consider an oil change an upgrade. The neglected car runs fine—until it doesn’t.
Perhaps in many cases it is a misnomer to call these actions system upgrades. They should simply be called control system preventive maintenance. These are not done for featured improvements, they are changes to prevent problems—problems like security patches, memory leaks, error fixes and blue-screen-of-death lockups. Some of these things many users might never see, while others are plagued; you never know which group you’ll be in until it is too late.
We all know that neglecting maintenance on our cars will shorten operating life. If you don’t change the oil and don’t check the level, the engine is sure to fail. But similar basic truths about factory maintenance are often ignored by owners hundreds or thousands of miles away. They are oblivious to the needs of their plants, and they have cut maintenance staffing so deeply that often no one on their staff is capable of doing what needs done.
Beyond simple maintenance is the question of system upgrades or replacements. There is significant cost associated with these for both software and hardware. Because of the cost, it can be difficult getting these upgrades approved. For many, the change is viewed as only risk reduction, not actual ROI. Perhaps by tying these upgrades to overall equipment effectiveness (OEE) and other analytics made possible by improved networking and visibility, a stronger ROI case can be made.
In recent conversations with automation engineers at Optimation, I’ve heard many stories about failures that could have been prevented had the appropriate system maintenance, upgrades or replacements taken place. These failures took place at inconvenient times and caused long work hours and unnecessary stress, all of which could have been avoided.
A food company called us to get a food plant back online after a system failure. While the system was down, production stopped. Food in process was aging, with risk of spoiling unless the system functionality was restored quickly. Managers were calling every hour to see when the plant was going to run. The stress and pressure were intense for both the idled operations management and for our engineers. The plant management had been unwilling to spend the money to do appropriate maintenance or the upgrades they had talked about for several years. The costs of the resulting production losses, added maintenance and rushed upgrade were high.
In another case, a roofing company had been using a system for years whose hardware and software were both obsolete. Custom code written for screen changes locked up the screens because of insufficient memory. As printed circuit boards died, they would try to get refurbished ones to work in their place. Though this was effective most of the time, it also took time and often production was lost for days. During slow times at the plant, there was no money for upgrades. During busy times, there was no downtime available for upgrades.
For a highway tunnel that opened for operation in 1995, redundant PLCs were installed at both ends. Eventually, the redundant cards began to fail, and replacements were no longer available. They are now buying multiples on eBay and hope at least some will function properly. We’ve quoted upgrades and replacement systems to them now for several years, but they have never considered it a priority. Other operational costs have always been the priority, and there was never cash left to use on the control system upgrades. Eventually, a complete failure of these systems will force an expensive rushed replacement and upgrade at a premium cost.
Corporate buyouts do not help in this situation. With companies changing owners, there is no reason to sink capital into unseen equipment when it will not positively impact the sale price. Clearly, the situation as viewed by the Optimation control system staff was not an optimistic one. I reached out to leaders of other member companies of the Control System Integrators Association (CSIA). I asked first if they were experiencing the same issue. I also asked if they had any ideas how we might be able to collectively address the problem in a positive way. I got answers to both questions.
“It’s an issue that bedevils us as well as our customers,” said Wright Sullivan, president of A&E Engineering. “When they suffer extensive downtime caused by failure of obsolete equipment, there is usually a very small number of people who can work on that equipment at all. When it comes at an unexpected time, this drains critical resources. For the customer, they are placing themselves at risk that with the plant down they will fail to meet delivery for their own customers.”
Sam Hoff, CEO of Patti Engineering, agreed on the magnitude of the problem. “It is amazing the ancient stuff I see at some manufacturers,” he said. “When they finally do an upgrade, it can be more expensive and cause more downtime than if they had just been gradually upgrading.”
Rick Pierro, co-founder and owner of Superior Controls, reflected on the concern: “A common occurrence is when the customer has a catastrophic failure of a 12-year-old database, MMI or PLC. They suddenly find there is no longer available hardware of software to repair the old equipment,” he said. “In this event, they are down for a much longer period while they frantically hire an SI to figure out the original scope and rebuild and test a new system. We have seen this happen many times. The increased cost to the customer due to unexpected and prolonged downtime can be enormous. We’ve seen it permanently reduce the customers’ sales and market share. Their customers choose another vendor and stay with them.”
It is commonplace for customers to drag their feet far too long, agreed PC Romano, president of Avid Solutions.
What could we at the CSIA do to constructively help clients avoid these problems? Sullivan suggested collecting and sharing stories of expensive failures and how they could be avoided might be helpful. “What if we had a Lessons Learned From Touching a Hot Stove—Legacy Automation Edition? This could gather some nightmare stories from CSIA integrators to share both the pain of not upgrading, as well as some of the benefits reaped by those who got out in front of this,” he said. An anthology could be created of things that have gone wrong and how much they cost. This might motivate some companies to move more effectively and avoid these problems.
Romano took it a step further. “CSIA’s vision and unique opportunity is to be the center of the SI universe,” he said. “I know that our automation suppliers spend a lot of time and money encouraging customers to update and upgrade, so how can we leverage this in a consolidated way? What if we were to develop some kind of center of information/portal on the real and opportunity costs based on failure to update/upgrade?”
Romano also noted the increased cybersecurity risks associated with not upgrading. CSIA could help to identify those. He suggested we might incorporate the National Cybersecurity & Communications Integration Center (NCCIC) data in an analysis we did for clients needing upgrades.
One aspect customers often overlook in an upgrade is the improvement in performance they will get out of their system, Hoff added. “Because of the speed of the modern-day processors compared to 20-30 years ago, the system will react to inputs faster,” he said. “We have had customers break production records within the first month of an upgrade.”
There is a lot to think about here and we have only scratched the surface. Perhaps a group at the CSIA can take this on and use some of these ideas to move this forward. A standard developed to audit a control system and suggest a cost-effective path forward would be a great value to many. We could help point out the risks, make it clear that savings made by neglect might be short term and false, evaluate potential production losses caused by extended downtime, and help clients consider what improved performance, increases in production and quality might be obtained.
Bill Pollock is president and CEO of Optimation Technology Inc., a certified member of the Control System Integrators Association (CSIA). For more information about Optimation, visit its profile on the Industrial Automation Exchange.
