The connectivity of plant floor operations technologies (OT) to the enterprise network has been an area of concern for manufacturers for more than two decades now since Ethernet’s initial forays into the manufacturing environment. Though these concerns about wide plant floor connectivity are not unfounded, the reality is that modern business requires this high-level of connectivity to meet partner and customer demands.
And while the OT and IT areas of industry have clearly grown more intertwined over the years, it’s not uncommon to encounter instances of ongoing IT/OT disputes.
Possibly the biggest indicator of the inevitable merger between IT and OT is the growing number of consumer-focused IT professionals now focusing on OT. As a case in point, Automation World recently highlighted Chef Robotics and its artificial intelligence-enabled robotic system designed for the food manufacturing industry. In our discussion with CEO Rajat Bhageria, he noted how Chef Robotics has focused on hiring engineers from consumer software companies to ensure its technology has an easy-to-use interface.
“We designed it this way because a lot of Chef users will be non-technical and often not native English speakers. We want Chef to be aligned with the workers it works alongside of, like it's a member of the staff,” he said.
Another example of this can be seen at Copia, which creates DevOps tools for controls and automation engineers. Adam Gluck helped found Copia to accelerate the digital revolution in U.S. manufacturing. Gluck worked on Uber's driver engineering team and became the founding engineer on Uber's engineering strategy team. With Copia, he looks to bring the same best practices that top-tier engineering organizations use to the manufacturing industries.
Filling a gap
A key aspect of Copia’s Git-based version control software is its design to streamline workflows so that engineers can spend less time searching for files, reviewing code and resolving issues. When asked why an engineer working in the manufacturing industry would want to use Copia’s software rather than that provided by a PLC’s manufacturer, Gluck said, “There's a lot of code in industrial environments where people make changes and sometimes, they break things. Copia offers code backup and change detection, which provides a set of interventions that makes this much less likely to happen and enables faster disaster recovery. Companies choose our solution because it provides visibility to unauthorized changes. Existing vendors don't offer this solution at all.”
He added that there are some source control and Git integrations within vendor solutions, but “there isn’t a full cloud-based, Git solution with web browser-based code review that enables these kinds of processes.”
Another industrial software gap that Gluck sees Copia filling comes via the company’s multi-vendor support which enables users to “have a standardized workflow across different industrial vendors,” he said. “That enables managers to treat the factory’s code layer even though there are disparate process environments among different vendors.”
Gluck noted the importance of this ability to “treat the factory’s code layer” as there are many standard processes already in place for making such changes in the IT environment. “For example, you can use Git when you make changes to go back to a previous version if you break something, and you have traceability to understand who changed what code and why,” he explained. “These processes were completely missing from code in industrial environments. I spoke to people in industrial spaces asking why these IT processes were not in place for OT and no one knew why despite being aware of them. It was a pretty obvious and immediate pain point that we knew we could address.”
Securing the plant floor
The most significant differences between IT and OT systems are the real-time communication requirements of key OT systems and the high cost of production system downtime. Beyond these key factors, the security requirements are not inseparable.
Gluck said there are also critical components of IT security that OT have yet to widely implement. He cited code review as an example of this.
“I've talked to site CIOs who have taken over OT cybersecurity, and it’s shocking what processes are missing,” Gluck said. “We've run into people literally having all their code on a laptop or a USB stick, and then losing them and all the backup code.”
Though there has been a focus on the networking layer to secure plant operations, Gluck suggested that OT professionals “need to move a layer up and pay close attention to what is happening with their code in the factory itself.”
He explained that Copia focuses on this area to provide manufacturers with “complete control and visibility over their code, enabling them to quickly recover in the event of a disaster. This includes ransomware attacks where companies are locked out of their machines. With the code automatically backed up to the cloud, disaster recovery becomes a proactive process where disasters are prevented before they occur.
“At the end of the day, the core problem is similar for IT and OT in that code is created and then things get broken,” said Gluck. That’s why he recommends use of Git-based source control in industry to ensure that manufacturers have a version log of all code changes in case something goes wrong.
He also recommends that manufacturers implement automated testing. “There are a number of solutions in the industrial space, such as a digital twins, which can be incorporated into the workflow,” he explained. “Having a system that enables visibility into the changes in the coding environment is key. This allows catching unauthorized changes and preventing them from happening automatically. All of these processes help ensure that changes to code are tracked and can be reviewed so there are no unauthorized changes and disaster recovery can happen quickly and easily.”
OT inclusion at the C-level Part of Gluck’s message to industry is that CIOs of manufacturing organizations need to shift from IT-focused roles to include manufacturing functions as well.
Key to doing this effectively, has said, is through the use of “blameless postmortems.”
“We have seen organizations that work really hard to implement top-down initiatives but the plant level employees just don't want to adopt the processes,” Gluck explained. “Often these projects fail because they are trying to implement something that's so nebulous or high level that employees on the floor don't understand the value of the initiative.”
Remedying this requires CIOs to “get low in the stack” to really understand what's happening at the ground level in their plants and the pain points of the organization as a whole.
“When we visit companies, there’s a big focus on outages but there's also a cultural difference in the OT space compared to the IT space in which CIOs can play an important role,” Gluck said. “CIOs should introduce blameless postmortems when there’s an outage. If something breaks, they should ask how the process can be improved. We run into a lot of organizations that are very blame-oriented, which results in employees making changes and then trying to hide them. Someone asks, ‘Who caused this?’ and employees feel like they're going to get in trouble as opposed to having effective conversations to build a set of processes to prevent problems. These cultural changes can be really valuable, and it also makes the quality of life much better for engineers.”
Even as manufacturers merge key functions of OT and IT and companies like Copia work to intertwine the best practices of the two, Gluck doesn’t exactly foresee a day when the two merge entirely.
“There are aspects of the OT space which are hyper operational in which CIOs won’t be deeply involved,” he said. “But if you look at the industrial space of 20 years ago, they were much more mechanical and physical with less code involved. We have seen a dramatic rise in the number of PLCs, robotics and sensors, making these environments more digital but they are still highly physical spaces. Because of this, I wouldn't be surprised if certain aspects of industry will continue to exist under operations or engineering management. If the [industrial] stack becomes very digital, we might see more responsibilities falling under CIOs.”
