Like many in the field of information technology (IT), I find that technology shifts before our eyes at a nosebleed pace. Technologies come and go. Some technologies are replaced quickly, while others seem to hang around for the long haul. It is important for us to stay relevant and knowledgeable about new technologies.
At each shift in my career—from support specialist on a global banks stock trading network to network and security trainer and then virtualized datacenter designing engineer—I spent some time thinking about my role and how I could best contribute, often in ways that are not confined to the title of the role. I would like to think that my hard work has led me to where I am today, as part of an information solutions team at Avid Solutions. At this point with Avid, I’m asking about the next thing that I should be learning, which led me right to the world of industrial control systems (ICSs).
I’m new to the ICS. But, contrary to popular belief, being new doesn’t mean being unproductive in a new industry. This is the busiest I’ve been in my entire career, and I’m enjoying it in ways that I didn’t expect. I firmly believe that, in this role, I can help our customers’ business IT organization to better understand and support the operations technology (OT) design and requirements. Customers in the industrial space need a translator between their IT and OT organizations.
Coming from the business IT side of things, I’ve seen how many of the IT organizations react to individuals or groups trying to run their own networks and servers. IT often feels that they are the de facto experts and should own all aspects of the server’s operation. Sometimes it’s a lack of trust by the IT group: They worry that a new server that they have not personally built is not secured properly or will become an unmanaged ad hoc rogue device that will become forgotten about until it becomes a problem.
There are also IT organizations that operate with a bit more hostility toward others. These exist and operate through resource power-grab primarily for reasons of fear or to justify their existence. In this situation, IT might resent the firewall team and vice versa. Private cloud virtualization teams don’t trust public cloud initiatives and cybersecurity trusts neither.
Where does this leave OT? In my short time at Avid, it’s become painfully apparent that OT is misunderstood. IT wants to run the OT networks the same way it runs the business networks. People in IT wonder if it’s possible to run the network on behalf of OT. For my part, I think it’s a completely plausible expectation, but I have not seen it work—yet.
An IT network is a relatively available infrastructure. But there is an argument that the highly available (HA) network infrastructure that has been built will failover and the client server stream will continue. I agree, but is it OT quality? IT-managed networks don’t conform to the needs of OT in many cases. OT needs the network to be up all the time. Rebooting a router to fix a problem is often not an option. If there is not a 30-250 ms failover between redundant paths, the network is not good enough. In a shared network between IT and OT, an employee’s large file transfer of an ISO file could bottleneck and impede OT telemetry. Of course, all efforts must be made to prevent this.
Similar arguments can be made for virtualization. Could an IT virtualization group run the OT virtual machines (VMs)? Sure, it could. But first, it’s important to understand that the cornerstone of IT virtualization in an enterprise is to put lots and lots of VMs on one or a few pieces of hardware to optimize resource sharing. This doesn’t work for OT without defining guaranteed uptime, reserved memory and CPU, real-time hitless failover, protection for OS changes (including not patching), and being secured from hostile business and Internet networks. Lastly, placing OT’s VMs on a distant IT network could lead to major problems if the real-time telemetry traffic from the controls network are dropped.
IT security compliance is also not directly compatible with OT at the lower levels of manufacturing. These systems are operating in real time, collecting records, controlling equipment and informing engineers of status on a human-machine interface (HMI). Interrupting this flow of traffic, even by accident, could damage property or, worse, it could harm the humans who are operating the system. Applying a patch on a live ICS network could impact data collection, controls and regulatory compliance. There is a need for IT to understand that it could be a year or even a decade before a plant is offline for maintenance and patching. The focus needs to be one of ensuring that only secured, authorized access is permitted from the outside, since the inside OT network needs to remain in a steady state for long periods of time.
In other words, it’s important for OT to explain its needs better to get the other groups to listen.
The National Institute of Science and Technology (NIST) has an excellent publication that defines the core differences between IT and OT. Reviewing this publication led me to imagine a person standing between the IT and ICS columns. This was my “aha” moment and where I discovered that I can uniquely contribute by helping customers in IT to stop, step back for a moment and review what they are attempting to take on. They often need to decide if it is within their scope to try and be successful at running an OT infrastructure or if a dedicated group would be best suited to do the job. If they are already operating the OT network and it’s not working well, a systems integrator can assist in identifying the changes needed to make the endeavor a success.
Another quick note about the NIST 800-82 publication: If you look at diagrams 5-1 through 5-5, there is a common control server that remains near the control network. In some of the designs today, this particular server has been moved out to a virtualization infrastructure.
My thought is that its importance is being overlooked since as a server, it’s being moved to the IT virtualization in the corporate network. This object server is the first in the line of real-time telemetry collection for the manufacturing processes. Consider what happens if this is moved across a best-effort network and multiple routers and switches.
Let’s take a look at how the disconnect formed between IT and OT.
For those unaware or not familiar with ICS requirements, the whole concept of the OT requirements seems backward compared with the rapidly shifting enterprise and consumer technologies. Someone familiar with IT networking and security best practices at the enterprise level will rightfully get agitated at how crazy OT sounds, at least until we explain why OT is this way and it’s not easy to change without breaking the manufacturing functionality.
First, they need to get past the gut reaction, ask questions and, most importantly, listen. Not listening can result in a tough situation where (as in one real customer case) an IT team thought it knew better than those providing the site requirements. They didn’t apply the stated requirements and it resulted in dropped traffic across a shared business/production network. Also, they incorrectly defined VMs on a shared infrastructure. The IT department is expert at enterprise best effort and shared resource design, but it just didn’t work for the ICS and had to be heavily modified.
So, my shared words of wisdom today for all IT and OT staff: Don’t assume you are more knowledgeable than the other. Listen. In other words, “The quieter you become, the more you will hear.”
For our customers, having a seasoned information solutions team as part of the solution integration process can have unexpected value, including clarification of any requirements early during the design process, and a set of educated eyes to identify any potential concerns that could cause issues in the final product.
Greg Guilmette is an information solutions consultant at Avid Solutions Inc., a certified member of the Control System Integrators Association (CSIA). For more information about Avid Solutions, visit its profile on the Industrial Automation Exchange.
