A few weeks ago, the latest cybersecurity ransomware attack—dubbed NotPetya or SortaPetya—affected many companies across the world, including some in the U.S. as well as one of our customers, a global leader in its industry with many facilities across the world. As the trusted system integrator partner for one of its facilities, we were called on to help with the recovery from this mess.
Upon arriving at the scene, we found that every Windows-based computer, virtual or local, was encrypted. Domain controllers, human-machine interface (HMI) servers and clients, historians and engineering workstations…all rendered useless.
Where to start? Paying the ransom and allowing the hackers to decrypt the compromised systems was not an option. The only choice was to rebuild each system either from scratch or, preferably, from a backup. We were about to find out how good—or not—the customer’s backup plan was. Most of the HMIs on the shop floor were easily recovered using a whole hard drive image. Some of the virtual servers were recovered with a backup. Not all of the backups were current; one was 2 years old.
Before systems were placed back on the shop floor network (SFN), each system was updated with the latest Windows patches and antivirus updates. Within two days, we had the critical systems back up and running. The automation and control systems were recovered and functioning within a week—even with the Fourth of July holiday in the midst of it. Other facilities within the company had not come even close to recovering and resuming production by then.
One of the keys to the automation systems’ quick recovery was the strong relationship built between integrator and (relatively new) customer. This partnership allowed us to gain knowledge of the customer’s industrial control systems (ICSs) and use that knowledge in its time of need. Our team understood the many control system applications that were in place and how they were being used, which made the learning curve a lot shorter.
Now what? What was learned during this crisis? The following security precautions are recommended to not just help prevent cybersecurity attacks, but to mitigate a variety of potential negative security events to your mission-critical systems:
- Backups! Ensure systems are backed up on a regular basis. Image all hard drives, back up virtual machines, and store configurations and programs on a non-Windows-based file server. No matter how secure you think your defense is, prevention is never 100 percent, so backups are key to a quick and painless recovery.
- Keep Windows-based systems up to date with the latest patches. In the ICS world, some applications don’t play nicely with every update, so it might not be possible to apply them all. This particular attack was patched by Microsoft update MS17-010, which was released on March 14. Disabling and/or removing the Server Message Block version 1 (SMBv1) networking protocol accomplishes the same action as the patch. If you have an app or hardware device that requires SMBv1, it’s probably time to ditch it. Too busy or don’t know how? Consider a service agreement with someone who does to help keep your data backed up and secure.
- Consider installing antivirus software. If it’s already installed, make sure it’s up to date. Only 16 out of 61 endpoint antivirus software packages were able to detect this particular ransomware, according to VirusTotal, so consider a non-signature-based antivirus package such as Cylance.
- Consider keeping your control system computers off the Internet. Sometimes that isn’t practical, especially with the emerging Industrial Internet of Things (IIoT). If the control systems are connected to the Internet, protect the SFN with a firewall. Don’t allow the use of ICS computers for personal Internet use or for checking email. Need remote access? Use a VPN for secure remote access.
- If possible, do not reuse passwords. It’s very common for all machines on the network to have the same local administrator password. How safe is that? Once a shared password is compromised (especially if it has administrator rights), an attack has the ability to propagate broadly. Although it might be inconvenient, consider using different local administrator passwords.
- Implement least-privilege administrative models. In other words, don’t put every domain user in the domain administrator group. Users should log on with a user account that has the absolute minimum permissions necessary to complete the current task and nothing more.
- Develop a relationship with a control system integrator. Another set of eyes that knows your automation systems inside and out is an asset. Though there are many integrators out there, finding an integrator certified by the Control System Integrators Association (CSIA) to partner with can be a great advantage. The CSIA certification provides an extra level of assurance that your integrator is operating a well-managed and stable business. Integrators with CSIA certification have been audited for their processes and adhere to best practices that reduce risk and contribute to successful partnerships with their clients.
Cyberattacks on ICSs are a nightmare, no question. Imagine going through the recovery process on your own; it’s probably your worst nightmare. In this particular case, the expertise of a CSIA-certified system integrator and the trusted partnership with its customer led to a quick and successful recovery of its mission-critical ICS—and a better night’s sleep for all.
Keith Mandachit is senior engineer at Huffman Engineering, a certified member of the Control System Integrators Association (CSIA). For more information about Huffman Engineering, visit its profile onThe Industrial Exchange.
