Complexity with enterprise architectures is keeping many IT directors and plant managers up at night as they wrestle with multiple platforms within a plant. These multiple layers of application platforms, often requiring proprietary protocols, have been added over the years as operations scaled through acquisition or organic growth.
Solutions for addressing this growth in plant complexity can include a unified automation strategy, industrial networking gateways and/or IT strategies. Pinellas County Utilities (PCU), which manages water and wastewater for the area around St. Petersburg, Fla., has focused on IT upgrades to deal with this continuous growth for more than decade.
PCU uses Stratus Technologies’ ftServers to run multiple application platforms for three water/wastewater plants and one reverse osmosis operation. On average, PCU supplies 66 million gallons of drinking water and treats 30 million gallons of wastewater at its facilities every day. The sprawling utility also includes more than 20,000 assets, 350 lift stations, five surface water sources, 13 well fields and 2,000 miles of pipe.
PCU enlisted the server technology in 2004 and bought six ftServer units to run individual applications tied to local control rooms. Since that time, plant modernization initiatives introduced a unified automation platform approach by way of Schneider Electric’s Wonderware platform, historians and human-machine interface (HMI) panels. In 2013, PCU purchased new servers to support the Wonderware applications and also introduced virtual machine architecture the same year.
“The utility had been using a cluster server approach up to that point, but realized the savings and efficiency potential that the new servers, partnered with Dell’s VMware, could provide,” says Mark Jankowski SCADA coordinator at PCU.
The consolidation of server technology and the move away from a cluster architecture allowed PCU to reduce licensing costs while still having critical redundancy for multiple historians and support for thin-client machines in four plants. The utility uses four Stratus ftServer 4500 systems to run a Windows server, the Wonderware platform, Microsoft products and the OPC-based KEPServerEX.
The technology houses two servers in one unit, and both perform the same instructions in real time for total redundancy. The design leverages field-programmable gate arrays (FPGAs) for cross-monitoring between servers and Intel chips for the precision communication.
“The two servers run in absolute lockstep, right down to the clock cycle,” says John Fryer, senior director of industry solutions for Stratus. “From an application perspective, the system just looks like a single server.” A firmware/middleware component for the system, called the automated uptime layer, executes about 500 checks across both server slices in the hardware system simultaneously.
“If one of the servers detects that there’s any problem—be it a hard failure, a disk, CPU or memory—then the unit will take itself out of service,” Fryer says. The water utility relies on Stratus to monitor the server 24/7/365 and to alert PCU of hard failures and part replacements. With its fault-tolerant architecture, the server system provides the ability to hot swap failed components during running operations and not have to restart. For PCU, the utility elects to have firmware updates done by a Stratus technician.
Downtime has not been an issue at the water/wastewater facilities, and the standard drivers that come with the system connect to the VMware operating system to help create a seamless architecture. The commissioning of the server project in 2013 took one week to update four water/wastewater plants.
According to Jankowski, the move to thin-client and fault-tolerant servers was an easy decision because of the success of the previous SCADA project and its ability to scale.
“PCU has more than 57 thin-client machines running across four servers at four separate locations, and we use various tools like VMware’s vSphere client to monitor and adjust the virtual machines,” Jankowski says. Over the past five years, “we haven’t changed the machine and have been trouble-free,” he adds. Rockwell Automation’s ACP ThinManager provides the centralized thin client and terminal server management software at PCU.
While these application-specific platforms add to the complexity, the utility is ready to handle more data. Some options moving forward could include bringing disparate historian data into one centralized area and then performing asset management analytics for better predictive operations and maintenance.
The complexity isn’t going away anytime soon at PCU, but the simplicity of many applications continuously working together is on schedule.
