Operator Training Simulators to the Rescue

The stock-market crash of 2008-’09 may have delayed the expected exodus of retiring baby boomers from the ranks of process industry personnel. But for manufacturers—which depend upon experienced process control operators to keep plants running safely and efficiently—it may be only a temporary reprieve. “Once the economy comes back and people’s 201(k)s become 401(k)s again and retirement is once again an option, I think we’re going to see people leaving en masse,” predicts Jeff Hackney, DeltaV OTS global sales manager at Emerson Process Management, the Austin, Texas-based process automaton vendor. Indeed, by some estimates, process industry manufacturers stand to lose half or more of their process control talent to retirement during the next five to 10 years.Companies are exploring various ways to capture or replace that knowledge before it walks out the door. And while no single solution is likely to suffice, many agree that Operator Training Simulator (OTS) systems hold major promise as one method for getting new, inexperienced operators quickly up to speed. Speed learning“I can take an operator who has no control room operations experience and have him sitting, by himself, on the panel, within four months. And that would be very comfortable,” says Stan Hamilton, simulation training specialist at an oil-sands upgrading plant operated by Nexen Inc., in Fort McMurray, Alberta, Canada. That’s no small feat, given the complexity of the Nexen process, which combines proprietary OrCrude upgrading technology, hydrocracking and gasification to produce premium synthetic crude oil from bitumen extracted using Steam Assisted Gravity Drainage (SAGD) techniques. Nexen uses an Emerson DeltaV distributed control system (DCS) to control the plant process, which was simulated prior to start-up using OTS technology supplied by SimSci-Esscor, part of automation supplier Invensys Operations Management (IOM), based in Plano, Texas. “Up here in Canada, it’s getting harder and harder to get qualified people to come in who have had experience with control rooms,” says Hamilton. “So we’re starting to take people from other fields, such as pulp and paper, whose process backgrounds are very limited. Yes, they may have run with a DCS, but not with the complexity of the plants that we’re running,” he observes. Hamilton credits the OTS as a major factor in quickly bringing these operators up to speed. Each console operator undergoes 160 hours on the simulator during the initial four-month training period, and will spend a minimum of 40 additional hours per year in OTS refresher training, Hamilton says.Like most greenfield projects these days, the simulator was built well ahead of the actual Nexen plant production start-up early this year. The OTS contract was awarded in 2005, and the SimSci-built, high-fidelity, dynamic simulation model required about two years to complete. In all, the OTS system incorporates “thousands” of input/output (I/O) points and came in at a price point of around $3 million.Prior to plant start-up, “we used the simulation to prove out our DCS configuration,” Hamilton notes, along with the control configuration for “a very involved SIS (safety integrated system) for the gas supplier.” OTS training of operators was also begun well ahead of plant start-up, so that operators could hit the ground running when the plant came online. The result, Hamilton believes, was a faster, smoother and problem-free plant start-up. “We feel that because we had this simulated, and because all our operations staff have gone through the training, we have probably eliminated any issues related to the start-up of new complex units,” he notes. “We haven’t had any fires, we haven’t had any operator errors...”Accident avoidanceOTS benefits also extend well beyond plant start-ups to help keep plants running smoothly and more productively once they are up. A strong OTS program helps eliminate errors by even experienced operators that can lead to accidents, product loss or costly plant shutdowns. Continuing refresher training can help keep operator skills up to date for infrequently performed tasks such as start-ups and shutdowns, and also prepare them for the unexpected. “One of the big things, of course, is that you train your operators in safety scenarios and upset conditions that you hope will never happen in your plant,” says Taher Aftab, senior engineer for dynamic simulation strategy at BP Chemicals, in Naperville, Ill. Simulator training is also particularly useful for practicing plant turnarounds—which today are performed less frequently than in the past—as a way to keep operator skills from getting rusty, Aftab adds.Aftab oversees OTS activities at a half dozen BP plants worldwide that produce paraxlene and purified terephthalic acid (PTA) used in making clothing and plastic bottles. These include three U.S. plants as well as plants in Belgium, China and Malaysia. BP Chemicals’ OTS program dates to the mid-1990s, says Aftab, and the technology is used to varying degrees at all six plants today. Five out of six units that make up the company’s plant in Decatur, Ala., have OTS systems that are used regularly for operator refresher training and certification, for example. Other plants use simulation not only for training, but also for control configuration check-out and optimization purposes when they make DCS or SIS modifications, for instance, or when they undertake major debottlenecking projects.“We have traditionally had the Honeywell TPS control system, and we are in the process of replacing that in various plants, so as the new DCS systems come in, the operators are trained using OTS on the new graphics and the new controls,” Aftab adds. Upgrade selections vary by plant, with some units so far going with Honeywell Experion control technology and others choosing Emerson DeltaV. OTS choices include Honeywell’s UniSim line or others with high-fidelity simulation capabilities, Aftab says.Quick savingsOTS benefits can be hard to quantify. “Unfortunately, the benefits are not tangible. You cannot really say that if you had not done simulator training that you would have been hit by something really disastrous,” Aftab observes.But some vendor representatives, such as Peter Henderson, product manager, simulation, for Phoenix-based supplier Honeywell Process Solutions, are quick to cite general numbers based on process production economics. Say a 50,000 barrels/day refinery operates on margins of $10 per barrel. If that refinery is able to cut a single day off the start-up time of the unit thanks to better OTS operator preparedness, that’s a savings of $500,000, Henderson points out. “If I can shorten a start-up, or if the operator can respond to an alarm before it becomes a trip, then it’s very quickly that you recognize the benefits of preparedness for operation,” Henderson relates.Emerson’s Hackney also cites regulatory compliance among OTS benefits for users in industries such as pharmaceuticals, food and beverage, and life sciences, who must conform to tight training regulations. Improved safety is another “huge” area of OTS benefits, Hackney says. “We’ve all seen tragedies over the years, and operator training will help reduce those types of incidents.”Engineering tooMartin Berutti, business director at Mynah Technologies, a Chesterfield, Mo.-based supplier of software for simulation model building, can cite a similar list of OTS benefits. Manufacturers are definitely looking toward simulation as one tool to help defuse today’s “demographic time bomb,” he says, by facilitating knowledge transfer from soon-to-retire baby boomers to a younger generation of operators. And Berutti adds that the best value of simulation comes through an “integrated approach” that involves not only operator training, but upfront testing of control configurations and other engineering uses. Tom Fiske, Ph.D., agrees. “The aging workforce and the influx of inexperienced workers” was one of the major factors that was driving “booming growth” in the OTS market before the current recession put a damper on simulation sales, says Fiske, a senior analyst at ARC Advisory Group Inc., in Dedham, Mass. Fiske expects that rapid growth to resume once the economic recovery hits stride. But he also believes that manufacturing users must look beyond just the training aspects of the technology. “I see the operator training simulator being an integral part of an operating facility, not just to train, but to do ‘what if’ scenarios, troubleshooting, engineering studies and continuous improvement,” Fiske says. In order capture all of that value, however, it’s imperative that OTS users make the investment to keep simulation models up to date, say industry sources. This means that when changes are made in plant equipment or control configurations, corresponding changes must be made in the simulation model.“Most companies will bring simulation online when they’re bringing up a brand new plant, and they’ll train all their people in simulation, but they don’t put the money into simulation to keep it updated,” says Nexen’s Hamilton. That not only dilutes the value of dynamic simulation training—which may no longer accurately reflect process responses to operator actions—but it also limits engineering uses of the system. At Nexen, says Hamilton, “our management has made the commitment that the simulation will be up to date.” As a result, he notes, the company is able to confidently test all control narrative changes on the simulator and make necessary tweaks and adjustments prior to live system installation. Sometimes, however, economic realities can get in the way. BP Chemical’s Aftab notes that his company launched a program about three years ago to upgrade simulation models across all of its plants. But budgetary constraints spawned by the economic downturn have slowed down that program, he says, limiting upgrades currently to just one model every six months.“The honest truth about operator training simulators is that they require a very serious commitment by a company’s senior management,” Aftab observes. “You not only need an instructor full time, but you have to take operators out of shift to train them. And then you need someone from the DCS group to keep good track of the management-of-change process so that can be brought into the simulator. And you need someone either from your own organization, or you need to outsource the actual effort of updating the simulator.”OTS pricingThe initial investment in an OTS can range anywhere from $20,000 for a generic, low-fidelity model used for basic process familiarization up to several million dollars for a large-scope, high-fidelity system customized to provide dynamic simulation for all or large sections of a specific process plant. As a way to encourage use of simulation technology, Emerson provides an entry-level system priced at $135,000 known as DeltaV OTS Express that uses MiMiC model technology provided by Mynah under a partnership arrangement. Suitable for use in limited-scope simulations—such as one column in a refining operation or a boiler in a utility application, for example—the Express package includes a virtual configuration controller, a process-specific dynamic model, customized courseware with workshops, five simple scenarios, a random event generator and scenario scoring, along with a DeltaV Simulate Pro standalone license and five eLearning seats, among other components. “It’s a true, complete operator-training solution,” says Hackney, and it is easily upgradeable to take on larger simulation tasks.As computing power has increased over the years, simulation systems are becoming more capable. “You used to need a whole room full of PCs (personal computers) to run a simulation of a cat cracker, but we can do it all now in a $5,000 high-end workstation,” observes Mynah’s Berutti. “I can buy an eight-core PC from Dell—because we’ve added multi-core capabilities to our MiMiC package—and we can offload different parts of it across different cores and different processors and just get incredible results.”Honeywell’s Henderson notes a similar trend, with users moving toward simulation of multiple process units per PC “to the point where you start to build toward total-plant simulation.” But Henderson points out too that PCs are generally the lowest-cost component of an OTS system. The process model typically accounts for about 90 percent of the value in an OTS, he says, with the simulation software and other deliverables as the remaining 10 percent. “So if I needed 30 PCs last year, and now this year, I only need five, I haven’t changed the overall cost of the system by all that much.”Team trainingNonetheless, technology advances are driving new uses and capabilities for process simulators. “With the increased computing horsepower, the coverage of the models can be more comprehensive and include more of the plant,” says ARC’s Fiske. “You can include different elements. You can train multiple operators. You can use a team-based approach with coordinated team exercises, with indoor operators and outdoor operators coordinating activities.”Indeed, the ability to train control room operators, field operators and maintenance personnel together is one of the major benefits touted for a new virtual reality (VR)-based system launched commercially in September by Invensys Operations Management. The company’s EyeSim system creates a computer-generated, photorealistic 3D representation of a real or proposed plant to enable trainees wearing stereoscopic headsets, or goggles, to enter a completely immersive environment in which they can move freely throughout the virtual plant, seeing exactly what they would see in a real plant. Using a game pad, they can interact with objects in the plant environment, opening and closing valves, for example. Trainees also hear appropriate sounds around them as equipment turns on and off. The experience has a video game-like feel that is certain to seem familiar to many new recruits and younger employees.Built using computer-added design (CAD) drawings, photos and other available plant information, the 3D virtual reality model is driven by IOM’s Dynsim high fidelity process simulator and other programs, and is integrated with the plant’s OTS dynamic model. In a training scenario, control room trainees may be interacting with the OTS through traditional displays and monitors in one room, while field operator trainees are in another room known as the “VR cave,” explains Maurizio Rovaglio, Ph.D., IOM head of innovation and emerging technologies. The VR cave room contains large screens on at least three of the walls to create the immersive environment. Field operator trainees in various parts of the room are interacting with different parts of the plant and can talk with each other and with the control room trainees via two-way radio. An instructor in a third room can follow what the control room operators are doing, and can also track the activities of field operators moving through the virtual plant via an overhead camera-like view.As with conventional OTS training, one major value of VR-based immersive training simulator (ITS) technology comes through the ability to prepare operators prior to new plant start-ups, says Rovaglio. Field operators and maintenance personnel can be hired ahead of time, and don’t have to wait until a plant is actually built to “walk through” the plant to learn and practice their jobs. Plant start-ups typically require coordination between control room operators and field plant personnel, while communication between the control room and outdoor workers can also be crucial in emergency scenarios, which can also be simulated.Twice the priceBased on early work with customers, IOM estimates that VR-based training can cut 20 percent to 25 percent off plant start-up times, says Rovaglio, while producing a 3 percent savings for a typical refinery maintenance budget. The price of an EyeSim system will fall “in the same order of magnitude” as that of an OTS system, he notes, though if both systems are developed in parallel for a plant, the resulting synergies might somewhat reduce the overall price. If an OTS system for a plant costs $1 million, for example, the tab for a combined OTS/EyeSim capability might come in at $1.8 million, Rovaglio suggests. Not surprisingly, perhaps, some IOM competitors question the value of the virtual-reality approach. “Our read is that our customers are really looking for training systems that are easy to maintain and implement and that give good value for the training environment,” says John Caldwell, Emerson’s DeltaV product marketing manager for advanced control and simulation products. “We don’t see the virtualization of a plant as a necessary requirement, and the maintenance costs seem to be very high in that area.”But ARC’s Fiske, for one, predicts that other vendors will soon be rolling out VR-based systems of their own. “In the future, I think it will be an option that everybody offers,” he says. VR technology will have particular appeal to younger workers, Fiske notes, “and if you want to attract the new workforce, you’re going to have to be commensurate with the technology that they’re used to.” When considered against the backdrop of some of the latest advances in interactive gaming technology, in fact, the EyeSim-type technology for industrial use is just “the tip of the iceberg,” Fiske adds.Rovaglio confirms that IOM has bigger plans for the technology. While the company is promoting EyeSim initially for operator training, it sees expanding uses for VR technology down the road. “Looking to the future, we have the vision that this tool can become the new generation of interface with the plant,” says Rovaglio. “Why should the control room of the future be flat?”Related Sidebar - DOE Taps Virtual RealityTo read the article accompanying this story, go towww.automationworld.com/feature-6313. 

Subscribe to Automation World's RSS Feeds for Feature Articles