All automation projects ideally follow a process that reflects the classic systems engineering V model, whereby a need is identified, which drives requirements. Requirements are analyzed and broken down so that they can be allocated to a functional solution. In this manner, the classic “How do you eat an elephant?” problem can be tackled with a “one bite at a time” strategy.
Once all requirements are allocated to an agreeable solution, the work to realize the solution can begin. Starting with the smallest subsystems identified in the analysis and allocation phases, the full solution comes together as subsystems are integrated to form a complete system that is not only built correctly, but provides a solution to the original need. Critical to the ultimate success of any automation project is appropriate testing as the system is being realized.
In the industrial automation world, two very common terms are factory acceptance testing (FAT) and site acceptance testing (SAT). The FAT is typically performed at the facility of the system provider, either by an OEM or a control system integrator. A properly performed FAT will provide a high level of assurance that the system has been developed as intended and that it will meet the requirements, allbeforethe system leaves the supplier’s shop. Issues identified at FAT are usually much easier and less expensive to address than if they are discovered later, during the highly stressful installation, commissioning and startup phases at the end user’s facility.
The SAT repeats most if not all of the testing performed during FAT, but with the system now installed in its operating environment and connected to real-world devices and interfaces. It is not always possible to simulate these real-world conditions during FAT, so SAT is a final test to further ensure no issues occur during startup and operation. In addition, SAT can identify issues such as incorrect installation or equipment that might have been damaged during shipping or storage.
A major process chemical manufacturer recently incurred a significant financial event when a large, VFD-driven motor driving a critical piece of newly commissioned process equipment experienced a motor bearing failure after only one month in service. Although the motor bearing temperature was being monitored by the control system and should have automatically tripped the motor prior to a catastrophic failure, the trip did not occur as designed. The system did present an indication of temperature to the operators, and a pre-trip alarm was correctly generated and acknowledged. Plant engineering was consulted, by phone, regarding the bearing temperature. However, the engineer believed the operator was referring to degrees Fahrenheit, when in fact the signal had been configured in degrees Celsius. Only after the motor bearing was observed to be smoking and sparking was the process manually tripped. The bearing was completely destroyed, thus making it impossible to determine the root cause of the bearing failure. The direct cost of recovery was significant, but the opportunity cost (lost sales revenue) was an order of magnitude higher.
Investigation revealed that the motor bearing temperature had been inadvertently ANDed with the temperature signal of the motor’s other bearing, which was functioning normally. This inadvertent 2oo2 voting scheme resulted in no trip. It was further determined that during both FAT and SAT, only the OTHER bearing trip logic had been tested. The OTHER bearing’s temperature trip logic was correct.
The project’s aggressive schedule, coupled with a lack of appropriate resources, contributed to a shortened, almost spot-check FAT and SAT. This drives home the importance of communicating to project management early and often the importance of planning adequate time and resources to conduct system testing. The copy-and-paste error that contributed to the process shutdown would have been caught and easily fixed at FAT had a 100 percent logic test been conducted.
Tom McGreevy, PE(SC), CAP, CFSE, PMP, is the southeast engineering manager at Applied Engineering Solutions Inc. (aeSolutions), a member of the Control System Integrators Association (CSIA). For more information about aeSolutions, visit its profile on the Industrial Automation Exchange.
