The PID example above illustrates the problem of validating machine learning algorithms where the number of inputs may be variable, the mathematical model is unknown, and the output includes a confidence factor. These new algorithms require a different approach to implicit testing. Developing a robust validation strategy should begin now.
A human Oracle will still be needed for the foreseeable future. But we can begin integrating the same advanced tools that have been used in other industries to address similar issues verifying the efficacy on machine learning models:
Metamorphic testing was developed to help mitigate the Oracle problem in application software testing. Grossly simplified, the methodology uses the relationship between a system’s output and multiple inputs to detect possible malfunctions. A possible application would be controlling pH when the volume of material to be controlled is known. If the system starts dumping excessive neutralization solution, it’s probably safe to assume something is wrong and we’re just making salt. Metamorphic testing, in this case, helps constrain testable values that cannot be precisely pre-defined in stochastic systems.
Cross Validation with an independent model is another possible tool. At its core, cross validation evaluates the output from two independent mathematical models of a system to determine if they are in agreement. The major drawback to this is that it doubles the work and processing required. But as our industry moves toward process modeling before equipment is built, this initial model may be used to validate the real-world process.
While not a solution to the Oracle problem, preparing now can help integrate and mold existing tools from other industries to progress implicit testing for validation. Machine learning models are being deployed now. The pursuit of advanced testing methodologies is a worthy aspirational goal for the automation industry.
Bill Mueller, is the founder and senior engineer of Lucid Automation and Security, an integrator member of the Control System Integrators Association (CSIA). For more information about Lucid Automation and Security, visit its profile on the Industrial Automation Exchange.