Cutting the Cost of Compliance with Connected Systems

The path that crude oil takes from extraction to conversion to consumption is long and carefully managed. For financial and environmental safety reasons, every step of the distribution process has to be monitored and controlled in a process called custody transfer.

Primary distribution terminals (PDTs) are critical parts of this system, where petroleum products are received—usually from pipelines or ships—and then discharged onto trucks, ships, or rail cars in structures called loading racks.

Many jobs need to be performed to keep these terminals and loading racks running efficiently and in compliance with government regulations and industry standards. Terminal Automation Systems (TAS) are commonly used to automate these functions. They bridge the supervisory control and data acquisition (SCADA) systems used to control and monitor field devices (electronic meter registers, tank level indicators, valves, pumps, etc.) with back-office systems. For example, a TAS might provide point-of-service authorization to permit fuel loading and then direct billing information to the terminal’s ERP system.

Recently, however, the American Petroleum Institute (API) introduced updates to API Standard 2350 that significantly changed the way tank receipts are handled. Critically, it added a requirement to implement a management system with formal processes for reconciling receipts and distributions.

The motivation behind the standard is to increase safety in places like PDTs where petrochemical spills risk vapor cloud explosions and other dangers to personnel and the environment, not to mention lawsuits, fines, and facility closure. But reconciliation is a major undertaking, requiring periodic accounting of incoming and outgoing flow meter registers and tank level gauges to identify potential discrepancies.

The challenge

Blendtech used groov EPIC to provide data processing and secure connectivity between corporate and control networks.

Blendtech is a division of PT Industrial Electric, suppliers of TAS for oil and gas distribution. The company offers expertise in design, engineering, installation, implementation, commissioning, training, and support of bulk product handling and loading equipment, additive and dye injection, ratio and wild stream blending, and a variety of custom process offerings for petrochemical and chemical industries.

One of Blendtech’s customers approached them seeking help to comply with API 2350. At the time, this customer was using a labor-intensive verification process every 24 hours, recording information in spreadsheets and reviewing them manually. In addition to improving compliance, the customer also hoped to improve operational efficiency, tank utilization, and accounting accuracy.

The fundamental challenge to reconciliation is the quantity of data required. Given this challenge, there are two common approaches to improving compliance: take manual inventory more often or bring more field data into the TAS.

For smaller terminal operators like Blendtech’s customer, neither option is attractive, since either hiring more personnel or expanding infrastructure might require an investment of hundreds of thousands of dollars.

The goal
Blendtech’s task was to find a third option—a way of automating this costly process that would be feasible for their customer.

Several years ago, Blendtech began working to converge information technology (IT) and operations technology (OT), as more of their projects required bringing data from field programmable logic controllers (PLCs) into TASs.

Initially, they experimented with Ignition SCADA from Inductive Automation to bridge these systems, but later used Node-Red on Opto 22’s groov Box because it didn’t require deploying and maintaining a PC. Blendtech had used Opto 22 products for over 25 years, integrating SNAP programmable automation controllers (PACs), G4/G1 I/O, and solid state relays into their own TAS product.

For this project, they envisioned a similar solution, and looked to the latest generation of industrial connectivity products, including Moxa gateways and Opto 22’s groov EPIC (edge programmable industrial controller).

Their goal was to build a transaction history in the customer’s site historian using the data reported to the TAS. In particular, they needed data from custody transfer-certified meter registers located at each loading rack and from pipeline manifolds where incoming fluid transfers are measured.

With an accurate transaction history in place, it would be possible to reallocate personnel currently used for reconciliation and have back-office operations staff audit tank usage more frequently. It would also be possible for accounting staff to perform more accurate internal audits, providing a basis for continuous improvement.

The system
So if meter register data was already reported to the TAS, why didn’t the customer already have an accurate transaction history they could use?

The short answer is that there simply wasn’t a bridge between where the data was available and where it needed to be. There was no connection between the TAS, which resided on a corporate network, and the site historian, which was part of the control network. Further, the data stored in the TAS required normalization—additional processing to format data items consistently— before it could be added to the site historian.

The control network contained an Allen-Bradley PLC that provided some aggregation of field I/O data, but it lacked connections to the loading rack meters and the functionality to read and process data from the TAS database. The flow meters themselves were controlled by their own embedded logic, independent of the field PLC, and reported their values back to the TAS using a proprietary protocol that wasn’t accessible to the rest of the control system.

The solution
To bridge these separate systems, Blendtech chose to build their system on Opto 22’s groov EPIC. It provided all the functions they needed in a single package—secure connectivity, data processing, and storage—which meant they didn’t need a PC or additional hardware to complete the design, and it was backed by excellent user documentation and free customer support.

With groov EPIC, Blendtech was able to present their customer with a reliable data aggregation solution that significantly reduced the cost of API 2350 compliance.

Here’s how they did it.

Step 1 involved bridging the networks. At its heart, groov EPIC is an industrial controller capable of high-speed I/O operations, but Blendtech focused on using its connectivity and processing power as a secure network gateway.

The groov EPIC processor (GRV-EPIC-PR1) has two completely segregated network interfaces, so that trusted and untrusted networks can be isolated. This segregation solved the basic problem of bridging the corporate and control networks without permitting unregulated traffic between them. The controller fit comfortably into the existing control cabinet using a zero-I/O chassis (GRV-EPIC-CHS0), a small-footprint housing (5.36” x 3.54”) for applications that don’t require I/O operations.

Step 2 was all about retrieving and normalizing data. While legacy PLCs may be capable of communicating using various industrial protocols, they are usually unable to interface with modern IT systems directly. With groov EPIC on the network, however, Blendtech could build a custom data processing interface for the TAS directly into the controller using the embedded Node-Red Internet of Things connectivity software. Node-Red boasts a large library of functions for connecting data sources, including drivers for many popular databases.

The ultimate goal was to provide time-sequenced data as a series of Modbus registers that the master PLC could read from the groov EPIC processor. That wasn’t the native format of the data in the TAS, however and, depending on which loading rack the data originated from, various operations had to be performed to achieve this consistent format.

Blendtech designed event triggers to recognize when data in the TAS had been updated, as well as the source of the data. Then they invoked logic to sequence, format, and scale appropriately for the specific data source. Blendtech used Node-Red to handle all of this, with custom formatting logic in embedded JavaScript functions.

Step 3 focused on moving data into the site historian. With 6 GB available on a fault-tolerant solid-state drive (SSD), the EPIC could also serve as the temporary storage location for everything it retrieved from the TAS, ensuring no additional hardware was necessary to complete their system.

After retrieval and processing in Node-Red, data was written to the groov EPIC’s internal hardware memory map, where it could be accessed via Modbus/TCP requests from the master PLC. Once there, the historian could poll the PLC at regular intervals to add this data to the site archives.

Blendtech also took advantage of groov EPIC’s built-in touchscreen and embedded groov View server to design a basic human-machine interface for controller and network health monitoring. Since groov View is mobile-ready, the status of the controller can be viewed from any smart device or computer connected to the network, using the built-in user access controls to maintain security.

The future
Ultimately, Blendtech was able to meet their customer’s goal, delivering a system to automate inventory reconciliation at a fraction of the cost of the typical alternatives. With significantly less manpower, they can audit their transaction history every hour instead of every day, improving their accounting granularity and increasing confidence that product is being safely managed.

What’s next? With around 30 similar installations in Canada and 120 across the U.S., Blendtech continues to present their reconciliation system to more potential customers. By installing the full Ignition Gateway and SQL-Bridge module on the EPIC, Blendtech plans to design even more robust transactional logic into their TAS database monitoring.

They are also expanding their portfolio of custom Industrial Internet of Things applications built on groov EPIC. For example, they are developing a traffic management system for use in distribution terminals. Using the EPIC’s I/O control functions, the system will control high-visibility lane and wait-time indicators to optimize the flow of tanker trucks through loading racks.