The Hume OPC Supervisor Application

Introduction

Feature Summary

• A working, polished, and complete application for configuring and managing OPC data collection.
• Connects to multiple Data Access Servers of any version.
  
• Leverages browsing and other introspection features of OPC to avoid configuration work.
• Built-in monitoring of connection health and automatic recovery on failure.
• Built-in connectivity to your custom SCADA and HMI applications running on other computers or platforms using the DMH.
• Direct support for tunneling, aggregation or distribution of your OPC data using the DMH.
• Integrates easily with the Hume SECS/GEM software or the Hume Data Collection application component.
• Simplifies working with OPC by managing low-level handles and type conversions automatically.
  
• Frees the developer from the low-level drudgery of working with COM and explicit memory management.  COM is a deadend technology so why waste your time?
• Frees the developer from needing to code the configuration of groups,  items, update subscriptions, and the startup or shutdown of data collection.
  
• Your applications only need to focus on processing collected data and supervisory control.
  

DMH Message System Integration

• SQL commands are accepted in mailbox SERVER_SQL for remote table data access.
• Tcl language commands are accepted in mailbox SERVER_RPC for remote control interaction.  The mailbox command processing can also be used by the remote introspection debugger, inspect.
  
• Subscription messages such as notification of OPC data changes can be sent to mailbox destinations, and specific procedures have been developed for optimized use of this feature.  There is also generic support to subscribe to changes in SQL table data and to distribute or replicate these changes as they occur.  The subscription mechanism provides an elegant interface to collected OPC data such that the integrator does not need to change the supervisor logic that acquires the data in order to use it in new ways.

Installation

OPC Server Installation

The OPC Standards

• OPC Common Definitions and Interfaces, Version 1.1, December 2002.
  
• Data Access Custom Interface Standard, Version 3.0, March 2003
• Data Access Custom Interface Standard, Version 2.05, December 2001.
• OPC Complex Data Specification, Version 1.0, December 2003
  

The OPC Supervisor Application User Guide

dmh_wish -tclargs "package require opcclient"

Supervisor Main Window

There is a list of your configured connections in the table display, and three rows of action buttons.  In general, the application uses the object-action style - you select one or more objects from a displayed list using the mouse, and press an action button to act upon the selection.  The shift and ctrl keys can be used with the mouse to extend a selection or edit the selected set. 

Another important general concept is that you are allowed to make configuration changes that are not saved to the file system unless you explicitly use the Save Config. action or Save action buttons.  So you can create temporary connections to servers, and experiment without creating or changing a configuration that is used at startup.  When you have unsaved configuration changes, you see a status message at the bottom of this window, and you are reminded of your unsaved changes when you initiate closing this window or you Exit the application.

Similarly, there is a deliberate break between your configuration and the actual state of a connection with the server.  You can perform configuration work without being online with the target server.   If you are online with a server and you make editing changes, you are often provided with a checkbox on the editing dialog so that you can explicitly specify whether changes should be applied to the connection.  There are action buttons such as Pause or Resume that affect the current state of a group without changing the configuration that is applied at startup.  Group structural changes such as creating groups, and adding or removing items are synchronized with the server if you are online.

The list of Connections is dynamic - it updates to display the time of the latest group data input as well as important status events such as connecting or disconnecting.  You usually will not see automatic connection recovery events because they happen quickly.  You may notice that the "connected at" timestamp is more recent.

The first row of buttons in the frame labeled Startup Configuration have actions related to editing connection configurations and the second row in the frame labeled Connection Actions has actions related to using a connection:

• New... is used to create a new connection definition
• Edit...  is used to edit or change a connection definition
• Copy... is used to make a "deep" copy of connection definition.  The New... and Edit... actions show an editing dialog that provides for configuring the server information and startup behavior.  The GUI button in the next row shows a Graphical User Interface which provides for configuring item, group, and memberships.  The Copy... action copies the entire configuration including the server properties and the item and group configuration to a new name.  The copied connection is not started.
• Delete...  is used to delete one or more connection configurations after a confirmation dialog.
• Start... and Stop... are used to connect to the server or disconnect.

The last row of buttons in the frame labeled Supervisor Actions are:

• Services  This brings up a dialog for you to configure starting as a DMH message system server.  You can have more than one instance of the OPC Supervisor running and acting as message system servers on a computer as long as each instance is configured with a unique DMH group name.  The default DMH group name is OPC which is mapped to the socket port of 5227.
• Save Config. This action saves your current configuration as a set of table files in the data subdirectory.  The table files are loaded when you start future sessions.  In a production setting, the administrator should make backup copies of the table data.  An integrator may want to copy the table files into a separate directory to preserve a configuration for backup or future reference.  The amount of table data, meaning the number of tables and table rows, does not increase as OPC data is collected.  So there is no ongoing table or file maintenance needed.
  
• Datahub  This action shows the Datahub table management GUI.  This window is mostly of use to a developer since the application windows provide for configuring and displaying the application table data.
  
• Tcl Console  This action shows a command console which can be used for interactive Tcl commands.
• Exit  The Datahub and Console windows have Exit actions which will not remind you of unsaved changes like this one will.

Connection Editor

Here is the Connection editing dialog that is show for the New... or Edit... actions.

• Auto Start specifies whether connecting and initializing data collection is desired when the program is started. 
  
• Retry Failed Start specifies whether the program should retry connecting if a connection attempt fails.  The Retry Connection Period entry specifies the retry interval period in seconds.
• Auto Detect specifies whether the application should perform smart polling to determine if the connection is functional.  The Max Inactivity Period specifies a time period in seconds used for smart polling.  If there has not been group input received within the previous period, the logic executes a method call to test the connection. 
  
• Auto Reconnect is relevant if Auto Detect is true.  If the smart polling determines that the connection is broken, the Auto Reconnect setting determines whether the logic attempts to reconnect.  The Retry Failed Start and Retry Connection Period are also applied to the reconnect attempt if necessary.
• Locale selection - the checkbox, entryfield, and Choose... button let you configure a choice of Locale from the list supported by the server.  When the connection is made, configuration choices such as Locale are synchronized with the server.
• Test Pressing this button tests whether you can connect to the configured server.
  
• Ok validates and applies your edits to the in-memory Datahub tables.  The changes are not saved to the file system
  

Connection User Interface

Group Editor

• Create Group when Connection is made - In most cases you will want your configured Group to be instantiated at the server when you connect, so usually this checkbox is checked.
• Activate for data reporting - OPC has a concept of a group being defined at the server but not being active for data access.  Make sure this button is checked to have data reporting enabled.
  
• Disable periodic OnDataChange callbacks.  This checkbox applies for an active Group.  You may pause or resume the periodic OnDataChange callback notifications using this checkbox.  When the periodic callback is disabled, you are still able to request a Refresh which causes the server to execute the OnDataChange callback in response.
• Reporting Period - specify a number of milliseconds for the periodic OnDataChange callback notifications.   The value of 0 instructs the server to use the minimum supported period.
  
• Time Bias - The default time offset is 0 minutes meaning reported time values are for the UTC timezone. Use this property to set a desired time offset in minutes for the group.
• Deadband Percent - A deadband percentage value may be established for the group to eliminate reporting of insignificant data changes. The deadband value only applies to Analog data items.
• Keep Alive - This property is only supported for a Version 3 server.  If set to a non-zero value of milliseconds, the OnDataChange callback is called even if none of the group values have changed sufficiently to warrant invoking the OnDataChange callback. The feature eliminates the need for polling to verify that the server is functional.  Setting the Keep Alive interval to the same as the Reporting Period would be typical. 
  
• Apply Changes Immediately - Usually you are editing the group to define properties for the next startup.  There are other action buttons on the Group pane such as Pause and Resume or Active Toggle that are for dynamic change.   When you are online, the dialog does support editing the properties and immediately synchronizing the changes with the server.
  

Browse Window

Group Membership

Application Integration and Using Collected Data

Example Applications

OPC Subscription Procedures

• clientID - a unique value to distinguish the subscribing process for any other DMH client process including the same application being executed multiple times.  A common value to use is the clientID value obtained from the DMH message system library such as the .NET DmhClient ClientID property value.
• destbox - the destination DMH mailbox for subscription notification messages.  This is the name of a mailbox that the subscribing application uses to receive DMH messages.
• ocname - the connection name to an OPC server as configured using the OPC Supervisor application.
• groupname - the name of the Group of items as configured using the OPC Supervisor application.
• max_messages - an optionally provided argument which specifies the maximum number of DMH messages that may be waiting for the subscriber to read.  If the safe subscription monitoring logic sees more than this number of pending messages in the mailbox queue, it closes the subscription and flushes and closes the mailbox.
  

Taking Control and Writing Data

The Application Source Code

The Tcl source code of the OPC Supervisor application is provided for your understanding and possible customization.

The Supervisor application dynamically loads a DLL that extends the Datahub process with new commands for using COM and OPC.  The Hume developed Tcl OPC commands are documented separately because they can be used independently by any application.   Similarly the Datahub Reference document is useful to understand detailed use of the SQL table management commands including the subscription features of the SQL open command. If you are new to Tcl, the syntax of the language is summarized in .../mann/Tcl.html.

Why are we using Tcl instead of a system programming language such as Java or C#?  A scripting language like Tcl is far more productive for this kind of work.  You may enjoy reading the IEEE Computer magazine article, Scripting: Higher Level Programming for the 21st Century, by John Ousterhout the creator of Tcl.

SECS/GEM Equipment Interface Integration

1. Front Panel Requirements - The tool user interface provides a control to enable or disable SECS communication.  There is a text field to display the current SECS communication state.  Another control is provided to enable or disable online control.  If remote commands are supported, or if there are host settable parameters that affect processing, then there is a control to choose between LOCAL or REMOTE control modes during online control.  A display of the current control state is not mandated but it is highly desirable.   If terminal display messages are supported, then there is an indicator for when a new message has been received from the host and there is a button for the operator to acknowledge the host display.  Terminal display messages are not used much and may be skipped in a minimal deployment.  Your tool may not have a physical operator console with a conventional user interface.  In this situation, apply the Front Panel Requirements to the main status screen that you provide to networked clients.  If you do not provide any kind of status screen, you are missing a strong opportunity to improve the competitive position of your equipment, and you are probably losing prospective customers.   Hume Integration can show you how to quickly deploy a small footprint web server with a suitable status screen.   This can give your marketing people something real to demonstrate; they can even show a status screen on their cell phone.
   
2. Startup Configuration - GEM specifies that the connection properties, timer values, communication enablement, and control state startup are all enduser configurable and saved in persistent storage.  Many tools provide a configuration dialog based on our example applications.  A faster deployment is possible for an OPC based tool by relying on the configuration dialog and persistence features provided by the Hume SECS Server.
   
3. Process State Model - Any batch tool has a process state model and GEM requires you to document it and to post data collection events on process state transitions.  Our example applications feature a simple model (INIT, IDLE, SETUP, EXECUTING, PAUSED), with the standard events being posted.
   
4. Variables - If your tool has measured values such as temperatures, pressures, and counters, these values should be served on the SECS/GEM interface as Status Variables.  Other important status or descriptive items should also be served as Status Variables.  If your tool uses process programs then the name of the currently selected program is served as variable PPExecName, and the name of the last used program (which may still be selected) is served as variable PPUsedName.   If you have a process state model, then you will support the ProcessState and PreviousProcessState built-in variables.
5. Parameters - Read/Write variables which are used to configure various aspects of your tool's behavior or processing are served on the SECS/GEM interface as ECV parameters.  Divide your parameters into two sets:  (1) values that affect current processing so the host is not allowed to change them in LOCAL control, and (2) other parameters.
   
6. Alarms - Alarms are an undesirable state with both a set and clear condition.  Each type of an alarm condition is defined as a SECS alarm type and the set and clear transitions are communicated on the SECS interface.
   
7. Events - An event is a notable occurrence such as the completion of a process step.  Your tool should post event occurrences on the SECS interface, which are used as triggers to send the host dynamically defined data collection reports containing Status Variable values.
   
8. Standard Documentation - SECS and GEM have strict enduser documentation requirements.  SEMI Standard PV2 for Photovoltaic equipment also adds to the documentation requirements.  Hume provides template enduser documentation to our customers for their modification and redistribution.
9. Acceptance - SECS and GEM have many complexities including the specification of many message types that are rarely used and pitfalls for naive implementations.  To insure your interface works well and is accepted by host users, focus on the essential requirements listed above and use a validated toolset to fill-in the low level implementation details and behavior models.  There are many other requirements of SECS and GEM such as implementing the communication and control state models.  These requirements are not listed because they are already provided by the Hume toolset.
   

OPC - SECS Equipment Interface Deployment Scenarios

      SECS                                DMH                      
[ Host ] <-----> [Eq SECS Server, OPC Super] <----->  [OEM SCADA Controller]
          \        OPC
                                     --------------< [OEM OPC Server(s)] 

 SECS                                 OPC            
 [ Host ] <-----> [Eq SECS Server, OPC Super] <------> [OEM OPC Server(s)]

 SECS                           OPC                           
[ Host ] <-----> [Eq SECS Server     ] <------> [OEM OPC Client SCADA Controller]
 [OPC Server & Client]                          |
              |                 OPC             |
                         \----------------------< [OEM OPC Server(s)]

SECS Integration Procedures

Alarms

Events

Parameters

Properties (Configuration)

Status Information

Remote Commands

Variables

Control

• tracewinShow - Setting this item to 1 or 0 causes the SECS Communication Trace window to be displayed or closed.  This is an important feature to make accessible from the tool.  You will want this feature to diagnose any communication problems in the field.
  
• tablewinShow - Setting this item to 1 or 0 causes the SECS Server Data Table window to be displayed or hidden.  This feature is only used during development.
• exitFlag - If this item transitions to a non-zero value while the SECS server is running, the SECS Server exits.   The SECS Server does not exit during startup if the value is already non-zero.
  
• configureShow - Setting this value to 1 or 0 causes the SECS property configuration dialog to be displayed or closed.  If your tool provides this variable, the SECS server logic provides persistence and editing of all of the SECS startup properties that are required to be persistent such as the HSMS socket port, HSMS active or passive, the SECS Device ID, timer values, the starting Control State, and whether to start with SECS communication enabled.  Including this variable provides for the quickest deployment since you do not have to provide an editing screen or provide persistence of SECS properties in the tool controller software.
  

SQL Table Schema

The table data is loaded from the file system during the application startup.  By default, a sub-directory, data, of the program's working directory is used for saving the table data.  The directory used for data saving can also be specified by setting an environment variable, HUME_OPC_DATA_DIR, to the desired directory using a Unix style slash delimited pathname such as c:/humeOPC/data.   Each table has its own file, tablename.tab, containing SQL statements for the table data.  As the application runs, new rows may be added, or rows may be updated in the in-memory table data structures.  By default, there is no logic to save the table data to the file system before shutting down.  Typically, if you make configuration changes, you need to explicitly press a save button on the user interface to preserve your changes for next time.  You can also save the data tables programmatically by calling the opc_client_save procedure.
 

Table Directory

opc_appstart .......... Application Startup Configuration

opc_client ............ Configured OPC Client Connections

opc_group ............. Configured Groups

opc_group_cfg_item .... Configuration of Group Membership

opc_group_item ........ Instance Data for Items in Groups

opc_item .............. Server Data Items

opc_item_property ..... Less Common Item Property Values


opc_secs_vartrace ..... SECS Interface, OPC Item Write Associations


opc_server ............ Connection Data for Available Servers

Schema Notes

Boolean Values

Boolean values are usually stored as integers (also referred to as int).  In this convention, 0 indicates false and 1 is the preferred value of true.  However it is usually the case that any non-zero value is acceptable as true.

Key column

Under Key the abbreviation PK indicates that the field is a primary key of the table.

The abbreviation PCK indicates that the field is part of a primary composite key for the table.  In other words, the field and one or more other fields, taken together, are the primary key for the table.  There can not be multiple rows in a table that have the same value of the primary key.

This table provides configuration records for the startup of the application particularly for non-interactive use such as being run as a Windows service.

At present, the defined fields provide for DMH message system initialization.  The table could easily include the startup of other features such as HTTP service but the current design is to deploy the OPC Supervisor as a simpler building-block application for data integration and leave more elaborate features to higher-level SCADA applications.

The application GUI does not feature configuring the startup as a DMH client.  This is because for most situations starting as a DMH server is the better choice since it does not require coordination with an external server process.

Each record in this table represents the configuration of a group for a particular OPC client connection.  There are a mix of data fields used for configuration purposes, and fields used for status information.  The configuration information is used at startup when creating groups and initializing.  The OPC application has features on the user interface with actions that change the group state to be inconsistent with the configuration data such as the ability to Pause and Resume OnDataChange callback notifications.

The application startup logic uses this table to know which groups to create, configure, and populate automatically when the parent connection is made.
 

Column Name	Key	Type	Description
ocname	PCK	varchar(32)	The OPC client connection name that a particular row is associated with.
groupname	PCK	varchar(32)	A name for the group assigned by the user which is unique for the particular connection.
cfg_active		int	Used as a boolean to configure whether the group should be active for IO operations.
cfg_period		int	The configuration of the periodic reporting interval in milliseconds which is requested.  The value 0 tells the server that the minimum supported period is desired.
cfg_disabled		int	Used as a boolean to specify whether the OnDataChange callbacks should be disabled.
cfg_handle		int	An integer handle value which is assigned by the OPC application to distinguish this group.  The handle is passed as an argument to callback methods.
cfg_locale		int	The configuration of a Locale ID for the group.  The value 0 tells the server to use the default.
cfg_timebias		int	The configuration of a time offset in minutes for the reported group timestamp values.
cfg_deadband		float	The configuration of a percentage deadzone for analog data items which provides for eliminating insignificant data changes from being reported.
cfg_klive_set		int	Used as a boolean to specify whether the IOPCGroupStateMgt2::SetKeepAlive method should be called with the cfg_klive_value.  This is a feature that is only supported by Version 3 servers.
cfg_klive_value		int	The configured keepalive period in milliseconds when it is relevant.  The value of 0 means that empty OnDataChange callbacks are not desired.  A positive value means that the server will invoke the OnDataChange callback at least as often as the supported keepalive period which may be different than the requested keepalive period.
cfg_callback		varchar(80)	Tcl code to execute for the OnDataCallbacks.  The OPC Supervisor application treats an empty value in this field as equivalent to the procedure name opc_OnDataChange.
auto_flags		int	Used as a bitfield to configuration startup actions.  At present only the 1 bit is defined with the meaning that the group should be instantiated and its item members added.  The cfg_active field lets you have groups that are added but are not active for data IO.
status		varchar(200)	This field is updated by the application logic to show major status events such as the group being added or paused.
ts_last_data		varchar(29)	This field holds the timestamp of the latest OnDataChange callback for this group as reported by the server with no timezone conversion.  The format is YYYY-MM-DDThh:mm:ss.mmm.  The field and this record is updated only after the OnDataChange logic has finished updating the individual item records in the opc_item table for the group.  Thus, a subscriber can use updates to this value as a completion signal.
last_qual		int	This group holds the master quality value of the latest OnDataChange callback for this group.  The value 0 means only good item quality values were reported if any item data was reported.
last_err		int	This group holds the master error value of the latest OnDataChange callback for this group.   The value 0 means no item error conditions were reported.

This table holds configuration records for the membership of items in groups.  An item can be a member of more than one group.  There are certain properties that can be configured separately for each instance of membership.
 

Column Name	Key	Type	Description
ocname	PCK	varchar(32)	The OPC client connection name that a particular row is associated with.
groupname	PCK	varchar(32)	The group name that a particular row is associated with.
item_id	PCK	varchar(200)	The item name that a particular row is associated with.
cfg_active		int	Used as a boolean to indicate whether the item is active for data reporting or data writing (IO).
cfg_sampling		int	Used as a bitfield to specify which of the optional Version 3 IOPCItemSamplingMgt methods should be called (some of the bits are mutually exclusive or redundant): 1 - set both the sampling period and buffering enabled values 2 - clear the sampling period 4 - clear the buffering enabled value 8 - set the sampling period 16 - set the buffering enabled value
cfg_samp_period		int	The sampling period in milliseconds.
cfg_samp_buff_enable		int	A boolean value for the SetItemBufferEnable() method call.
sampling_status		varchar(80)	Status information for the result of applying the cfg_sampling options at startup.
cfg_deadband		int	Used as a bitfield to control use of the Version 3 IOPCItemDeadbandMgt interface at startup. 1 - set the per item deadband value 2 - clear the per item deadband value
cfg_deadband_value		int	A percentage deadband value.  The standards specify that deadband values only apply to analog items.
deadband_status		varchar(80)	Status information for the result of applying the deadband options at startup.

Most of the usual property values of an item such as units and description are provided for as columns in the opc_item table.  This table is used to save less common property values as they are discoved by browsing.
 

Column Name	Key	Type	Description
ocname	PCK	varchar(32)	The OPC client connection name that a particular row is associated with.
item_id	PCK	varchar(200)	The fully-qualified name for the item as defined by the server.
prop_id	PCK	int	The property type identifier.  Values from 1-4999 are defined by the OPC standards.  Values of 5000 and greater are defined by the server provider.
prop_description		varchar(80)	A description of the property type.
prop_type		int	The variant type code of the property value.
prop_value		varchar(60000)	The property value.

This table supports deploying a SECS interface which works with OPC items.  Each row in the table specifies an OPC item that is written to with the changing data of a SECS connection array element.  This association is used to communicate important status information such as the current communication and control states.  You can see from the table keying that the design is to write each value change to only one OPC item.
 

Column Name	Key	Type	Description
spname	PCK	varchar(32)	The SECS interface name that a particular row is associated with.
subscript	PCK	varchar(32)	The name of the SECS interface array element such as comm_state.
ocname		varchar(32)	The OPC client connection name that a particular row is associated with.
groupname		varchar(32)	The OPC group name that a particular row is associated with.
item_id		varchar(200)	The fully-qualified name for the item as defined by the server.
svr_handle		int	The handle for the item assigned by the OPC server.
value_type		int	The  variant type code used to write the item value.  The usual initialization is to copy the server's canonical type.
string_maxlen		int	An approximate length limit imposed on writing string values.  This is needed if large process programs are transferred over the SECS interface and the communication activity trace messages are written by this logic.

Document Version

This document describes the configurable OPC Supervisor Application provided by Hume Integration Software as part of its Datahub SDK.  The document is current for the Tcl 8.5/Tk 8.5 versions for Windows.