Hume Datahub SDK Recent Changes

SECS/GEM Software Changes

• HSMS receiving logic for large recipe transfers (ppbodyFileMode is true) or when using the file_receive option have been revised to use asynchronous receiving instead of blocking reads. (11/5/24). 
  
• An application can install a callback that is executed when HSMS linktest messages are received.  See the hsms command documentation. (10/31/23).
  
• New python equipment and host libraries have been added to the SDK.  (11/18/22).
  
• The Tcl localtime command was revised to no longer provide unique, increasing time values if the system clock is set backwards.  Higher resolution time values are now provided on Windows version 8 or newer. (9/27/22).
• Optional diagnostic messages when sending the HSMS Select.req control message are newly added so that diagnostic output is similar for all HSMS control messages. (6/15/22).
• Better error reply messages for invalid message data have been added to the ::dmh::secs_to_TSN command.  Also, during SECS communication, the last conversion error and last partially-converted SECS list are saved by the SECS Server as variables ::dmh::secs_to_TSN_last_err and ::dmh::secs_to_TSN_partial_msg to provide after-the-fact diagnosis of received invalid message data. (10/28/21).
• There is a new procedure secs_CLOCK_reset that can be called to deal with the system clock being changed while an app is running.  The app uses an enhancement to the localtime command; a reset subcommand which causes refreshing of cached system time values. (7/13/21).
• A fix was made to the startup initialization of custom Conversation receiving items for Tcl applications. (5/20/21).
• Stream 13 dataset transfer logic was improved to workaround the issue of the other party specifying a maximum RECLEN of 0. (5/14/21).
  
• The Tcl hsms command has new subcommands for receiving or sending messages directly to files or from files.  The update supports communicating messages that are too large to handle in memory.  There is newly added Tcl application code for deploying Stream 21 in the gem2/lib/filesend.tcl file which can be used from the Tcl/Tk host and equipment applications, or used by compiled language applications. The Tcl code also provides a worked example of how to deploy custom large message types. (11/9/2020).
• Timestamp values in message tracing now use the procedure ::dmh::tracetimestamp which can be replaced by an app. (6/30/20).
• An issue with the appearance of table data for Tcl/Tk apps on high DPI displays was fixed. (5/20/20).
  
• Features to query and set RS-232 serial port signals have been added to the secsport command.  These features are not needed by standard SECS-I; they are used for custom requirements. (4/17/20).
  
• Tcl/Tk Linux application users may notice improved font appearances and more font choices due to Tk being compiled with support for X FreeType fonts.  Users with very large displays can limit the width of table display windows by setting an environment variable HUB_MAX_WIDTH to the maximum pixel width desired. (1/2/2020).
  
• Receiving of non-standard HSMS message function values greater than 127 was fixed. (10/2/19).
  
• If errors occur when appending to a SECS trace log file, the console window is raised if available, and error messages are written to stderr. Usually messages to stderr are printed in the console window with color highlighting. (4/17/18).
  
• Customers who build custom versions of the SecsServer.exe program are advised that the file system paths contained within the SecsServer program file have been revised.  New custom versions should be built using the newer Hume provided SecsServer.exe file as the starting point, and care should be taken not to mix older and new version building scripts. (11/21/17).
• The logic for periodic "file saving active" messages when logging SECS communication has been revised to fix an issue with Daylight Savings Time changes. (3/20/17).
  
• The Conversation feature of the Tcl/Tk applications has been updated to guard against overly long messages being displayed by the Validate message dialog. (8/1/16).
  
• The Trace windows and Status windows of the Tcl/Tk SECS apps have been enhanced to simplify copy-and-paste usage of the SECS communication display. (1/20/16).
  
• Environment variables DMH_CONNECT_DEBUG, DMH_CONNECT_WAIT, and DMH_CONNECT_RETRY may be defined and set to override the default timing during startup when a .NET, C++, or Java app connects to the newly started SECS Server background process.  See the .NET SecsPort DebugEvent or the ServerError event documentation for details. (1/6/16).
  
• The HSMS connection management logic was improved to retry and recover certain socket exceptions that formerly resulted in a disconnection.  (12/22/14).
  

Host Side

• By default HSMS client connection setups use asynchronous calls that perform better when tools are down.  There may be a new issue with Windows 10 closing these connections when the tool hostname cannot be resolved even though the connection gets successfully setup using the IP address.  To workaround the issue, the user can define and set an environment variable HUME_HSMS_ASYNC to 0, which causes the connection setup to be made with synchronous calls and avoids making calls to resolve the hostname associated with the IP address. (5/5/23).
• When receiving event reports with >60k of data, new checks are made so that inserting a row into the ei_event_log table does not fail.  Large variable values received in event reports can exceed this limit and still be passed to compiled apps.  Tcl apps can use the procedure ei_var_query to obtain cached variable values.  (11/18/22).
• The Tcl/Tk Host Supervisor has new features of being able to sort the Defined Reports by CEID or Event name, similarly the Parameters and Status Variables tables have new features to sort by name or varID by clicking on the desired column header. (10/11/22).
• A fix was made to the .NET SecsHost EventReportConfigs() method. (7/9/21).
  
• Minor changes were made to the .NET SecsHost and gem2/SecsHost.tcl files so that custom applications can use VFEI features that have been available for Tcl applications. (11/4/18).
  
• The .NET, C++, and Java host libraries have new features to set a base working directory for application data files including *.tab files, log files, and process program transfer directories.  A custom application can set the SupervisorDirectory property before starting.  Programs that call the SupervisorStart method, such as the example .NET and Java host applications can be told to use a specific relative or absolute directory by setting an environment variable, HUME_SUPERVISOR_DIR.  The Tcl Host Supervisor uses a different environment variable TCL_SUPERVISOR_DIR for the same purpose so the different apps can be separately configured. (10/21/18).
  
• The Tcl Host Supervisor PROC_STATE_NAMES Conversation item was revised to fix a minor bug with displaying the correct initial process state for equipment that deploys the ProcessStateNames variable. (1/17/18)
  
• The Tcl Host Supervisor has new 300mm Conversation Items that are loadable from the ../gem2/custom directory that provide for defining event reports and enum variable value mappings for the E40, E87, E90, and E94 300mm standards.  The Conversation Items in 300mm_host.cvt can be loaded and the first item, 300mm-1, can be "Sent" as a shortcut to load and "Send" several Conversation Items.  We show "Send" in quotes because the Send action is used to execute the Tcl code in these Items but SECS messages are not always sent.  "Sending" the 300mm-UI item creates a User Interface for exercising 300mm features including working with Ports, Carriers, Process Jobs, and Control Jobs. (12/12/17 and 1/20/18).
• Handling of alarm information during initialization was revised to synchronize the set/clear state using S5F8 reply data.  Also, S5F5R is sent or not sent per the initialization configuration whereas previously there was optimizing logic that would conditionally skip sending the message. (12/8/17).
  
• The Tcl Host Supervisor supports adding a variable EVENT_DESCRIPTION to event reports with the value being set by the host logic using event descriptions known from the equipment variable EventDescriptions or configured.  Also, a new checkbox provides for adding the formatted event reports seen in the Recent Reports list to the SECS trace for enhanced viewing or logging. (11/21/17).
• The ei_event_log table schema has been updated.  The length of the data column was changed to 60000 from 20000 which allows for processing larger event reports in this table.  Received event report data is parsed and individual variable values are saved in the ei_variable table, so the limits of the ei_event_log table only affects certain modes of processing event report data such as the display to the Tcl Host Supervisor UI.  New error catching logic has been added in case the length limit  is exceeded. (12/19/16).
  
• New Conversation items were added to the Tcl/Tk Host Supervisor for interactive use of S1F21R (DVVALS_QUERY) and S1F23R (EVENTS_QUERY).  The latter item can be used to create Report Definitions for the Event DVVAL associations. (12/14/15).
  

Equipment Side

• There is stricter checking for invalid message data for message types S1F11, S1F21, and S2F29. (11/28/24).
  
• A fix has been made for .NET and other compiled apps which change the communication configuration dynamically without restarting.  There needed to be preservation of array element values set by custom code or by using the PropertySet() method. (9/27/23).
  
• A new option to enforce strict variable type usage for message types S1F3 and S2F13 can be set by assigning element value stricttypes to 1 in the Tcl SECS connection array.  Setting the option prevents the host from obtaining non-SV values when using S1F3 or obtaining non-ECV values when using S2F13.  Users of compiled language libraries can use the PropertySet method during initialization to configure this option. (8/28/23).
  
• Built-in support was added for optionally deploying a ControlStatePrevious SV Variable as varID 302.  The Tcl procedure eq_var_add_ControlStatePrevious $spname  can be called during initialization to add the variable.  OEMs that deploy this variable will typically also call eq_ControlStateModeSet described below to configure how the ControlStateOFFLINE event report data is handled. (8/23/23).
• An additional built-in data value variable (DVVAL), ECIDChangeValue, was added to communicate operator ECV parameter value changes along with the existing variable ECIDChangeName.  These variable values are set when the OperatorECVChange event is posted. (7/23/23).
• Using a connection name that includes hyphen characters no longer causes errors with certain built-in variable values. (6/24/23).
• Built-in logic has been added to save and restore the enabled/disabled settings of alarms.  A compiled app can override the built-in behavior by setting the connection array element AlarmsAreSaved to 0 using the PropertySet method. (4/19/23).
• The replies to S1F3 and S1F11 for an empty list input are now sorted in ascending numeric order of varID.  Previously the reply sequence was the text-sorted order of the variable table select query. (3/7/23).
  
• New options have been added which can be set by assigning element values in the Tcl SECS connection array.  Setting the option wbit_s5f1 to 1 causes S5F1 alarm report messages to be sent with the reply-wanted header bit set.  Setting the option wbit_s6f1 causes S6F1 trace report messages to be sent asking for a reply.  Setting the option spool_no_reply to 1 causes activation of spooling to occur if the host fails to reply to a primary message and spooling is enabled and configured.  Setting all these options to 1 can help to detect a lost host connection sooner and help prevent the loss of messages.  A new method PropertySet has been added to the .NET SecsPort, and C++ SecsEquipCpp for convenient setting of these and similar options. (2/27/23).
• Built-in support has been added to respond to introspection message types, S2F51 through S2F63. (12/22/22).
  
• Spooling logic was improved for large messages.  The previous length limit of 70,000 characters per spooled message has been removed. (9/16/21).
  
• Hume has addressed a logical flaw in the standards that the ControlStateOFFLINE event report can only be sent when online, but certain customers expect the ControlState variable value to show an offline state.  The OEM can call the eq_ControlStateModeSet procedure during initialization to have reported values be for the intended state and not the actual state.  The procedure is defined in gem2/equip.tcl with calling signature eq_ControlStateModeSet {spname {isLogical 0}}. (9/15/20).
• Improved diagnostic information is provided in error notifications and added to Tracing and logging for errors that occur when formatting event reports. (9/15/20).
  
• Changes are occasionally made to files in the gem2/equip directory for the 300mm SDK, and those changes are not itemized in this document.
  
• Use of the mbx_xact procedure was revised to use a version in the SDK lib directory and the version in gem2/server was removed. (9/7/20).
  
• The EventName DVVAL is now set for S6F15 and S6F17 event report request message types. (5/6/20).
  
• A fix was made so that non-SECS protocol connection attempts to an HSMS server no longer break an existing host connection. (5/6/20).
  
• An error affecting custom Tcl applications not using the default gemsim name was fixed in the Conversation initialization code. (10/14/19).
  
• Field sizes in the ei_trace_active SQL table used to manage Trace Reports (S6F1) have been increased. (6/15/19).
  
• More checks for illegal data resulting in S9F7 error messages have been added to the handling of event report setup messages S2F33, S2F35, and S2F37. (5/29/19).
  
• Edits have been made to support the optional deployment of SEMI E58, the ARAMS standard.  (4/12/19).
  
• Value changes of the control_mode array element are newly passed as StateChange events to support apps where the ControlMode property setting is not set using an operator GUI. (3/9/19).
  
• The logic for handling S2F15 messages that set the value of ECV Parameters has been updated.  If a numeric value type is specified without a size indication (eg., I2 and not I2:1), and the ECMIN and ECMAX values are empty strings, then the host is allowed to set an empty value.  If the host sets a non-empty numeric value, it should be a scalar value (size 1) when there is not a size specified, otherwise the received vector size must match the specified size.  In the usual situation where the host is setting numeric type values, either the ECMIN and ECMAX values may be empty strings in which case the corresponding check against a min or max value is skipped.  However, the passed numeric data value must be within the possible range of the configured value type. (6/9/17).   
  
• The example Tool UI Tcl application has been updated to avoid replacing customized versions of the custom online procedure and the operator spooling alert procedures.  This surfaced as an issue when using Tcl OPC integration.  The startup logic in gemsim was changed so that the Tool UI sets a different value of MODLN, the model name.  (5/9/17).
  
• Methods to send HSMS linktest messages have been added to the C++, .NET, and Java APIs.  The Tcl secs_xact procedure has been enhanced to handle linktests too.  An example use would be to force discovery that the SECS communication is up or down just before an important event report.  Said another way, polling with linktest messages can be used to force discovery of a broken connection, so that spooling is activated sooner or so that the app can synchronize with the link state. (1/27/16).
  
• When the host sets ECV Parameters using S2F15, the t_latest field in the ei_variable table row is now updated to enable notifications whenever the host sets the same variable value.  An interested client can open a subscription to changes in the t_latest field in addition to the usual subscription on variable value updates. (11/4/15).
  
• Additional error trapping was added to the event posting logic in order to provide diagnostic messages for improper variable values. (9/10/15).
  
• The maximum length of an EventName was raised to 256 from 32. (2/23/15).
  
• The property PPFilesByRequest and a file request event have been added to the .NET, Java and C++ libraries.  In March 2015, transfer logic was revised to allow using process program names that are not valid filenames when using the FilesByRequest feature. (11/7/14) and (3/27/15).
  

OPC Software

• There is a design flaw in the OPC DA (Classic) API such that certain browse API calls cannot work properly in a 64-bit application.  Customers that require OPC browse features should use the Hume 32-bit DMH85 SDK and not the 64-bit DMH86 SDK.
  
• The OPC Supervisor app has newly-added configurable limits when using the Browse or Discover features to enable working with servers that have very large numbers of items. Also, the Browse windows have a new menu action to discovery Items underneath a selected branch of the tree. (10/23/17).
  
• The OPC Supervisor application was updated to disconnect client connections when exit is called so that the application exits cleanly. (5/29/12).
• The OPC client software is being successfully deployed by SECS/GEM equipment OEMs.  New features have been added to directly use OPC data items to manage displaying SECS Server windows including a configuration dialog.  An improvement has been made to handle version 2 discovery finding new items but ending before completion with an error result. (6/29/12).
• The conversion of received data timestamps to the local timezone is now done in a catch in case the data is not valid. (11/5/15).
  

Tcl/Tk Application and API Changes

• The DMH Server logic can be configured to use non-blocking reads with DMH clients by defining and setting the environment variable DMH_SERVER_NONBLOCKING to 1. This addresses a theoretical circumstance where a client is sending a large message and is somehow interrupted without a disconnection being noticed, or there is improper use of the DMH socket connection by non-Hume software. (9/11/23).
  
• Improvements for high DPI (dots-per-inch) displays have been made for Tk 8.6 applications.  Hume has added 2X and 3X larger built-in bitmaps on all platforms that are automatically chosen based on the value of tk scaling when bitmaps are initialized.  Hume has also added 2X and 3X larger checkbutton and radiobutton bitmaps that are automatically selected based on the value of tk scaling during initialization.  Use of tk scaling to determine sizes works well for Windows.  On Linux the tk scaling value is not reliable. As a workaround, the environment variable TK_BITMAP_SIZE may be set to the desired scaling value and this overrides use of the tk scaling value.    A fix was also made to the Windows scaling of ttk checkbuttons and ttk radiobuttons per the change at SourceForge for new patch releases of Tk.  These changes are very noticeable on new laptops featuring high resolution 4k displays.  Tk 8.6 for Windows is "dpiAware" and displays crisp, fine resolution text and widgets on high DPI displays.  Tk 8.5 apps are not "dpiAware" and by default Windows automatically scales their windows for high DPI screens to prevent bitmaps and text from being too small.  The Windows system-performed scaling is noticeably less sharp than seen with Tk 8.6 on high DPI screens. (12/9/21).
  
• The procedure ::dmh::hub_table_SQL was added to the SDK library.  It was formerly part of the Humelib Components package. (9/7/20).
• The ::dmh::localtime and ::dmh::mktime commands now support millisecond timestamps with format codes 17 and 18. (6/30/20).
• Hume has added a traditional code debugger for Tcl interpreters that can be used on demand to provide stepping, display of local data values, breakpoints, and execution abort.  See the ::dmh::dbDebug page. (4/14/20).
• The Hume ::dmh::dmh_dde command which implements the send command on Windows has been updated to support child interpreters. This makes it possible to use the inspect application to debug parent and child interpreters that co-exist in the same Tcl/Tk application. (3/30/20).
  
• The client disconnect action featured on the DMH status window was revised. (12/28/18).
• A fix was made to window placement logic in the Tk procedures dmh_gets and dmh_win_shrink to fix an issue with dual monitors on Windows when the right monitor is defined as the main screen. (2/13/18).
• The unzip.exe program included with the Windows  SDK installer was updated to version 6.0 after a report that the older version failed to run during an install on Windows 10. (10/13/17).
  
• The inspect application has been updated to include introspection of TclOO Objects and Classes that are newly included in Tcl 8.6.
  
• The dmh86 (dmh_wish) executables on Unix now accept the command line option -console with different behavior.  This option now provides a console window using Tk which replaces use of the terminal shell.  The Tk console features use of the cursor for command history recall as well as consolidated input and output. Prior to this change, use of the -console option would enable terminal use when there was an input file and terminal use would ordinarily be disabled. This feature was revised in 2017 so that when -notk is specified with -console on POSIX systems, X-Windows is not used; instead the terminal window is enabled for console IO. (9/15/14) and (6/30/2017).
  
• The Hume SDK license check logic has been modified to be case insensive when matching the system hostname to a dmh_license record. (5/14/14).
• The Tcl/Tk SECS/GEM applications were revised to inform the user of possible licensing issues at startup. (3/25/14).

Tcl/Tk 8.6 Compatibility Changes

• The current release of the Hume Datahub SDK includes Tcl/Tk applications that are based on Tcl & Tk patch level 8.6.9 as of February, 2019.  From a compatibility point of view, Tcl 8.6 differences from Tcl 8.5 should be minor and present few problems (more details).  However, customers are advised that the new Stackless Evaluation internal implementation is radically different from prior versions and differences may become apparent for busy applications that previously had deeply-nested evaluations.
  
• The Hume Windows 8.6 build is 64 bit with threading enabled.  Prior windows versions were always 32 bit without threading enabled.  Customers that use compiled Tcl extensions (packages) will need to obtain 64 bit versions.  Users of the OPC features will need to install the 64 bit "Core Components Redistributable (x64)" download from the OPC Foundation website.  With threading enabled, Hume no longer compiles in the memory_stats command in the windows versions.
  
• Since Tcl version 8.4, the Hume dmh package Tcl extension has been revised to use the ::dmh namespace to prevent name conflicts such as occurred with the Tcl 8.4 lset command.  Read more about the new Tcl 8.4 namespace and compatibility issues in the dmh_import document.  The Hume provided Tcl commands dmh_import, dmh_var, and vset were added to the 8.3 (DMH83) distribution and newer Tcl/Tk distributions to support deploying software that is compatible with Tcl 8.3 and newer.
  
• The Datahub SDK Tcl/Tk 8.6 distribution includes updated versions of the packages [incr Tcl],  [incr Tk], and Iwidgets.   Hume provides modified index files for these packages so that the correct versions are chosen when multiple versions are installed.  Recent Tcl/Tk versions provide the ttk widget package which is superior to Iwidgets and should be used for new development.  To discourage further use of Iwidgets, the Hume installation installation no longer features an Iwidgets demo or the package documentation.  Now that Tcl 8.6 includes both [Incr Tcl] and TclOO object-oriented programming (OOP) packages, guidance is appropriate for choosing an OOP package. For a developer who is familiar with Java or C#, TclOO appears to be a non-starter because it lacks integration of common (aka static) variables whose values are shared by instances of a class.  TclOO also lacks public/protected/private variable modifiers which leads to tedious extra work to provide get and set methods for data items that would be public in another system.  On the other hand, TclOO is loaded with state-of-the-art OOP constructs such as mixins, filters, and Slot classes, and is well-suited for the self-amusement of computer science professionals.  Incr Tcl has been widely used for production applications and provides comparable features to Java and C#.  The syntax of coding a method call within a method is as expected - the implied reference to the current object is implicit and does not have to be explicitly coded.  In other words, in Java, C#, or [Incr Tcl] you would code "someMethodCall" but in TclOO you would need to code "my someMethodCall" to explicitly reference the current object.
  

New with Tcl/Tk 8.6

• Stackless Evaluation: The evaluation of many levels of nested proc calls are no longer implemented as a stack of nested C routine calls. This revision in the internal implementation of Tcl evaluation makes deep recursion in Tcl scripts safe to do. But there's more...This new implementation enables a collection of new commands, coroutine,  tailcall, yield, and  yieldto  that provide new capabilities and models of concurrency for Tcl scripts that are rarely found in a popular programming language.
  
• Object Oriented Programming: The commands of the TclOO package are now part of Tcl itself. This gives Tcl an additional object system that includes advanced features such as meta-classes, filters, and mixins.
• IPv6 Networking: Both client and server socket connections can use IPv6 network addresses alongside IPv4 as supported by the underlying platform.
  
• Built-in Zlib Compression: New command zlib provides utilities to handle compression of data and streams.
• List Processing: New commands lmap and dict map enable the elegant expression of transformations over Tcl containers.
• Additional New Tcl Features: Temporary file creation, enhancements to list sorting and setting, dict filtering, encoding and decoding of binary sequences, and many more.
• Built-in PNG Image Support: Photo images now support read/write in the PNG format, with the ability to set the alpha channel.
• Busy Windows: New command tk busy is a variant of blt::busy that lets the interactivity of windows be suspended and restored as required by the needs of the program.
• New Font Selection Dialog Interface: New command tk fontchooser provides a portable interface to the standard font selector of the platform, whether that is modal or not.
• Angled Tk Canvas Text: New option -angle $degrees to $canvas create text for rotating displayed text.
• Moving Things on a Canvas: New commands $canvas moveto, $canvas imove and $canvas rchars for moving and manipulating canvas items.
• Many more improvements: See the Changes in Tc/Tk 8.6 page on the wiki.tcl.tk site for details.

Hume Changes to the 8.6 Tcl/Tk SourceForge Distribution

• See the comments above in the Tcl/Tk Application Changes regarding Hume-added 2X and 3X bitmaps for radiobuttons, checkbuttons, and built-in bitmaps that improve appearances for high DPI displays.
  
• The Windows console menu has a new item to display errorInfo, and a new item to display the HTML documentation. Bindings are added for Motif style cut-and-paste and Windows style right button paste. These changes are actually done at runtime without changing the core code.  The changes that Hume has made in the past to make the console safe for memory use are no longer needed - the core has similar logic.  However, Hume has revised the console trimming logic for better performance, and has added a "fast mode" which optionally shortens long output lines.  Hume also makes the Windows console available on Linux.
  
• The Tk C code is patched for Windows so that the send command and winfo interps work using the Hume DDE command.
• The Tcl library is patched so that index files of the name tclindex.txt are used if tclIndex is not found. This enables Tcl to execute from an ISO 9660 CDROM image where tclIndex is not a valid filename.
• The Tk C code is patched to create an option to the winfo command, timelastevent. Executing winfo timelastevent returns a value that changes with each new window event at the display. This value will be unchanged over time when there has been no user interaction with the window interface. An application can poll this value using after, and determine if the user has been inactive.  Users should migrate to the  Tk command tk inactive for this purpose.
  
• Hume compiles the Tcl DLL on Windows so that 256 socket connections are possible. The default compilation results in a limit of 64.
• Hume edits the Windows built-in command processing to allow more commands and to be case-insensitive.
  
• Hume converts the Tcl/Tk man pages to HTML, adds new pages, and provides custom indexes.