Welcome to the Red Book for HPCC Systems® 7.6.x series. There are several sections in this Red Book as follows:
- General HPCC Systems core platform
- Significant new features core platform
- ECL IDE
Users may benefit from glancing at other Red Book entries when making a large jump between releases.
Here's how to contact us if you find an issue or want to add something to the Red Book:
- To raise an issue, use our Community Issue Tracker. Please create yourself an account if you don't already have one, to get automatic updates as your issue progresses through the workflow.
- To ask a developer a technical question about something you are doing or have encountered, post in the Developer Forum.
- To add a note to the RedBook, please contact Lorraine Chapman with full details.
General HPCC Systems core platform
Keyed Join fix for the remote handling of parts from othr clusters
When a smaller Thor cluster read a file from a larger Thor where the larger Thor file overlapped some slaves of the smaller cluster, it could find non-primary copies of the file parts local to the slaves. The effect of this was that other slaves working on KJ would send requests to those local slaves, but did not map to the correct (non-primary) copy, resulting in a 'Failed to open' file error.
A workaround is to add this option:
which forces all KJ part thread handlers to access the keyed parts directly.
Thor jobs locking up when using SMART join
This may happen in specific circumstances, when using a SMART join that has failed over to a local hash join and the LHS is stopped early. The workaround if this issue is being hit in builds prior to this fix (7.4.34), would be to switch from using a SMART join, to either a regular HASH join, or a standard JOIN.
Note: Because this only happens when it has already run out of memory (and failed over to a local hash join), coding specifically as a HASH join is no worse performance wise.
Log file format change
The default log file format has changed in 7.6.0 to add an Audience field. This is configurable in the environment.conf file.
Any existing tools you use for parsing or extracting information from log files may need to be modified.
The new standard message format is:
MessageId - Unique line number
Audience - One of the following: Operator, User, Prog (for internal messages), Audit, and Legacy
Date - YYYY-MM-DD
Time - HH:MM:SS.mmm (where mmm is the milliseconds)
PID - Process Id
TID - Thread ID
Then actual log message
Python 3 is now the default
Beginning in 7.6.0, the platform supports Python3 by default. It still includes plugins for both Python2 and
Python3, but only one may be safely enabled at a time as the Python libraries export the same symbols for both versions.
Enabling both may lead to unpredictable results including segmentation faults or undefined symbol errors.
By default, the Python2 plugin is present but disabled, and the Python3 plugin is present and enabled. If you
want to use Python2 modify your environment.conf file according to the example in the file. A cluster updated from
an older version may not have the appropriate entries in the environment.conf file to support either version of Python.
Fix to HASH functions changes behavior
Prior to 7.6.0, the HASH32 and HASH64 operations on a VARSTRING did not ignore trailing spaces as they should have.
Beginning in version 7.6.0, trailing spaces ARE ignored.
For versions prior to 7.6.0, you can use TRIM to remove trailing spaces. For example: