Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

Welcome to the Red Book for HPCC Systems® 6.0.0. There are several sections in this Red Book as follows:

  • General HPCC Systems Core Platform
  • Significant New Features Core Platform
  • MySQL

This is a minor release including some new features and performance enhancements.

Users upgrading from releases prior to HPCC Systems 6.x.x may benefit from glancing at the Red Book entries from previous releases.

Here's how to contact us if you find an issue or want to add something to the Red Book:

  • To raise an issue, use our Community Issue Tracker. Please create yourself an account if you don't already have one, to get automatic updates as your issue progresses through the workflow.
  • To ask a developer a technical question about something you are doing or have encountered, post in the Developer Forum.
  • To add a note into the RedBook, please contact Lorraine Chapman with full details.

General HPCC Systems Core Platform

NEW - New supported/retired distributions in HPCC Systems 6.4.0

We now support Ubuntu 17.04. The following distributions are no longer supported:

  • Precise 12/04
  • CENTOS 5

For a full list of supported distributions and how to build the HPCC Systems platform from the sources for these distributions, visit the HPCC Systems Developer Wiki on GitHub.

NEW - Using a single colon in logical files names generates incorrect filenames which can't be renamed in ECL Watch/ECLIDE

The recommended way to reference the path of a logical file is to use a double colon following example:


A case was reported to us recently where a user (either by design or by accident) had included a single colon in the path as shown in this example:


On writing to the file system, this was interpreted as:


Once this has happened, it is not possible to rename the file correctly using ECL Watch or ECL IDE. Users who find themselves in this situation may use the following workaround to manually rename the file(s) on the file system to %3A using this bash command:

for BADNAME in /var/lib/HPCCSystems/thor/thor_cluster/parent_folder/child_folder%3Alogical_filename* ; do mv $BADNAME ${BADNAME/\%3A/\%3a}; done

This makes it possible to rename the file(s) in ECL Watch or ECL IDE in the usual way.

NEW - Simplified installation process for the Rembed plugin package

The Rembed plugin package now includes RInside to prevent installation failure and obscure error messages on systems previously using an incompatible version of RInside. More information about installing the Rembed plugin can be found in the Embedded Languages section of the Installing and Running the HPCC Systems guide.

NEW - The default behavior for the R plugin has changed

Previously, embedded R code used persistent, shared environment for R execution. From 6.4.0, each embedded R function will be run in a local environment with options to persist/share this environment between embed calls in the same way as for Python. This improvement resolves issues experienced when using embedded R on Roxie/Thor where the R environment from other queries would still be loaded. More information about the options available can be found in the Embedded Languages section of the Installing and Running the HPCC Systems guide.

NEW - Changes to denormalize to correctly preserve values from the left record if there are no matches

In versions prior to HPCC Systems 6.4.0, any constants assignments to fields inside the transform for the denormalize would be propagated into any following operations. The was incorrect behaviour. If there are no matching records then the left row is passed as-is to the output.

From HPCC Systems 6.4.0, values are correctly preserved from the left record if there are no matches.

NEW - TBB (Thread Building Blocks) libraries are now included as a GitHub sub module in our source code

Previously, we used a Linux vendor distribution version of the TBB libraries which did not always provide the most current version. Since older versions of the TBB libraries can core with more recent versions of HPCC System, we have decided to include the libraries as a sub-module in our source code, to ensure that the most current version is always used. 

NEW - Couchbase plugin xpath parse error when processing dataset (fixed in HPCC Systems 6.4.2)

Parsing the 'schedule' child dataset when there isn't an explicit XPATH defined will result in the following error:

Error: IPropertyTree: xpath parse error
XPath Exception: Qualifier expected e.g. [..]
in xpath = ?Row
^ (0, 0), 1,

NEW - 64-bit Client Tools installation location

HPCC Systems users who are accustomed to locating Client Tools in the C:\Program Files\HPCCSystems\version_no\clienttools folder, should note that while this is the correct location for the 32-bit version, the 64-bit version will be located in C:\Program Files (x86)\HPCCSystems\version_no\clienttools folder.

NEWIncorrect code generation on some aggregation operations on inline datasets

Previously, when an operation on an inline dataset (e.g. a filter) included a min/max (or other aggregate operation) of a filter/limit of the same dataset, the resulting values from the inline dataset were being incorrectly substituted into the min/max operation.

E.g. catch3 which has the following structure: 

ds := inline-dataset
child := limit(ds, 4); (a filter might also trigger it)
m := max(child, some-field);
output(ds(some-other-field != m));

This was incorrect behaviour which has now been corrected. Be aware that the behaviour of some existing queries that were relying on this incorrect behaviour, may now change.

NEW (string)(string100)string50value no longer converted to string50value

Previously (string)(string100)string50field would have removed both casts. If the value was then being compared against another string, it would not cause any problems because trailing spaces are ignored. However if for some reason the untrimmed length was being used, that length will now change.



would return 50, now it will correctly return 100.

NEW - LOOKUP attribute available in HPCC Systems 6.4.0

While the layout of a file is specified in the ECL Code, if the file on disk is using a different layout, version control problems can be caused. This feature allows file resolution to be done at compile time. Two language constructs have been added which are associated with this new feature.

  1. A DATASET declaration can be given a lookup attribute to indicate that the filename should be looked up at compile time.
  2. LOOKUP can also be used in the RECORDOF function taking a filename rather than a dataset. The result will be expanded at compile time to the record layout stored in the named file's metadata.

More details are available about  this new feature and how to use it in our blog post File layout resolution at compile time.

NEW - Change in Dropzone configuration in HPCC Systems 6.4.0

You can now supply a list of preferred dropzone destinations but keep some destinations hidden from users using the ECLWatchVisible attribute in the HPCC Systems Configuration Manager. When set to true, a specified landing zone will be shown in the dropdown list of available droopiness and ECL Watch. When set to false, the drop zone is hidden. This may be useful in situations where you want to maintain access to a number of dropzones for your users but protect some locations from being used as general drop zone locations. You can also add notes for other administrators to keep a record of why your dropzones are configured the way they are. More information about dropzone configuration settings an be found in the HPCC Systems Configuration Guide.

  • No labels