Page tree
Skip to end of metadata
Go to start of metadata

Welcome to the Red Book for HPCC Systems® 7.0.0 Beta. There are several sections in this Red Book as follows:

  • General HPCC Systems core platform
  • Significant new features core platform

This beta release provides a preview of features and enhancements to be included in HPCC Systems 7.0.0. Please give them a test drive and and provide feedback to help us fine tune the gold release targeted for later in 2018.

Users may benefit from glancing at the Red Book entries when making a large jump between releases.

Here's how to contact us if you find an issue or want to add something to the Red Book:

  • To raise an issue, use our Community Issue Tracker. Please create yourself an account if you don't already have one, to get automatic updates as your issue progresses through the workflow.
  • To ask a developer a technical question about something you are doing or have encountered, post in the Developer Forum.
  • To add a note into the RedBook, please contact Lorraine Chapman with full details.

General HPCC Systems core platform

The default implementation of GROUP(dataset, fields, ,ALL) has changed

Previously the output of the operation happened to be globally sorted by the grouping fields, but this will no longer be the case.  If any ECL code had been relying on this side-effect, that ECL may generate different results in HPCC Systems 7.0.0  Adding  ,SORTED to the GROUP statement will cause the system to use the pre 7.0 implementation and generate globally sorted results.

Uninstall WS-SQL before downloading HPCC Systems 7.0.0 Beta 1 to avoid compatibility errors

WS-SQL is no longer a stand alone module with a separate installation process. It has now been integrated into the HPCC Systems platform as a core feature. If you previously installed WS-SQL as a standalone add on to HPCC Systems that predates version 7.0.0 Beta 1, please uninstall the earlier version of WS-SQL before installing HPCC Systems 7.0.0 Beta 1, to avoid compatibility issues causing an error.

Updates and new features to WS-SQL will now be recorded in the HPCC Systems changelog.

Internet Explorer 8, 9 and 10 are not supported in HPCC Systems 7.0.0.

Microsoft stopped supporting these browser versions in January 2016, leading us to stop supporting them from HPCC Systems 7.0.0 Beta .

Unicode TRIM now removes only space characters rather than all whitespace

Prior to HPCC Systems 7.0.0, the implementation of TRIM on unicode strings related to removing whitespace, was inconsistent with the implementation applied to standard STRING values. This inconsistency has been fixed in HPCC Systems 7.0.0. In STRING, this is space (0x20), horizontal tab, vertical tab, line feed, form feed, carriage return (0x09 to 0x0D) and non-breaking space (0xA0). In UNICODE, it is all characters with the white space property.

However, it’s possible some code may have been relying on the previous behaviour. If so, you can specify, WHITESPACE on any TRIM expression where you wish to use the prior semantics of removing other whitespace characters as well as space characters.

New option to disable built-in scheduler in eclserver

A 'schedulerDisabled' option has been added to the eclserver configuration which defaults to FALSE. 

If an eclscheduler component has been added to the environment, schedulerDisabled should be set to TRUE in eclserver components to disable the built in scheduler to avoid the situation where multiple schedulers are dealing with the same eclserver. The same problem may also be an issue where multiple load-balanced eclservers are used.

Eventually, we plan to remove the eclscheduler from eclserver completely.

Converting to a varstring from an integer is now always done via a string

This change has one known effect, (varstring2)123 used to produce ' **', it now produces '12'.

Root scope for workunit is now blank

If a statistic or timer previously had a scope of "workunit" in HPCC Systems 6.x.x, it will now have a scope of "" in HPCC Systems 7.x.x. This may affect code that is processing timings from existing workunits.

Assigning to the same field more than once in a transform now produces an error instead of a warning

Inside a transform:

SELF.x := ...
SELF.x.y := ....

Now generates an error that SELF.x.y has already been assigned.

Significant new features core platform

New attribute in EMBED treating it as an activity,  allowing it to stream output

This feature allows embeds to behave in similar ways to the inbuilt activities.  When the activity attribute is specified on an EMBED the system will generate an activity in the graph for that embed.  The inputs to the activity are any leading parameters which have streamed dataset type.  (Any streamed dataset parameters that follow parameters of other types are not processed as input activities.)  If the result has a dataset type, then the output from the activity will be fed into the activtiies that use that dataset.

The code for implementing the embed will need to behave differently depending on whether it is a global activity (only generating a portion of the dataset on each node) or executed within a child dataset, where each activity will generate the complete dataset.  The way this information is passed to the embeded code depends on the language:

  • C++ - An extra parameter IThorActivityContext * activity is passed to the function.  It has the following members:

    interface IThorActivityContext
        virtual bool isLocal() const = 0;         // is the activity local
        virtual unsigned numSlaves() const = 0;   // How many slaves is this activity executed on
        virtual unsigned numStrands() const = 0;  // How many strands per slave (currently 1)
        virtual unsigned querySlave() const = 0;  // 0 based 0..numSlaves-1
        virtual unsigned queryStrand() const = 0; // 0 based 0..numStrands-1
  • Python and other embedded languages - An extra local variable _activity_ is passed to the function.  It has the following members:

    isLocal         - is the activity local
    numSlaves       - How many slaves is this activity executed on
    numStrands      - How many strands per slave (currently 1)
    slave           - which slave is this call for (0..numSlaves-1)
    strand          - which strand is the call for (0..numStrands-1) 

Examples are included in the regression suite, for example, testing/regress/ecl/embedactivity*.ecl and testing/regress/ecl/pyembedactivity.ecl.

  • No labels