Get Started Using the HPCC Systems Cloud Native Platform

Find out more about our Cloud Native platform:

Resource

Description

Author

Resource

Description

Author

Helm Chart GitHub Repository

This Helm chart can be used to deploy an entire HPCC environment to a K8s cluster.

HPCC Systems Platform Team

Readme - Using and Configuring the Chart

HPCC Local (host) storage

HPCC Systems Platform Team

Helm Chart Examples Repository

Includes examples for a number of different areas such as Azure Storage, Cert Manager, Logging, Metrics, Secrets Storage Planes and more.

HPCC Systems Platform Team

Running HPCC Systems on Microsoft Hyper-V

If you are a PC user and want to run the HPCC Systems Platform, the simplest and most natural environment may be to use a Hyper-V Virtual Machine.  Hyper-V is standard on many versions of Windows, and tends to work better than most add-on virtualization systems.  It also has good support for multiple CPUs, and is quite reliable and performant.

Lili Xu
Software Engineer III
LexisNexis® Risk Solutions Group

Cloud Cost Savings using ECL Watch

Learn about the most efficient ways to track and trend cloud usage costs, as well as how to set configuration parameters.

Shamser Ahmed
Senior Consulting Software Engineer
LexisNexis® Risk Solutions Group

Optimising the Cloud Operation Cost of HPCC Systems using Azure Spot Instances

Processing data on the cloud using Virtual Machines can be expensive. Read about the results of some research carried out by Roshan Bhandari about how to save a considerable amount of money by running your VM using Azure Spot instances. 

Roshan Bhandari
HPCC Systems Intern 2021
Senior Systems Engineer
LexisNexis® Risk Solutions Group

Developing and Training a GNN Model on Azure

In today’s world, security in our schools is critical for a healthy learning environment. The need for that security has inspired a High School student to design a facial recognition interface using HPCC Systems GNN bundle, to identify these potential threats by utilizing the camera built into the school’s security robot.

Carina Wang
HPCC Systems Intern 2021

Getting Started with the HPCC Systems Cloud Native Platform Using Terraform and Azure

Includes step-by-step instructions, showing how to deploy the HPCC Systems Cloud Native Platform into Azure using Terraform open source and additional modules from the LexisNexis Risk Solutions Terraform open-source registry.

Godson Fortil
Software Engineer I
LexisNexis® Risk Solutions Group

Configuring a HashiCorp Vault as a Certificate Manager on HPCC Systems

The importance of certificate management and instructions for setting up and configuring a Hashicorp Vault.

Nikita Jha
2021 HPCC Systems Intern Program

Features new and Improved in HPCC Systems 8.2.0 Gold

HPCC Systems Cloud Native and Bare Metal users can take a tour of the highlights included in this release.

Lorraine Chapman
Consulting Business Analyst
LexisNexis® Risk Solutions Group

File Storage on the HPCC Systems Cloud Native Platform

The method of defining storage on our Cloud Native platform has been rationalized and simplified in HPCC Systems 8.2.0.  This blog takes you through the new way storage is defined in the values.yaml file, providing some examples of how it can be used.

Gavin Halliday
Enterprise/Lead Architect
LexisNexis® Risk Solutions Group

Setting up a default HPCC Systems cluster on Microsoft Azure Cloud Using HPCC Systems 7.8.x and Kubernetes

Walk through the steps required to setup a default cluster on Microsoft Azure

Jake Smith
Enterprise/Lead Architect
LexisNexis® Risk Solutions Group

Persisting Data in an HPCC Systems Cloud native environment

How to persist your data and get your data on to your cluster in a cloud native environment.

Gavin Halliday
Enterprise/Lead Architect
LexisNexis® Risk Solutions Group

Configuring Storage in the Cloud Native HPCC Systems Platform

This blog provides more details about configuring storage on a cloud native HPCC Systems cluster.

Gavin Halliday
Enterprise/Lead Architect
LexisNexis® Risk Solutions Group

Importing Data into the Cloud Native HPCC Systems Platform

This blog continues the theme of storing and persisting data, focusing on how to use storage planes.

Gavin Halliday
Enterprise/Lead Architect
LexisNexis® Risk Solutions Group

Exporting Data from an HPCC Systems Bare Metal platform to our Cloud Native Platform

A tutorial style blog providing the steps to take to export files to a Kubernetes system using azure blob storage.

Gavin Halliday
Enterprise/Lead Architect
LexisNexis® Risk Solutions Group

Installing HPCC Systems with Istio on Microsoft Azure

Can we use a service mesh for internode communication within HPCC Systems?

This blog provides a tutorial style guide to the answer to this question, focusing on installing a Service Mesh (specifically Istio) with HPCC Systems and making it work.

Manish Kumar Jaychand
Technology Lead
Infosys

Installing HPCC Systems with Linkerd on Microsoft Azure

In this tutorial style blog, Nandhini shares the steps needed to install HPCC Systems with Linkerd on Microsoft Azure. Find out more about Linkerd, which is a service mesh for Kubernetes. It makes running services easier and safer by providing runtime debugging, observability, reliability and security without requiring any changes to code.

Nandhini Velu
Technology Analyst
Infosys

HPCC Systems Logs in the Cloud -Update Nov 2021

Log visualizations help identify, track and predict important events and trends on HPCC Systems clusters, by spotting interesting patterns and giving you visual clues which are easier to interpret than reading through the log file itself. Starting in HPCC Systems 8.0.0, a mechanism will be provided to deploy a lightweight Elastic Stack instance as a simple solution for the logs on-cloud processing challenges. In this blog, Rodrigo Pastrana walks through how to get started, view the logs, apply filter conditions and configure the Elastic Stack components.

Rodrigo Pastrana
Architect
LexisNexis® Risk Solutions Group

Containerized HPCC Systems Platform

This book focuses on containerized deployments. The first section is about using Docker containers and Helm charts locally. The second part uses the same techniques in the cloud.

Jim DeFabia
Consulting Software Engineer
LexisNexis® Risk Solutions Group

What's new in HPCC Systems 7.10.0

New features now available in both our Cloud native and bare metal HPCC Systems platforms.

Lorraine Chapman
Consulting Business Analyst
LexisNexis® Risk Solutions Group

What's new in HPCC Systems 7.12.0

New features now available in both our Cloud native and bare metal HPCC Systems platforms.

Lorraine Chapman
Consulting Business Analyst
LexisNexis® Risk Solutions Group

Interns Contributing to the HPCC Systems Cloud Native Platform

Some of our Class of 2020 interns have been working on cloud specific projects, while others have added new items to their project task list which focus on setting up and testing HPCC Systems using different Cloud platforms. This blog shares the details of the valuable contributions our interns are making to this ongoing development project.

Lorraine Chapman
Consulting Business Analyst
LexisNexis® Risk Solutions Group

Leveraging and Evaluating Kubernetes Support on Microsoft Azure with the HPCC Systems Cloud Native Platform.

Yash Mishra is studying for a Masters in Computer Science at Clemson University. This summer, he joined the HPCC Systems intern program to work on a project that contributes to our ongoing cloud native platform development project. In this blog, Yash shares the results of his work focusing on cost considerations, storage and troubleshooting.

Yash Mishra
Masters of Computer Science
Clemson University

Setting up an HPCC Systems cluster on AWS EKS

Our 'Going Cloud Native' blog series continues with this demonstration, showing how to setup and deploy an HPCC Systems Platform Cluster on the AWS Elastic Kubernetes Service (EKS). This tutorial style blog is a truly collaborative effort featuring contributions from Lucas Varella who is an intern studying at the University of Santa Catarina (UFSC) in Brazil, our LexisNexis Risk Solutions Group colleagues, Hugo Watanuki and Xiaoming Wang and Akash Gheewala a Solutions Architect from Amazon Web Services.

Lucas Varella
Bachelor of Information Systems
Federal University of Santa Catarina, (UFSC), Brazil

Hugo Watanuki
Senior Technical Support Engineer
LexisNexis Risk Solutions Group

Xiaoming Wang
Senior Consulting Software Engineer
LexisNexis® Risk Solutions Group

Akash Gheewala
Solutions Architect
Amazon Web Services (AWS)

How to... Videos

A series of short videos providing some quick tips and tricks for completing specific tasks.

Various HPCC Systems Developers

2021 HPCC Systems Community Day Presentations about our Cloud Native Platform 

Description

Presenter(s)

2021 HPCC Systems Community Day Presentations about our Cloud Native Platform 

Description

Presenter(s)

A Customized HPCC Systems Cloud Deployment for Your Company

Focuses on the approach and methods used in a custom Azure cloud deployment, based on our own company experience. This includes how to perform robust deployments of HPCC Systems environments on the cloud, with a focus on security, maintainability and supportability, and using the precepts of zero touch and zero trust.

Jon Burger
LexisNexis Risk Solutions Group

HPCC Systems Logging in the Cloud and an Elastic Stack Solution

As HPCC Systems continues its journey to the cloud, one major challenge faced is the ephemeral nature of log data and the accessibility of distributed application-level logs. This presentation discusses these challenges, the HPCC Systems logging architecture, and a simple Elastic Stack-based solution to the challenge. We'll demonstrate in detail the end-to- end solution, which includes Helm-based deployment, Kibana configuration, HPCC Systems log exploration, querying, and filtering. We'll also discuss an advanced topic that improves log data query performance by utilizing Elastic Search Ingest Pipelines. Finally, we'll touch on other possible solutions such as Azure Log Analytics.

Greg Panagiotatos & Rodrigo Pastrana
LexisNexis Risk Solutions Group

A Simple HPCC Systems Cloud Deployment for Open-Source Users

How to setup and implement a basic HPCC Systems cluster in the cloud using Azure Kubernetes. We will walk through the deployment configuration leveraging Terraform, GitOps/Flux2 and storage settings. Note: This talk is intended for the wider open source HPCC Systems community. It is advised to check with your organization for any specific security protocols

Xiaoming Wang & Godson Fortil
LexisNexis Risk Solutions Group

Running a Containerized HPCC Systems Platform Locally


Includes a demo where you can learn how to start up and use a containerized single-node HPCC Systems Platform cluster on a local machine running Docker Desktop or Minikube to use for ECL Development, testing, or training. This provides a platform for hands-on experience with HPCC Systems to experiment with and even create real-world data analytics applications--all on your desktop or laptop PC.

Jim DeFabia & Christopher Lo
LexisNexis Risk Solutions Group

Terasort with HPCC Systems on Azure Kubernetes Service and High Performance Storage

Challenges, AKS considerations and storage options are discussed, including a demo covering the setup and configuration of HPCC Systems on AKS with Blob NFS 3.0 and performing a Terasort.

Shrikrishna Khose & Steve Griffith
Microsoft

Design Considerations for Migrating Your HPCC Systems Data Lake to the Cloud


Lessons learned and design best practices through our own cloud migration experience are shared. The beginning of our presentation is a simple installation of our cluster on Azure using the community helm charts. During this demo we hit topics such as how the HPCC Systems platform differs between the Kubernetes cluster that we are deploying and the bare metal installations that community members are familiar with. Dive into helm for HPCC Systems, the value of .yaml files and a few different ways that the cluster can be configured and explain storage in the cloud compared to bare metal. Then learn about ROXIE and Thor usage in the cloud. Krishna covers some details about getting query lists, suspended queries, and doing package file deployments. Michael expands on basic security features that end users will want to enable in the cloud, including encryption in transit and at rest in a cloud environment such as Azure.

Krishna Turlapathi & Michael Gardner
LexisNexis Risk Solutions Group

Cloud Security & Authentication in HPCC Systems


Covers a discussion of some current and future technology enhancements around HPCC Systems platform security, with a primary focus on cloud deployments. This session will include a look at support for mutual transport layer security between internal components within an HPCC Systems environment, external facing TLS for securing access into the HPCC Systems environment, using cert-manager to generate TLS certificates for HPCC Systems services and components, installing externally created TLS certificates, secrets management, and a look at our plans for future support of OAUTH2 based authentication and authorization, with an initial focus on support for OAUTH2 integration with Azure Active Directory services.

Russ Whitehead, Tony Fishbeck & Mark Kelly
LexisNexis Risk Solutions Group

Securing Your Cloud Native HPCC Systems with Service Mesh


With the advent of Cloud and Kubernetes over the years, machines are no longer considered as attached to a data center. Machines are more ephemeral than ever before. The traditional architecture of the HPCC Systems environment harnessed the physical storage of each node and that in turn gave certain performance benefits. But with cloud, it is no longer necessary to have a fixed machine for a process. The true power of cloud can be harnessed only when we treat them as ephemeral. Therefore, a Thor worker node which is always on in a traditional HPCC Systems environment is spun up only when it is required in a cloud environment. With the latest cloud native version of HPCC Systems, we now have the flexibility to spin up the clusters only when required. In this session, we will cover how the latest cloud native platform is different from the bare metal version, explain service mesh and how it fits into the HPCC Systems scheme of things, and a comparison of service mesh Istio and Linkerd.

Manish Kumar Jaychand
Technology Lead
Infosys

What’s New in HPCC Systems and the Cloud Native Roadmap

An update on the new features in the latest release including data handling in the cloud native version

Gavin Halliday
Enterprise/Lead Architect
LexisNexis® Risk Solutions Group

HPCC Systems on Azure: Present and Future Possibilities

The Microsoft team will discuss the strong collaboration between Microsoft and LexisNexis Risk Solutions. See how HPCC Systems can be the foundation for a Data Lake and explore Azure services integration possibilities. Bonus: high level overview on Azure certification paths.

Shrikrishna Khose, Senior Cloud Solution Architect, and Steve Griffith, Principal Technical Specialist
Cloud Native Global Black Belt Team, Microsoft

HPCC Systems Tech Talk Summer Series - Path to the Cloud Interviews

Who's in the room

HPCC Systems Tech Talk Summer Series - Path to the Cloud Interviews

Who's in the room

An interview with Richard Chapman - Part 1
Hear Richard Chapman talk about this ongoing development project

Flavio Villanustre
CISO and VP of Technology 
LexisNexis® Risk Solutions Group

and

Richard Chapman
VP Technology
HPCC Systems Development Lead
LexisNexis® Risk Solutions Group

An interview with Jake Smith
Jake talks about the latest developments and the challenges faced in going Cloud native with HPCC Systems

Flavio Villanustre
CISO and VP of Technology 
LexisNexis® Risk Solutions Group

and

Jake Smith
Enterprise/Lead Architect 
LexisNexis® Risk Solutions Group

An interview with Gavin Halliday
Find out more about file storage on the HPCC Systems Cloud native platform

Flavio Villanustre
CISO and VP of Technology
LexisNexis® Risk Solutions Group

and

Gavin Halliday
Enterprise/Lead Architect
LexisNexis® Risk Solutions Group

An interview with Ming Wang
Listen to Ming talk about Docker desktop, Kubernetes and an ongoing project looking into setting up HPCC Systems on the Google Cloud platform, including the new Google Anthos GKE

Flavio Villanustre
CISO and VP of Technology
LexisNexis® Risk Solutions Group

and

Ming Wang
Senior Consulting Software Engineer
LexisNexis® Risk Solutions Group

An Interview with Shamser Ahmed

Shamser talks about Cloud Provisioning and the latest future development work planned for the Workunit Analysis Tool. 

Flavio Villanustre
CISO and VP of Technology
LexisNexis® Risk Solutions Group

and

Shamser Ahmed
Senior Consulting Software Engineer
LexisNexis® Risk Solutions Group

An Interview with Godji Fortil

Godji talks about the different types of testing taking place on our cloud native platform, including configuration, functional and performance testing. 

Flavio Villanustre
CISO and VP of Technology
LexisNexis® Risk Solutions  Group

and

Godson Fortil
Software Engineer I
LexisNexis® Risk Solutions Group

An Interview with Anthony Fishbeck

Tony talks about streamlining the configuration and deployment of ESP for cloud deployment.

Flavio Villanustre
CISO and VP of Technology
LexisNexis® Risk Solutions Group

and

Anthony Fishbeck
Senior Architect 
LexisNexis® Risk Solutions Group

An Interview with Lorraine Chapman

Lorraine talks about the contribution our Class of 2020 interns are making to our Cloud Native development project and introduces the HPCC Systems Roadmap.

Flavio Villanustre
CISO and VP of Technology,
LexisNexis® Risk Solutions Group

and


Lorraine Chapman
Consulting Business Analyst 
LexisNexis® Risk Solutions Group

An Interview with Richard Chapman - Part 2

Richard completes this series with an update providing the latest information about our new Cloud Native platform.

Flavio Villanustre
CISO and VP of Technology
LexisNexis® Risk Solutions Group

and

Richard Chapman
VP Technology
HPCC Systems Development Lead
LexisNexis® Risk Solutions Group

All pages in this wiki are subject to our site usage guidelines.