Build System as Code

The proposal period for 2022 internships is now closed
The proposal period for 2023 internships will open in November 2022

Find out about the HPCC Systems Summer Internship Program.

Project Description

The aims of this project include codifying build configurations and infrastructures for the various HPCC Systems Platform builds, plugins, and other components. This internship project is the first of its kind as we seek to move from a software-based build environment to infrastructure as code (IaC).

Currently, the HPCC Systems Platform builds are done on Jenkins through an automated process. While that is the way we have been building the platform and its components for a long time, it is no longer the ideal way in today’s innovative tech space. Some of the issues with such an environment are that configurations are not version controlled, hard to manage in numbers, not very consistent, and their lifetime does not exceed the lifetime of the server. What we hope to achieve with IaC is a version-controlled build system as code that offers a lifetime that exceeds the lifetime of the underlying infrastructures, improves consistency and stability, boosts productivity, minimizes user errors, while being cloud agnostic.

If you are interested in this project, please contact the mentor of this project.

Completion of this project involves:

Week 1 – onboarding / design
Week 2 – design / initial build images (Docker Images for each support Distro, I.e, Ubuntu, CentOS, Mac and Windows)
Week 3 – prototyping locally for baremetal platform builds
Week 4 – cmake/build optimizations based on week 3 input
Week 5 – plugin baremetal package builds
Week 6 – cmake/build optimizations based on week 5 input
Week 7 - setup matrix of builds in github actions
Week 8, 9, 10 – setup a jenkins x environment to test IaC flexibility in multiple cloud environments using helm/terraform
Week 11 – write ups documenting the BSaC, possible wiki entries
Week 12 - Demonstrate full build setup in github actions, azure, Jenkins

The role each of these resources plays in this project:

Docker	Image creation
Bash	HPCC build configuration
CMake	HPCC build configuration
Git	Version control
GitHub Action	Automation
Helm	Kubernetes configuration
Kubernetes	Container orchestration
Azure/AKS	Infrastructure deployment
AWS/EKS	Infrastructure deployment
Jenkins X	Pipeline
Terraform	Infrastructure management
GitOps	Build configuration

Mentor(s)

Michael Gardner

Backup Mentor(s):
Ming Wang, Godji Fortil

Skills needed

Docker, Bash, CMake, Git, GitHub Action, Azure Cloud, AWS Cloud, Linux, Jenkins X, Kubernetes, Helm, Terraform, GitOps
Ability to build and test the HPCC system (guidance will be provided).
Ability to write test code. Knowledge of ECL is not a requirement since it should be possible to re-use existing code with minimal changes for this purpose. Links are provided below to our ECL training documentation and online courses should you wish to become familiar with the ECL language.

Other resources

HPCC Systems website
Learning ECL documentation and on-line training courses
Docker - https://docs.docker.com/
Bash - https://www.redhat.com/sysadmin/learn-bash-scripting
CMake - https://cmake.org/cmake/help/latest/guide/tutorial/index.html
Git - https://www.atlassian.com/git/tutorials
GitHub Action - https://docs.github.com/en/actions/guides
Azure - https://azure.microsoft.com/en-us/get-started/
AWS - https://aws.amazon.com/getting-started/
Linux - https://training.linuxfoundation.org/resources/?_sft_content_type=tutorial
Jenkins X - https://jenkins-x.io/v3/about/what/

Page tree

Build System as Code

Mentor(s)

Skills needed

Other resources