HPCC Systems to SQL Store Integration

The proposal period for 2022 internships is now closed
The proposal period for 2023 internships will open in November 2022

Find out about the HPCC Systems Summer Internship Program.

Project Description

SQL Databases and Data Warehouse technologies provide the best reporting experience as a backend for BI tools. While HPCC is great at data ETL and query materialization, it is not designed to perform SQL query workloads. A solution would be to move HPCC data after HPCC processes it into a SQL database. While this is workable currently, it takes multiple steps that are customized to each use case. We need a system that treats a SQL database table like an HPCC logical file. Hence, the proposal is to create a new file type (SQL) that triggers the ECL OUTPUT action to write to a database instead of the HPCC filesystem.

While writing to an SQL database is the immediate need, we could also incorporate reading data from the SQL database at a later point.

Benefits of this project

• Increase HPCC Systems adoption. This feature will enable HPCC to incorporate ETL and Database workflows seamlessly.
• Superb support for BI tools. In addition, HPCC can bundle database technology that enables these tools easily.
• Will eliminate handholding of data as the HPCC platform automates the IO. Hence, a single system can enforce security. This will be a huge win for our cloud journey.

If you are interested in this project, please contact the mentor.