Page tree
Skip to end of metadata
Go to start of metadata

The proposal period for 2022 internships is now closed
The proposal period for 2023 internships will open in November 2022

This project is available as a student work experience opportunity with HPCC Systems this summer. Curious about other projects we are offering? Take a look at our Ideas List

Find out about the HPCC Systems Summer Internship Program.

Project Description

Currently HPCC Systems uses its own message interface (libmp) built on top of Berkeley sockets to send and receive messages. HPCC Systems uses its MP layer in the Thor, Eclagent, Dafilesrv, Sasha, Backupnode and Dali components. We would like to explore if using a different message layer offers improved performance, robustness and code maintainability. We have, in the recent past, had a project to do this with MPI (See the poster about this project/listen to Saminda Wijeratne speak about this intern project from 2018), the industry standard Message Passing Interface layer. This intern project successfully demonstrated that the existing HPCC Systems interface could be replaced.

HPCC System is highly distributed and messages are often sent and received concurrently from many threads, making synchronization and performance key design considerations. One such message layer we are interested in is ZeroMQ (0MQ) (zeromq.org). Possible optimizations the transport layer offers, such as RDMA and shared memory for intranode communication are potentially interesting.

Completion of this project involves:

Provide details about the following:

  • Checked in code
    Replacement of our libmp layer with a new layer implementing all existing functionality 
  • Documentation
    Document code directly and provide test guide 
  • Test code
    Can use existing mptest and regression suite, could extend mptest for more performance benchmarks 
  • Regression tests
    Can use existing mptest and regression suite 

Expected feature list:

All the same features and functions as the existing libmp.

By the midterm review we would expect you to have:

mptest program working within a single machine using > 1 slaves and > 1 virtual slaves

Please add details below including the JIRA ticket details:

Mentor

Mark Kelly
Contact details

Backup Mentor: Jake Smith
Contact Details 

Skills needed
  • Ability to code in C++
  • Ability to build and test the HPCC system (guidance will be provided)
  • Ability to write test code
Deliverables
  • Checked in code
  • Test cases demonstrating the correct behaviour and performance
  • Documentation
Other resources
  • No labels