Replace existing socket-based message passing interface with an open source package

The proposal period for 2022 internships is now closed
The proposal period for 2023 internships will open in November 2022

This project is available as a student work experience opportunity with HPCC Systems this summer. Curious about other projects we are offering? Take a look at our Ideas List. 

Find out about the HPCC Systems Summer Internship Program.

Project Description

Currently HPCC Systems uses its own message interface (libmp) built on top of Berkeley sockets to send and receive messages. HPCC Systems uses its MP layer in the Thor, Eclagent, Dafilesrv, Sasha, Backupnode and Dali components. We would like to explore if using a different message layer offers improved performance, robustness and code maintainability. We have, in the recent past, had a project to do this with MPI (See the poster about this project/listen to Saminda Wijeratne speak about this intern project from 2018), the industry standard Message Passing Interface layer. This intern project successfully demonstrated that the existing HPCC Systems interface could be replaced.

HPCC System is highly distributed and messages are often sent and received concurrently from many threads, making synchronization and performance key design considerations. One such message layer we are interested in is ZeroMQ (0MQ) (zeromq.org). Possible optimizations the transport layer offers, such as RDMA and shared memory for intranode communication are potentially interesting.

Completion of this project involves:

Provide details about the following:

• Checked in code
  Replacement of our libmp layer with a new layer implementing all existing functionality 
• Documentation
  Document code directly and provide test guide 
• Test code
  Can use existing mptest and regression suite, could extend mptest for more performance benchmarks 
• Regression tests
  Can use existing mptest and regression suite 

Expected feature list:

All the same features and functions as the existing libmp.

By the midterm review we would expect you to have:

mptest program working within a single machine using > 1 slaves and > 1 virtual slaves

Please add details below including the JIRA ticket details: