Page tree
Skip to end of metadata
Go to start of metadata

The proposal period for 2022 internships is now closed
The proposal period for 2023 internships will open in November 2022

Student work experience opportunities also exist for students who want to suggest their own project idea. Project suggestions must be relevant to HPCC Systems and of benefit to our open source community. 

Find out about the HPCC Systems Summer Internship Program.

Project Description

HPCC Systems clients frequently need to ensure that shared resources are not changed whilst they are being accessed, or that no one has access whilst they are being changed. For this we need a distributed locking engine that allows HPCC Systems client components to lock a named resource in an exclusive or shared manner. Currently we have an implementation of this built into our distributed meta data component Dali. It manages distributed locks established by clients that are currently closely coupled with elements within the meta store.

For example, each logical file is located via a psuedo xpath in Dali’s meta store. A distributed lock is usually associated with this node in the meta store, so that contention with other clients to a logical file can be managed.

The Dali implementation is not load balanced and is a single point of failure, if Dali stops, all clients lose their locking state.

This project is to investigate and create a proof of concept c++ client library, that implements the required semantics using 3rd party distributed locking servers (e.g. zookeeper / HashiCorp’s Consul, Etcd), that demonstrated benefits in terms of load-balancing and redundancy.

The basic semantics required are:

  • Ability to lock a named lock exclusively (no other client will be able to lock the same lock).
  • Ability to lock a named lock non-exclusively (other clients can also lock this named lock non-exclusively, but none will be able to lock it exclusively).
  • Ability to test lockability with a timeout, e.g. try-lock(<timeout>)
  • Ability to unlock a previously locked named lock.
  • Automatically release all locks when client terminates.

From the clients point of view the interface should look something like this:

   interface ILockService
   {
   // attempt to lock the named lock until the timeout is hit
       virtual LockId lock(const char *lockName, bool exclusive, unsigned timeout) = 0;
   // unlock an existing lock. NB: the lock may still remain locked by others.
       virtual bool unlock(LockId lockId) = 0;
   // change the exclusivity of an existing lock
       virtual bool changeLock(LockId lockId, bool exclusive) = 0;
   };

There should also be related services that allow lock discovery, e.g. to list all locks on the system, or ones that match a pattern, or ones that belong to a certain client.

If you are interested in this project, please contact Jake Smith.

Completion of this project involves:

  • <4+ high level tasks to be completed>

By the mid term review we would expect you to have:

  • <What must be completed to pass the evaluation and continue on to complete the project>
Mentor

Jake Smith
Contact Details

Backup Mentor: TBD
Contact Details

Skills needed
  • Proficiency in C++ required
  • Ability to build and test the HPCC system (guidance will be provided).
  • Ability to write test code. Knowledge of ECL is not a requirement since it should be possible to re-use existing code with minimal changes for this purpose. Links are provided below to our ECL training documentation and online courses should you wish to become familiar with the ECL  language.
Deliverables

Midterm

  • <Deliverable(s) to be achieved>

End of project

  • <Deliverables expected by the end of the internship>
Other resources
  • No labels