The proposal period for 2022 internships is now closed
The proposal period for 2023 internships will open in November 2022
Student work experience opportunities also exist for students who want to suggest their own project idea. Project suggestions must be relevant to HPCC Systems and of benefit to our open source community.
Find out about the HPCC Systems Summer Internship Program.
HPCC Systems clients frequently need to ensure that shared resources are not changed whilst they are being accessed, or that no one has access whilst they are being changed. For this we need a distributed locking engine that allows HPCC Systems client components to lock a named resource in an exclusive or shared manner. Currently we have an implementation of this built into our distributed meta data component Dali. It manages distributed locks established by clients that are currently closely coupled with elements within the meta store.
For example, each logical file is located via a psuedo xpath in Dali’s meta store. A distributed lock is usually associated with this node in the meta store, so that contention with other clients to a logical file can be managed.
The Dali implementation is not load balanced and is a single point of failure, if Dali stops, all clients lose their locking state.
This project is to investigate and create a proof of concept c++ client library, that implements the required semantics using 3rd party distributed locking servers (e.g. zookeeper / HashiCorp’s Consul, Etcd), that demonstrated benefits in terms of load-balancing and redundancy.
The basic semantics required are:
- Ability to lock a named lock exclusively (no other client will be able to lock the same lock).
- Ability to lock a named lock non-exclusively (other clients can also lock this named lock non-exclusively, but none will be able to lock it exclusively).
- Ability to test lockability with a timeout, e.g. try-lock(<timeout>)
- Ability to unlock a previously locked named lock.
- Automatically release all locks when client terminates.
From the clients point of view the interface should look something like this:
// attempt to lock the named lock until the timeout is hit
virtual LockId lock(const char *lockName, bool exclusive, unsigned timeout) = 0;
// unlock an existing lock. NB: the lock may still remain locked by others.
virtual bool unlock(LockId lockId) = 0;
// change the exclusivity of an existing lock
virtual bool changeLock(LockId lockId, bool exclusive) = 0;
There should also be related services that allow lock discovery, e.g. to list all locks on the system, or ones that match a pattern, or ones that belong to a certain client.
If you are interested in this project, please contact Jake Smith.
Completion of this project involves:
- <4+ high level tasks to be completed>
By the mid term review we would expect you to have:
- <What must be completed to pass the evaluation and continue on to complete the project>
Backup Mentor: TBD
End of project