[GitHub] [incubator-druid] himanshug opened a new issue #9053: [Proposal] Druid discovery extension for Kubernetes

[GitBox <gi...@apache.org>]

himanshug opened a new issue #9053: [Proposal] Druid discovery extension for Kubernetes
URL: https://github.com/apache/incubator-druid/issues/9053
 
 
   ### Motivation
   
   Druid has provisions for writing an extension that could be used for "node discovery" and "leader election". Such extension coupled with "HTTP" based "segment management" and "remote task runner" could be used to obviate the need of operating a Zookeeper cluster for Druid.
   However, no such extension exists and above is just theory.
   
   Proposal is to write an extension based on APIs exposed by Kubernetes API server (which itself is backed by [etcd](https://github.com/etcd-io/etcd) which has sufficient primitives to be able to do  "node discovery" and "leader election" .
   
   ### Proposed changes
   
   A new extension, "druid-kubernetes-extensions" , would be added with implementations of various discovery related interfaces such as "DruidNodeDiscoveryProvider", "DruidLeaderSelector", "DruidNodeAnnouncer" etc.
   Additionally, since this is first such extension, there might be some changes needed in core as well to enable writing the extension.
   
   ### Rationale
   
   etcd based extension could also be written, but in many Kubernetes deployments, access to etcd is guarded and users of K8s clusters are not expected to use it. So for users(I happen to be one of those) deploying Druid on kubernetes, it is required to not depend directly on etcd. So, I decided not to pursue that.
   it is also possible to have Druid implement one of the consensus algorithms, but that is a lot more work and is very error prone. Consensus algorithms are notoriously hard to implement. In the current use case, that is not strictly needed.
   
   ### Operational impact
   
   None
   
   ### Test plan (optional)
   
   I would be testing the extension on my own Druid clusters deployed in Kubernetes.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org