[GitHub] samarthjain commented on issue #7088: Improve parallelism of zookeeper based segment change processing

[GitBox <gi...@apache.org>]

samarthjain commented on issue #7088: Improve parallelism of zookeeper based segment change processing
URL: https://github.com/apache/incubator-druid/pull/7088#issuecomment-464933495
 
 
   Overview of the main changes:
   
   1) I have added a new class CuratorBasedLoadQueuePeonV2. The class borrows from CuratorBasedLoadQueuePeon. The biggest different between the two classes is that I have removed the notion of tracking the segment being currently processed. Also, we now use a fixed sized threadpool for creating nodes in zookeeper. The default size of the pool is twenty threads which can be changed by the setting the ```druid.coordinator.curator.create.zknode.numThreads``` in coordinator config. I have also introduced a random delay in these threads so that there is some jitter in zookeeper create node requests.
   
   For now, I have left the CuratorBasedLoadQueuePeon class as it is. We can either get rid of it or we can introduce an additional config which controls which version of Curator based peon to use. Right now I have hardcoded to use CuratorBasedLoadQueuePeonV2 in LoadQueueTaskMaster.
   
   2) In ZkCoordinator we now use a multi threaded pool to process segment change requests. The size of this thread pool can be configured via ```druid.segmentCache.numLoadingThreads```. Yes, this change brings back the earlier non effective config.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org