[GitHub] [incubator-druid] samarthjain commented on a change in pull request #7088: Improve parallelism of zookeeper based segment change processing

[GitBox <gi...@apache.org>]

samarthjain commented on a change in pull request #7088: Improve parallelism of zookeeper based segment change processing
URL: https://github.com/apache/incubator-druid/pull/7088#discussion_r272482707
 
 

 ##########
 File path: docs/content/configuration/index.md
 ##########
 @@ -1251,7 +1251,11 @@ These Historical configurations can be defined in the `historical/runtime.proper
 |`druid.segmentCache.infoDir`|Historical nodes keep track of the segments they are serving so that when the process is restarted they can reload the same segments without waiting for the Coordinator to reassign. This path defines where this metadata is kept. Directory will be created if needed.|${first_location}/info_dir|
 |`druid.segmentCache.announceIntervalMillis`|How frequently to announce segments while segments are loading from cache. Set this value to zero to wait for all segments to be loaded before announcing.|5000 (5 seconds)|
 |`druid.segmentCache.numLoadingThreads`|How many segments to drop or load concurrently from from deep storage.|10|
-|`druid.segmentCache.numBootstrapThreads`|How many segments to load concurrently from local storage at startup.|Same as numLoadingThreads|
+|`druid.coordinator.loadqueuepeon.curator.numCreateThreads`|Number of threads creating zk nodes corresponding to segments that need to be loaded or dropped.|10|
+|`druid.coordinator.loadqueuepeon.curator.numCallbackThreads`|Number of threads for executing callback actions associated with loading or dropping of segments.|2|
+|`druid.coordinator.loadqueuepeon.curator.numMonitorThreads`|Number of threads to use for monitoring deletion of zk nodes|1|
+|`druid.coordinator.curator.create.zknode.batchSize`|Number of zk nodes to create in one iteration.|5000|
+|`druid.coordinator.curator.create.zknode.repeatDelay`|Delay before creating next batch of zk nodes|PT1M|
 
 Review comment:
   If we adopt max_network_throughput, then we would need to measure the rate at which we are adding segments to the processing queue. It is doable and there are rate limiter implementations out there including in Guava. Having said that, not all of the max_network_throughput configured by user can be directly used for rate limiting purposes. Time is spent not only downloading segments, but also in decompressing and memory mapping. So the real rate to limit would be a factor of the throughput. Would that mean we would need to make the factor configurable (which isn't too bad really, just wanted to call it out though). 
   
   Any other feedback you have @egor-ryashin? . I would like to get this pull request in before it goes stale beyond recovery :).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org