[GitHub] samarthjain commented on a change in pull request #7088: Improve parallelism of zookeeper based segment change processing

[GitBox <gi...@apache.org>]

samarthjain commented on a change in pull request #7088: Improve parallelism of zookeeper based segment change processing
URL: https://github.com/apache/incubator-druid/pull/7088#discussion_r258334781
 
 

 ##########
 File path: server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderConfig.java
 ##########
 @@ -46,7 +47,7 @@
   private int announceIntervalMillis = 0; // do not background announce
 
   @JsonProperty("numLoadingThreads")
-  private int numLoadingThreads = 10;
+  private int numLoadingThreads = JvmUtils.getRuntimeInfo().getAvailableProcessors();
 
 Review comment:
   Before, numLoading threads config wasn't being used. These threads also do the CPU intensive work of uncompressing the segment and memory mapping them. But you are right, the work is probably more I/O bound than CPU. The change I made was to provide more of a lower bound on minimum number of threads which should be safe for this pool. But I could be swayed to change it back to what it was. FWIW, in our internal setup, I have set the config to 2 * number of CPUs and it has been holding up fine.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org