[GitHub] samarthjain commented on issue #7068: zk nodes for segments to load are created serially causing slow download of segments

[GitBox <gi...@apache.org>]

samarthjain commented on issue #7068: zk nodes for segments to load are created serially causing slow download of segments
URL: https://github.com/apache/incubator-druid/issues/7068#issuecomment-463429473
 
 
   @fjy or @gianm - would you happen to remember if this was a deliberate design decision? Before I go down the path of making changes, I want to be sure I am not missing out on some tricky issues here. 
   
   The issue is fairly severe. In my logs, I see that it took over 5 hours for the coordinator to create zk entries and roughly the same time for historicals to download 3TB of segments from s3.
   
   We have also seen that in some cases, after a largish batch ingestion of a few hundred GBs, the segments are not available on the cluster for almost a couple of hours. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org