[GitHub] pzhdfy commented on issue #6988: [Improvement] historical fast restart by lazy load columns metadata(20X faster)

[GitBox <gi...@apache.org>]

pzhdfy commented on issue #6988: [Improvement] historical fast restart by lazy load columns metadata(20X faster)
URL: https://github.com/apache/incubator-druid/pull/6988#issuecomment-463524867
 
 
   historical size:   100k segments and 10TB size
   Each time, We will drop the page cache
   
   **1. without this patch**
       _1) druid.segmentCache.numBootstrapThreads = 1_
           40min
       _2) druid.segmentCache.numBootstrapThreads = 10_
           also 40 min, because when numBootstrapThreads=1, reading all columns metadata has cost 100% disk util, setting numBootstrapThreads a  higher number makes no sense
   
   **2. with this patch and lazyLoadOnStart = false**
      the result is very similar with scenario 1，there are not any odd performance effects， becauce we don't use  memoized suppliers when  lazyLoadOnStart = false
   
   **3. with this patch and lazyLoadOnStart = true**
      _1) druid.segmentCache.numBootstrapThreads = 1_
           8min
       _2) druid.segmentCache.numBootstrapThreads = 10_
           2 min, 4 times faster than numBootstrapThreads = 1, because when numBootstrapThreads=1 and  lazyLoadOnStart = true , we don't read all columns metadata , disk util will less than 100%, setting numBootstrapThreads a  higher number will benefit
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org