[GitHub] [incubator-druid] himanshug commented on a change in pull request #6988: [Improvement] historical fast restart by lazy load columns metadata(20X faster)

[GitBox <gi...@apache.org>]

himanshug commented on a change in pull request #6988: [Improvement] historical fast restart by lazy load columns metadata(20X faster)
URL: https://github.com/apache/incubator-druid/pull/6988#discussion_r352946425
 
 

 ##########
 File path: docs/configuration/index.md
 ##########
 @@ -1359,6 +1359,7 @@ These Historical configurations can be defined in the `historical/runtime.proper
 |`druid.segmentCache.announceIntervalMillis`|How frequently to announce segments while segments are loading from cache. Set this value to zero to wait for all segments to be loaded before announcing.|5000 (5 seconds)|
 |`druid.segmentCache.numLoadingThreads`|How many segments to drop or load concurrently from deep storage. Note that the work of loading segments involves downloading segments from deep storage, decompressing them and loading them to a memory mapped location. So the work is not all I/O Bound. Depending on CPU and network load, one could possibly increase this config to a higher value.|Number of cores|
 |`druid.segmentCache.numBootstrapThreads`|How many segments to load concurrently during historical startup.|`druid.segmentCache.numLoadingThreads`|
+|`druid.segmentCache.lazyLoadOnStart`|Whether or not to load segment columns metadata lazily during historical startup. When set to true, Historical startup time will be dramatically improved by deferring segment loading until the first time that segment takes part in a query, which will incur this cost instead.|false|
 
 Review comment:
   It is worth mentioning the catch mentioned in https://github.com/apache/incubator-druid/pull/6988#issuecomment-463528940 in the docs so that user can make a informed decision on whether to enable this flag or not.
   e.g. add following blurb ....
   
   "One catch is that if historical crashes while in the process of downloading and creating segment files on local disk, it is possible to end up with a corrupted segment on disk, this requires manual intervention to delete corrupted files. When the flag is set to true, historical startup would complete successfully and queries using this segment would fail at runtime."

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org