[GitHub] [incubator-druid] eranmeir edited a comment on issue #7838: Improve IncrementalIndex concurrency scalability

[GitBox <gi...@apache.org>]

eranmeir edited a comment on issue #7838: Improve IncrementalIndex concurrency scalability
URL: https://github.com/apache/incubator-druid/pull/7838#issuecomment-502591292
 
 
   @leventov Thanks for the review. I will soon add a commit that addresses the issues you raise. I have updated the attached pdf with raw numbers of pre- and post-fix benchmarks.
   
   @jihoonson Thanks for the feedback. We used Oak for indexing large amounts of data and saw significant performance gains from off-heap storage and multi-threaded indexing. This separate PR may be thought of as ground work for Oak, but multi-threaded performance gains are obviously independent. 
   
   The way I understand what you write, queries that happen before segment merge may benefit from increase in indexing throughput. 
   
   We haven't tried comparing costs of index building and segment merging. Perhaps we can try and leverage multi-threading in the merge process. I'll be happy to discuss this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org