[GitHub] [druid] liran-funaro edited a comment on pull request #10001: Optimizing incremental-index ingestion using off-heap key/value map (OakMap)

[GitBox <gi...@apache.org>]

liran-funaro edited a comment on pull request #10001:
URL: https://github.com/apache/druid/pull/10001#issuecomment-687738552


   We continue to evaluate Oak incremental-index on our (Verizon Media) production workload and seeing interesting results.
   For example, we run Druid's middle-manager on two identical VMs that consumed events from the same Kafka feed for an entire day; one with Oak and one with the "vanilla" Druid (on-heap incremental index).
   
   ### Oak vs. on-heap ("vanilla") incremental-index comparison:
   * Oak required 35% less flush operations for the entire duration
   * Which produced 50% larger partitions
   * As a result, Oak spent 50 **minutes** less time in flush operations; i.e., almost 40% reduction compared to the on-heap implementation
   * This also reduced the time it took to merge and push the segments by almost a minute.
   
   In the latest Druid Summit, we've seen that these kinds of optimizations are vital.
   Companies invest resources in post-compaction, thus, creating larger partitions during ingestion can reduce these efforts cost. 
   
   We appreciate any effort by the community for moving this forward, starting by reviewing our design proposal (#10321).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org