[GitHub] [incubator-druid] sanastas commented on issue #7676: Add OakIncrementalIndex to Druid

[GitBox <gi...@apache.org>]

sanastas commented on issue #7676: Add OakIncrementalIndex to Druid
URL: https://github.com/apache/incubator-druid/pull/7676#issuecomment-503631910
 
 
   Hi Slim (@b-slim ),
   
   We have taken your comment about Netty Allocator seriously and implemented and measured OakIncrementalIndex performance based on Netty. The comparison was running IncrementalIngestionBenchmark and inserting 3 million rows which is about 3.8GGB of off-heap data. We have tried giving different off-heap size limits, the result can be seen in the following table.
   
   [Oak Native Allocator vs Netty.pdf](https://github.com/apache/incubator-druid/files/3306983/Oak.Native.Allocator.vs.Netty.pdf)
   
   To summarize, Netty Allocator requires up to twice memory and when its memory request is satisfied the latency is a bit slower. Looking closer on Netty implementation it is understandable, because Netty Allocator Pools hold buffers in different (exponentially increasing) sizes. This is reasonable for network protocol buffers but less reasonable for data storage.
   
   Bottom line, based on empirical results we would like to continue with the primitive OakNativeAllocator which satisfies the Druid needs and gives better performance. The experiments with Netty Allocator will continue. Thank you for you valuable input!
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org