[GitHub] [incubator-druid] b-slim edited a comment on issue #7676: Add OakIncrementalIndex to Druid

[GitBox <gi...@apache.org>]

b-slim edited a comment on issue #7676: Add OakIncrementalIndex to Druid
URL: https://github.com/apache/incubator-druid/pull/7676#issuecomment-494628884
 
 
   > Exactly! This is the mostly important! And here we need your input! How does Druid manages Direct memory? Is there some policy? Documentation to read? Pointers to code?
   
   I don't think there is bunch of docs but in nutshell Druid uses 3 kind of memory resources.
   
   1.  Mapped memory for all the immutable data and it is managed by the OS
   2. DirectByte Buffers used for Decompression and Query Execution (Aggregate / merging) managed by common resource pool.  
   3. OnHeap stuff used by the rest including what you are working on the realtime in memory DS managed by the JVM
   I recommend you take a look on how we use those buffers for query processing, it is all coming from io.druid.guice.DruidProcessingModule you can track this `io.druid.guice.DruidProcessingModule#getMergeBufferPool`
   
   Not sure how you want to hook into this but maybe the Druid Processing module can be one of your block providers.
   This doc might help as well-> http://druid.io/docs/latest/operations/performance-faq.html
   As a side note: 
   - Netty buffers are backed by byteBuffers, they just add some wrappers around the buffer but you still can get the reference to it thought.
   - You might wanna have a flag to turn off unsafe reads most of the Druid users i know they prefer to run 10% slower and not wanting to deal with JVM seg fault stuff. 
   
   Sorry for short answer! will try touch base. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org