[GitHub] leventov commented on issue #6016: Druid 'Shapeshifting' Columns

[GitBox <gi...@apache.org>]

leventov commented on issue #6016: Druid 'Shapeshifting' Columns
URL: https://github.com/apache/incubator-druid/pull/6016#issuecomment-406901637
 
 
   Some high-level thoughts:
    - It would be really nice if along or before this PR something was done to make the segment format system more structured: #5347.
    - The query processing monomorphization framework (#4079) should now work per-block rather than per-interval, unless all encodings in the shape-shifting framework always do something like filling an `int[]` array or a ByteBuffer.
    - Could you please elaborate why `Memory` didn't work for you well?
    - > A downside of the current pooling approach I have in place is that each different block size has it's own set of pools to support it, so if multiple block sizes are used in practice, it will be a larger amount of heap and direct overhead.
   
      Why the same maximum-sized objects couldn't be used for blocks of any size, similarly how it is currently done with column compression buffers?
   
   Also, in case you didn't see this article: https://lemire.me/blog/2018/04/17/iterating-in-batches-over-data-structures-can-be-much-faster/, it might give you some ideas about block sizing in general.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org