[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #6422: Allow segment generation with a bounded memory

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on issue #6422:
URL: https://github.com/apache/incubator-pinot/issues/6422#issuecomment-756977627


   I think you are referring to the hash-set for deduplicating the values? We also stores the values in an array for sorting purpose.
   We can use the off-heap data structure to reduce the memory usage, but that will also be inefficient.
   I have plan to unify the segment creation for offline and real-time segment, and we might be able to borrow the off-heap data structure in real-time segments.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org