[GitHub] [pinot] richardstartin commented on issue #7929: Performance problem in segment build

[GitBox <gi...@apache.org>]

richardstartin commented on issue #7929:
URL: https://github.com/apache/pinot/issues/7929#issuecomment-997279739


   @sajjad-moradi the purpose of the change was to reduce the amount of memory required for variable length data when some of the values are very long, which results in very large buffers. An impact on build time was expected, but the impact on small fixed size chunks wasn’t considered. Memory mapping 4-8KB at a time doesn’t make sense and will result in very high syscall overhead, so this is less than ideal. We now have the V4 raw format for variable length data, so the benefits (reduced memory consumption) of the change are no longer relevant important, so it makes sense to me to reintroduce the compressed chunk buffer. However, some of the changes in that PR were important, so the change should be limited to reintroducing the compression buffer and removing the memory mapping per chunk.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org