You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/11 06:43:26 UTC
[GitHub] [pinot] kishoreg commented on pull request #8879: [Draft] extract sort fields into a temp file for better data locality for sorting
kishoreg commented on PR #8879:
URL: https://github.com/apache/pinot/pull/8879#issuecomment-1152868478
Another idea along the lines of what @richardstartin mentioned but it does not have to be dictionary encoded and potentially avoid the need to sort the file on disk and sort complexity reduced from a function of num_rows to cardinality of the sort column
In the first pass,
create a map of <key, bitmap> sort this map/array on key.
you can now create a sorted doc id array (this is also optional)
you can simply create a record reader and pass the hashmap that will dictate the reading order.
We do something like this in real-time segment converter when we flush real-time to disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org