[GitHub] justinborromeo opened a new pull request #7133: 6088 - Time Ordering On Scans

[GitBox <gi...@apache.org>]

justinborromeo opened a new pull request #7133: 6088 - Time Ordering On Scans
URL: https://github.com/apache/incubator-druid/pull/7133
 
 
   This PR addresses points 2 and 3 of issue #6088, is based on proposal #7036, and stems from a refactor of the spaghetti in PR #7024.
   
   This PR introduces two strategies for time-ordering results from scan queries:
   
   1. Priority Queue: Each segment on a Historical is opened sequentially.  Every row is added to a bounded priority queue which is ordered by timestamp.  For every row above the result set limit, the row with the earliest (if descending) or latest (if ascending) timestamp will be dequeued.  After every row has been processed, the sorted contents of the priority queue are streamed back to the Broker(s) in batches.  
   
   2. K-Way/N-Way Merge: Each segment on a Historical is opened in parallel.  Since each segment's rows are already time-ordered, a k-way merge can be performed on the results from each segment.  This approach doesn't persist the entire result set in memory (like the Priority Queue) as it streams back batches as they are returned from the merge function.
   
   Each strategy has a corresponding configurable limit to protect historicals from OOMEs:
   
   1. The `druid.query.scan.maxRowsQueuedForTimeOrdering` property protects
   from OOMEs by limiting the number of rows in the query result set when time ordering is used.
   
   2. The `druid.query.scan.maxSegmentsTimeOrderedInMemory` limit protects from creating too many decompression/decoding buffers by capping the number of segments opened per historical when time ordering is used.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org