[GitHub] [pinot] Jackie-Jiang commented on issue #8931: Broker routing time is impact by the time pruner with 10k+ segments

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on issue #8931:
URL: https://github.com/apache/pinot/issues/8931#issuecomment-1160781895

   There are 2 part of the issue:
   1. Partition pruning (`SinglePartitionColumnSegmentPruner`): we re-calculate the matching partition on a per segment per query basis, which can be very costly when there are lots of segments. Instead, we should be able to pre-group the segments to each partition id so that we only need to calculate the matching partition once
   2. Time pruning (`TimeSegmentPruner`): This is where the interval tree is introduced. Currently we first get all the available segments matching the time range, then check if the segment is selected. Instead, we can probably only match the selected segments from the previous step.
   
   Some profile number can help identify the hotspot of the algorithm


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org