You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/20 14:43:53 UTC
[GitHub] [pinot] jadami10 opened a new issue, #8931: Broker routing time is impact by the time pruner with 10k+ segments
jadami10 opened a new issue, #8931:
URL: https://github.com/apache/pinot/issues/8931
We seen for tables where we have 10 thousand or more partitions, broker routing time will increase to 100ms or more. We have not had a chance to profile yet, but the theory is that serializing/deserializing interval trees with large numbers of segments is causing issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jadami10 commented on issue #8931: Broker routing time is impact by the time pruner with 10k+ segments
Posted by GitBox <gi...@apache.org>.
jadami10 commented on issue #8931:
URL: https://github.com/apache/pinot/issues/8931#issuecomment-1161899428
I've DMed you a profile from the broker where you can see the time spent on both partition and time pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8931: Broker routing time is impact by the time pruner with 10k+ segments
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8931:
URL: https://github.com/apache/pinot/issues/8931#issuecomment-1160781895
There are 2 part of the issue:
1. Partition pruning (`SinglePartitionColumnSegmentPruner`): we re-calculate the matching partition on a per segment per query basis, which can be very costly when there are lots of segments. Instead, we should be able to pre-group the segments to each partition id so that we only need to calculate the matching partition once
2. Time pruning (`TimeSegmentPruner`): This is where the interval tree is introduced. Currently we first get all the available segments matching the time range, then check if the segment is selected. Instead, we can probably only match the selected segments from the previous step.
Some profile number can help identify the hotspot of the algorithm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org