You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "praveenc7 (via GitHub)" <gi...@apache.org> on 2025/02/14 04:01:27 UTC
[I] Significant Latency Overhead due to empty Response Handling [pinot]
praveenc7 opened a new issue, #15064:
URL: https://github.com/apache/pinot/issues/15064
Recent improvements to ensure a valid data type is returned when queries result in empty responses (due to broker pruning or segment pruning) have introduced substantial query overhead.
• In some cases, we observed a 200x increase in query latency (e.g., from 6ms to 1200ms).
• On average, the overhead was at least 100x, leading to high CPU utilization and even host failures.
### Root Cause
This issue appears to be linked to the following PRs:
1. [PR #13831](https://github.com/apache/pinot/pull/13831)
2. [PR #14918](https://github.com/apache/pinot/pull/14918)
### Findings
A/B testing and profiling point to a this operation compileQuery(https://github.com/apache/pinot/blob/ad6662fba8101c622a3a33ad5ea7771d3f5fe55f/pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java#L347) in the method as the bottleneck.
<img width="865" alt="Image" src="https://github.com/user-attachments/assets/d23850ea-afb6-4a77-aa22-17d702a4b824" />
This can be easily reproduced from one of the integ-test with and without the improvements from the above PRs
### **Open Question**
We’ve observed that this logic is commonly used in the multi-stage query engine for query compilation.
• Is this latency a known issue, or has something changed downstream that we may not be aware of?
• The current Calcite library has been in place for nearly eight months, so it’s unclear if a recent change is causing this behavior.
cc : @vvivekiyer @Jackie-Jiang @albertobastos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Significant Latency Overhead due to empty Response Handling [pinot]
Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on issue #15064:
URL: https://github.com/apache/pinot/issues/15064#issuecomment-2663860662
@yashmayya Yes we do see this in queries having large IN clauses. However this was observed in some simple queries as well
Query pattern
```
SELECT col_a, MAX(col_b)
FROM table_x
WHERE col_b >= 10000
AND col_c NOT IN ('value_x')
AND col_d = 123456789
// Large IN clause
AND col_a IN (
'x1',
x2',
'x3',
'x4'
.......
'x1000'
);
```
```
SELECT col_a, col_b, col_c, SUM(col_d)
FROM table_y
WHERE col_a IN ('XXXXX') -- High cardinality
AND col_c >= 10000
GROUP BY col_a, col_b, col_c
ORDER BY SUM(col_d) DESC
LIMIT 20000;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Significant Latency Overhead due to empty Response Handling [pinot]
Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on issue #15064:
URL: https://github.com/apache/pinot/issues/15064#issuecomment-2660607457
For example [decorrelation](https://github.com/search?q=repo%3Aapache%2Fpinot%20RelDecorrelator.decorrelateQuery&type=code) is un-necessary overhead for simple single-stage queries that no joins, no subqueries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Significant Latency Overhead due to empty Response Handling [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15064:
URL: https://github.com/apache/pinot/issues/15064#issuecomment-4580953897
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Significant Latency Overhead due to empty Response Handling [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15064: Significant Latency Overhead due to empty Response Handling
URL: https://github.com/apache/pinot/issues/15064
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Significant Latency Overhead due to empty Response Handling [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on issue #15064:
URL: https://github.com/apache/pinot/issues/15064#issuecomment-2662389263
@praveenc7 can you share some of the queries where you saw disproportionately large latency overheads due to the MSQE compilation? Do they have really large `IN` clauses? Based on the image of the profile that you've shared, it looks like the root cause is this known issue - https://github.com/apache/pinot/issues/13617. There are currently some attempts at solving it (https://github.com/apache/pinot/pull/14615, https://github.com/apache/pinot/pull/15027) but we're still discussing the cleanest option to fix that issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Significant Latency Overhead due to empty Response Handling [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15064:
URL: https://github.com/apache/pinot/issues/15064#issuecomment-4348591992
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org