You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "dang-stripe (via GitHub)" <gi...@apache.org> on 2025/06/05 00:37:53 UTC
[I] Summary broker query log for multistage engine queries [pinot]
dang-stripe opened a new issue, #16001:
URL: https://github.com/apache/pinot/issues/16001
We've found it difficult to do post-hoc analysis of query timeouts/failures on the multistage engine without access to the stage stats. Debugging these queries have involved:
1. Piecing together distributed server logs to understand where the query failed or slowed down
2. Re-running the query, raising the timeout as needed, and collecting the stage stats adhoc
Given the complexity of the multistage engine, it'd be ideal for these queries to have a concise summary log from the broker similar to how single stage does it [here](https://github.com/apache/pinot/blob/master/pinot-broker/src/main/java/org/apache/pinot/broker/querylog/QueryLogger.java#L184). Some metadata that'd be helpful for debugging:
1. Query success or failure
2. Query latency
3. Concise representation of the stage graph (1->[2,3],2->4,etc) including which stages are leafs
4. Which stages of the query succeeded or failed
5. How many servers were involved in each stage
6. How long (wall clock) time did each stage take
This would make it much easier to debug production issues on the fly and provide centralized visibility into query execution across all stages.
cc @gortiz @Jackie-Jiang @jadami10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Summary broker query log for multistage engine queries [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #16001:
URL: https://github.com/apache/pinot/issues/16001#issuecomment-2943796722
I've created #16007, where I'm modifying log in case of error to include the stats. I've also included another log that creates the same summary in case of successful execution, but that log can be very spammy, so it is logged at DEBUG level to be disabled by default. A new marker can be used to enable just that log instead of all debug logs from the class. See the PR to see an example. If needed, Log4j2's [BusterFilter](https://logging.apache.org/log4j/2.x/manual/filters.html#BurstFilter) can be used to apply a rate to these logs.
The stats already contain most of the information requested here:
1. Query success or failure: Different messages are logged on each case
2. Query latency: Included in the summary
3. Concise representation of the stage graph (1->[2,3],2->4,etc) including which stages are leafs: This is included in the stats.
4. Which stages of the query succeeded or failed: We cannot include that right now, more info below.
5. For stages that failed, which servers failed the complete the stage: In order to have that info we need to merge https://github.com/apache/pinot/pull/15919 first.
6. How long (wall clock) time did each stage take: Stats for each operator include `clockTimeMs`, which is calculated as `executionTimeMs / parallelism`. Right now, `executionTimeMs` is 0 for stages that fail, but that is something we will try to improve.
It is not easy to include how many servers have failed because Pinot aborts the execution as soon as the first error is received. We may decide to change that in the future, but it would require significant changes to the code and likely increase latency in error handling, so it is unlikely that we will invest resources in this area in the short term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Summary broker query log for multistage engine queries [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16001: Summary broker query log for multistage engine queries
URL: https://github.com/apache/pinot/issues/16001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Summary broker query log for multistage engine queries [pinot]
Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on issue #16001:
URL: https://github.com/apache/pinot/issues/16001#issuecomment-2985592274
Thanks @gortiz!
> Which stages of the query succeeded or failed: We cannot include that right now, more info below.
> It is not easy to include how many servers have failed because Pinot aborts the execution as soon as the first error is received.
Is it possible to derive this from the stage stats? Or is that lacking information as well?
> For stages that failed, which servers failed the complete the stage: In order to have that info we need to merge https://github.com/apache/pinot/pull/15919 first.
How does this behave when queries timeout at a leaf stage vs. intermediate stage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Summary broker query log for multistage engine queries [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16001:
URL: https://github.com/apache/pinot/issues/16001#issuecomment-4113208765
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Summary broker query log for multistage engine queries [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16001:
URL: https://github.com/apache/pinot/issues/16001#issuecomment-4300860168
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org