You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/07/13 22:38:20 UTC
[GitHub] [pinot] jasperjiaguo opened a new issue, #9055: Improving Query Preemption on Brokers/Servers
jasperjiaguo opened a new issue, #9055:
URL: https://github.com/apache/pinot/issues/9055
Currently, the broker/server pre-empt a query at certain points during scheduling/execution when the remainingTime reaches 0. However, this is not sufficient for a couple of use-cases:
- When a user wants to initiate some ad-hoc queries for data analysis on a large production table but doesn't specify a tight data range.
- When unexpected slow queries are saturating CPU/memory on broker and servers.
- When a large report generation query (especially with large groupby+orderby/select*/other compute intensive functions) is initiated.
- ...
Any of these cases can cause a broker/server to be very slow or bring down a host. Meanwhile, the recovery time can be quite long as the servers will struggle to finish the computation.
Our pre-emption logic can be extended to both brokers and servers to pre-empt queries based on two factors
- CPU Time
Broker currently has logic to return queries that have reached timeout. However, this can be improved at the broker/server. We can potentially account for the total CPU workload on all worker threads on a server.
- Memory consumption - to avoid out-of-memory errors
We want to estimate the heap memory usage and pre-empt based on a threshold.
We would like to find the correct instrument of measuring these factors and improve preemption with least overhead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jasperjiaguo commented on issue #9055: Improving Query Preemption on Brokers/Servers
Posted by GitBox <gi...@apache.org>.
jasperjiaguo commented on issue #9055:
URL: https://github.com/apache/pinot/issues/9055#issuecomment-1188485159
> cc @atris since this is related to the query resiliency and cost estimation
Listed are some potential approaches we are evaluating/benchmarking. It's just a sketch pad please don't review.
https://docs.google.com/document/d/1O_YKoDpyWjdxd55gE6zAuOjN8_wh0nNBAn9ocIGq25E/edit?usp=sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #9055: Improving Query Preemption on Brokers/Servers
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #9055:
URL: https://github.com/apache/pinot/issues/9055#issuecomment-1259853148
FYI @lakshmanan-v
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #9055: Improving Query Preemption on Brokers/Servers
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #9055:
URL: https://github.com/apache/pinot/issues/9055#issuecomment-1225188133
Update -- @jasperjiaguo is working on writing the design doc. Will share here once we have the first draft.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #9055: Improving Query Preemption on Brokers/Servers
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #9055:
URL: https://github.com/apache/pinot/issues/9055#issuecomment-1329962107
Implementation for runtime query killing on server has been merged as part of https://github.com/apache/pinot/pull/9727
@jasperjiaguo is working to extend the support on broker as well. More or less similar design shared and reviewed earlier here but just extending it to work with broker execution code and thread model.
Here are the user docs to leverage the feature on server - https://docs.pinot.apache.org/operators/operating-pinot/oom-protection-using-automatic-query-killing
We did testing on actual production use case where expensive queries in the past have caused OOM. With the change enabled, we can see culprit queries getting killed live and server is protected from OOM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia closed issue #9055: Runtime query killing on brokers and servers using accounting
Posted by "siddharthteotia (via GitHub)" <gi...@apache.org>.
siddharthteotia closed issue #9055: Runtime query killing on brokers and servers using accounting
URL: https://github.com/apache/pinot/issues/9055
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #9055: Runtime query killing on brokers and servers using accounting
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #9055:
URL: https://github.com/apache/pinot/issues/9055#issuecomment-1376615246
PR should be out in a week or two for broker query killing support. @jasperjiaguo actively working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #9055: Runtime query killing on brokers and servers using accounting
Posted by "siddharthteotia (via GitHub)" <gi...@apache.org>.
siddharthteotia commented on issue #9055:
URL: https://github.com/apache/pinot/issues/9055#issuecomment-1411638150
Support for broker query killing has been merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #9055: Runtime query killing on brokers and servers using accounting
Posted by "siddharthteotia (via GitHub)" <gi...@apache.org>.
siddharthteotia commented on issue #9055:
URL: https://github.com/apache/pinot/issues/9055#issuecomment-1405363637
PR is out for broker query killing support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9055: Improving Query Preemption on Brokers/Servers
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9055:
URL: https://github.com/apache/pinot/issues/9055#issuecomment-1188396168
cc @atris since this is related to the query resiliency and cost estimation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #9055: Improving Query Preemption on Brokers/Servers
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #9055:
URL: https://github.com/apache/pinot/issues/9055#issuecomment-1278524897
@vvivekiyer @jasperjiaguo and I met with @Jackie-Jiang and @walterddr to review the design doc and some code walkthrough.
Overall looks good. One main suggestion received was to unify / encapsulate the different kinds of info being held in different places (e.g Trace related info, the stats being introduced in this PR etc) into some form of ExecutionContext. Will be making those changes and send out the PR / first PR in a week or so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #9055: Runtime query killing on brokers and servers using accounting
Posted by "siddharthteotia (via GitHub)" <gi...@apache.org>.
siddharthteotia commented on issue #9055:
URL: https://github.com/apache/pinot/issues/9055#issuecomment-1411641568
@jasperjiaguo - please make sure to add user docs asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org