You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/20 14:09:46 UTC
[GitHub] [pinot] jadami10 opened a new issue, #8928: GroupByOrderByCombineOperator does not bound the number of threads it uses
jadami10 opened a new issue, #8928:
URL: https://github.com/apache/pinot/issues/8928
When the user does not specify the number of threads for a query, the CombinePlan* operators both use a minimum of 1 threads, maximum of 10 threads, and a default of num_cpus/2 threads for their tasks.
The GroupByOrderByCombineOperator on the other hand just sets the number of threads to the number of operators.
I haven't actually profiled this or logged how many operators there are in practice, but this feels like it could be a bug when comparing to how the CombinePlan code treats this flag. I'm curious if you feel this is of concern as well before we dig into it further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8928: GroupByOrderByCombineOperator does not bound the number of threads it uses
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8928:
URL: https://github.com/apache/pinot/issues/8928#issuecomment-1160768350
This is a legacy behavior which is carried over to prevent performance regression. Basically for group-by queries, we will use all CPU resources if not explicitly configured. `maxExecutionThreads` query option will still be honored if specified.
It is a trade off between latency and stability: using all CPU cores can give the best latency if query rate is low, but one expensive query can exhaust the CPU resource.
There might be a way to get both world. We may dynamically decide the threads to use based on the query load. I think this is a topic worth more discussion. cc @kishoreg @siddharthteotia @yupeng9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jadami10 commented on issue #8928: GroupByOrderByCombineOperator does not bound the number of threads it uses
Posted by GitBox <gi...@apache.org>.
jadami10 commented on issue #8928:
URL: https://github.com/apache/pinot/issues/8928#issuecomment-1161896073
thank you for taking this up. For now we will manually set that config using the same logic CombinePlan uses to prefer stability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org