You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "walterddr (via GitHub)" <gi...@apache.org> on 2023/06/07 21:19:04 UTC
[GitHub] [pinot] walterddr opened a new issue, #10871: [multistage] run planner for routing on controller
walterddr opened a new issue, #10871:
URL: https://github.com/apache/pinot/issues/10871
Description
==
when query broker for a specific query. there's no need to parse out the tenant info.
however when using the Pinot controller UI. controller needs to first detect all the tables accessed by the query, then find the common tenant of a broker to route the query to.
Status Quo
==
currently this is done via a util done in #10336. however this util has lots of drawback and overhead. b/c it is not running the actual planner.
Proposed solution
==
1. run the calcite planner on controller (`QueryEnvironment.compileQuery`)
2. and use `RelOptUtils.findTables` to find all tables accessed by this query.
3. use the same util currently available to find common broker tenant list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #10871: [multistage] run planner for routing on controller
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on issue #10871:
URL: https://github.com/apache/pinot/issues/10871#issuecomment-1581528046
CC @tibrewalpratik17 and @abhioncbr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhioncbr commented on issue #10871: [multistage] run planner for routing on controller
Posted by "abhioncbr (via GitHub)" <gi...@apache.org>.
abhioncbr commented on issue #10871:
URL: https://github.com/apache/pinot/issues/10871#issuecomment-1581669237
Sounds good to me. I can work on this if it's not urgent. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tibrewalpratik17 commented on issue #10871: [multistage] run planner for routing on controller
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on issue #10871:
URL: https://github.com/apache/pinot/issues/10871#issuecomment-1582161315
Hey @abhioncbr is it okay if i pick this up and get this assigned to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tibrewalpratik17 commented on issue #10871: [multistage] run planner for routing on controller
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on issue #10871:
URL: https://github.com/apache/pinot/issues/10871#issuecomment-1581585198
> And what will we pass as WorkerManager to the QueryEnvironment object as it looks specific to broker / server configs?
On deeper look I see we don't need WorkerManager in the `compileQuery` path.
Seems doable to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #10871: [multistage] run planner for routing on controller
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on issue #10871:
URL: https://github.com/apache/pinot/issues/10871#issuecomment-1581751011
> @walterddr Just curious will this not add an additional dependency of pinot-controller module on `pinot-query-planner`?And what will we pass as `WorkerManager` to the QueryEnvironment object as it looks specific to broker / server configs?
Controller is never designed for sustained QPS workload --> in fact each pending query will hog up a thread/cpu thus it can handle maximum `no_of_core` query at a time.
>
> Also for 2. we can use `RelToPlanNodeConverter.getTableNamesFromRelRoot(relRoot.rel)` which internally uses `RelOptUtils.findAllTableQualifiedNames(relRoot)` at the moment. This is a part of pinot-query-planner module itself.
yes, that's what I meant, we dont need to configure `WorkerManager` as we are not assigning worker at the moment, we only want to do the planning part.
> Sounds good to me. I can work on this if it's not urgent. Thanks.
Thank you @abhioncbr, sorry I was originally intended to @ankitsultana but yes this is not urgent. please feel free to pick this one up at your convenience. (assigned to you)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tibrewalpratik17 commented on issue #10871: [multistage] run planner for routing on controller
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on issue #10871:
URL: https://github.com/apache/pinot/issues/10871#issuecomment-1581566627
@walterddr Just curious will this not add an additional dependency of pinot-controller module on `pinot-query-planner`?And what will we pass as `WorkerManager` to the QueryEnvironment object as it looks specific to broker / server configs?
Also for 2. we can use `RelToPlanNodeConverter.getTableNamesFromRelRoot(relRoot.rel)` which internally uses `RelOptUtils.findAllTableQualifiedNames(relRoot)` at the moment. This is a part of pinot-query-planner module itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] abhioncbr commented on issue #10871: [multistage] run planner for routing on controller
Posted by "abhioncbr (via GitHub)" <gi...@apache.org>.
abhioncbr commented on issue #10871:
URL: https://github.com/apache/pinot/issues/10871#issuecomment-1582358174
> Hey @abhioncbr is it okay if i pick this up and get this assigned to me?
Sure. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org