You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "walterddr (via GitHub)" <gi...@apache.org> on 2023/04/20 20:44:40 UTC
[GitHub] [pinot] walterddr opened a new issue, #10657: [draft][multistage] decouple plan/runtime API abstracts.
walterddr opened a new issue, #10657:
URL: https://github.com/apache/pinot/issues/10657
Background
===
Currently we have multiple abstractions reused with different components in planner and runtime. it causes several problems
- when trying to add partition-based routing and planning it is super complex
- information only required in plan time and dispatch time get leaked to runtime which is not useful, but somehow usage is mixed and hard to change
- mailbox uses information way more than necessary and makes it hard to identify b/c the mailboxIdentifier equal requires all those to be identical.
- ... many other issues
Proposed changes
===
Several abstract is being introduced and will replace the current abstract
1. Step 1a: replace `VirtualServer`
`VirtualServer` is now a `ServerInstance + VirtualID`, it will be replaced with
`Worker` which is indicating parallelism of work. It:
(1) is globally indexed per stage;
(2) mapped to a single `ServerInstance` stored in `StageMetadata`,
(3) contains partition or segment info which will be put into a new abstract called: `WorkerMetadata`
with this `VirtualServer` is completely removed, and we decoupled `ServerInstance` which is not useful in runtime from `VirtualID` or `workerID` which is used in runtime.
- Step 1b: replace identifiers:
- `MailboxIdentifier` will use `workerID` which is globally indexed to uniquely identify a stream as:
`reqID|sendingStageID|sendingWorkerID|receivingStageID|receivingStageWorkerID`
- `OpChainID` will use `WorkerID` as well `reqID|stageID|workerID`
- Step 2: support Hash-Partitioned data distribution
see: https://docs.google.com/document/d/1CdvxmOOctk6kS5PdgCy7f5KVh5urw4YY0YZGbwuPJt4/edit#
- Step 3: support worker assignment based on data partition and worker/parallelism
see: https://docs.google.com/document/d/1SKDKV6LXr4uFFUsR3djz5BWWMqcSJIYEqJBoL1zeDD8/edit]
CC @Jackie-Jiang @xiangfu0 @ankitsultana @somandal @siddharthteotia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #10657: [draft][multistage] decouple plan/runtime API abstracts.
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on issue #10657:
URL: https://github.com/apache/pinot/issues/10657#issuecomment-1532100863
collecting some feedback from folks, please feel free to comment below as well:
- Parser should be part of the external of planner so that
- it can be used in controller and broker to parse out tables for routing
- parser is also used by v1 engine so it make sense to single it out (and it is invariant with the Planner)
- We should consider making the component modules so that it doesn't need to force dependency on some other packages, for example:
- parser doens't need to depend on planner
- dispatcher doesn't need to depend on calcite.
- decision here is to make sure interfaces are clear, whether to group them into separated modules can be refactor later if interfaces have clear boundaries.
- `Planner` -> `LogicalPlanner`:
- there should be a conversion regarding such as RelTrait, or RelExchange is being the only node looked at by dispatcher
- where does some planning info to
- *information that only affects the node->operator* then this should be node-level info and make it used in PhysicalPlanner (on the server) (e.g. use concurrent index map group-by or 2-way reducer/merger group-by, this decision might not be able to be made on broker b/c it might need info like column cardinality on segment level)
- *Information that not only affects the node but also the upstream/downstream* then this decision should be made on the broker (e.g. using sort-merge join requires sorting operation on both inputs; using hash join requires materializing hash table on either left or right side)
- hint can always be attached, but the decision on where to use that hint is decided with the rule above^
- PinotQueryPlanner and Plan should be a separate module
- the only goal achieved here is de-calcite --> convert RelNode into format that doesn't depend on Calcite
- PinotQueryPlanner + LogicalPlanner can be replaced with a Cache later on if we decide to run periodic tasks that doesn't have changes to the SQL itself (e.g. speed-up planning)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on issue #10657: [draft][multistage] decouple plan/runtime API abstracts.
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on issue #10657:
URL: https://github.com/apache/pinot/issues/10657#issuecomment-1529832117
step 1a and 1b are done via #10673 and #10681
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on issue #10657: [draft][multistage] decouple plan/runtime API abstracts.
Posted by "siddharthteotia (via GitHub)" <gi...@apache.org>.
siddharthteotia commented on issue #10657:
URL: https://github.com/apache/pinot/issues/10657#issuecomment-1517108493
Nice. Will review
FYI @vvivekiyer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org