You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2021/11/03 10:19:59 UTC
[GitHub] [shardingsphere] strongduanmu opened a new issue #13441: Discussion on splitting multiple sub modules of infra-executor
strongduanmu opened a new issue #13441:
URL: https://github.com/apache/shardingsphere/issues/13441
## Background
This issue is a problem encountered during the issue #13091. In order to route accurately for federation executor and hide the actual tables and actual columns, we need to move the route engine, rewrite engine, executor, and merge engine into the federation execution engine, which requires us to add dependencies to the federation execution engine.
The current dependencies are as follows. If we add `infra-merge` dependency in the infra-executor, it will cause a circular dependency.
```
org.apache.shardingsphere:shardingsphere-infra-executor:jar:5.0.0-RC1-SNAPSHOT
|----org.apache.shardingsphere:shardingsphere-infra-common:jar:5.0.0-RC1-SNAPSHOT:compile
|----org.apache.shardingsphere:shardingsphere-infra-rewrite:jar:5.0.0-RC1-SNAPSHOT:compile
|----org.apache.shardingsphere:shardingsphere-infra-optimize:jar:5.0.0-RC1-SNAPSHOT:compile
|----org.apache.calcite:calcite-core:jar:1.27.0:compile
|----org.apache.shardingsphere:shardingsphere-sql-parser-mysql:jar:5.0.0-RC1-SNAPSHOT:test
org.apache.shardingsphere:shardingsphere-infra-merge:jar:5.0.0-RC1-SNAPSHOT
|----org.apache.shardingsphere:shardingsphere-infra-binder:jar:5.0.0-RC1-SNAPSHOT:compile
|----org.apache.shardingsphere:shardingsphere-infra-executor:jar:5.0.0-RC1-SNAPSHOT:compile
```
## Adjust dependencies
So I want to discuss whether `infra-executor` can be split into multiple sub-modules, the structure is as follows:
```
shardingsphere-infra-executor
|----shardingsphere-infra-executor-driver
|----shardingsphere-infra-executor-federation
|----shardingsphere-infra-context
|----shardingsphere-infra-optimize
|----shardingsphere-infra-merge
```
The `shardingsphere-infra-executor-driver` module is responsible for handling standard kernel execution. The `shardingsphere-infra-executor-federation` module is responsible for handling query optimization statement execution.
Existing modules that rely on `shardingsphere-infra-executor` can be replaced with `shardingsphere-infra-executor-driver`. If there are some modules that rely on `shardingsphere-infra-executor-federation`, just add dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on issue #13441: Discussion on splitting multiple sub modules of infra-executor
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on issue #13441:
URL: https://github.com/apache/shardingsphere/issues/13441#issuecomment-958867000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on issue #13441: Discussion on splitting multiple sub modules of infra-executor
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on issue #13441:
URL: https://github.com/apache/shardingsphere/issues/13441#issuecomment-958931861
@terrymanu Thank you for your suggestions. In the long run, these works are very valuable. I will consider it and then update this plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu closed issue #13441: Discussion on splitting multiple sub modules of infra-executor
Posted by GitBox <gi...@apache.org>.
strongduanmu closed issue #13441:
URL: https://github.com/apache/shardingsphere/issues/13441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on issue #13441: Discussion on splitting multiple sub modules of infra-executor
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on issue #13441:
URL: https://github.com/apache/shardingsphere/issues/13441#issuecomment-977776001
After some discussion, we decided to rename `shardingsphere-infra-optimize` to `shardingsphere-infra-federation` and add two submodules `shardingsphere-infra-federation-optimizer` and `shardingsphere-infra-federation-executor` , which correspond to query optimization and optimization execution.
```
shardingsphere-infra-federation
|----shardingsphere-infra-federation-optimizer
|----shardingsphere-infra-federation-executor
```
The new module dependency is as follow:
The `shardingsphere-infra-executor` module is positioned as an execution engine for the underlying storage, and will support MPP and other executors in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] terrymanu commented on issue #13441: Discussion on splitting multiple sub modules of infra-executor
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #13441:
URL: https://github.com/apache/shardingsphere/issues/13441#issuecomment-958924974
I just share some ideas:
1. It is better to spilt optimizer and executor, the `infra-executor-federation` maybe does not make sense, federation may belong to optimizer.
2. In the future, the executor may accept more input types, such as: SQL(for now), relnode for query plan (In future).
3. The executor itself should be pluggable and may split by multiple-thread executor and multiple-process executor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on issue #13441: Discussion on splitting multiple sub modules of infra-executor
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on issue #13441:
URL: https://github.com/apache/shardingsphere/issues/13441#issuecomment-958867000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on issue #13441: Discussion on splitting multiple sub modules of infra-executor
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on issue #13441:
URL: https://github.com/apache/shardingsphere/issues/13441#issuecomment-958867000
@terrymanu @tristaZero Can you help judge whether this plan is feasible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] strongduanmu commented on issue #13441: Discussion on splitting multiple sub modules of infra-executor
Posted by GitBox <gi...@apache.org>.
strongduanmu commented on issue #13441:
URL: https://github.com/apache/shardingsphere/issues/13441#issuecomment-984445349
Currently, the split of infra-executor has been completed, I will close this issue. The mpp executor discussed in the issue will be tracked in a other issue later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] terrymanu commented on issue #13441: Discussion on splitting multiple sub modules of infra-executor
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #13441:
URL: https://github.com/apache/shardingsphere/issues/13441#issuecomment-958924974
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] terrymanu commented on issue #13441: Discussion on splitting multiple sub modules of infra-executor
Posted by GitBox <gi...@apache.org>.
terrymanu commented on issue #13441:
URL: https://github.com/apache/shardingsphere/issues/13441#issuecomment-958924974
I just share some ideas:
1. It is better to spilt optimizer and executor, the `infra-executor-federation` maybe does not make sense, federation may belong to optimizer.
2. In the future, the executor may accept more input types, such as: SQL(for now), relnode for query plan (In future).
3. The executor itself should be pluggable and may split by multiple-thread executor and multiple-process executor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org