You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2021/04/04 15:41:19 UTC

[GitHub] [shardingsphere] guimingyue opened a new pull request #9934: Query optimization for federated SQL query

guimingyue opened a new pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934


   Add a new module `shardingsphere-infra-optimizer`, and this module is based on Apache Calcite framework. The main packages are as follows:
   
   0. `org.apache.shardingsphere.infra.optimizer.converter`, the converter of converting ShardingSphere ast to Calcite ast, or SqlNode in Calcite world, and the Converter of converting SqlNode to RelNode of Calcite.
   1. `org.apache.shardingsphere.infra.optimizer.planner`, the core of the optimizer based on Calcite Planner framework. It contains the planner rules(`org.apache.shardingsphere.infra.optimizer.planner.rule`) and the planner process which is response for converting logical plan to physical plan, also be known as execution plan, please refer to `DefaultPlanner` for this process. 
   2. `org.apache.shardingsphere.infra.optimizer.rel`, definition and operation of the `RelNode`.
   3. `org.apache.shardingsphere.infra.optimizer.schema`, the bridge of ShardingSphere and Calcite metadata.
   
   For the whole execution process, method `org.apache.shardingsphere.infra.optimize.planner.Compiler#compileQuery` is the entrance of converting ast and optimizing relnode, and this method also return the physical `RelNode`(execution plan).
   
   This module is different from the existing module`shardingsphere-infra-optimize` which can be regraded as an adaptor of calcite, and cannot optimize the join operator which can be pushdown to the sharding database. The new module does not denpend on the `shardingsphere-infra-executor` which is different from `shardingsphere-infra-optimize` too. 
   
   This module has not been finished yet, there still many `TODO` be left in some classes of this module, `org.apache.shardingsphere.infra.optimizer.converter.SqlNodeConverter` for example. 
   
   If you have any problems about this module, let me know.
   
   In the next PR, I will commit the executor of the execution plan, which is based on the volcano execution model, after this PR is merged.
   
   Fixes #8284.
   
   Changes proposed in this pull request:
   - Add a new module `shardingsphere-infra-optimizer`, and the optimizer is base on the Apache Calcite framework.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] guimingyue closed pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
guimingyue closed pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] codecov-commenter commented on pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#issuecomment-830816603


   # [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#9934](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b39d760) into [master](https://codecov.io/gh/apache/shardingsphere/commit/11320534213183e9716cd867c4d4b3fda02ef4f4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1132053) will **increase** coverage by `0.25%`.
   > The diff coverage is `76.66%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/shardingsphere/pull/9934/graphs/tree.svg?width=650&height=150&src=pr&token=ZvlXpWa7so&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #9934      +/-   ##
   ============================================
   + Coverage     67.96%   68.21%   +0.25%     
   - Complexity      683      722      +39     
   ============================================
     Files          1703     1752      +49     
     Lines         29201    30088     +887     
     Branches       5250     5381     +131     
   ============================================
   + Hits          19845    20525     +680     
   - Misses         7826     7971     +145     
   - Partials       1530     1592      +62     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...ngsphere/infra/optimizer/rel/TableNameVisitor.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL1RhYmxlTmFtZVZpc2l0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...sphere/infra/optimizer/rel/physical/SSProject.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL3BoeXNpY2FsL1NTUHJvamVjdC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...here/infra/optimizer/tools/PushdownRelBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvdG9vbHMvUHVzaGRvd25SZWxCdWlsZGVyLmphdmE=) | `40.00% <40.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...nfra/optimizer/planner/rule/PushAggToScanRule.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcGxhbm5lci9ydWxlL1B1c2hBZ2dUb1NjYW5SdWxlLmphdmE=) | `50.00% <50.00%> (ø)` | `1.00 <1.00> (?)` | |
   | [...e/infra/optimizer/rel/physical/SSAbstractJoin.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL3BoeXNpY2FsL1NTQWJzdHJhY3RKb2luLmphdmE=) | `50.00% <50.00%> (ø)` | `1.00 <1.00> (?)` | |
   | [...sphere/infra/optimizer/sql/ExtendedSqlShuttle.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvc3FsL0V4dGVuZGVkU3FsU2h1dHRsZS5qYXZh) | `50.00% <50.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...here/infra/optimizer/sql/SqlDynamicValueParam.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvc3FsL1NxbER5bmFtaWNWYWx1ZVBhcmFtLmphdmE=) | `50.00% <50.00%> (ø)` | `2.00 <2.00> (?)` | |
   | [...here/infra/optimizer/rel/physical/SSMergeSort.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL3BoeXNpY2FsL1NTTWVyZ2VTb3J0LmphdmE=) | `57.14% <57.14%> (ø)` | `1.00 <1.00> (?)` | |
   | [...mizer/planner/rule/PushSortToMultiRoutingRule.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcGxhbm5lci9ydWxlL1B1c2hTb3J0VG9NdWx0aVJvdXRpbmdSdWxlLmphdmE=) | `59.09% <59.09%> (ø)` | `1.00 <1.00> (?)` | |
   | [...sphere/infra/optimizer/rel/TableAndRexShuttle.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL1RhYmxlQW5kUmV4U2h1dHRsZS5qYXZh) | `63.49% <63.49%> (ø)` | `0.00 <0.00> (?)` | |
   | ... and [88 more](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1132053...b39d760](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] guimingyue commented on pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
guimingyue commented on pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#issuecomment-864961831


   > @guimingyue Cool, I will give it a look.
   > Could you look at the latest master branch? I suppose 1 and 2 are supported now, and in terms of 3, could you give more clarification?
   > 
   > ```
   > 1. join type is missing.
   > 2. having clause of group by is not supported.
   > 3. expression has not been pared.
   > ```
   
   I have tested in the latest branch, the result follows.
   - joinType of JoinTableSegment is still null for inner join and left outer join according testcase of `org.apache.shardingsphere.infra.optimize.core.convert.SqlNodeConverterTest`  in my PR.
   - having clause is supported now, but the inner expression of AggregationProjectionSegment is String, take this sql  text as an example, `select order_id, user_id from t_order where order_id = 10 group by order_id having sum(order_id) > 0`, the type of expr field of `HavingSegment` is `BinaryOperationExpression` and the left parameter type of `BinaryOperationExpression` object is `AggregationProjectionSegment`. As a result, the type of `innerExpression` field is String whose value is `(order_id)`, which should be parsed as `ColumnSegment` object.
   - the field text of `ExpressionProjectionSegment` is a Stirng type, which cannot be converted to a AST expression, for example, in the select list of this sql text `select 10 + 30, order_id + 10, order_id, user_id from t_order`, the expression of `10+30`  will be parsed as an `ExpressionProjectionSegment` object.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] codecov-io edited a comment on pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#issuecomment-813071453


   # [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=h1) Report
   > Merging [#9934](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=desc) (10c6f0e) into [master](https://codecov.io/gh/apache/shardingsphere/commit/00b1703646aa8d144e73d4e56ae8514036bf690c?el=desc) (00b1703) will **increase** coverage by `0.26%`.
   > The diff coverage is `76.66%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/shardingsphere/pull/9934/graphs/tree.svg?width=650&height=150&src=pr&token=ZvlXpWa7so)](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #9934      +/-   ##
   ============================================
   + Coverage     67.99%   68.26%   +0.26%     
   - Complexity      677      716      +39     
   ============================================
     Files          1684     1733      +49     
     Lines         28281    29183     +902     
     Branches       5017     5153     +136     
   ============================================
   + Hits          19230    19921     +691     
   - Misses         7628     7776     +148     
   - Partials       1423     1486      +63     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...ngsphere/infra/optimizer/rel/TableNameVisitor.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL1RhYmxlTmFtZVZpc2l0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...sphere/infra/optimizer/rel/physical/SSProject.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL3BoeXNpY2FsL1NTUHJvamVjdC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...here/infra/optimizer/tools/PushdownRelBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvdG9vbHMvUHVzaGRvd25SZWxCdWlsZGVyLmphdmE=) | `40.00% <40.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...nfra/optimizer/planner/rule/PushAggToScanRule.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcGxhbm5lci9ydWxlL1B1c2hBZ2dUb1NjYW5SdWxlLmphdmE=) | `50.00% <50.00%> (ø)` | `1.00 <1.00> (?)` | |
   | [...e/infra/optimizer/rel/physical/SSAbstractJoin.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL3BoeXNpY2FsL1NTQWJzdHJhY3RKb2luLmphdmE=) | `50.00% <50.00%> (ø)` | `1.00 <1.00> (?)` | |
   | [...sphere/infra/optimizer/sql/ExtendedSqlShuttle.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvc3FsL0V4dGVuZGVkU3FsU2h1dHRsZS5qYXZh) | `50.00% <50.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...here/infra/optimizer/sql/SqlDynamicValueParam.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvc3FsL1NxbER5bmFtaWNWYWx1ZVBhcmFtLmphdmE=) | `50.00% <50.00%> (ø)` | `2.00 <2.00> (?)` | |
   | [...here/infra/optimizer/rel/physical/SSMergeSort.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL3BoeXNpY2FsL1NTTWVyZ2VTb3J0LmphdmE=) | `57.14% <57.14%> (ø)` | `1.00 <1.00> (?)` | |
   | [...mizer/planner/rule/PushSortToMultiRoutingRule.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcGxhbm5lci9ydWxlL1B1c2hTb3J0VG9NdWx0aVJvdXRpbmdSdWxlLmphdmE=) | `59.09% <59.09%> (ø)` | `1.00 <1.00> (?)` | |
   | [...sphere/infra/optimizer/rel/TableAndRexShuttle.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL1RhYmxlQW5kUmV4U2h1dHRsZS5qYXZh) | `63.49% <63.49%> (ø)` | `0.00 <0.00> (?)` | |
   | ... and [108 more](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=footer). Last update [00b1703...10c6f0e](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] guimingyue commented on pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
guimingyue commented on pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#issuecomment-864528714


   > Hi @guimingyue Do you think it is possible to firstly raise PRs into [SqlNodeConverter](https://github.com/apache/shardingsphere/blob/master/shardingsphere-infra/shardingsphere-infra-optimize/src/main/java/org/apache/shardingsphere/infra/optimize/core/convert/SqlNodeConverter.java)
   
   It‘s OK. I will try to raise another PR with SqlNodeConverter.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] tristaZero commented on pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
tristaZero commented on pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#issuecomment-864725989


   @guimingyue Cool, I will give it a look.
   Could you look at the latest master branch? I suppose 1 and 2 are supported now, and in terms of 3, could you give more clarification?
   
   ```
   1. join type is missing.
   2. having clause of group by is not supported.
   3. expression has not been pared.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] tristaZero commented on a change in pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
tristaZero commented on a change in pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#discussion_r607579829



##########
File path: shardingsphere-infra/pom.xml
##########
@@ -40,5 +40,6 @@
         <module>shardingsphere-infra-context</module>
         <module>shardingsphere-infra-datetime</module>
         <module>shardingsphere-infra-optimize</module>
+        <module>shardingsphere-infra-optimizer</module>

Review comment:
       What's the difference between `shardingsphere-infra-optimize` and `shardingsphere-infra-optimizer`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] guimingyue commented on a change in pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
guimingyue commented on a change in pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#discussion_r607599406



##########
File path: shardingsphere-infra/pom.xml
##########
@@ -40,5 +40,6 @@
         <module>shardingsphere-infra-context</module>
         <module>shardingsphere-infra-datetime</module>
         <module>shardingsphere-infra-optimize</module>
+        <module>shardingsphere-infra-optimizer</module>

Review comment:
       The module `shardingsphere-infra-optimize` can be regarded as an adaptor of calcite, and cannot optimize the join operator which can be pushdown to the sharding database. `shardingsphere-infra-optimizer` module does not depend on the `shardingsphere-infra-executor` which is different from `shardingsphere-infra-optimize` too. The `shardingsphere-infra-executor` will depend on the `shardingsphere-infra-optimizer` for the execution of the physical plan.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] guimingyue commented on pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
guimingyue commented on pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#issuecomment-864571542


   @tristaZero I have raised a pr to merge SqlNodeConverter。
   There are still many TODO for both ss ast and converter tool, I think the list blow ought to be given priority.
   - join type is missing.
   - having clause of group by is not supported.
   - expression has not been pared.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] codecov-io commented on pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#issuecomment-813071453


   # [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=h1) Report
   > Merging [#9934](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=desc) (6cdcfee) into [master](https://codecov.io/gh/apache/shardingsphere/commit/00b1703646aa8d144e73d4e56ae8514036bf690c?el=desc) (00b1703) will **decrease** coverage by `0.06%`.
   > The diff coverage is `68.32%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/shardingsphere/pull/9934/graphs/tree.svg?width=650&height=150&src=pr&token=ZvlXpWa7so)](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #9934      +/-   ##
   ============================================
   - Coverage     67.99%   67.93%   -0.07%     
   - Complexity      677      710      +33     
   ============================================
     Files          1684     1733      +49     
     Lines         28281    29183     +902     
     Branches       5017     5153     +136     
   ============================================
   + Hits          19230    19825     +595     
   - Misses         7628     7881     +253     
   - Partials       1423     1477      +54     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | [...pache/shardingsphere/infra/optimizer/ExecStmt.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvRXhlY1N0bXQuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...ardingsphere/infra/optimizer/planner/Compiler.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcGxhbm5lci9Db21waWxlci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...ngsphere/infra/optimizer/rel/TableNameVisitor.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL1RhYmxlTmFtZVZpc2l0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [.../infra/optimizer/rel/physical/SSHashAggregate.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL3BoeXNpY2FsL1NTSGFzaEFnZ3JlZ2F0ZS5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...here/infra/optimizer/rel/physical/SSLimitSort.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL3BoeXNpY2FsL1NTTGltaXRTb3J0LmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...sphere/infra/optimizer/rel/physical/SSProject.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL3BoeXNpY2FsL1NTUHJvamVjdC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...ingsphere/infra/optimizer/rel/physical/SSSort.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcmVsL3BoeXNpY2FsL1NTU29ydC5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
   | [...nfra/optimizer/planner/rule/PushAggToScanRule.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcGxhbm5lci9ydWxlL1B1c2hBZ2dUb1NjYW5SdWxlLmphdmE=) | `35.71% <35.71%> (ø)` | `1.00 <1.00> (?)` | |
   | [...timizer/planner/rule/SSLimitSortConverterRule.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvcGxhbm5lci9ydWxlL1NTTGltaXRTb3J0Q29udmVydGVyUnVsZS5qYXZh) | `38.46% <38.46%> (ø)` | `1.00 <1.00> (?)` | |
   | [...here/infra/optimizer/tools/PushdownRelBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtb3B0aW1pemVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9vcHRpbWl6ZXIvdG9vbHMvUHVzaGRvd25SZWxCdWlsZGVyLmphdmE=) | `40.00% <40.00%> (ø)` | `0.00 <0.00> (?)` | |
   | ... and [112 more](https://codecov.io/gh/apache/shardingsphere/pull/9934/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=footer). Last update [00b1703...6cdcfee](https://codecov.io/gh/apache/shardingsphere/pull/9934?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] tristaZero commented on pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
tristaZero commented on pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#issuecomment-864521460


   Hi @guimingyue Do you think it is possible to firstly raise PRs into [SqlNodeConverter](https://github.com/apache/shardingsphere/blob/master/shardingsphere-infra/shardingsphere-infra-optimize/src/main/java/org/apache/shardingsphere/infra/optimize/core/convert/SqlNodeConverter.java)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] guimingyue commented on a change in pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
guimingyue commented on a change in pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#discussion_r623917497



##########
File path: shardingsphere-infra/shardingsphere-infra-optimizer/src/main/java/org/apache/shardingsphere/infra/optimizer/converter/RelNodeConverter.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.shardingsphere.infra.optimizer.converter;
+
+import org.apache.calcite.config.CalciteConnectionConfigImpl;
+import org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.plan.Contexts;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCostFactory;
+import org.apache.calcite.plan.RelOptCostImpl;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelRoot;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql.type.SqlTypeFactoryImpl;
+import org.apache.calcite.sql.validate.SqlConformanceEnum;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorUtil;
+import org.apache.calcite.sql.validate.SqlValidatorWithHints;
+import org.apache.calcite.sql2rel.SqlToRelConverter;
+import org.apache.calcite.sql2rel.StandardConvertletTable;
+import org.apache.calcite.tools.Frameworks;
+import org.apache.shardingsphere.infra.metadata.schema.ShardingSphereSchema;
+import org.apache.shardingsphere.infra.optimizer.schema.ShardingSphereCalciteSchema;
+
+import java.util.Properties;
+
+/**
+ * convert calcite SqlNode to calcite RelNode.
+ */
+public class RelNodeConverter {
+
+    private static boolean topDownOpt = Boolean.parseBoolean(System.getProperty("calcite.planner.topdown.opt", "true"));
+    
+    private static final RelDataTypeFactory TYPE_FACTORY = new SqlTypeFactoryImpl(RelDataTypeSystem.DEFAULT);
+    
+    private final RelOptCostFactory costFactory = RelOptCostImpl.FACTORY;
+
+    private final SqlToRelConverter.Config converterConfig;
+
+    private final SqlValidatorWithHints validator;
+    
+    private final CalciteCatalogReader catalog;
+    
+    private final ShardingSphereSchema shardingSphereSchema;
+
+    public RelNodeConverter(final String schemaName, final ShardingSphereSchema shardingSphereSchema) {
+        this.shardingSphereSchema = shardingSphereSchema;
+        this.converterConfig = SqlToRelConverter.config()
+                .withInSubQueryThreshold(Integer.MAX_VALUE)
+                .withExpand(false);
+        CalciteSchema rootSchema = createRootSchema(schemaName);
+        this.catalog = new CalciteCatalogReader(rootSchema,
+                rootSchema.path(schemaName),
+                new JavaTypeFactoryImpl(),
+                new CalciteConnectionConfigImpl(new Properties()));
+
+        validator = SqlValidatorUtil.newValidator(SqlStdOperatorTable.instance(), catalog, TYPE_FACTORY, SqlValidator.Config.DEFAULT
+                .withLenientOperatorLookup(false)
+                .withSqlConformance(SqlConformanceEnum.DEFAULT)
+                .withIdentifierExpansion(true));
+    }
+
+    /**
+     * validate calcite SqlNode and convert SqlNode to RelNode.
+     * @param sqlNode SqlNode to be validated and converted
+     * @return relational algebra of sql
+     */
+    public RelNode validateAndConvert(final SqlNode sqlNode) {
+        final RelOptCluster cluster = createRelOptCluster();
+        final SqlToRelConverter sqlToRelConverter = new SqlToRelConverter(null, validator, catalog, cluster,
+                StandardConvertletTable.INSTANCE, converterConfig);
+

Review comment:
       done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] zhujunxxxxx commented on a change in pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
zhujunxxxxx commented on a change in pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#discussion_r606892495



##########
File path: shardingsphere-infra/shardingsphere-infra-optimizer/src/main/java/org/apache/shardingsphere/infra/optimizer/converter/RelNodeConverter.java
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.shardingsphere.infra.optimizer.converter;
+
+import org.apache.calcite.config.CalciteConnectionConfigImpl;
+import org.apache.calcite.jdbc.CalciteSchema;
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.plan.Contexts;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCostFactory;
+import org.apache.calcite.plan.RelOptCostImpl;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.prepare.CalciteCatalogReader;
+import org.apache.calcite.rel.RelCollationTraitDef;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelRoot;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rel.type.RelDataTypeSystem;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.fun.SqlStdOperatorTable;
+import org.apache.calcite.sql.type.SqlTypeFactoryImpl;
+import org.apache.calcite.sql.validate.SqlConformanceEnum;
+import org.apache.calcite.sql.validate.SqlValidator;
+import org.apache.calcite.sql.validate.SqlValidatorUtil;
+import org.apache.calcite.sql.validate.SqlValidatorWithHints;
+import org.apache.calcite.sql2rel.SqlToRelConverter;
+import org.apache.calcite.sql2rel.StandardConvertletTable;
+import org.apache.calcite.tools.Frameworks;
+import org.apache.shardingsphere.infra.metadata.schema.ShardingSphereSchema;
+import org.apache.shardingsphere.infra.optimizer.schema.ShardingSphereCalciteSchema;
+
+import java.util.Properties;
+
+/**
+ * convert calcite SqlNode to calcite RelNode.
+ */
+public class RelNodeConverter {
+
+    private static boolean topDownOpt = Boolean.parseBoolean(System.getProperty("calcite.planner.topdown.opt", "true"));
+    
+    private static final RelDataTypeFactory TYPE_FACTORY = new SqlTypeFactoryImpl(RelDataTypeSystem.DEFAULT);
+    
+    private final RelOptCostFactory costFactory = RelOptCostImpl.FACTORY;
+
+    private final SqlToRelConverter.Config converterConfig;
+
+    private final SqlValidatorWithHints validator;
+    
+    private final CalciteCatalogReader catalog;
+    
+    private final ShardingSphereSchema shardingSphereSchema;
+
+    public RelNodeConverter(final String schemaName, final ShardingSphereSchema shardingSphereSchema) {
+        this.shardingSphereSchema = shardingSphereSchema;
+        this.converterConfig = SqlToRelConverter.config()
+                .withInSubQueryThreshold(Integer.MAX_VALUE)
+                .withExpand(false);
+        CalciteSchema rootSchema = createRootSchema(schemaName);
+        this.catalog = new CalciteCatalogReader(rootSchema,
+                rootSchema.path(schemaName),
+                new JavaTypeFactoryImpl(),
+                new CalciteConnectionConfigImpl(new Properties()));
+
+        validator = SqlValidatorUtil.newValidator(SqlStdOperatorTable.instance(), catalog, TYPE_FACTORY, SqlValidator.Config.DEFAULT
+                .withLenientOperatorLookup(false)
+                .withSqlConformance(SqlConformanceEnum.DEFAULT)
+                .withIdentifierExpansion(true));
+    }
+
+    /**
+     * validate calcite SqlNode and convert SqlNode to RelNode.
+     * @param sqlNode SqlNode to be validated and converted
+     * @return relational algebra of sql
+     */
+    public RelNode validateAndConvert(final SqlNode sqlNode) {
+        final RelOptCluster cluster = createRelOptCluster();
+        final SqlToRelConverter sqlToRelConverter = new SqlToRelConverter(null, validator, catalog, cluster,
+                StandardConvertletTable.INSTANCE, converterConfig);
+

Review comment:
       remove this empty line please.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] guimingyue commented on pull request #9934: Query optimization for federated SQL query

Posted by GitBox <gi...@apache.org>.
guimingyue commented on pull request #9934:
URL: https://github.com/apache/shardingsphere/pull/9934#issuecomment-864961831


   > @guimingyue Cool, I will give it a look.
   > Could you look at the latest master branch? I suppose 1 and 2 are supported now, and in terms of 3, could you give more clarification?
   > 
   > ```
   > 1. join type is missing.
   > 2. having clause of group by is not supported.
   > 3. expression has not been pared.
   > ```
   
   I have tested in the latest branch, the result follows.
   - joinType of JoinTableSegment is still null for inner join and left outer join according testcase of `org.apache.shardingsphere.infra.optimize.core.convert.SqlNodeConverterTest`  in my PR.
   - having clause is supported now, but the inner expression of AggregationProjectionSegment is String, take this sql  text as an example, `select order_id, user_id from t_order where order_id = 10 group by order_id having sum(order_id) > 0`, the type of expr field of `HavingSegment` is `BinaryOperationExpression` and the left parameter type of `BinaryOperationExpression` object is `AggregationProjectionSegment`. As a result, the type of `innerExpression` field is String whose value is `(order_id)`, which should be parsed as `ColumnSegment` object.
   - the field text of `ExpressionProjectionSegment` is a Stirng type, which cannot be converted to a AST expression, for example, in the select list of this sql text `select 10 + 30, order_id + 10, order_id, user_id from t_order`, the expression of `10+30`  will be parsed as an `ExpressionProjectionSegment` object.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org