[jira] [Updated] (FLINK-12371) Add support for converting (NOT) IN/ (NOT) EXISTS to SemiJoin, and generating optimized logical plan

["godfrey he (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-12371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

godfrey he updated FLINK-12371:
-------------------------------
    Description: 
This issue aims to convert IN/EXISTS to semi-join, and NOT IN/NOT EXISTS to anti-join.

In Calcite, [SemiJoin|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/SemiJoin.java] only represents semi-join, (could not represent anti-join) and requires equi join condition. Queries like `select * from left where left.a1 in (select right.a2 from right where left.b1 > right.b2)` and `select * from left where not exists (select * from right)` could not be converted to Calcite SemiJoin operator.

To solve the above problem, We need copy the {{SemiJoin}} class to Flink, and make the following changes:
1. make {{SemiJoin}} extending from {{Join}}, not from {{EquiJoin}}. (to support non-equi join condition) 
2. add {{isAnti}} field attribute to represent anti-join.

Currently, there are no rules to convert (NOT) IN/ (NOT) EXISTS to SemiJoin, so we need a whole new rule (named {{FlinkSubQueryRemoveRule}}) to meet our requirement.

  was:
This issue aims to convert IN/EXISTS to semi-join, and NOT IN/NOT EXISTS to anti-join.

In Calcite, [SemiJoin|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/SemiJoin.java] only represents semi-join, (could not represent anti-join) and requires equi join condition. Queries like `select * from left where left.a1 in (select right.a2 from right where left.b1 > right.b2)` and `select * from left where not exists (select * from right)` could not be converted to Calcite SemiJoin operator.

To solve the above problem, We need copy the {{SemiJoin}} class to Flink, and make the following changes:
1. make {{SemiJoin}} extending from {{Join}}, not from {{EquiJoin}}. (to support non-equi join condition) 
2. add {{isAnti}} field attribute to represent anti-join.

Currently, there are no rules to convert (NOT) IN/ (NOT) EXISTS to SemiJoin, so we need a whole new set of rules to meet our requirement.
1. {{FlinkSubQueryRemoveRule}}, a planner rule that converts IN and EXISTS into semi-join, converts NOT IN and NOT EXISTS into anti-join
2. {{FlinkRewriteSubQueryRule}}, a planner rule that converts filter condition like: {{(select count(*) from T) > 0}} to {{exists(select * from T)}}


> Add support for converting (NOT) IN/ (NOT) EXISTS to SemiJoin, and generating optimized logical plan
> ----------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-12371
>                 URL: https://issues.apache.org/jira/browse/FLINK-12371
>             Project: Flink
>          Issue Type: New Feature
>          Components: Table SQL / Planner
>            Reporter: godfrey he
>            Assignee: godfrey he
>            Priority: Major
>
> This issue aims to convert IN/EXISTS to semi-join, and NOT IN/NOT EXISTS to anti-join.
> In Calcite, [SemiJoin|https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/rel/core/SemiJoin.java] only represents semi-join, (could not represent anti-join) and requires equi join condition. Queries like `select * from left where left.a1 in (select right.a2 from right where left.b1 > right.b2)` and `select * from left where not exists (select * from right)` could not be converted to Calcite SemiJoin operator.
> To solve the above problem, We need copy the {{SemiJoin}} class to Flink, and make the following changes:
> 1. make {{SemiJoin}} extending from {{Join}}, not from {{EquiJoin}}. (to support non-equi join condition) 
> 2. add {{isAnti}} field attribute to represent anti-join.
> Currently, there are no rules to convert (NOT) IN/ (NOT) EXISTS to SemiJoin, so we need a whole new rule (named {{FlinkSubQueryRemoveRule}}) to meet our requirement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)