You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Sergey Nuyanzin (Jira)" <ji...@apache.org> on 2024/11/15 08:35:00 UTC
[jira] [Commented] (FLINK-36715) Changed behavior for filter push down for IS NOT DISTINCT FROM after Calcite upgrade to 1.33
[ https://issues.apache.org/jira/browse/FLINK-36715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17898515#comment-17898515 ]
Sergey Nuyanzin commented on FLINK-36715:
-----------------------------------------
The main issue is that after such optimization plan for some queries changed e.g. for
{code:sql}
SELECT * FROM
(SELECT * FROM src WHERE k = 0) src1
LEFT OUTER JOIN
(SELECT * from src WHERE k = 0) src2
ON (src1.k = src2.k AND src2.k > 10)
{code}
before the change
{code}
LogicalProject(k=[$0], v=[$1], k0=[$2], v0=[$3])
+- LogicalJoin(condition=[AND(=($0, $2), >($2, 10))], joinType=[left])
:- LogicalProject(k=[$0], v=[$1])
: +- LogicalFilter(condition=[=($0, 0)])
: +- LogicalTableScan(table=[[default_catalog, default_database, src, source: [TestTableSource(k, v)]]])
+- LogicalProject(k=[$0], v=[$1])
+- LogicalFilter(condition=[=($0, 0)])
+- LogicalTableScan(table=[[default_catalog, default_database, src, source: [TestTableSource(k, v)]]])
{code}
after the change
{code}
LogicalProject(exprs=[[CAST(0:BIGINT):BIGINT, $0, $1, $2]])
+- LogicalJoin(condition=[true], joinType=[left])
:- LogicalProject(exprs=[[$1]])
: +- LogicalFilter(condition=[=($0, 0)])
: +- LogicalTableScan(table=[[default_catalog, default_database, src, source: [TestTableSource(k, v)]]])
+- LogicalValues(type=[RecordType(BIGINT k, VARCHAR(2147483647) v)], tuples=[[]])
{code}
So from one side it was able to detect that from one side {{k}} should be 0 and could be moved to projection and {{src2.k > 10}} could be pushed down.
From the other side join condition became {{true}} and some joins stopped working e.g. SortMergeJoin since they expect {{equi}} join condition which is now not present here...
> Changed behavior for filter push down for IS NOT DISTINCT FROM after Calcite upgrade to 1.33
> ---------------------------------------------------------------------------------------------
>
> Key: FLINK-36715
> URL: https://issues.apache.org/jira/browse/FLINK-36715
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API, Table SQL / Planner
> Reporter: Sergey Nuyanzin
> Priority: Major
>
> After moving to 1.33 need to put RexUtil into Filnk repo to prevent a number of failures (more than 30)
> like
> {noformat}
> FlinkLogicalCalc(select=[CAST(0:BIGINT AS BIGINT) AS k, v, k AS k0, v0])
> +- FlinkLogicalJoin(condition=[true], joinType=[left])
> :- FlinkLogicalCalc(select=[v], where=[=(k, 0:BIGINT)])
> : +- FlinkLogicalLegacyTableSourceScan(table=[[default_catalog, default_database, src, source: [TestTableSource(k, v)]]], fields=[k, v])
> +- FlinkLogicalValues(type=[RecordType(BIGINT k, VARCHAR(2147483647) v)], tuples=[[]])
> This exception indicates that the query uses an unsupported SQL feature.
> Please check the documentation for the set of currently supported SQL features.
> {noformat}
> This is a result of https://issues.apache.org/jira/browse/CALCITE-5336
> the jira issue is to adapt Flink behavior and to remove RexUtil after that from Flink repo
--
This message was sent by Atlassian Jira
(v8.20.10#820010)