[Impala-ASF-CR] IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE

["Csaba Ringhofer (Code Review)" <ge...@cloudera.org>]

Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18731 )

Change subject: IMPALA-11424: Support pushdown non-equi join predicate from OUTER/INNER JOIN to SCANNODE
......................................................................


Patch Set 17:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
File testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test:

http://gerrit.cloudera.org:8080/#/c/18731/11/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test@5
PS11, Line 5: alltype
> Yes, you are right, it will be better. I added.
About Parquet + ORC stat/dictionary filters:
these predicates are only visible in the plan from explain_level=2

The Kudu predicates and partition pruning is visible at level 1.


http://gerrit.cloudera.org:8080/#/c/18731/17/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test
File testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test:

http://gerrit.cloudera.org:8080/#/c/18731/17/testdata/workloads/functional-planner/queries/PlannerTest/none-equal-predicate-push-down.test@587
PS17, Line 587:   HDFS partitions=24/24
Partition pruning does not seem to work here - t2.`month` <= 2 should reduce the number of partitions.

It would be great to investigate this in this patch or at least a TODO should be added to the test.

AFAIK partition pruning happens here:
https://github.com/apache/impala/blob/c38afdf50f652dcc454eb8f20da45224e3c9bfca/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L1585

This is done before calling HashJoinNode.init(), so these new conjuncts do not exist at that point yet.

I think that it would be the best to generate the new conjuncts earlier so that the scan nodes can pick them up automatically, but I am not yet sure about the right place to do this.



-- 
To view, visit http://gerrit.cloudera.org:8080/18731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3ce23cbd7522a209c830504f329b972d67bc263
Gerrit-Change-Number: 18731
Gerrit-PatchSet: 17
Gerrit-Owner: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Baike Xia <xi...@163.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 02 Jan 2023 13:28:44 +0000
Gerrit-HasComments: Yes