[jira] [Created] (FLINK-24139) Push down more predicates through Join in stream mode

["Alexander Trushev (Jira)" <ji...@apache.org>]

Alexander Trushev created FLINK-24139:
-----------------------------------------

             Summary: Push down more predicates through Join in stream mode
                 Key: FLINK-24139
                 URL: https://issues.apache.org/jira/browse/FLINK-24139
             Project: Flink
          Issue Type: Improvement
          Components: Table SQL / Planner
            Reporter: Alexander Trushev
         Attachments: q13_after.json, q13_after.png, q13_after.txt, q13_before.json, q13_before.png, q13_before.txt

h3. Context

Rule {{JoinDependentConditionDerivationRule}} is introduced in FLINK-12509. This rule rewrites join condition in such way that more predicates can be pushed down through join. For example,
 # Source A = [a0, a1, a2], source B = [b0, b1]
 # 
{code:sql}
select * from A join B on a0 = b0 where (a1 = 0 and b1 = 0) or a2 = 0{code}

 # {{JoinDependentConditionDerivationRule}} transforms condition ((a1 and b1) or a2) to (((a1 and b1) or a2) and (a1 or a2))
 # {{JoinConditionPushRule}} pushes (a1 or a2) to A source

It is a good optimization that can lead to performance improvement of query execution.
 Currently, {{JoinDependentConditionDerivationRule}} is used only in batch mode.
h3. Proposal

Enable {{JoinDependentConditionDerivationRule}} in stream mode.
h3. Benefit

Experiment based on [https://github.com/ververica/flink-sql-benchmark]
 Cluster – 4 nodes each 2 slots
 Dataset – tpcds_bin_orc_20
 Before – 1.14.0-rc0
 After – 1.14.0-rc0 + patched {{FlinkStreamProgram}} including {{JoinDependentConditionDerivationRule}}
||TPC-DS 20 GB||Before||After||
|q13 stream mode|83 s|8 s|

Query plan, stream graph, dashboard visualization before and after the patch are in the attachment

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)