[jira] [Updated] (SPARK-57039) Fold InnerJoin with single-row LocalRelation/OneRowRelation into Project

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-57039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-57039:
-----------------------------------
    Labels: pull-request-available  (was: )

> Fold InnerJoin with single-row LocalRelation/OneRowRelation into Project
> ------------------------------------------------------------------------
>
>                 Key: SPARK-57039
>                 URL: https://issues.apache.org/jira/browse/SPARK-57039
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Kent Yao
>            Priority: Major
>              Labels: pull-request-available
>
> h2. Background
> Currently {{ConvertToLocalRelation}} (Optimizer.scala) folds {{Project}} / {{Filter}} / {{Limit}} over {{LocalRelation}}, but does *not* fold {{InnerJoin(LocalRelation(single-row), other, Inner)}} or {{InnerJoin(OneRowRelation(), other, Inner)}} — even when the single-row side has no streaming / Unevaluable hazards and the user gave no join hint.
> This leaves a redundant join node that:
> * Forces broadcast/shuffle planning for what is logically a constant attach.
> * Blocks downstream constant-folding and column pruning of the single-row side's projected attributes.
> * Is structurally identical to what {{DecorrelateInnerQuery.scala:435}} flags as a TODO: {{// TODO add a more general rule to optimize join with OneRowRelation}}.
> h2. Proposal
> Add two complementary rules in {{Optimizer.scala}}:
> # *{{ConvertToLocalRelation}} case 5/6* (left/right symmetric arms) — fold {{Inner Join}} where one side is {{LocalRelation(out, [row], false, _)}} (single-row, ≥1 column) into {{Project(literals ++ other.output, other)}}, optionally wrapped in {{Filter(cond, ...)}}. Literal values preserve the original {{ExprId}} via {{Alias(Literal.create(...))(exprId)}}.
> # *{{FoldInnerJoinWithOneRowRelation}}* (new independent rule, tree-pattern {{INNER_LIKE_JOIN}}) — fold {{Inner Join}} where one side is {{OneRowRelation()}} (single-row, 0 columns) into {{other}} or {{Filter(cond, other)}}. {{OneRowRelation}} does *not* publish {{LOCAL_RELATION}} tree-pattern, so a single combined rule using {{transformWithPruning(_.containsPattern(LOCAL_RELATION))}} would silently miss it — hence two rules.
> Four narrowings (all strictly conservative):
> || Dimension || Accepted || Rejected || Reason ||
> | JoinType | {{Inner}} only | LeftOuter / RightOuter / FullOuter / LeftSemi / LeftAnti / Cross | Outer needs null-pad; Semi/Anti needs existence semantics; Cross has no condition (already handled). |
> | JoinHint | {{JoinHint.NONE}} only | BROADCAST / SHUFFLE_HASH / SHUFFLE_MERGE / SHUFFLE_REPLICATE_NL | User hints take priority over plan rewrite. |
> | Row count | {{data.length == 1}} | 0 (covered by {{PropagateEmptyRelation}}) / >1 | Inlining >1 rows as Literals would LoC-explode and require Cartesian materialization. |
> | Condition | {{!cond.exists(hasUnevaluableExpr)}} && {{!other.isStreaming}} | ScalarSubquery / DynamicPruning / streaming | Unevaluable expressions can't safely move into Filter; streaming has state-store ordering constraints. |
> h2. Test coverage
> * New {{FoldInnerJoinWithOneRowRelationSuite}}: 9 tests (OneRow × table left/right, no-cond, LeftOuter negative, ArrayType / MapType / nested StructType columns preserved, nested struct-of-array, Unevaluable-condition negative).
> * Existing {{ConvertToLocalRelationSuite}}: +4 tests (T7-T10) including a strong-assertion ExprId preservation test ({{collectFirst { case _: Join }.isEmpty && output.length == 4}}).
> * All 19 new unit tests verified RED-sensitive (fail under deliberately-broken rule, pass under correct impl).
> * Plan-stability regression: {{TPCDSV1_4 + TPCDSV1_4Stats + TPCDSV2_7 + TPCDSV2_7Stats + TPCDSModified + TPCDSModifiedStats + TPCH}} = *322 / 322 PASS, 0 plan-diff*.
> h2. Backward compatibility
> No user-facing change. Query results are bit-identical; only the optimized logical plan shape changes (one less Join, one more Project/Filter). No new SQLConf knob — the four narrowings make this strictly safe.
> h2. Related work
> Trino: {{RemoveRedundantCrossJoin}} + {{EvaluateZeroLimit}}. DuckDB: {{ConstantPropagation}} over cartesian. Doris: handled in FE constant-folding pass.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org