[jira] [Created] (SPARK-53143) Fix self join in DataFrame API - Join is not the only expected output from analyzer

["David Milicevic (Jira)" <ji...@apache.org>]

David Milicevic created SPARK-53143:
---------------------------------------

             Summary: Fix self join in DataFrame API - Join is not the only expected output from analyzer
                 Key: SPARK-53143
                 URL: https://issues.apache.org/jira/browse/SPARK-53143
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 4.0.0
            Reporter: David Milicevic


There is an edge case with self join when multiple joins are present.
 
Example: two joins, where the latter one is self join.
The first one is the "using" join - in this case, analyzer's
`ResolveNaturalAndUsingJoin` will add `Project` as the top node.
The second join is a self join, but with specified join condition (i.e. `joinExprs`) -
if the join condition uses columns that are not part of the project list (of the first
join), `AddMetadataColumns` rule will be hit to add metadata for those columns. As a
consequence, `Project` will be added to the top of joined plan to return the
original/expected list of projected columns.

Whereas similar (i.e. `Project` node on top) can happen in multiple other cases,
from `Dataset` perspective the issue is specific to self joins only, since
`resolveSelfJoinCondition` assumed that the analyzed plan will be always of `Join` type.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org