You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/09/12 20:39:00 UTC
[jira] [Updated] (SPARK-49618) Union ( & UnionExec) nodes equality not take into account unaligned positions of branches causing NO ( reuse of exchange and cached plans)
[ https://issues.apache.org/jira/browse/SPARK-49618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-49618:
-----------------------------------
Labels: pull-request-available (was: )
> Union ( & UnionExec) nodes equality not take into account unaligned positions of branches causing NO ( reuse of exchange and cached plans)
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-49618
> URL: https://issues.apache.org/jira/browse/SPARK-49618
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.3.4, 3.4.3
> Reporter: Asif
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.4
>
>
> Ideally a Union( plan1, plan2) and Union (plan2, plan1) are logically equal, so long as the output attributes of plan1 and plan2 are matching in terms of name , data type, metdadata etc ( though differing in terms of exprId).
> But because current equality and hashCode is dependent on the order of the children, the canonicalizations do not match.
> This causes reuse of exchange not happening in following situations:
>
> Exchange 1. = Union( plan1, plan2)
> Exchange 2 = Union( plan2, plan1)
>
> similarly the cached lookup also misses picking the InMemoryRelation.
>
> Will be submitting a PR and bug tests for the above scenarios.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org