You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dilip Biswal (JIRA)" <ji...@apache.org> on 2018/08/21 22:48:00 UTC
[jira] [Comment Edited] (SPARK-25168) PlanTest.comparePlans may
make a supplied resolved plan unresolved.
[ https://issues.apache.org/jira/browse/SPARK-25168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587956#comment-16587956 ]
Dilip Biswal edited comment on SPARK-25168 at 8/21/18 10:47 PM:
----------------------------------------------------------------
[~cloud_fan] OK.. Let me close this then since we are unable to do anything about this at present and since nothing is seriously impacted. Perhaps we can consider making normalizedPlan private so callers are not handed out unresolved plans ?
was (Author: dkbiswal):
[~cloud_fan] OK.. Let me close this then since we are unable to do anything about this at present and since nothing is seriously impacted. Perhaps we can consider making normalizedPlan private so callers are not handled out unresolved plans ?
> PlanTest.comparePlans may make a supplied resolved plan unresolved.
> -------------------------------------------------------------------
>
> Key: SPARK-25168
> URL: https://issues.apache.org/jira/browse/SPARK-25168
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.1
> Reporter: Dilip Biswal
> Priority: Minor
>
> Hello,
> I came across this behaviour while writing unit test for one of my PR. We use the utlility class PlanTest for writing plan verification tests. Currently when we call comparePlans and the resolved input plan contains a Join operator, the plan becomes unresolved after the call to `normalizePlan(normalizeExprIds(plan1))`. The reason for this is that, after cleaning the expression ids, duplicateResolved returns false making the Join operator unresolved.
> {code}
> def duplicateResolved: Boolean = left.outputSet.intersect(right.outputSet).isEmpty
> // Joins are only resolved if they don't introduce ambiguous expression ids.
> // NaturalJoin should be ready for resolution only if everything else is resolved here
> lazy val resolvedExceptNatural: Boolean = {
> childrenResolved &&
> expressions.forall(_.resolved) &&
> duplicateResolved &&
> condition.forall(_.dataType == BooleanType)
> }
> {code}
> Please note that, plan verification actually works fine. It just looked awkward to compare two unresolved plan for equality.
> I am opening this ticket to discuss if it is an okay behaviour. If its an tolerable behaviour then we can close the ticket.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org