You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/06/17 10:32:00 UTC
[jira] [Commented] (SPARK-7088) [REGRESSION] Spark 1.3.1 breaks
analysis of third-party logical plans
[ https://issues.apache.org/jira/browse/SPARK-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589484#comment-14589484 ]
Apache Spark commented on SPARK-7088:
-------------------------------------
User 'smola' has created a pull request for this issue:
https://github.com/apache/spark/pull/6853
> [REGRESSION] Spark 1.3.1 breaks analysis of third-party logical plans
> ---------------------------------------------------------------------
>
> Key: SPARK-7088
> URL: https://issues.apache.org/jira/browse/SPARK-7088
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.3.1
> Reporter: Santiago M. Mola
> Priority: Critical
> Labels: regression
>
> We're using some custom logical plans. We are now migrating from Spark 1.3.0 to 1.3.1 and found a few incompatible API changes. All of them seem to be in internal code, so we understand that. But now the ResolveReferences rule, that used to work with third-party logical plans just does not work, without any possible workaround that I'm aware other than just copying ResolveReferences rule and using it with our own fix.
> The change in question is this section of code:
> {code}
> }.headOption.getOrElse { // Only handle first case, others will be fixed on the next pass.
> sys.error(
> s"""
> |Failure when resolving conflicting references in Join:
> |$plan
> |
> |Conflicting attributes: ${conflictingAttributes.mkString(",")}
> """.stripMargin)
> }
> {code}
> Which causes the following error on analysis:
> {code}
> Failure when resolving conflicting references in Join:
> 'Project ['l.name,'r.name,'FUNC1('l.node,'r.node) AS c2#37,'FUNC2('l.node,'r.node) AS c3#38,'FUNC3('r.node,'l.node) AS c4#39]
> 'Join Inner, None
> Subquery l
> Subquery h
> Project [name#12,node#36]
> CustomPlan H, u, (p#13L = s#14L), [ord#15 ASC], IS NULL p#13L, node#36
> Subquery v
> Subquery h_src
> LogicalRDD [name#12,p#13L,s#14L,ord#15], MapPartitionsRDD[1] at mapPartitions at ExistingRDD.scala:37
> Subquery r
> Subquery h
> Project [name#40,node#36]
> CustomPlan H, u, (p#41L = s#42L), [ord#43 ASC], IS NULL pred#41L, node#36
> Subquery v
> Subquery h_src
> LogicalRDD [name#40,p#41L,s#42L,ord#43], MapPartitionsRDD[1] at mapPartitions at ExistingRDD.scala:37
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org