You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Armbrust (JIRA)" <ji...@apache.org> on 2015/09/09 01:17:45 UTC
[jira] [Resolved] (SPARK-10327) Cache Table is not working while
subquery has alias in its project list
[ https://issues.apache.org/jira/browse/SPARK-10327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Armbrust resolved SPARK-10327.
--------------------------------------
Resolution: Fixed
Fix Version/s: 1.6.0
Issue resolved by pull request 8494
[https://github.com/apache/spark/pull/8494]
> Cache Table is not working while subquery has alias in its project list
> -----------------------------------------------------------------------
>
> Key: SPARK-10327
> URL: https://issues.apache.org/jira/browse/SPARK-10327
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Cheng Hao
> Fix For: 1.6.0
>
>
> Code to reproduce that:
> {code}
> import org.apache.spark.sql.hive.execution.HiveTableScan
> sql("select key, value, key + 1 from src").registerTempTable("abc")
> cacheTable("abc")
> val sparkPlan = sql(
> """select a.key, b.key, c.key from
> |abc a join abc b on a.key=b.key
> |join abc c on a.key=c.key""".stripMargin).queryExecution.sparkPlan
> assert(sparkPlan.collect { case e: InMemoryColumnarTableScan => e }.size === 3) // failed
> assert(sparkPlan.collect { case e: HiveTableScan => e }.size === 0) // failed
> {code}
> The query plan like:
> {code}
> == Parsed Logical Plan ==
> 'Project [unresolvedalias('a.key),unresolvedalias('b.key),unresolvedalias('c.key)]
> 'Join Inner, Some(('a.key = 'c.key))
> 'Join Inner, Some(('a.key = 'b.key))
> 'UnresolvedRelation [abc], Some(a)
> 'UnresolvedRelation [abc], Some(b)
> 'UnresolvedRelation [abc], Some(c)
> == Analyzed Logical Plan ==
> key: int, key: int, key: int
> Project [key#14,key#61,key#66]
> Join Inner, Some((key#14 = key#66))
> Join Inner, Some((key#14 = key#61))
> Subquery a
> Subquery abc
> Project [key#14,value#15,(key#14 + 1) AS _c2#16]
> MetastoreRelation default, src, None
> Subquery b
> Subquery abc
> Project [key#61,value#62,(key#61 + 1) AS _c2#58]
> MetastoreRelation default, src, None
> Subquery c
> Subquery abc
> Project [key#66,value#67,(key#66 + 1) AS _c2#63]
> MetastoreRelation default, src, None
> == Optimized Logical Plan ==
> Project [key#14,key#61,key#66]
> Join Inner, Some((key#14 = key#66))
> Project [key#14,key#61]
> Join Inner, Some((key#14 = key#61))
> Project [key#14]
> InMemoryRelation [key#14,value#15,_c2#16], true, 10000, StorageLevel(true, true, false, true, 1), (Project [key#14,value#15,(key#14 + 1) AS _c2#16]), Some(abc)
> Project [key#61]
> MetastoreRelation default, src, None
> Project [key#66]
> MetastoreRelation default, src, None
> == Physical Plan ==
> TungstenProject [key#14,key#61,key#66]
> BroadcastHashJoin [key#14], [key#66], BuildRight
> TungstenProject [key#14,key#61]
> BroadcastHashJoin [key#14], [key#61], BuildRight
> ConvertToUnsafe
> InMemoryColumnarTableScan [key#14], (InMemoryRelation [key#14,value#15,_c2#16], true, 10000, StorageLevel(true, true, false, true, 1), (Project [key#14,value#15,(key#14 + 1) AS _c2#16]), Some(abc))
> ConvertToUnsafe
> HiveTableScan [key#61], (MetastoreRelation default, src, None)
> ConvertToUnsafe
> HiveTableScan [key#66], (MetastoreRelation default, src, None)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org