You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2016/11/23 17:16:00 UTC
[jira] [Comment Edited] (SPARK-18549) Failed to Uncache a View that
References a Dropped Table.
[ https://issues.apache.org/jira/browse/SPARK-18549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15690765#comment-15690765 ]
Xiao Li edited comment on SPARK-18549 at 11/23/16 5:15 PM:
-----------------------------------------------------------
Are you suggesting name-based resolution? This is risky. You know, cache manager is cross session. It is very hard for us to determine which temporary views, views, tables, or global temporary tables are the original cached data.
Cross session unique ID might be needed. However, I am not sure this is a right direction.
BTW, this could become worse if we re-implement the views in Spark 2.2.
was (Author: smilegator):
Are you suggesting name-based resolution? This is risky. You know, cache manager is cross session. It is very hard for us to determine the original temporary views, views, tables, or global temporary tables.
Cross session unique ID might be needed. However, I am not sure this is a right direction.
BTW, this could become worse if we re-implement the views in Spark 2.2.
> Failed to Uncache a View that References a Dropped Table.
> ---------------------------------------------------------
>
> Key: SPARK-18549
> URL: https://issues.apache.org/jira/browse/SPARK-18549
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.2
> Reporter: Xiao Li
> Priority: Critical
>
> {code}
> spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
> spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
> sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
> // Cache is empty at the beginning
> assert(spark.sharedState.cacheManager.isEmpty)
> sql("CACHE TABLE testView")
> assert(spark.catalog.isCached("testView"))
> // Cache is not empty
> assert(!spark.sharedState.cacheManager.isEmpty)
> {code}
> {code}
> // drop a table referenced by a cached view
> sql("DROP TABLE jt1")
> -- So far everything is fine
> // Failed to unache the view
> val e = intercept[AnalysisException] {
> sql("UNCACHE TABLE testView")
> }.getMessage
> assert(e.contains("Table or view not found: `default`.`jt1`"))
> // We are unable to drop it from the cache
> assert(!spark.sharedState.cacheManager.isEmpty)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org