You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2016/06/15 22:13:09 UTC
[jira] [Resolved] (SPARK-9689) Cache doesn't refresh for
HadoopFsRelation based table
[ https://issues.apache.org/jira/browse/SPARK-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin resolved SPARK-9689.
--------------------------------
Resolution: Fixed
Assignee: (was: Cheng Hao)
Fix Version/s: 2.0.0
I think this one has been fixed in 2.0 already.
> Cache doesn't refresh for HadoopFsRelation based table
> ------------------------------------------------------
>
> Key: SPARK-9689
> URL: https://issues.apache.org/jira/browse/SPARK-9689
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.1, 1.5.0
> Reporter: Cheng Hao
> Fix For: 2.0.0
>
>
> {code:title=example|borderStyle=solid}
> // create a HadoopFsRelation based table
> sql(s"""
> |CREATE TEMPORARY TABLE jsonTable (a int, b string)
> |USING org.apache.spark.sql.json.DefaultSource
> |OPTIONS (
> | path '${path.toString}'
> |)""".stripMargin)
>
> // give the value from table jt
> sql(
> s"""
> |INSERT OVERWRITE TABLE jsonTable SELECT a, b FROM jt
> """.stripMargin)
> // cache the HadoopFsRelation Table
> sqlContext.cacheTable("jsonTable")
>
> // update the HadoopFsRelation Table
> sql(
> s"""
> |INSERT OVERWRITE TABLE jsonTable SELECT a * 2, b FROM jt
> """.stripMargin)
> // Even this will fail
> sql("SELECT a, b FROM jsonTable").collect()
> // This will fail, as the cache doesn't refresh
> checkAnswer(
> sql("SELECT a, b FROM jsonTable"),
> sql("SELECT a * 2, b FROM jt").collect())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org