[jira] [Commented] (SPARK-11777) HiveContext.saveAsTable does not update the metastore on overwrite

["Stanislav Hadjiiski (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15008588#comment-15008588 ] 

Stanislav Hadjiiski commented on SPARK-11777:
---------------------------------------------

I was to change the HDFS contents manually I would have issued a refresh after that. IMHO spark leaving the metastore in non-updated and non-readable (through Impala) state should not be acceptable. Furthermore, there seems not to be a spark-specific way to issue the update.

> HiveContext.saveAsTable does not update the metastore on overwrite
> ------------------------------------------------------------------
>
>                 Key: SPARK-11777
>                 URL: https://issues.apache.org/jira/browse/SPARK-11777
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.5.1
>            Reporter: Stanislav Hadjiiski
>
> Consider the following code:
> {quote}
> case class Bean(cdata: String)
> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sparkContext)
> val df = hiveContext.createDataFrame(Bean("test10") :: Bean("test20") :: Nil)
> df.write.mode(SaveMode.Overwrite).saveAsTable("db_name.table")
> {quote}
> This works as expected - if the table does not exist it is created, otherwise it's content is replaced. However, only in the first case the data is accessible through impala (i.e. outside of spark environment). To get it working after overwriting a
> {quote}
> REFRESH db_name.table
> {quote}
> should be issued in impala-shell. Neither
> {quote}
> hiveContext.refreshTable("db_name.table")
> {quote}
> nor
> {quote}
> hiveContext.sql("REFRESH TABLE db_name.table")
> {quote}
> fixes the issue. The same applies if the {{default}} database is used (and {{db_name.}} is omiited everywhere)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org