[jira] [Updated] (SPARK-18544) Append with df.saveAsTable writes data to wrong location

["Yin Huai (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yin Huai updated SPARK-18544:
-----------------------------
    Priority: Blocker  (was: Major)

> Append with df.saveAsTable writes data to wrong location
> --------------------------------------------------------
>
>                 Key: SPARK-18544
>                 URL: https://issues.apache.org/jira/browse/SPARK-18544
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Eric Liang
>            Priority: Blocker
>
> When using saveAsTable in append mode, data will be written to the wrong location for non-managed Datasource tables. The following example illustrates this.
> It seems somehow pass the wrong table path to InsertIntoHadoopFsRelation from DataFrameWriter. Also, we should probably remove the repair table call at the end of saveAsTable in DataFrameWriter. That shouldn't be needed in either the Hive or Datasource case.
> {code}
> scala> spark.sqlContext.range(10000).selectExpr("id", "id as A", "id as B").write.partitionBy("A", "B").mode("overwrite").parquet("/tmp/test_10k")
> scala> sql("msck repair table test_10k")
> scala> sql("select * from test_10k where A = 1").count
> res6: Long = 1
> scala> spark.sqlContext.range(10).selectExpr("id", "id as A", "id as B").write.partitionBy("A", "B").mode("append").parquet("/tmp/test_10k")
> scala> sql("select * from test_10k where A = 1").count
> res8: Long = 1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org