[jira] [Created] (SPARK-9600) DataFrameWriter.saveAsTable always writes data to "/user/hive/warehouse"

["Cheng Lian (JIRA)" <ji...@apache.org>]

Cheng Lian created SPARK-9600:
---------------------------------

             Summary: DataFrameWriter.saveAsTable always writes data to "/user/hive/warehouse"
                 Key: SPARK-9600
                 URL: https://issues.apache.org/jira/browse/SPARK-9600
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 1.4.1, 1.5.0
            Reporter: Cheng Lian
            Priority: Critical


Having a {{hive-site.xml}} with a non-default {{hive.metastore.warehouse.dir}} value, Spark SQL still writes to the default warehouse location {{/user/hive/warehouse}} when saving data source tables using {{DataFrameWriter.saveAsTable()}}:
{noformat}
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost/metastore_hive13_hadoop2</value>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>password</value>
  </property>

  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>hdfs://localhost:9000/user/hive/warehouse_hive13</value>
  </property>
</configuration>
{noformat}
Spark shell snippet to reproduce:
{noformat}
sqlContext.range(10).write.saveAsTable("xxx")
{noformat}
Running {{DESC EXTENDED xxx}} in Hive to check SerDe propertyies:
{noformat}
...
location:hdfs://localhost:9000/user/hive/warehouse_hive13/xxx
...
parameters:{path=hdfs://localhost:9000/user/hive/warehouse/xxx, serialization.format=1})
...
{noformat}
We are probably using execution Hive configuration when calling {{HiveMetastoreCatalog.hiveDefaultTableFilePath()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org