You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Sebastian (JIRA)" <ji...@apache.org> on 2016/11/26 19:53:58 UTC
[jira] [Comment Edited] (SPARK-15034) Use the value of
spark.sql.warehouse.dir as the warehouse location instead of using
hive.metastore.warehouse.dir
[ https://issues.apache.org/jira/browse/SPARK-15034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15698424#comment-15698424 ]
Thomas Sebastian edited comment on SPARK-15034 at 11/26/16 7:53 PM:
--------------------------------------------------------------------
Hi [~yhuai]
Could you suggest to resolve this:
I want my local mode spark metastore point to the hdfs location: /user/hive/warehouse.
Could you please suggest, what solution works here.
- I am using spark 2.0 in local mode.
- Hive is already pointing to /user/hive/warehouse in hdfs location
-I do not see spark not getting pointed to /user/hive/warehouse in hdfs, instead it points to a local file location-where spark is installed.
-I already tried below, still not working.
SparkSession.builder().appName("eg").config("spark.sql.warehouse.dir", "hdfs://localhost:9000/user/hive/warehouse").enableHiveSupport().getOrCreate()
- Error thrown is:
-----------------------
16/11/27 01:06:26 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/11/27 01:06:26 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
16/11/27 01:06:31 ERROR metastore.RetryingHMSHandler: AlreadyExistsException(message:Database default already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:891)
was (Author: thomastechs):
Hi [~yhuai]
Could you suggest to resolve this:
I want my local mode spark metastore point to the hdfs location: /user/hive/warehouse.
Could you please suggest, what solution works here.
- I am using spark in local mode.
- Hive is already pointing to /user/hive/warehouse in hdfs location
-I do not see spark not getting pointed to /user/hive/warehouse in hdfs, instead it points to a local file location-where spark is installed.
-I already tried below, still not working.
SparkSession.builder().appName("eg").config("spark.sql.warehouse.dir", "hdfs://localhost:9000/user/hive/warehouse").enableHiveSupport().getOrCreate()
- Error thrown is:
-----------------------
16/11/27 01:06:26 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/11/27 01:06:26 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
16/11/27 01:06:31 ERROR metastore.RetryingHMSHandler: AlreadyExistsException(message:Database default already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:891)
> Use the value of spark.sql.warehouse.dir as the warehouse location instead of using hive.metastore.warehouse.dir
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-15034
> URL: https://issues.apache.org/jira/browse/SPARK-15034
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Yin Huai
> Assignee: Yin Huai
> Labels: release_notes, releasenotes
> Fix For: 2.0.0
>
>
> Starting from Spark 2.0, spark.sql.warehouse.dir will be the conf to set warehouse location. We will not use hive.metastore.warehouse.dir.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org