You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "YoungWoo Kim (JIRA)" <ji...@apache.org> on 2015/08/20 10:33:48 UTC
[jira] [Comment Edited] (BIGTOP-1971) Support Spark SQL CLI with
Apache Hive out of the box
[ https://issues.apache.org/jira/browse/BIGTOP-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704499#comment-14704499 ]
YoungWoo Kim edited comment on BIGTOP-1971 at 8/20/15 8:33 AM:
---------------------------------------------------------------
Without SPARK_HOME/conf/hive-site.xml, "$ ./spark-sql --master yarn" failed:
{noformat}
...
Caused by: java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 76 more
Caused by: java.sql.SQLException: Directory /usr/lib/spark/metastore_db cannot be created.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
... 73 more
Caused by: ERROR XBM0H: Directory /usr/lib/spark/metastore_db cannot be created.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.services.monitor.StorageFactoryService$10.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source)
... 73 more
{noformat}
Users can create own hive-site.xml or symlink existing one but it should work out of the box. Currently Apache Hive on Bigtop just works without additional configuration by default.
was (Author: warwithin):
Without SPARK_HOME/conf/hive-site.xml, "$ ./spark-sql --master yarn" failed:
{noformat}
...
Caused by: java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
... 76 more
Caused by: java.sql.SQLException: Directory /usr/lib/spark/metastore_db cannot be created.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
... 73 more
Caused by: ERROR XBM0H: Directory /usr/lib/spark/metastore_db cannot be created.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.services.monitor.StorageFactoryService$10.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source)
... 73 more
{noformat}
> Support Spark SQL CLI with Apache Hive out of the box
> -----------------------------------------------------
>
> Key: BIGTOP-1971
> URL: https://issues.apache.org/jira/browse/BIGTOP-1971
> Project: Bigtop
> Issue Type: Improvement
> Components: spark
> Affects Versions: 1.0.0
> Reporter: YoungWoo Kim
> Assignee: YoungWoo Kim
> Fix For: 1.1.0
>
>
> Spark SQL supports reading and writing Apache Hive. To run the spark-sql CLI with Hive support, It's required $\{SPARK_HOME\}/conf/hive-site.xml file, so, Bigtop should package the hive-site.xml for Spark SQL (with Hive support)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)