[jira] [Commented] (BIGTOP-3776) spark-thriftserver should work without external hive-metastore

["Masatake Iwasaki (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17579329#comment-17579329 ] 

Masatake Iwasaki commented on BIGTOP-3776:
------------------------------------------

If Spark is deployed without Hive, spark-thriftserver fails to start since embedded derby can not create database files.

{noformat}
Wed Aug 03 04:35:49 UTC 2022 Thread[main,5,main] java.io.FileNotFoundException: derby.log (Permission denied)
Wed Aug 03 04:35:49 UTC 2022 Thread[main,5,main] Cleanup action starting
ERROR XBM0H: Directory /metastore_db cannot be created.
        at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
        at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
        at org.apache.derby.impl.services.monitor.StorageFactoryService$10.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source)
        at org.apache.derby.impl.services.monitor.FileMonitor.createPersistentService(Unknown Source)
        at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection$5.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.derby.impl.jdbc.EmbedConnection.createPersistentService(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.createDatabase(Unknown Source)
        ...
{noformat}


I'm expecting conditionally adding following configuration to /etc/spark/conf/spark-defalt.conf is the fix.

{noformat}
spark.hadoop.javax.jdo.option.ConnectionURL jdbc:derby:;databaseName=/var/lib/spark/metastore_db;create=true
{noformat}


> spark-thriftserver should work without external hive-metastore
> --------------------------------------------------------------
>
>                 Key: BIGTOP-3776
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-3776
>             Project: Bigtop
>          Issue Type: Bug
>          Components: deployment
>            Reporter: Masatake Iwasaki
>            Priority: Major
>
> Puppet manifest to deploy spark-thriftserver implicitly depend on external hive-metastore (and hive-site.xml pointing it). It should work even if spark is deployed alone.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)