You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@bigtop.apache.org by "Konstantinos Tsakalozos (JIRA)" <ji...@apache.org> on 2016/05/26 09:13:12 UTC

[jira] [Created] (BIGTOP-2462) Make Spark events and history logs directory configurable

Konstantinos Tsakalozos created BIGTOP-2462:
-----------------------------------------------

             Summary: Make Spark events and history logs directory configurable
                 Key: BIGTOP-2462
                 URL: https://issues.apache.org/jira/browse/BIGTOP-2462
             Project: Bigtop
          Issue Type: Improvement
          Components: spark
    Affects Versions: 1.1.0
            Reporter: Konstantinos Tsakalozos


Currently spark events and history logs are hard coded to a path inside HDFS. Our spark-defaults.conf contains:

spark.eventLog.dir hdfs:///var/log/spark/apps
spark.history.fs.logDirectory hdfs:///var/log/spark/apps

As a result deploying Bigtop Spark forces you to also have HDFS. This is an unnecessary dependency since Spark can operate with a variety of storage solutions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)