[jira] [Created] (SPARK-6751) Spark History Server support multiple application attempts

["Thomas Graves (JIRA)" <ji...@apache.org>]

Thomas Graves created SPARK-6751:
------------------------------------

             Summary: Spark History Server support multiple application attempts
                 Key: SPARK-6751
                 URL: https://issues.apache.org/jira/browse/SPARK-6751
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core, YARN
    Affects Versions: 1.3.0
            Reporter: Thomas Graves


Spark on Yarn supports running multiple application attempts (configurable number) in case the first (or second..) attempts fail.  The Spark History server only supports one history file though.  Under the default configs it keeps the first attempts history file. You can set the undocumented config spark.eventLog.overwrite to allow the follow on attempts to overwrite the first attempts history file.

Note that in spark 1.2 not having the overwrite config set causes any following attempts to actually fail to run, in spark 1.3 they run and you just see a warning at the end of the attempts.

It would be really nice to have an option that keeps all the attempts history files.  This way a user can go back and look at each one individually.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org