[jira] [Created] (SPARK-18716) Restrict the disk usage of spark event log.

["Genmao Yu (JIRA)" <ji...@apache.org>]

Genmao Yu created SPARK-18716:
---------------------------------

             Summary: Restrict the disk usage of spark event log. 
                 Key: SPARK-18716
                 URL: https://issues.apache.org/jira/browse/SPARK-18716
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.0.2
            Reporter: Genmao Yu


We've had reports of overfull disk usage of spark event log file. Current implementation has following drawbacks:
1. If we did not start Spark HistoryServer or Spark HistoryServer just failed, there is no chance to do clean work.
2. Spark HistoryServer is cleaning event log file based on file age only. If there are abundant applications constantly, the disk usage in every {{spark.history.fs.cleaner.maxAge}} can still be very large.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org