[jira] [Comment Edited] (SPARK-1940) Enable rolling of executor logs (stdout / stderr)

["Tathagata Das (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010413#comment-14010413 ] 

Tathagata Das edited comment on SPARK-1940 at 6/4/14 1:50 AM:
--------------------------------------------------------------

This is solved in https://github.com/apache/spark/pull/895

This PR solves this by implementing a simple `RollingFileAppender` within Spark (disabled by default). When enabled (using configuration parameter `spark.executor.rollingLogs.enabled`), the logs can get rolled over either by time interval (set with `spark.executor.rollingLogs.interval`, set to daily by default), or by size of logs (set with  `spark.executor.rollingLogs.size`). Finally, old logs can be automatically deleted by specifying how many of the latest log files to keep (set with `spark.executor.rollingLogs.keepLastN`). The web UI can show the logs across the rolled over files.




was (Author: tdas):
This is solved in https://github.com/apache/spark/pull/895

This PR solves this by implementing a simple RollingFileAppender within Spark (disabled by default). When enabled (using configuration parameter spark.executor.rollingLogs.enabled), the logs can get rolled over by time interval (set with spark.executor.rollingLogs.cleanupTtl, set to daily by default). Old logs (older than 7 days, by default) will get deleted automatically. The web UI can show the logs across the rolled over files.



> Enable rolling of executor logs (stdout / stderr)
> -------------------------------------------------
>
>                 Key: SPARK-1940
>                 URL: https://issues.apache.org/jira/browse/SPARK-1940
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Tathagata Das
>            Assignee: Tathagata Das
>
> Currently, in the default log4j configuration, all the executor logs get sent to the file <code>[executor-working-dir]/stderr</code>. This does not all log files to be rolled, so old logs cannot be removed. 
> Using log4j RollingFileAppender allows log4j logs to be rolled, but all the logs get sent to a different set of files, other than the files <code>stdout</code> and <code>stderr</code> . So the logs are not visible in the Spark web UI any more as Spark web UI only reads the files <code>stdout</code> and <code>stderr</code>. Furthermore, it still does not allow the stdout and stderr to be cleared periodically in case a large amount of stuff gets written to them (e.g. by explicit println inside map function).
> Solving this requires rolling of the logs in such a way that Spark web UI is aware of it and can retrieve the logs across the rolled-over files.



--
This message was sent by Atlassian JIRA
(v6.2#6252)