[jira] [Commented] (SPARK-5210) Support log rolling in EventLogger

["Tao Wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14626325#comment-14626325 ] 

Tao Wang commented on SPARK-5210:
---------------------------------

Hi [~joshrosen], we found same problem in streaming / Thrift Server logs. The HistoryServer usually collapses because of OOM exception when it read too large event log written by long-running application.

Even we can tune memory settings for it, but it is not a elegant way as logs generated by Streaming/Thrift Server could increase infinitely.

We now plan to write event log to separate files according to their job id, say 50 jobs a file. Then HistoryServer could read small file relatively which has a low probability to cause OOM.

> Support log rolling in EventLogger
> ----------------------------------
>
>                 Key: SPARK-5210
>                 URL: https://issues.apache.org/jira/browse/SPARK-5210
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core, Web UI
>            Reporter: Josh Rosen
>
> For long-running Spark applications (e.g. running for days / weeks), the Spark event log may grow to be very large.
> As a result, it would be useful if EventLoggingListener supported log file rolling / rotation.  Adding this feature will involve changes to the HistoryServer in order to be able to load event logs from a sequence of files instead of a single file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org