You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2019/08/25 23:48:00 UTC
[jira] [Comment Edited] (SPARK-28594) Allow event logs for running
streaming apps to be rolled over.
[ https://issues.apache.org/jira/browse/SPARK-28594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915357#comment-16915357 ]
Jungtaek Lim edited comment on SPARK-28594 at 8/25/19 11:47 PM:
----------------------------------------------------------------
Coincidentally I was working on the design of this feature for 2 weeks. Looks like reporter doesn't seem to work on this feature, I'll taking up this issue and go forward.
Only POC done. Just started implementing. Here's design doc to describe the approach:
[https://docs.google.com/document/d/12bdCC4nA58uveRxpeo8k7kGOI2NRTXmXyBOweSi4YcY/edit?usp=sharing]
was (Author: kabhwan):
Coincidentally I was working on the design of this feature for 2 weeks. Looks like reporter doesn't seem to work on this feature, I'll taking up this issue and go forward.
Only POC done. Just started implementing. Here's design doc to describe the approach:
[https://docs.google.com/document/d/12bdCC4nA58uveRxpeo8k7kGOI2NRTXmXyBOweSi4YcY/edit#heading=h.7bmfccqq7ozy]
> Allow event logs for running streaming apps to be rolled over.
> --------------------------------------------------------------
>
> Key: SPARK-28594
> URL: https://issues.apache.org/jira/browse/SPARK-28594
> Project: Spark
> Issue Type: Improvement
> Components: Input/Output
> Affects Versions: 3.0.0
> Environment: This has been reported on 2.0.2.22 but affects all currently available versions.
> Reporter: Stephen Levett
> Priority: Major
>
> At all current Spark releases when event logging on spark streaming is enabled the event logs grow massively. The files continue to grow until the application is stopped or killed.
> The Spark history server then has difficulty processing the files.
> https://issues.apache.org/jira/browse/SPARK-8617
> Addresses .inprogress files but not event log files that are still running.
> Identify a mechanism to set a "max file" size so that the file is rolled over when it reaches this size?
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org