[jira] [Commented] (SPARK-9924) checkForLogs and cleanLogs are scheduled at fixed rate and can get piled up

["Thomas Graves (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737416#comment-14737416 ] 

Thomas Graves commented on SPARK-9924:
--------------------------------------

[~vanzin] Any reason this wasn't picked back into spark 1.5 branch?

> checkForLogs and cleanLogs are scheduled at fixed rate and can get piled up
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-9924
>                 URL: https://issues.apache.org/jira/browse/SPARK-9924
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 1.4.0
>            Reporter: Rohit Agarwal
>            Assignee: Rohit Agarwal
>             Fix For: 1.6.0
>
>
> {{checkForLogs}} and {{cleanLogs}} are scheduled using {{ScheduledThreadPoolExecutor.scheduleAtFixedRate}}. When their execution takes more time than the interval at which they are scheduled, they get piled up.
> This is a problem on its own but the existence of SPARK-7189 makes it even worse. Let's say there is an eventLog which takes 15s to parse and which happens to be the last modified file (that gets reloaded again and again due to SPARK-7189.) If this file stays the last modified file for, let's say, an hour, then a lot of executions of that file would have piled up as the default {{spark.history.fs.update.interval}} is 10s. If there is a new eventLog file now, it won't show up in the history server ui for a long time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org