[jira] [Updated] (SPARK-16395) Show warning if CheckpointWriteHandlers are queued up in the fixed thread pool

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-16395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated SPARK-16395:
------------------------------
    Priority: Major  (was: Critical)

Is this related to my comment just now on https://issues.apache.org/jira/browse/SPARK-13288?focusedCommentId=15364319&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15364319 or a huge coincidence? If so perhaps it's better to not split this off as a separate JIRA.

> Show warning if CheckpointWriteHandlers are queued up in the fixed thread pool
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-16395
>                 URL: https://issues.apache.org/jira/browse/SPARK-16395
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>            Reporter: Yan Chen
>
> When the batch interval is small and time spent writing to HDFS for checkpointing is always longer than the batch interval, the CheckpointWriteHandlers could be queued up in the fixed thread pool. 
> Memory issue could happen in this situation.
> So if we can log a warning if this situation happens, it will help a lot of people when they encounter the same problem.
> The idea is if the size of the queue exceeds some number, we log a warning saying it's queued up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org