You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dominique De Vito (JIRA)" <ji...@apache.org> on 2019/07/06 15:51:00 UTC
[jira] [Commented] (SPARK-19977) Scheduler Delay (in UI Advanced
Metrics) for a task gradually increases from 5 ms to 30 seconds in Spark
Streaming application
[ https://issues.apache.org/jira/browse/SPARK-19977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879711#comment-16879711 ]
Dominique De Vito commented on SPARK-19977:
-------------------------------------------
Same pb, but with Spark 2.0.2 (from DataStax Enterprise Edition 5.1.x).
My streaming app reads from Kafka.
During a period of time without ANY received data (0 msg read from any partition), the scheduling delay is very close to 0 ms. Then, while there is still no msg read, the scheduling delay increases up to 4 sec, then 7 sec, then 12 sec....
So far, I have not crafted a way to reproduce the bug...
[~rayqiu] : have you met again the bug ? or has it disappeared while upgrading ? Thanks.
> Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to 30 seconds in Spark Streaming application
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-19977
> URL: https://issues.apache.org/jira/browse/SPARK-19977
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 2.1.0
> Reporter: Ray Qiu
> Priority: Major
>
> Scheduler Delay (in UI Advanced Metrics) for a task gradually increases from 5 ms to 30+ seconds in a Spark Streaming application, where multiple Kafka direct streams are processed. These kafka streams are processed separately (not combined via union).
> It causes the task processing time to increase greatly and eventually stops working.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org