[jira] [Resolved] (SPARK-4537) Add 'processing delay' and 'totalDelay' to the metrics reported by the Spark Streaming subsystem

["Tathagata Das (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tathagata Das resolved SPARK-4537.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 1.2.1
                   1.3.0

> Add 'processing delay' and 'totalDelay' to the metrics reported by the Spark Streaming subsystem
> ------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-4537
>                 URL: https://issues.apache.org/jira/browse/SPARK-4537
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 1.1.0
>            Reporter: Gerard Maas
>              Labels: metrics
>             Fix For: 1.3.0, 1.2.1
>
>
> As the Spark Streaming tuning guide indicates, the key indicators of a healthy streaming job are:
> - Processing Time
> - Total Delay
> The Spark UI page for the Streaming job [1] shows these two indicators but the metrics source for Spark Streaming (StreamingSource.scala)  [2] does not.
> Adding these metrics will allow external monitoring systems that consume the Spark metrics interface to track these two critical pieces of information on a streaming job performance.
> [1] https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala#L127
> [2] https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/StreamingSource.scala



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org