You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Renyi Xiong <re...@gmail.com> on 2016/04/05 20:02:05 UTC

Spark Streaming UI reporting a different task duration

Hi TD,

We noticed that Spark Streaming UI is reporting a different task duration
from time to time.

e.g. here's the standard output of the application which reports the
duration of the longest task is about 3.3 minutes:

16/04/01 16:07:19 INFO TaskSetManager: Finished task 1077.0 in stage 0.0
(TID 1077) in 125425 ms on CH1SCH080051460 (1562/1563)

16/04/01 16:08:30 INFO TaskSetManager: Finished task 926.0 in stage 0.0
(TID 926) in 196776 ms on CH1SCH080100841 (1563/1563)


but on spark streaming UI it's about 2.3 minutes




*Summary Metrics for 1563 Completed Tasks*


*Metric*

*Min*

*25th percentile*

*Median*

*75th percentile*

*Max*

Duration

12 s

21 s

24 s

29 s

2.3 min

GC Time

0 ms

0 ms

0 ms

0 ms

62 ms

Shuffle Write Size / Records

0.0 B / 0

532.9 KB / 131

1198.9 KB / 299

3.8 MB / 1161

9.0 MB / 2865

I wonder if you have any quick idea about where the missing 1 minute could
be?


thanks a lot,

Renyi.