Spark application Runtime Measurement

[Fei Hu <hu...@gmail.com>]

Dear all,

I have a question about how to measure the runtime for a Spak application.
Here is an example:


   - On the Spark UI: the total duration time is 2.0 minutes = 120 seconds
   as following

[image: Screen Shot 2016-07-09 at 11.45.44 PM.png]

   - However, when I check the jobs launched by the application, the time
   is 13s + 0.8s + 4s = 17.8 seconds, which is much less than 120 seconds. I
   am not sure which time I should choose to measure the performance of the
   Spark application.

[image: Screen Shot 2016-07-09 at 11.48.26 PM.png]

   - I also check the event timeline as following. There is a big gap
   between the second job and the third job. I do not know what happened
   during that gap.

[image: Screen Shot 2016-07-09 at 11.53.29 PM.png]

Is there anyone who can help explain which time is the exact time to
measure the performance of a Spark application.

Thanks in advance,
Fei