[Spark Streaming] Unknown delay in event timeline

[Zhiwen Sun <pe...@gmail.com>]

Env : Spark 1.6.2 + kafka 0.8.2 DirectStream.

Spark streaming job with 1s interval.
Process time of micro batch suddenly became to 4s while is is usually 0.4s .
When we check where the time spent, we find a unknown delay in job.
There is no executor computing or shuffle reading. It is about 4s blank in
event timeline,
Event timeline snapshot is in attachment.
It seems that executors have different launch time.

Any one can give some suggestion?

Thanks!




Zhiwen Sun