[Spark Web Interface] What is Task Deserialization Time?

[Никита Романов <ni...@mail.ru.INVALID>]

Hi!
 
I’m puzzled by my Spark application’s behaviour.  Task Deserialization Time dominates Executor Computing Time and pretty much everything else:
 
You can see here that the network load is very small. The above is one Stage of my application, I do a couple dozens of such and even the numbers aggregated over the course of my application’s runtime are quite small:
 
Here’s the DAG of one Stage, you can see all I do is  mapPartitions a cached RDD and then  treeReduce the result:
 
What could be the reason for such a long Task Deserialization Time? How can the executor take longer to obtain its task than to compute it? Or is my understanding somehow flawed and this isn’t what’s happening during Task Deserialization? I submit my application’s jar in spark-submit, and all the dependencies jars are present in SPARK_HOME on all the cluster nodes, and they’re all quite standard (spark-core, spark-mllib, breeze).
 
I’m using Kryo serializer, and all the classes my RDDs consist of are registered with it. Although, if my understanding of  https://stackoverflow.com/questions/40259196/understanding-sparks-closures-and-their-serialization/40261550#40261550  is correct, the tasks for the workers/closures are serialized using Java serializer anyway. So perhaps this information is not relevant for the question.
 
Hope the experts here will be able to shed some light on this mystery :)
 
Best regards,
Nick