Large variation in spark in Task Deserialization Time

[Pulasthi Supun Wickramasinghe <pu...@gmail.com>]

Hi All,

I am seeing a huge variation on spark Task Deserialization Time for my
collect and reduce operations. while most tasks complete within 100ms a few
take mote than a couple of seconds which slows the entire program down. I
have attached a screen shot of the web ui where you can see the variation


As you can see the Task Deserialization Time time has a Max of 7s and 75th
percentile at 0.3 seconds.

Does anyone know the reasons that may cause these kind of numbers. Any help
would be greatly appreciated.

Best Regards,
​
-- 
Pulasthi S. Wickramasinghe
Graduate Student  | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
cell: 224-386-9035

Fwd: Large variation in spark in Task Deserialization Time

[Pulasthi Supun Wickramasinghe <pu...@gmail.com>]

Hi Devs/All,

I am seeing a huge variation on spark Task Deserialization Time for my
collect and reduce operations. while most tasks complete within 100ms a few
take mote than a couple of seconds which slows the entire program down. I
have attached a screen shot of the web UI where you can see the variation


As you can see the Task Deserialization Time time has a Max of 7s and 75th
percentile at 0.3 seconds.

Does anyone know the reasons that may cause these kind of numbers. Any help
would be greatly appreciated.

Best Regards,
Pulasthi
-- 
Pulasthi S. Wickramasinghe
Graduate Student  | Research Assistant
School of Informatics and Computing | Digital Science Center
Indiana University, Bloomington
cell: 224-386-9035