Maximum jobs finish very soon, some of them take longer time.

[Sarthak Dash <da...@gmail.com>]

Hello everyone,

I am trying out Spark for the first time, and after a month of work - I am
stuck with an issue. I have a very simple program that, given a directed
Graph with nodes/edges parameters and a particular node, tries to figure out
all the siblings(in the traditional sense) of the given node.

Right now, I have 1200 partitions, and I see that while most of the tasks(on
an average 1190-1195) tasks finish within 500 ms, a few tasks (5-10 of them)
take about 1-2 seconds to finish. I am aiming for a scenario wherein all the
tasks finish under a second, and hence trying to figure out why a few
tasks(5-10 of them) take longer time to complete as opposed to the remaining
(1190-1195) tasks ?

Also, Please let me know whether its possible to change some settings, so as
to achieve my target scenario ?
Any help would be much appreciated.

My configurations:
1. Tried with both FAIR/FIFO scheduler.
2. Tried playing around with Spark.locality.wait settings. Currently I have
a max scheduler delay of 300 ms.
3. Version: Apache Spark 1.0.0 on a 50 node cluster, 14GB each RAM, 8
cores/node.

Thanks,
Sarthak



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Maximum-jobs-finish-very-soon-some-of-them-take-longer-time-tp10750.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.