You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by on 2015/07/07 00:36:30 UTC

How does executor cores change the spark job behavior ?

I have a simple job , that reads data => union => filter => map and the
count

1 Job started 2402 tasks read 149G of input.

I started the job with different number of executors

1) 1 -->  8.3 mins
2) 2 --> 5.6 mins
3) 3 --> 3.1 mins

1) Why is increasing the cores speading up this app ?
2) I started the job with --num-executors 9973 but when i click executors
tab i see 330 executors. So can i start the job with --num-executors 330 as
i get only that from YARN cluster ?
3) I had set the split size to 64 MB but when i start the job
with --executor-memory 14g , how do i decide how much memory i need ? also
as the cores are increasing how do i get that into the calculations ?
4) as the speed is getting better how far can i go with increasing
executors ?

-- 
Deepak