Why the length of each task varies

[Gavin Liu <il...@gmail.com>]

I am implementing wordcount on the spark cluster (1 master, 3 slaves) in
standalone mode. I have 546G data, and the dfs.blocksize I set is 256MB.
Therefore, the amount of tasks are 2186. My 3 slaves each uses 22 cores and
72 memory to do the processing, so the computing ability of each slave
should be same. 

Since wordcount just has two parts, map and reduce, therefore, I think in
each stage, each task takes care of one partition, so the length of each
task should be nearly same right?

However, from the event timeline I saw in job UI, I found that the length of
each task in mapToPair stage varies much and there were many small tasks. I
don't know if it is normal or it is my own problem ? 

Here is the pic of event timeline,
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n24008/QQ%E6%88%AA%E5%9B%BE20150727172511.png> 

And the amount of the tasks assigned to each slave are also different,
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n24008/QQ%E6%88%AA%E5%9B%BE20150727172739.png> 

Anybody has any idea with this? Thanks in advance.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Why-the-length-of-each-task-varies-tp24008.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Why the length of each task varies

[Gylfi <gy...@berkeley.edu>]

Hi. 

Have you ruled out that this may just be I/O time?
Word count is a very "light-wight" task for the CPU but you will be needing
to read the initial data from what ever storage device you have your HDFS
running on. 
As you have 3 machines, 22 cores each but perhaps just one or a few HDD /
SSD / NAS the 22 cores may be saturating your I/O capacity and thus I/O
determines the running time or your task?
If it is some form of NAS storage you may be saturating the network
capacity. 

If this is the case, that would explain fluctuations in the observed running
times. A given Map-task may have been lucky, and the data was read when the
I/O was not busy, or unlucky, many machine cores (map-tasks) starting a new
block at about the same time. 

Also, 22*256MB = 5632 MB: This is the RAM you need to cache a block of data
for each map-task running in parallel on the same machine. 
Depending on how much RAM you have per node, you may want to re-block the
data on HDFS for optimal performance.

Hope this helps, 
   Gylfi. 



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Why-the-length-of-each-task-varies-tp24008p24014.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org