You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Rares Vernica <rv...@gmail.com> on 2009/08/27 21:51:58 UTC
understand merge phase performance
Hello,
I need some help in understanding the performance of the merge phase.
I am surprised how a merge phase for a task processing a smaller
amount of data takes more that the merge phase for a task processing a
larger amount of data.
I have two jobs. Both are executed on the same 1 Node cluster (4 Map
and 4 Reduce in parallel). Job 1 is doing word count while Job 2 is
building an inverted list index (from word to records). They both take
the same input data. As you can imagine, the amount of data transfered
between the Job 1 Map and the Job 1 Reduce is smaller that the amount
of data transfered between Job 2 Map and Job 2 Reduce. Still, the
merge phase of Job 1 takes 2 seconds while the merge phase of Job 2
takes less that 1 second (it is reported as taking 0 seconds). Another
difference is the fact that the keys for Job 1 are Text, while the
keys for Job 2 are IntWritable (words are converted to integers).
Here are some relevant statistics.
Job 1
---
Reduce input groups 450,693
Map output bytes 3,152,970,981
Map input bytes 3,169,519,473
Job 2
---
Reduce input groups 450,633
Map output bytes 8,349,926,960
Map input bytes 3,169,519,473
Here are some excerpts from the logs which might be relevant:
Job 1
---
2009-08-27 12:10:04,171 INFO org.apache.hadoop.mapred.ReduceTask:
Interleaved on-disk merge complete: 0 files left.
2009-08-27 12:10:04,172 INFO org.apache.hadoop.mapred.ReduceTask:
In-memory merge complete: 4 files left.
2009-08-27 12:10:04,197 INFO org.apache.hadoop.mapred.Merger: Merging
4 sorted segments
2009-08-27 12:10:04,197 INFO org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 4 segments left of total size: 13139424
bytes
2009-08-27 12:10:04,203 INFO org.apache.hadoop.io.compress.CodecPool:
Got brand-new compressor
2009-08-27 12:10:06,695 INFO org.apache.hadoop.mapred.ReduceTask:
Merged 4 segments, 13139424 bytes to disk to satisfy reduce memory
limit
2009-08-27 12:10:06,696 INFO org.apache.hadoop.mapred.ReduceTask:
Merging 1 files, 2952848 bytes from disk
2009-08-27 12:10:06,696 INFO org.apache.hadoop.mapred.ReduceTask:
Merging 0 segments, 0 bytes from memory into reduce
2009-08-27 12:10:06,696 INFO org.apache.hadoop.mapred.Merger: Merging
1 sorted segments
2009-08-27 12:10:06,699 INFO org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor
2009-08-27 12:10:06,700 INFO org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 1 segments left of total size: 2952844 bytes
Job 2
---
2009-08-27 11:01:42,851 INFO org.apache.hadoop.mapred.ReduceTask:
Interleaved on-disk merge complete: 4 files left.
2009-08-27 11:01:42,851 INFO org.apache.hadoop.mapred.ReduceTask:
In-memory merge complete: 0 files left.
2009-08-27 11:01:42,862 INFO org.apache.hadoop.mapred.ReduceTask:
Merging 4 files, 508528572 bytes from disk
2009-08-27 11:01:42,862 INFO org.apache.hadoop.mapred.ReduceTask:
Merging 0 segments, 0 bytes from memory into reduce
2009-08-27 11:01:42,866 INFO org.apache.hadoop.mapred.Merger: Merging
4 sorted segments
2009-08-27 11:01:42,873 INFO org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor
2009-08-27 11:01:42,874 INFO org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor
2009-08-27 11:01:42,876 INFO org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor
2009-08-27 11:01:42,876 INFO org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor
2009-08-27 11:01:42,877 INFO org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 4 segments left of total size: 508528556
bytes
>From the performance, I feel that the Job 1 sort is done on disk while
the Job 2 sort is done in memory. This is strange because Job 1 Map
output is smaller than Job 2 Map output.
Thanks a lot!
Rares Vernica
Re: understand merge phase performance
Posted by Rares Vernica <rv...@gmail.com>.
On Thu, Aug 27, 2009 at 12:51 PM, Rares Vernica<rv...@gmail.com> wrote:
>
> Another difference is the fact that the keys for Job 1 are Text, while the
> keys for Job 2 are IntWritable (words are converted to integers).
I changed the key type of Job 2 from IntWritable to Text and the merge
phase performance is the same. The Map output bytes increased
slightly.
It seems that merging data from disk only is faster than merging data
from memory and disk...
Cheers,
Rares Vernica
Re: understand merge phase performance
Posted by Rares Vernica <rv...@gmail.com>.
On Fri, Aug 28, 2009 at 10:13 AM, Rares Vernica<rv...@gmail.com> wrote:
>
> On Fri, Aug 28, 2009 at 4:46 AM, Jothi Padmanabhan<jo...@yahoo-inc.com> wrote:
>>
>> Could you attach the complete reducer logs for the two runs?
>
> Attached.
Forgot the attachments... Here they are.
Cheers!
Rares Vernica
Re: understand merge phase performance
Posted by Rares Vernica <rv...@gmail.com>.
On Fri, Aug 28, 2009 at 4:46 AM, Jothi Padmanabhan<jo...@yahoo-inc.com> wrote:
>
> Could you attach the complete reducer logs for the two runs?
Attached.
> How many maps did the Jobs have?
Both jobs had 4 maps.
> And what were there map output sizes?
The sizes in bytes are in the first email. Here are the number of records.
Job 1: Map output records 181,755,130
Combine output records 31,641,028
Job 2: Map output records 43,538,720
(does not have a combiner)
Thanks!
Rares Vernica