You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Donovan Hide <do...@gmail.com> on 2010/10/19 14:02:30 UTC

io.sort.mb maximum limit

Hi,
is there a reason why the io.sort.mb setting is hard-coded to the
maximum of 2047MB?

MapTask.java 789-791

      if ((sortmb & 0x7FF) != sortmb) {
        throw new IOException("Invalid \"io.sort.mb\": " + sortmb);
      }

Given that the EC2 High-Memory Quadruple Extra Large Instance has
68.4GB of memory and 8 cores, it would make sense to be able to set
the io.sort.mb to close to 8GB. I have map task that outputs
144,586,867 records of average size 12 bytes, and a greater than
2047MB sort buffer would allow me to prevent the inevitable spills. I
know I can reduce the size of the map inputs to solve the problem, but
2047MB seems a bit arbitrary given the spec of EC2 instances.

Cheers,
Donovan.

Re: io.sort.mb maximum limit

Posted by Donovan Hide <do...@gmail.com>.
Hi Ed,

thanks for the reply. The art of evaluating performance is a black one indeed:)

I'm basing my interest on the advice from Todd Lipcon, referenced here:

http://www.mentby.com/todd-lipcon/iosortmb-configuration.html

Specifically:

     "you're unlikely to see a big difference in performance unless
you cut down the number of spills from >1 to 1"

So, given an io.sort.mb of 2047MB and a io.sort.record.percent of 0.57 I get:

2010-10-19 13:59:45,596 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2010-10-19 13:59:45,900 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2010-10-19 13:59:46,161 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 2047
2010-10-19 13:59:48,134 INFO org.apache.hadoop.mapred.MapTask: data
buffer = 738373632/922967040
2010-10-19 13:59:48,134 INFO org.apache.hadoop.mapred.MapTask: record
buffer = 61173404/76466752
2010-10-19 13:59:48,160 INFO
org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2010-10-19 14:03:18,803 INFO org.apache.hadoop.mapred.MapTask:
Spilling map output: record full = true
2010-10-19 14:03:18,803 INFO org.apache.hadoop.mapred.MapTask:
bufstart = 0; bufend = 734080848; bufvoid = 922967040
2010-10-19 14:03:18,803 INFO org.apache.hadoop.mapred.MapTask: kvstart
= 0; kvend = 61173404; length = 76466752
2010-10-19 14:07:00,799 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0
2010-10-19 14:09:35,992 INFO org.apache.hadoop.mapred.MapTask:
Spilling map output: record full = true
2010-10-19 14:09:35,992 INFO org.apache.hadoop.mapred.MapTask:
bufstart = 734080848; bufend = 545194644; bufvoid = 922967040
2010-10-19 14:09:35,992 INFO org.apache.hadoop.mapred.MapTask: kvstart
= 61173404; kvend = 45880055; length = 76466752
2010-10-19 14:14:29,105 INFO org.apache.hadoop.mapred.MapTask: Finished spill 1
2010-10-19 14:14:59,017 INFO org.apache.hadoop.mapred.MapTask:
Starting flush of map output
2010-10-19 14:15:54,328 INFO org.apache.hadoop.mapred.MapTask: Finished spill 2
2010-10-19 14:15:54,365 INFO org.apache.hadoop.mapred.Merger: Merging
3 sorted segments
2010-10-19 14:15:54,387 INFO org.apache.hadoop.mapred.Merger: Down to
the last merge-pass, with 3 segments left of total size: 2024217502
bytes
2010-10-19 14:18:03,284 INFO org.apache.hadoop.mapred.Task:
Task:attempt_201010180321_0028_m_000000_0 is done. And is in the
process of commiting
2010-10-19 14:18:03,363 INFO org.apache.hadoop.mapred.Task: Task
'attempt_201010180321_0028_m_000000_0' done.

The two minutes 9 seconds spent doing the merge could presumably be
saved if there was no spill? The trouble is I can't test this given
the 2047MB cap. I'm not set up for building from source yet ...

Cheers,
Donovan.




On 19 October 2010 14:44, ed <ha...@gmail.com> wrote:
> HI Donovan,
>
> This is sort of tangential to your question but we tried upping our
> io.sort.mb to a really high value and it actually resulted in slower
> performance (I think we bumped it up to 1400MB and this was slower than
> leaving it at 256 on a machine with 32GB of RAM).  I'm not entirely sure why
> this was the case.  It could have been a garbage collection issue or some
> other secondary effect that was slowing things down.
>
> Keep in mind that Hadoop will always spill map outputs to disk no matter how
> large your sort buffer is in case the reducer crashes, the data needs to
> exist on disk somewhere for the next reducer making the attempt so it might
> be counterproductive to try and "eliminate" spills.
>
> ~Ed
>
>
> On Tue, Oct 19, 2010 at 8:02 AM, Donovan Hide <do...@gmail.com> wrote:
>
>> Hi,
>> is there a reason why the io.sort.mb setting is hard-coded to the
>> maximum of 2047MB?
>>
>> MapTask.java 789-791
>>
>>       if ((sortmb & 0x7FF) != sortmb) {
>>         throw new IOException("Invalid \"io.sort.mb\": " + sortmb);
>>       }
>>
>> Given that the EC2 High-Memory Quadruple Extra Large Instance has
>> 68.4GB of memory and 8 cores, it would make sense to be able to set
>> the io.sort.mb to close to 8GB. I have map task that outputs
>> 144,586,867 records of average size 12 bytes, and a greater than
>> 2047MB sort buffer would allow me to prevent the inevitable spills. I
>> know I can reduce the size of the map inputs to solve the problem, but
>> 2047MB seems a bit arbitrary given the spec of EC2 instances.
>>
>> Cheers,
>> Donovan.
>>
>

Re: io.sort.mb maximum limit

Posted by ed <ha...@gmail.com>.
HI Donovan,

This is sort of tangential to your question but we tried upping our
io.sort.mb to a really high value and it actually resulted in slower
performance (I think we bumped it up to 1400MB and this was slower than
leaving it at 256 on a machine with 32GB of RAM).  I'm not entirely sure why
this was the case.  It could have been a garbage collection issue or some
other secondary effect that was slowing things down.

Keep in mind that Hadoop will always spill map outputs to disk no matter how
large your sort buffer is in case the reducer crashes, the data needs to
exist on disk somewhere for the next reducer making the attempt so it might
be counterproductive to try and "eliminate" spills.

~Ed


On Tue, Oct 19, 2010 at 8:02 AM, Donovan Hide <do...@gmail.com> wrote:

> Hi,
> is there a reason why the io.sort.mb setting is hard-coded to the
> maximum of 2047MB?
>
> MapTask.java 789-791
>
>       if ((sortmb & 0x7FF) != sortmb) {
>         throw new IOException("Invalid \"io.sort.mb\": " + sortmb);
>       }
>
> Given that the EC2 High-Memory Quadruple Extra Large Instance has
> 68.4GB of memory and 8 cores, it would make sense to be able to set
> the io.sort.mb to close to 8GB. I have map task that outputs
> 144,586,867 records of average size 12 bytes, and a greater than
> 2047MB sort buffer would allow me to prevent the inevitable spills. I
> know I can reduce the size of the map inputs to solve the problem, but
> 2047MB seems a bit arbitrary given the spec of EC2 instances.
>
> Cheers,
> Donovan.
>