You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Manoj Babu <ma...@gmail.com> on 2012/12/10 14:09:03 UTC
Reg: Map output copy failure
Hi All
I got the below exception, Is the issue related to
https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?
Am using CDH3U1
2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task:
attempt_201211120903_9197_r_000024_0 : Map output copy failure :
java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1593)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1453)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1234)
Cheers!
Manoj.
Re: Reg: Map output copy failure
Posted by Bharath Mundlapudi <bh...@yahoo.com>.
What was the job or query you were running?
Couple of suggestions:
1. Reduce data set size with job chaining
2. Increase Reduce task heap
3. If you are using Hive/Pig, you may want to tune your query.
-Bharath
________________________________
From: Manoj Babu <ma...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, December 10, 2012 5:09 AM
Subject: Reg: Map output copy failure
Hi All
I got the below exception, Is the issue related to https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?
Am using CDH3U1
2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task: attempt_201211120903_9197_r_000024_0 : Map output copy failure : java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1593) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1453) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1234)Cheers!
Manoj.
Re: Reg: Map output copy failure
Posted by Manoj Babu <ma...@gmail.com>.
Please ignore the above comment.
Cheers!
Manoj.
On Mon, Dec 10, 2012 at 6:59 PM, Manoj Babu <ma...@gmail.com> wrote:
> In the ReduceTask.java having the below code
>
> maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
> (int)Math.min(Runtime.getRuntime().maxMemory(),
> Integer.MAX_VALUE))
> * maxInMemCopyUse);
>
> But in the patch
> maxSize = (long)Math.min(
> Runtime.getRuntime().maxMemory() * maxInMemCopyUse,
> Integer.MAX_VALUE);
>
> Whether cdh3u1 not holding this fix?
> Kindly Advice.
>
> Cheers!
> Manoj.
>
>
>
> On Mon, Dec 10, 2012 at 6:39 PM, Manoj Babu <ma...@gmail.com> wrote:
>
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
>>
>>
>
Re: Reg: Map output copy failure
Posted by Manoj Babu <ma...@gmail.com>.
Please ignore the above comment.
Cheers!
Manoj.
On Mon, Dec 10, 2012 at 6:59 PM, Manoj Babu <ma...@gmail.com> wrote:
> In the ReduceTask.java having the below code
>
> maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
> (int)Math.min(Runtime.getRuntime().maxMemory(),
> Integer.MAX_VALUE))
> * maxInMemCopyUse);
>
> But in the patch
> maxSize = (long)Math.min(
> Runtime.getRuntime().maxMemory() * maxInMemCopyUse,
> Integer.MAX_VALUE);
>
> Whether cdh3u1 not holding this fix?
> Kindly Advice.
>
> Cheers!
> Manoj.
>
>
>
> On Mon, Dec 10, 2012 at 6:39 PM, Manoj Babu <ma...@gmail.com> wrote:
>
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
>>
>>
>
Re: Reg: Map output copy failure
Posted by Manoj Babu <ma...@gmail.com>.
Please ignore the above comment.
Cheers!
Manoj.
On Mon, Dec 10, 2012 at 6:59 PM, Manoj Babu <ma...@gmail.com> wrote:
> In the ReduceTask.java having the below code
>
> maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
> (int)Math.min(Runtime.getRuntime().maxMemory(),
> Integer.MAX_VALUE))
> * maxInMemCopyUse);
>
> But in the patch
> maxSize = (long)Math.min(
> Runtime.getRuntime().maxMemory() * maxInMemCopyUse,
> Integer.MAX_VALUE);
>
> Whether cdh3u1 not holding this fix?
> Kindly Advice.
>
> Cheers!
> Manoj.
>
>
>
> On Mon, Dec 10, 2012 at 6:39 PM, Manoj Babu <ma...@gmail.com> wrote:
>
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
>>
>>
>
Re: Reg: Map output copy failure
Posted by Manoj Babu <ma...@gmail.com>.
Please ignore the above comment.
Cheers!
Manoj.
On Mon, Dec 10, 2012 at 6:59 PM, Manoj Babu <ma...@gmail.com> wrote:
> In the ReduceTask.java having the below code
>
> maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
> (int)Math.min(Runtime.getRuntime().maxMemory(),
> Integer.MAX_VALUE))
> * maxInMemCopyUse);
>
> But in the patch
> maxSize = (long)Math.min(
> Runtime.getRuntime().maxMemory() * maxInMemCopyUse,
> Integer.MAX_VALUE);
>
> Whether cdh3u1 not holding this fix?
> Kindly Advice.
>
> Cheers!
> Manoj.
>
>
>
> On Mon, Dec 10, 2012 at 6:39 PM, Manoj Babu <ma...@gmail.com> wrote:
>
>> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
>>
>>
>
Re: Reg: Map output copy failure
Posted by Manoj Babu <ma...@gmail.com>.
In the ReduceTask.java having the below code
maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
(int)Math.min(Runtime.getRuntime().maxMemory(),
Integer.MAX_VALUE))
* maxInMemCopyUse);
But in the patch
maxSize = (long)Math.min(
Runtime.getRuntime().maxMemory() * maxInMemCopyUse,
Integer.MAX_VALUE);
Whether cdh3u1 not holding this fix?
Kindly Advice.
Cheers!
Manoj.
On Mon, Dec 10, 2012 at 6:39 PM, Manoj Babu <ma...@gmail.com> wrote:
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
>
>
RE: Map output copy failure
Posted by David Parks <da...@yahoo.com>.
I had the same problem yesterday, it sure does look to be dead on that
issue. I found another forum discussion on AWS that suggested more memory as
a stop-gap way to deal with it, or apply the patch. I checked the code on
hadoop 1.0.3 (the version on AWS) and it didn't have the fix, so it looks
like it's only in the newer builds. I actually have an AWS ticket opened for
it seeing if their engineers can offer any guidance as well.
My understanding is that it should be doing a shuffle on disk in this case,
it appeared to be just a small fix (a few lines) to apply the patch to
src/mapred/org/apache/hadoop/mapred/ReduceTask.java
Dave
From: Manoj Babu [mailto:manoj444@gmail.com]
Sent: Monday, December 10, 2012 8:09 PM
To: user@hadoop.apache.org
Subject: Reg: Map output copy failure
Hi All
I got the below exception, Is the issue related to
https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?
Am using CDH3U1
2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task:
attempt_201211120903_9197_r_000024_0 : Map output copy failure :
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMe
mory(ReduceTask.java:1593)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1453)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1302)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1234)
Cheers!
Manoj.
Re: Reg: Map output copy failure
Posted by Manoj Babu <ma...@gmail.com>.
In the ReduceTask.java having the below code
maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
(int)Math.min(Runtime.getRuntime().maxMemory(),
Integer.MAX_VALUE))
* maxInMemCopyUse);
But in the patch
maxSize = (long)Math.min(
Runtime.getRuntime().maxMemory() * maxInMemCopyUse,
Integer.MAX_VALUE);
Whether cdh3u1 not holding this fix?
Kindly Advice.
Cheers!
Manoj.
On Mon, Dec 10, 2012 at 6:39 PM, Manoj Babu <ma...@gmail.com> wrote:
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
>
>
Re: Reg: Map output copy failure
Posted by Bharath Mundlapudi <bh...@yahoo.com>.
What was the job or query you were running?
Couple of suggestions:
1. Reduce data set size with job chaining
2. Increase Reduce task heap
3. If you are using Hive/Pig, you may want to tune your query.
-Bharath
________________________________
From: Manoj Babu <ma...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, December 10, 2012 5:09 AM
Subject: Reg: Map output copy failure
Hi All
I got the below exception, Is the issue related to https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?
Am using CDH3U1
2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task: attempt_201211120903_9197_r_000024_0 : Map output copy failure : java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1593) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1453) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1234)Cheers!
Manoj.
Re: Reg: Map output copy failure
Posted by Manoj Babu <ma...@gmail.com>.
In the ReduceTask.java having the below code
maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
(int)Math.min(Runtime.getRuntime().maxMemory(),
Integer.MAX_VALUE))
* maxInMemCopyUse);
But in the patch
maxSize = (long)Math.min(
Runtime.getRuntime().maxMemory() * maxInMemCopyUse,
Integer.MAX_VALUE);
Whether cdh3u1 not holding this fix?
Kindly Advice.
Cheers!
Manoj.
On Mon, Dec 10, 2012 at 6:39 PM, Manoj Babu <ma...@gmail.com> wrote:
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
>
>
Re: Reg: Map output copy failure
Posted by Bharath Mundlapudi <bh...@yahoo.com>.
What was the job or query you were running?
Couple of suggestions:
1. Reduce data set size with job chaining
2. Increase Reduce task heap
3. If you are using Hive/Pig, you may want to tune your query.
-Bharath
________________________________
From: Manoj Babu <ma...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, December 10, 2012 5:09 AM
Subject: Reg: Map output copy failure
Hi All
I got the below exception, Is the issue related to https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?
Am using CDH3U1
2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task: attempt_201211120903_9197_r_000024_0 : Map output copy failure : java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1593) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1453) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1234)Cheers!
Manoj.
RE: Map output copy failure
Posted by David Parks <da...@yahoo.com>.
I had the same problem yesterday, it sure does look to be dead on that
issue. I found another forum discussion on AWS that suggested more memory as
a stop-gap way to deal with it, or apply the patch. I checked the code on
hadoop 1.0.3 (the version on AWS) and it didn't have the fix, so it looks
like it's only in the newer builds. I actually have an AWS ticket opened for
it seeing if their engineers can offer any guidance as well.
My understanding is that it should be doing a shuffle on disk in this case,
it appeared to be just a small fix (a few lines) to apply the patch to
src/mapred/org/apache/hadoop/mapred/ReduceTask.java
Dave
From: Manoj Babu [mailto:manoj444@gmail.com]
Sent: Monday, December 10, 2012 8:09 PM
To: user@hadoop.apache.org
Subject: Reg: Map output copy failure
Hi All
I got the below exception, Is the issue related to
https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?
Am using CDH3U1
2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task:
attempt_201211120903_9197_r_000024_0 : Map output copy failure :
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMe
mory(ReduceTask.java:1593)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1453)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1302)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1234)
Cheers!
Manoj.
Re: Reg: Map output copy failure
Posted by Manoj Babu <ma...@gmail.com>.
In the ReduceTask.java having the below code
maxSize = (int)(conf.getInt("mapred.job.reduce.total.mem.bytes",
(int)Math.min(Runtime.getRuntime().maxMemory(),
Integer.MAX_VALUE))
* maxInMemCopyUse);
But in the patch
maxSize = (long)Math.min(
Runtime.getRuntime().maxMemory() * maxInMemCopyUse,
Integer.MAX_VALUE);
Whether cdh3u1 not holding this fix?
Kindly Advice.
Cheers!
Manoj.
On Mon, Dec 10, 2012 at 6:39 PM, Manoj Babu <ma...@gmail.com> wrote:
> at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302)
>
>
Re: Reg: Map output copy failure
Posted by Bharath Mundlapudi <bh...@yahoo.com>.
What was the job or query you were running?
Couple of suggestions:
1. Reduce data set size with job chaining
2. Increase Reduce task heap
3. If you are using Hive/Pig, you may want to tune your query.
-Bharath
________________________________
From: Manoj Babu <ma...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, December 10, 2012 5:09 AM
Subject: Reg: Map output copy failure
Hi All
I got the below exception, Is the issue related to https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?
Am using CDH3U1
2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task: attempt_201211120903_9197_r_000024_0 : Map output copy failure : java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1593) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1453) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1302) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1234)Cheers!
Manoj.
RE: Map output copy failure
Posted by David Parks <da...@yahoo.com>.
I had the same problem yesterday, it sure does look to be dead on that
issue. I found another forum discussion on AWS that suggested more memory as
a stop-gap way to deal with it, or apply the patch. I checked the code on
hadoop 1.0.3 (the version on AWS) and it didn't have the fix, so it looks
like it's only in the newer builds. I actually have an AWS ticket opened for
it seeing if their engineers can offer any guidance as well.
My understanding is that it should be doing a shuffle on disk in this case,
it appeared to be just a small fix (a few lines) to apply the patch to
src/mapred/org/apache/hadoop/mapred/ReduceTask.java
Dave
From: Manoj Babu [mailto:manoj444@gmail.com]
Sent: Monday, December 10, 2012 8:09 PM
To: user@hadoop.apache.org
Subject: Reg: Map output copy failure
Hi All
I got the below exception, Is the issue related to
https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?
Am using CDH3U1
2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task:
attempt_201211120903_9197_r_000024_0 : Map output copy failure :
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMe
mory(ReduceTask.java:1593)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1453)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1302)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1234)
Cheers!
Manoj.
RE: Map output copy failure
Posted by David Parks <da...@yahoo.com>.
I had the same problem yesterday, it sure does look to be dead on that
issue. I found another forum discussion on AWS that suggested more memory as
a stop-gap way to deal with it, or apply the patch. I checked the code on
hadoop 1.0.3 (the version on AWS) and it didn't have the fix, so it looks
like it's only in the newer builds. I actually have an AWS ticket opened for
it seeing if their engineers can offer any guidance as well.
My understanding is that it should be doing a shuffle on disk in this case,
it appeared to be just a small fix (a few lines) to apply the patch to
src/mapred/org/apache/hadoop/mapred/ReduceTask.java
Dave
From: Manoj Babu [mailto:manoj444@gmail.com]
Sent: Monday, December 10, 2012 8:09 PM
To: user@hadoop.apache.org
Subject: Reg: Map output copy failure
Hi All
I got the below exception, Is the issue related to
https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?
Am using CDH3U1
2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task:
attempt_201211120903_9197_r_000024_0 : Map output copy failure :
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMe
mory(ReduceTask.java:1593)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1453)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1302)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1234)
Cheers!
Manoj.