You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ved Prakash <me...@gmail.com> on 2008/03/10 06:10:01 UTC
MapReduce failure
Hi friends,
I have made a cluster of 3 machines, one of them is master, and other 2
slaves. I executed a mapreduce job on master but after Map, the execution
terminates and Reduce doesn't happen. I have checked dfs and no output
folder gets created.
this is the error I see
08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
task_200803101001_0001_m_000064_0, Status : FAILED
java.lang.OutOfMemoryError: Java heap space
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
:95)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.Text.write(Text.java:243)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
MapTask.java:347)
at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
:72)
at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
:59)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
:1787)
08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce 17%
08/03/10 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
08/03/10 10:35:51 INFO mapred.JobClient: map 57% reduce 17%
08/03/10 10:36:04 INFO mapred.JobClient: map 58% reduce 17%
08/03/10 10:36:07 INFO mapred.JobClient: map 57% reduce 17%
08/03/10 10:36:07 INFO mapred.JobClient: Task Id :
task_200803101001_0001_m_000071_0, Status : FAILED
java.lang.OutOfMemoryError: Java heap space
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
:95)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.Text.write(Text.java:243)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
MapTask.java:347)
at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
:72)
at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
:59)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
:1787)
though it tries to overcome this problem but the mapreduce application
doesn't create output, can anyone tell me why is this happening?
Thanks
Re: [memory leak?] Re: MapReduce failure
Posted by Stefan Groschupf <sg...@101tec.com>.
ups sorry I forgot to mention I use 0.16.0. I will try to update to
16.1 tomorrow and see if this helps, but i couldn't find an closed
issue in jira that might be related.
On Mar 15, 2008, at 8:37 PM, Stefan Groschupf wrote:
> Hi there,
>
> we see the same situation and browsing the posts there are quite a
> lot of people running into this OOM problem.
> We run a own Mapper and our mapred.child.java.opts is -Xmx3048m, I
> think that should be more then enough.
> Also I changed io.sort.mb to 10, which had also no impact.
>
> Any ideas what might cause the OutOfMemoryError ?
> Thanks.
> Stefan
>
>
>
>
> On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote:
>
>> What is the heap size you are using for your tasks? Check
>> 'mapred.child.java.opts' in your hadoop-default.xml. Try increasing
>> it. This will happen if you try running the random-writer + sort
>> examples with default parameters. The maps are not able to spill
>> the data to the disk. Btw what version of HADOOP are you using?
>> Amar
>> On Mon, 10 Mar 2008, Ved Prakash wrote:
>>
>>> Hi friends,
>>>
>>> I have made a cluster of 3 machines, one of them is master, and
>>> other 2
>>> slaves. I executed a mapreduce job on master but after Map, the
>>> execution
>>> terminates and Reduce doesn't happen. I have checked dfs and no
>>> output
>>> folder gets created.
>>>
>>> this is the error I see
>>>
>>> 08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
>>> task_200803101001_0001_m_000064_0, Status : FAILED
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
>>> :95)
>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>> at org.apache.hadoop.io.Text.write(Text.java:243)
>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
>>> MapTask.java:347)
>>> at org.apache.hadoop.examples.WordCount
>>> $MapClass.map(WordCount.java
>>> :72)
>>> at org.apache.hadoop.examples.WordCount
>>> $MapClass.map(WordCount.java
>>> :59)
>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
>>> at org.apache.hadoop.mapred.TaskTracker
>>> $Child.main(TaskTracker.java
>>> :1787)
>>>
>>> 08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce 17%
>>> 08/03/10 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
>>> 08/03/10 10:35:51 INFO mapred.JobClient: map 57% reduce 17%
>>> 08/03/10 10:36:04 INFO mapred.JobClient: map 58% reduce 17%
>>> 08/03/10 10:36:07 INFO mapred.JobClient: map 57% reduce 17%
>>> 08/03/10 10:36:07 INFO mapred.JobClient: Task Id :
>>> task_200803101001_0001_m_000071_0, Status : FAILED
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
>>> :95)
>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>> at org.apache.hadoop.io.Text.write(Text.java:243)
>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
>>> MapTask.java:347)
>>> at org.apache.hadoop.examples.WordCount
>>> $MapClass.map(WordCount.java
>>> :72)
>>> at org.apache.hadoop.examples.WordCount
>>> $MapClass.map(WordCount.java
>>> :59)
>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
>>> at org.apache.hadoop.mapred.TaskTracker
>>> $Child.main(TaskTracker.java
>>> :1787)
>>>
>>> though it tries to overcome this problem but the mapreduce
>>> application
>>> doesn't create output, can anyone tell me why is this happening?
>>>
>>> Thanks
>>>
>>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> Menlo Park, California, USA
> http://www.101tec.com
>
>
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com
RE: [memory leak?] Re: MapReduce failure
Posted by Devaraj Das <dd...@yahoo-inc.com>.
Wordcount doesn't work? Which platform are you on? I just ran the wordcount
on a RHEL linux machine with hadoop-0.16.0 and that worked, and my -Xmx is
512m.
Could you pls share your config file (hadoop-site.xml) as well?
> -----Original Message-----
> From: Stefan Groschupf [mailto:sg@101tec.com]
> Sent: Sunday, March 16, 2008 12:46 PM
> To: core-user@hadoop.apache.org
> Subject: Re: [memory leak?] Re: MapReduce failure
>
> I do not instantiate 3 GB of objects, that is for sure. The
> wordcount example does not run anymore so I dont think this
> is something releated to my code, beside the wordcount
> example many other users report the same problem:
> See:
> http://markmail.org/search/?q=org.apache.hadoop.mapred.MapTask
> %24MapOutputBuffer.collect+order%3Adate-backward
> Thanks for your help!
>
> Stefan
>
>
> On Mar 15, 2008, at 11:02 PM, Devaraj Das wrote:
>
> > It might have something to do with your application itself. By any
> > chance are you doing a lot of huge object allocation (directly or
> > indirectly) within the map method? Which version of hadoop
> are you on?
> >
> >> -----Original Message-----
> >> From: Stefan Groschupf [mailto:sg@101tec.com]
> >> Sent: Sunday, March 16, 2008 9:07 AM
> >> To: core-user@hadoop.apache.org
> >> Subject: [memory leak?] Re: MapReduce failure
> >>
> >> Hi there,
> >>
> >> we see the same situation and browsing the posts there are quite a
> >> lot of people running into this OOM problem.
> >> We run a own Mapper and our mapred.child.java.opts is -Xmx3048m, I
> >> think that should be more then enough.
> >> Also I changed io.sort.mb to 10, which had also no impact.
> >>
> >> Any ideas what might cause the OutOfMemoryError ?
> >> Thanks.
> >> Stefan
> >>
> >>
> >>
> >>
> >> On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote:
> >>
> >>> What is the heap size you are using for your tasks? Check
> >>> 'mapred.child.java.opts' in your hadoop-default.xml. Try
> increasing
> >>> it. This will happen if you try running the random-writer + sort
> >>> examples with default parameters. The maps are not able to
> >> spill the
> >>> data to the disk. Btw what version of HADOOP are you using?
> >>> Amar
> >>> On Mon, 10 Mar 2008, Ved Prakash wrote:
> >>>
> >>>> Hi friends,
> >>>>
> >>>> I have made a cluster of 3 machines, one of them is
> >> master, and other
> >>>> 2 slaves. I executed a mapreduce job on master but after
> Map, the
> >>>> execution terminates and Reduce doesn't happen. I have
> checked dfs
> >>>> and no output folder gets created.
> >>>>
> >>>> this is the error I see
> >>>>
> >>>> 08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
> >>>> task_200803101001_0001_m_000064_0, Status : FAILED
> >>>> java.lang.OutOfMemoryError: Java heap space
> >>>> at
> >>>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
> >>>> :95)
> >>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> >>>> at org.apache.hadoop.io.Text.write(Text.java:243)
> >>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
> >>>> MapTask.java:347)
> >>>> at org.apache.hadoop.examples.WordCount
> >>>> $MapClass.map(WordCount.java
> >>>> :72)
> >>>> at org.apache.hadoop.examples.WordCount
> >>>> $MapClass.map(WordCount.java
> >>>> :59)
> >>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> >>>> at org.apache.hadoop.mapred.TaskTracker
> >>>> $Child.main(TaskTracker.java
> >>>> :1787)
> >>>>
> >>>> 08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce
> >> 17% 08/03/10
> >>>> 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
> >> 08/03/10 10:35:51
> >>>> INFO mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:04 INFO
> >>>> mapred.JobClient: map 58% reduce 17% 08/03/10 10:36:07 INFO
> >>>> mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:07 INFO
> >>>> mapred.JobClient: Task Id :
> >>>> task_200803101001_0001_m_000071_0, Status : FAILED
> >>>> java.lang.OutOfMemoryError: Java heap space
> >>>> at
> >>>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
> >>>> :95)
> >>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> >>>> at org.apache.hadoop.io.Text.write(Text.java:243)
> >>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
> >>>> MapTask.java:347)
> >>>> at org.apache.hadoop.examples.WordCount
> >>>> $MapClass.map(WordCount.java
> >>>> :72)
> >>>> at org.apache.hadoop.examples.WordCount
> >>>> $MapClass.map(WordCount.java
> >>>> :59)
> >>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> >>>> at org.apache.hadoop.mapred.TaskTracker
> >>>> $Child.main(TaskTracker.java
> >>>> :1787)
> >>>>
> >>>> though it tries to overcome this problem but the mapreduce
> >>>> application doesn't create output, can anyone tell me
> why is this
> >>>> happening?
> >>>>
> >>>> Thanks
> >>>>
> >>>
> >>
> >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >> 101tec Inc.
> >> Menlo Park, California, USA
> >> http://www.101tec.com
> >>
> >>
> >>
> >
> >
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> Menlo Park, California, USA
> http://www.101tec.com
>
>
>
Re: [memory leak?] Re: MapReduce failure
Posted by Stefan Groschupf <sg...@101tec.com>.
I do not instantiate 3 GB of objects, that is for sure. The wordcount
example does not run anymore so I dont think this is something
releated to my code, beside the wordcount example many other users
report the same problem:
See:
http://markmail.org/search/?q=org.apache.hadoop.mapred.MapTask%24MapOutputBuffer.collect+order%3Adate-backward
Thanks for your help!
Stefan
On Mar 15, 2008, at 11:02 PM, Devaraj Das wrote:
> It might have something to do with your application itself. By any
> chance
> are you doing a lot of huge object allocation (directly or indirectly)
> within the map method? Which version of hadoop are you on?
>
>> -----Original Message-----
>> From: Stefan Groschupf [mailto:sg@101tec.com]
>> Sent: Sunday, March 16, 2008 9:07 AM
>> To: core-user@hadoop.apache.org
>> Subject: [memory leak?] Re: MapReduce failure
>>
>> Hi there,
>>
>> we see the same situation and browsing the posts there are
>> quite a lot of people running into this OOM problem.
>> We run a own Mapper and our mapred.child.java.opts is
>> -Xmx3048m, I think that should be more then enough.
>> Also I changed io.sort.mb to 10, which had also no impact.
>>
>> Any ideas what might cause the OutOfMemoryError ?
>> Thanks.
>> Stefan
>>
>>
>>
>>
>> On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote:
>>
>>> What is the heap size you are using for your tasks? Check
>>> 'mapred.child.java.opts' in your hadoop-default.xml. Try increasing
>>> it. This will happen if you try running the random-writer + sort
>>> examples with default parameters. The maps are not able to
>> spill the
>>> data to the disk. Btw what version of HADOOP are you using?
>>> Amar
>>> On Mon, 10 Mar 2008, Ved Prakash wrote:
>>>
>>>> Hi friends,
>>>>
>>>> I have made a cluster of 3 machines, one of them is
>> master, and other
>>>> 2 slaves. I executed a mapreduce job on master but after Map, the
>>>> execution terminates and Reduce doesn't happen. I have checked dfs
>>>> and no output folder gets created.
>>>>
>>>> this is the error I see
>>>>
>>>> 08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
>>>> task_200803101001_0001_m_000064_0, Status : FAILED
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at
>>>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
>>>> :95)
>>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>> at org.apache.hadoop.io.Text.write(Text.java:243)
>>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
>>>> MapTask.java:347)
>>>> at org.apache.hadoop.examples.WordCount
>>>> $MapClass.map(WordCount.java
>>>> :72)
>>>> at org.apache.hadoop.examples.WordCount
>>>> $MapClass.map(WordCount.java
>>>> :59)
>>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
>>>> at org.apache.hadoop.mapred.TaskTracker
>>>> $Child.main(TaskTracker.java
>>>> :1787)
>>>>
>>>> 08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce
>> 17% 08/03/10
>>>> 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
>> 08/03/10 10:35:51
>>>> INFO mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:04 INFO
>>>> mapred.JobClient: map 58% reduce 17% 08/03/10 10:36:07 INFO
>>>> mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:07 INFO
>>>> mapred.JobClient: Task Id :
>>>> task_200803101001_0001_m_000071_0, Status : FAILED
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at
>>>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
>>>> :95)
>>>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>>>> at org.apache.hadoop.io.Text.write(Text.java:243)
>>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
>>>> MapTask.java:347)
>>>> at org.apache.hadoop.examples.WordCount
>>>> $MapClass.map(WordCount.java
>>>> :72)
>>>> at org.apache.hadoop.examples.WordCount
>>>> $MapClass.map(WordCount.java
>>>> :59)
>>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
>>>> at org.apache.hadoop.mapred.TaskTracker
>>>> $Child.main(TaskTracker.java
>>>> :1787)
>>>>
>>>> though it tries to overcome this problem but the mapreduce
>>>> application doesn't create output, can anyone tell me why is this
>>>> happening?
>>>>
>>>> Thanks
>>>>
>>>
>>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> 101tec Inc.
>> Menlo Park, California, USA
>> http://www.101tec.com
>>
>>
>>
>
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com
RE: [memory leak?] Re: MapReduce failure
Posted by Devaraj Das <dd...@yahoo-inc.com>.
It might have something to do with your application itself. By any chance
are you doing a lot of huge object allocation (directly or indirectly)
within the map method? Which version of hadoop are you on?
> -----Original Message-----
> From: Stefan Groschupf [mailto:sg@101tec.com]
> Sent: Sunday, March 16, 2008 9:07 AM
> To: core-user@hadoop.apache.org
> Subject: [memory leak?] Re: MapReduce failure
>
> Hi there,
>
> we see the same situation and browsing the posts there are
> quite a lot of people running into this OOM problem.
> We run a own Mapper and our mapred.child.java.opts is
> -Xmx3048m, I think that should be more then enough.
> Also I changed io.sort.mb to 10, which had also no impact.
>
> Any ideas what might cause the OutOfMemoryError ?
> Thanks.
> Stefan
>
>
>
>
> On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote:
>
> > What is the heap size you are using for your tasks? Check
> > 'mapred.child.java.opts' in your hadoop-default.xml. Try increasing
> > it. This will happen if you try running the random-writer + sort
> > examples with default parameters. The maps are not able to
> spill the
> > data to the disk. Btw what version of HADOOP are you using?
> > Amar
> > On Mon, 10 Mar 2008, Ved Prakash wrote:
> >
> >> Hi friends,
> >>
> >> I have made a cluster of 3 machines, one of them is
> master, and other
> >> 2 slaves. I executed a mapreduce job on master but after Map, the
> >> execution terminates and Reduce doesn't happen. I have checked dfs
> >> and no output folder gets created.
> >>
> >> this is the error I see
> >>
> >> 08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
> >> task_200803101001_0001_m_000064_0, Status : FAILED
> >> java.lang.OutOfMemoryError: Java heap space
> >> at
> >> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
> >> :95)
> >> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> >> at org.apache.hadoop.io.Text.write(Text.java:243)
> >> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
> >> MapTask.java:347)
> >> at org.apache.hadoop.examples.WordCount
> >> $MapClass.map(WordCount.java
> >> :72)
> >> at org.apache.hadoop.examples.WordCount
> >> $MapClass.map(WordCount.java
> >> :59)
> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> >> at org.apache.hadoop.mapred.TaskTracker
> >> $Child.main(TaskTracker.java
> >> :1787)
> >>
> >> 08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce
> 17% 08/03/10
> >> 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
> 08/03/10 10:35:51
> >> INFO mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:04 INFO
> >> mapred.JobClient: map 58% reduce 17% 08/03/10 10:36:07 INFO
> >> mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:07 INFO
> >> mapred.JobClient: Task Id :
> >> task_200803101001_0001_m_000071_0, Status : FAILED
> >> java.lang.OutOfMemoryError: Java heap space
> >> at
> >> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
> >> :95)
> >> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> >> at org.apache.hadoop.io.Text.write(Text.java:243)
> >> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
> >> MapTask.java:347)
> >> at org.apache.hadoop.examples.WordCount
> >> $MapClass.map(WordCount.java
> >> :72)
> >> at org.apache.hadoop.examples.WordCount
> >> $MapClass.map(WordCount.java
> >> :59)
> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> >> at org.apache.hadoop.mapred.TaskTracker
> >> $Child.main(TaskTracker.java
> >> :1787)
> >>
> >> though it tries to overcome this problem but the mapreduce
> >> application doesn't create output, can anyone tell me why is this
> >> happening?
> >>
> >> Thanks
> >>
> >
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec Inc.
> Menlo Park, California, USA
> http://www.101tec.com
>
>
>
[memory leak?] Re: MapReduce failure
Posted by Stefan Groschupf <sg...@101tec.com>.
Hi there,
we see the same situation and browsing the posts there are quite a lot
of people running into this OOM problem.
We run a own Mapper and our mapred.child.java.opts is -Xmx3048m, I
think that should be more then enough.
Also I changed io.sort.mb to 10, which had also no impact.
Any ideas what might cause the OutOfMemoryError ?
Thanks.
Stefan
On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote:
> What is the heap size you are using for your tasks? Check
> 'mapred.child.java.opts' in your hadoop-default.xml. Try increasing
> it. This will happen if you try running the random-writer + sort
> examples with default parameters. The maps are not able to spill the
> data to the disk. Btw what version of HADOOP are you using?
> Amar
> On Mon, 10 Mar 2008, Ved Prakash wrote:
>
>> Hi friends,
>>
>> I have made a cluster of 3 machines, one of them is master, and
>> other 2
>> slaves. I executed a mapreduce job on master but after Map, the
>> execution
>> terminates and Reduce doesn't happen. I have checked dfs and no
>> output
>> folder gets created.
>>
>> this is the error I see
>>
>> 08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
>> task_200803101001_0001_m_000064_0, Status : FAILED
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
>> :95)
>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>> at org.apache.hadoop.io.Text.write(Text.java:243)
>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
>> MapTask.java:347)
>> at org.apache.hadoop.examples.WordCount
>> $MapClass.map(WordCount.java
>> :72)
>> at org.apache.hadoop.examples.WordCount
>> $MapClass.map(WordCount.java
>> :59)
>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
>> at org.apache.hadoop.mapred.TaskTracker
>> $Child.main(TaskTracker.java
>> :1787)
>>
>> 08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce 17%
>> 08/03/10 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
>> 08/03/10 10:35:51 INFO mapred.JobClient: map 57% reduce 17%
>> 08/03/10 10:36:04 INFO mapred.JobClient: map 58% reduce 17%
>> 08/03/10 10:36:07 INFO mapred.JobClient: map 57% reduce 17%
>> 08/03/10 10:36:07 INFO mapred.JobClient: Task Id :
>> task_200803101001_0001_m_000071_0, Status : FAILED
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
>> :95)
>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>> at org.apache.hadoop.io.Text.write(Text.java:243)
>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
>> MapTask.java:347)
>> at org.apache.hadoop.examples.WordCount
>> $MapClass.map(WordCount.java
>> :72)
>> at org.apache.hadoop.examples.WordCount
>> $MapClass.map(WordCount.java
>> :59)
>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
>> at org.apache.hadoop.mapred.TaskTracker
>> $Child.main(TaskTracker.java
>> :1787)
>>
>> though it tries to overcome this problem but the mapreduce
>> application
>> doesn't create output, can anyone tell me why is this happening?
>>
>> Thanks
>>
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101tec Inc.
Menlo Park, California, USA
http://www.101tec.com
Re: MapReduce failure
Posted by Ved Prakash <me...@gmail.com>.
I increased the heap size as you have suggested, and I could run a map
reduce job on it.
thanks
On Mon, Mar 10, 2008 at 10:58 AM, Amar Kamat <am...@yahoo-inc.com> wrote:
> What is the heap size you are using for your tasks? Check
> 'mapred.child.java.opts' in your hadoop-default.xml. Try increasing it.
> This will happen if you try running the random-writer + sort examples with
> default parameters. The maps are not able to spill the data to the disk.
> Btw what version of HADOOP are you using?
> Amar
> On Mon, 10 Mar 2008, Ved Prakash
> wrote:
>
> > Hi friends,
> >
> > I have made a cluster of 3 machines, one of them is master, and other 2
> > slaves. I executed a mapreduce job on master but after Map, the
> execution
> > terminates and Reduce doesn't happen. I have checked dfs and no output
> > folder gets created.
> >
> > this is the error I see
> >
> > 08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
> > task_200803101001_0001_m_000064_0, Status : FAILED
> > java.lang.OutOfMemoryError: Java heap space
> > at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
> > :95)
> > at java.io.DataOutputStream.write(DataOutputStream.java:90)
> > at org.apache.hadoop.io.Text.write(Text.java:243)
> > at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
> > MapTask.java:347)
> > at org.apache.hadoop.examples.WordCount$MapClass.map(
> WordCount.java
> > :72)
> > at org.apache.hadoop.examples.WordCount$MapClass.map(
> WordCount.java
> > :59)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> > at org.apache.hadoop.mapred.TaskTracker$Child.main(
> TaskTracker.java
> > :1787)
> >
> > 08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce 17%
> > 08/03/10 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
> > 08/03/10 10:35:51 INFO mapred.JobClient: map 57% reduce 17%
> > 08/03/10 10:36:04 INFO mapred.JobClient: map 58% reduce 17%
> > 08/03/10 10:36:07 INFO mapred.JobClient: map 57% reduce 17%
> > 08/03/10 10:36:07 INFO mapred.JobClient: Task Id :
> > task_200803101001_0001_m_000071_0, Status : FAILED
> > java.lang.OutOfMemoryError: Java heap space
> > at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
> > :95)
> > at java.io.DataOutputStream.write(DataOutputStream.java:90)
> > at org.apache.hadoop.io.Text.write(Text.java:243)
> > at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
> > MapTask.java:347)
> > at org.apache.hadoop.examples.WordCount$MapClass.map(
> WordCount.java
> > :72)
> > at org.apache.hadoop.examples.WordCount$MapClass.map(
> WordCount.java
> > :59)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> > at org.apache.hadoop.mapred.TaskTracker$Child.main(
> TaskTracker.java
> > :1787)
> >
> > though it tries to overcome this problem but the mapreduce application
> > doesn't create output, can anyone tell me why is this happening?
> >
> > Thanks
> >
>
Re: MapReduce failure
Posted by Amar Kamat <am...@yahoo-inc.com>.
What is the heap size you are using for your tasks? Check
'mapred.child.java.opts' in your hadoop-default.xml. Try increasing it.
This will happen if you try running the random-writer + sort examples with
default parameters. The maps are not able to spill the data to the disk.
Btw what version of HADOOP are you using?
Amar
On Mon, 10 Mar 2008, Ved Prakash
wrote:
> Hi friends,
>
> I have made a cluster of 3 machines, one of them is master, and other 2
> slaves. I executed a mapreduce job on master but after Map, the execution
> terminates and Reduce doesn't happen. I have checked dfs and no output
> folder gets created.
>
> this is the error I see
>
> 08/03/10 10:35:21 INFO mapred.JobClient: Task Id :
> task_200803101001_0001_m_000064_0, Status : FAILED
> java.lang.OutOfMemoryError: Java heap space
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
> :95)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
> MapTask.java:347)
> at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
> :72)
> at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
> :59)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
> :1787)
>
> 08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce 17%
> 08/03/10 10:35:31 INFO mapred.JobClient: map 56% reduce 17%
> 08/03/10 10:35:51 INFO mapred.JobClient: map 57% reduce 17%
> 08/03/10 10:36:04 INFO mapred.JobClient: map 58% reduce 17%
> 08/03/10 10:36:07 INFO mapred.JobClient: map 57% reduce 17%
> 08/03/10 10:36:07 INFO mapred.JobClient: Task Id :
> task_200803101001_0001_m_000071_0, Status : FAILED
> java.lang.OutOfMemoryError: Java heap space
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java
> :95)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(
> MapTask.java:347)
> at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
> :72)
> at org.apache.hadoop.examples.WordCount$MapClass.map(WordCount.java
> :59)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
> :1787)
>
> though it tries to overcome this problem but the mapreduce application
> doesn't create output, can anyone tell me why is this happening?
>
> Thanks
>