You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2008/01/22 15:40:34 UTC
[jira] Commented: (HADOOP-2438) In streaming, jobs that used to
work, crash in the map phase -- even if the mapper is /bin/cat
[ https://issues.apache.org/jira/browse/HADOOP-2438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561360#action_12561360 ]
Arun C Murthy commented on HADOOP-2438:
---------------------------------------
Arkady, what is the max java heapsize for the child when you notice this OOM?
> In streaming, jobs that used to work, crash in the map phase -- even if the mapper is /bin/cat
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-2438
> URL: https://issues.apache.org/jira/browse/HADOOP-2438
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.15.1
> Reporter: arkady borkovsky
>
> The exception is either "out of memory" of or "broken pipe" -- see both stack dumps bellow.
> st Hadoop input: |null|
> last tool output: |[B@20fa83|
> Date: Sat Dec 15 21:02:18 UTC 2007
> java.io.IOException: Broken pipe
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:124)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:96)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)
> at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:107)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)
> -------------------------------------------------
> java.io.IOException: MROutput/MRErrThread
> failed:java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect (MapTask.java:347)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run (PipeMapRed.java:344)
> at org.apache.hadoop.streaming.PipeMapper.map(PipeMapper.java:76)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:
> 1760)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HADOOP-2438) In streaming, jobs that used to work, crash in the map phase -- even if the mapper is /bin/cat
Posted by arkady borkovsky <ar...@yahoo-inc.com>.
Arun,
this is a Streaming job.
I should not even know what "max java heapsize" means...
-- ab
On Jan 22, 2008, at 6:40 AM, Arun C Murthy (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/HADOOP-2438?
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=12561360#action_12561360 ]
>
> Arun C Murthy commented on HADOOP-2438:
> ---------------------------------------
>
> Arkady, what is the max java heapsize for the child when you notice
> this OOM?
>
>> In streaming, jobs that used to work, crash in the map phase --
>> even if the mapper is /bin/cat
>> ---------------------------------------------------------------------
>> -------------------------
>>
>> Key: HADOOP-2438
>> URL: https://issues.apache.org/jira/browse/
>> HADOOP-2438
>> Project: Hadoop
>> Issue Type: Bug
>> Affects Versions: 0.15.1
>> Reporter: arkady borkovsky
>>
>> The exception is either "out of memory" of or "broken pipe" -- see
>> both stack dumps bellow.
>> st Hadoop input: |null|
>> last tool output: |[B@20fa83|
>> Date: Sat Dec 15 21:02:18 UTC 2007
>> java.io.IOException: Broken pipe
>> at java.io.FileOutputStream.writeBytes(Native Method)
>> at java.io.FileOutputStream.write(FileOutputStream.java:260)
>> at java.io.BufferedOutputStream.flushBuffer
>> (BufferedOutputStream.java:65)
>> at java.io.BufferedOutputStream.flush
>> (BufferedOutputStream.java:123)
>> at java.io.BufferedOutputStream.flush
>> (BufferedOutputStream.java:124)
>> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>> at org.apache.hadoop.streaming.PipeMapper.map
>> (PipeMapper.java:96)
>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
>> at org.apache.hadoop.mapred.TaskTracker$Child.main
>> (TaskTracker.java:1760)
>> at org.apache.hadoop.streaming.PipeMapper.map
>> (PipeMapper.java:107)
>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
>> at org.apache.hadoop.mapred.TaskTracker$Child.main
>> (TaskTracker.java:1760)
>> -------------------------------------------------
>> java.io.IOException: MROutput/MRErrThread
>> failed:java.lang.OutOfMemoryError: Java heap space
>> at java.util.Arrays.copyOf(Arrays.java:2786)
>> at java.io.ByteArrayOutputStream.write
>> (ByteArrayOutputStream.java:94)
>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>> at org.apache.hadoop.io.Text.write(Text.java:243)
>> at org.apache.hadoop.mapred.MapTask
>> $MapOutputBuffer.collect (MapTask.java:347)
>> at org.apache.hadoop.streaming.PipeMapRed
>> $MROutputThread.run (PipeMapRed.java:344)
>> at org.apache.hadoop.streaming.PipeMapper.map
>> (PipeMapper.java:76)
>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
>> at org.apache.hadoop.mapred.TaskTracker$Child.main
>> (TaskTracker.java:
>> 1760)
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>