You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "lohit vijayarenu (JIRA)" <ji...@apache.org> on 2007/10/13 20:35:50 UTC
[jira] Created: (HADOOP-2053) OutOfMemoryError : Java heap space
errors in hadoop 0.14
OutOfMemoryError : Java heap space errors in hadoop 0.14
--------------------------------------------------------
Key: HADOOP-2053
URL: https://issues.apache.org/jira/browse/HADOOP-2053
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.14.1
Reporter: lohit vijayarenu
Fix For: 0.15.0
In recent hadoop 0.14 we are seeing few jobs where map taskf fail with java.lang.OutOfMemoryError: Java heap space problem
These were the same jobs which used to work fine with 0.13
<stack>
task_200710112103_0001_m_000015_1: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.Text.write(Text.java:243)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:340)
</stack>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2053) OutOfMemoryError : Java heap space
errors in hadoop 0.14
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-2053:
----------------------------------
Attachment: HADOOP-2053_1_20071015.patch
Here is a patch which frees the reference to the large {{DataOutputBuffer}} that {{BasicTypeSorterBase}} has in it's {{close}} method... this lets the GC collect away the keyValBuffer.
In absence of this patch, there is a window where both the currently active keyValBuffer and the one that should have been freed in the previous iteration are both active i.e. doubling the required amount of memory, which leads to the OutOfMemoryException.
All credit to this goes to Koji!
> OutOfMemoryError : Java heap space errors in hadoop 0.14
> --------------------------------------------------------
>
> Key: HADOOP-2053
> URL: https://issues.apache.org/jira/browse/HADOOP-2053
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0, 0.14.1, 0.14.2
> Reporter: lohit vijayarenu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.3
>
> Attachments: HADOOP-2053_1_20071015.patch
>
>
> In recent hadoop 0.14 we are seeing few jobs where map taskf fail with java.lang.OutOfMemoryError: Java heap space problem
> These were the same jobs which used to work fine with 0.13
> <stack>
> task_200710112103_0001_m_000015_1: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:340)
> </stack>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2053) OutOfMemoryError : Java heap space
errors in hadoop 0.14
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535247 ]
Hudson commented on HADOOP-2053:
--------------------------------
Integrated in Hadoop-Nightly #274 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/274/])
> OutOfMemoryError : Java heap space errors in hadoop 0.14
> --------------------------------------------------------
>
> Key: HADOOP-2053
> URL: https://issues.apache.org/jira/browse/HADOOP-2053
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0, 0.14.1, 0.14.2
> Reporter: lohit vijayarenu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.3
>
> Attachments: HADOOP-2053_1_20071015.patch
>
>
> In recent hadoop 0.14 we are seeing few jobs where map taskf fail with java.lang.OutOfMemoryError: Java heap space problem
> These were the same jobs which used to work fine with 0.13
> <stack>
> task_200710112103_0001_m_000015_1: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:340)
> </stack>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2053) OutOfMemoryError : Java heap space
errors in hadoop 0.14
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534581 ]
Devaraj Das commented on HADOOP-2053:
-------------------------------------
If fixing HADOOP-2043 leads us to doing a 0.14.3 release, we should include the fix for this issue in that as well.
> OutOfMemoryError : Java heap space errors in hadoop 0.14
> --------------------------------------------------------
>
> Key: HADOOP-2053
> URL: https://issues.apache.org/jira/browse/HADOOP-2053
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0, 0.14.1, 0.14.2
> Reporter: lohit vijayarenu
> Priority: Blocker
>
> In recent hadoop 0.14 we are seeing few jobs where map taskf fail with java.lang.OutOfMemoryError: Java heap space problem
> These were the same jobs which used to work fine with 0.13
> <stack>
> task_200710112103_0001_m_000015_1: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:340)
> </stack>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2053) OutOfMemoryError : Java heap space
errors in hadoop 0.14
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-2053:
----------------------------------
Fix Version/s: 0.14.3
Assignee: Arun C Murthy
> OutOfMemoryError : Java heap space errors in hadoop 0.14
> --------------------------------------------------------
>
> Key: HADOOP-2053
> URL: https://issues.apache.org/jira/browse/HADOOP-2053
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0, 0.14.1, 0.14.2
> Reporter: lohit vijayarenu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.3
>
>
> In recent hadoop 0.14 we are seeing few jobs where map taskf fail with java.lang.OutOfMemoryError: Java heap space problem
> These were the same jobs which used to work fine with 0.13
> <stack>
> task_200710112103_0001_m_000015_1: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:340)
> </stack>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2053) OutOfMemoryError : Java heap space
errors in hadoop 0.14
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-2053:
--------------------------------
Fix Version/s: (was: 0.15.0)
Priority: Blocker (was: Major)
Affects Version/s: 0.14.0
0.14.2
Marking this a blocker since apps that were working with 0.13 release fails with 0.14
> OutOfMemoryError : Java heap space errors in hadoop 0.14
> --------------------------------------------------------
>
> Key: HADOOP-2053
> URL: https://issues.apache.org/jira/browse/HADOOP-2053
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0, 0.14.1, 0.14.2
> Reporter: lohit vijayarenu
> Priority: Blocker
>
> In recent hadoop 0.14 we are seeing few jobs where map taskf fail with java.lang.OutOfMemoryError: Java heap space problem
> These were the same jobs which used to work fine with 0.13
> <stack>
> task_200710112103_0001_m_000015_1: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:340)
> </stack>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-2053) OutOfMemoryError : Java heap space
errors in hadoop 0.14
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HADOOP-2053.
-----------------------------------
Resolution: Fixed
I just committed this. Thanks, Arun!
> OutOfMemoryError : Java heap space errors in hadoop 0.14
> --------------------------------------------------------
>
> Key: HADOOP-2053
> URL: https://issues.apache.org/jira/browse/HADOOP-2053
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.0, 0.14.1, 0.14.2
> Reporter: lohit vijayarenu
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.14.3
>
> Attachments: HADOOP-2053_1_20071015.patch
>
>
> In recent hadoop 0.14 we are seeing few jobs where map taskf fail with java.lang.OutOfMemoryError: Java heap space problem
> These were the same jobs which used to work fine with 0.13
> <stack>
> task_200710112103_0001_m_000015_1: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:243)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:340)
> </stack>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.