You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Suresh Srinivas (JIRA)" <ji...@apache.org> on 2009/03/24 19:34:51 UTC
[jira] Created: (HADOOP-5564) hadoop command uses large JVM heap
size
hadoop command uses large JVM heap size
---------------------------------------
Key: HADOOP-5564
URL: https://issues.apache.org/jira/browse/HADOOP-5564
Project: Hadoop Core
Issue Type: Bug
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Fix For: 0.20.0
Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5564) hadoop command uses large JVM heap
size
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-5564:
------------------------------------
Attachment: 5564.trunk.patch
Updated trunk patch
> hadoop command uses large JVM heap size
> ---------------------------------------
>
> Key: HADOOP-5564
> URL: https://issues.apache.org/jira/browse/HADOOP-5564
> Project: Hadoop Core
> Issue Type: Bug
> Components: build
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: 5564.patch, 5564.trunk.patch, 5564.trunk.patch
>
>
> Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5564) hadoop command uses large JVM heap
size
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688792#action_12688792 ]
Suresh Srinivas commented on HADOOP-5564:
-----------------------------------------
I have tested this manually by making sure {{hadoop}} command takes the patch of changed code and it prints the {{JAVA_PLATFORM}} successfully.
> hadoop command uses large JVM heap size
> ---------------------------------------
>
> Key: HADOOP-5564
> URL: https://issues.apache.org/jira/browse/HADOOP-5564
> Project: Hadoop Core
> Issue Type: Bug
> Components: build
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: 5564.patch
>
>
> Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5564) hadoop command uses large JVM heap
size
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688815#action_12688815 ]
Steve Loughran commented on HADOOP-5564:
----------------------------------------
has anyone tested that everyting works ok on a 64 bit JVM? one with pointers twice the size?
> hadoop command uses large JVM heap size
> ---------------------------------------
>
> Key: HADOOP-5564
> URL: https://issues.apache.org/jira/browse/HADOOP-5564
> Project: Hadoop Core
> Issue Type: Bug
> Components: build
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Priority: Blocker
> Fix For: 0.20.0, 0.21.0
>
> Attachments: 5564.patch, 5564.trunk.patch, 5564.trunk.patch
>
>
> Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5564) hadoop command uses large JVM heap
size
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-5564:
-------------------------------------------
Hadoop Flags: [Reviewed]
+1 patch looks good
Talked to Koji and Suresh offline. We have considered using -client. However, the [tool doc|http://java.sun.com/javase/6/docs/technotes/tools/solaris/java.html] says that the -client option may be ignored by the jdk.
> hadoop command uses large JVM heap size
> ---------------------------------------
>
> Key: HADOOP-5564
> URL: https://issues.apache.org/jira/browse/HADOOP-5564
> Project: Hadoop Core
> Issue Type: Bug
> Components: build
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: 5564.patch
>
>
> Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5564) hadoop command uses large JVM heap
size
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-5564:
-------------------------------------------
Component/s: build
Priority: Blocker (was: Major)
As Koji suggested, we probably should add -Xmx to limit the memory usage.
> hadoop command uses large JVM heap size
> ---------------------------------------
>
> Key: HADOOP-5564
> URL: https://issues.apache.org/jira/browse/HADOOP-5564
> Project: Hadoop Core
> Issue Type: Bug
> Components: build
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Priority: Blocker
> Fix For: 0.20.0
>
>
> Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5564) hadoop command uses large JVM heap
size
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689227#action_12689227 ]
Hudson commented on HADOOP-5564:
--------------------------------
Integrated in Hadoop-trunk #790 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/790/])
. Limit the JVM heap size in the java command for initializing JAVA_PLATFORM. (Suresh Srinivas via szetszwo)
> hadoop command uses large JVM heap size
> ---------------------------------------
>
> Key: HADOOP-5564
> URL: https://issues.apache.org/jira/browse/HADOOP-5564
> Project: Hadoop Core
> Issue Type: Bug
> Components: build
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Priority: Blocker
> Fix For: 0.20.0, 0.21.0
>
> Attachments: 5564.patch, 5564.trunk.patch, 5564.trunk.patch
>
>
> Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5564) hadoop command uses large JVM heap
size
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-5564:
------------------------------------
Attachment: 5564.trunk.patch
Attaching patch for the trunk. Tested the patch manually on trunk.
> hadoop command uses large JVM heap size
> ---------------------------------------
>
> Key: HADOOP-5564
> URL: https://issues.apache.org/jira/browse/HADOOP-5564
> Project: Hadoop Core
> Issue Type: Bug
> Components: build
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: 5564.patch, 5564.trunk.patch
>
>
> Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5564) hadoop command uses large JVM heap
size
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-5564:
------------------------------------
Attachment: 5564.patch
Setting the heap size to 32MB
> hadoop command uses large JVM heap size
> ---------------------------------------
>
> Key: HADOOP-5564
> URL: https://issues.apache.org/jira/browse/HADOOP-5564
> Project: Hadoop Core
> Issue Type: Bug
> Components: build
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: 5564.patch
>
>
> Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5564) hadoop command uses large JVM heap
size
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE resolved HADOOP-5564.
--------------------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
I have committed this to 0.20 and above. Thanks, Suresh!
> hadoop command uses large JVM heap size
> ---------------------------------------
>
> Key: HADOOP-5564
> URL: https://issues.apache.org/jira/browse/HADOOP-5564
> Project: Hadoop Core
> Issue Type: Bug
> Components: build
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Priority: Blocker
> Fix For: 0.20.0, 0.21.0
>
> Attachments: 5564.patch, 5564.trunk.patch, 5564.trunk.patch
>
>
> Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.