You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Doug Cutting (JIRA)" <ji...@apache.org> on 2007/09/10 18:55:29 UTC
[jira] Created: (HADOOP-1867) use single parameter to specify a
node's available ram
use single parameter to specify a node's available ram
------------------------------------------------------
Key: HADOOP-1867
URL: https://issues.apache.org/jira/browse/HADOOP-1867
Project: Hadoop
Issue Type: Improvement
Reporter: Doug Cutting
To simplify configuration, we should use a single parameter to indicate a node's available RAM. Sites should not need to adjust more than this single parameter to configure nodes available memory. In task JVMs, some significant percentage of the memory should be reserved for application code, with the remainder divided among various system buffers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1867) use single parameter to specify a
node's available ram
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12527803 ]
eric baldeschwieler commented on HADOOP-1867:
---------------------------------------------
it would be great if we could automagically choose reasonable defaults. We'll still want a memory for M-R override since issues like what other code you run aon a node will dictate your preferred RAM allocated to M-R.
> use single parameter to specify a node's available ram
> ------------------------------------------------------
>
> Key: HADOOP-1867
> URL: https://issues.apache.org/jira/browse/HADOOP-1867
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Doug Cutting
>
> To simplify configuration, we should use a single parameter to indicate a node's available RAM. Sites should not need to adjust more than this single parameter to configure nodes available memory. In task JVMs, some significant percentage of the memory should be reserved for application code, with the remainder divided among various system buffers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1867) use single parameter to specify a
node's available ram
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526218 ]
Doug Cutting commented on HADOOP-1867:
--------------------------------------
> Isn't memory allowed for JVM already available?
Good point. So perhaps we should base things on Runtime#maxMemory()? We could set the JVM heap size based on /proc/meminfo, then base things on that.
Parameters whose values might be automatically determined include:
- fs.inmemory.size.mb
- io.sort.factor
- io.sort.mb
- io.file.buffer.size
What about mapred.child.java.opts?
> use single parameter to specify a node's available ram
> ------------------------------------------------------
>
> Key: HADOOP-1867
> URL: https://issues.apache.org/jira/browse/HADOOP-1867
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Doug Cutting
>
> To simplify configuration, we should use a single parameter to indicate a node's available RAM. Sites should not need to adjust more than this single parameter to configure nodes available memory. In task JVMs, some significant percentage of the memory should be reserved for application code, with the remainder divided among various system buffers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1867) use single parameter to specify a
node's available ram
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526204 ]
Raghu Angadi commented on HADOOP-1867:
--------------------------------------
Isn't memory allowed for JVM already available?
> use single parameter to specify a node's available ram
> ------------------------------------------------------
>
> Key: HADOOP-1867
> URL: https://issues.apache.org/jira/browse/HADOOP-1867
> Project: Hadoop
> Issue Type: Improvement
> Reporter: Doug Cutting
>
> To simplify configuration, we should use a single parameter to indicate a node's available RAM. Sites should not need to adjust more than this single parameter to configure nodes available memory. In task JVMs, some significant percentage of the memory should be reserved for application code, with the remainder divided among various system buffers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.