You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Alejandro Abdelnur (JIRA)" <ji...@apache.org> on 2012/10/15 18:38:03 UTC
[jira] [Created] (YARN-160) nodemanagers should obtain cpu/memory
values from underlying OS
Alejandro Abdelnur created YARN-160:
---------------------------------------
Summary: nodemanagers should obtain cpu/memory values from underlying OS
Key: YARN-160
URL: https://issues.apache.org/jira/browse/YARN-160
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.0.3-alpha
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.0.3-alpha
As mentioned in YARN-2
*NM memory and CPU configs*
Currently these values are coming from the config of the NM, we should be able to obtain those values from the OS (ie, in the case of Linux from /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have an interface that obtains this information. In addition implementations of this interface should be able to specify a mem/cpu offset (amount of mem/cpu not to be avail as YARN resource), this would allow to reserve mem/cpu for the OS and other services outside of YARN containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-160) nodemanagers should obtain cpu/memory
values from underlying OS
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476330#comment-13476330 ]
Alejandro Abdelnur commented on YARN-160:
-----------------------------------------
Radim, thanks for pointing that. This it seems it is not wired for the NMs to report such info back to the RM, looking at the code NodeManager creates a NodeStatusUpdater using values from configuration files. Only the ContainersMonitorIMpl reads the underlying memory. So this JIRA would be to get the NodeStatusUpdater to use a resourcecalculator. Makes sense?
> nodemanagers should obtain cpu/memory values from underlying OS
> ---------------------------------------------------------------
>
> Key: YARN-160
> URL: https://issues.apache.org/jira/browse/YARN-160
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.0.3-alpha
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.3-alpha
>
>
> As mentioned in YARN-2
> *NM memory and CPU configs*
> Currently these values are coming from the config of the NM, we should be able to obtain those values from the OS (ie, in the case of Linux from /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have an interface that obtains this information. In addition implementations of this interface should be able to specify a mem/cpu offset (amount of mem/cpu not to be avail as YARN resource), this would allow to reserve mem/cpu for the OS and other services outside of YARN containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-160) nodemanagers should obtain cpu/memory
values from underlying OS
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480408#comment-13480408 ]
Radim Kolar commented on YARN-160:
----------------------------------
My proposal is to make entire new framework for resource management - MAPREDUCE-4256
> nodemanagers should obtain cpu/memory values from underlying OS
> ---------------------------------------------------------------
>
> Key: YARN-160
> URL: https://issues.apache.org/jira/browse/YARN-160
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.0.3-alpha
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.3-alpha
>
>
> As mentioned in YARN-2
> *NM memory and CPU configs*
> Currently these values are coming from the config of the NM, we should be able to obtain those values from the OS (ie, in the case of Linux from /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have an interface that obtains this information. In addition implementations of this interface should be able to specify a mem/cpu offset (amount of mem/cpu not to be avail as YARN resource), this would allow to reserve mem/cpu for the OS and other services outside of YARN containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (YARN-160) nodemanagers should obtain cpu/memory values from underlying OS
Posted by Alejandro Abdelnur <tu...@gmail.com>.
thanks Radim, unless i'm missing something this is not wired, correct? yes being able to override makes sense
Alejandro
On Oct 15, 2012, at 10:36 AM, "Radim Kolar (JIRA)" <ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/YARN-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476288#comment-13476288 ]
>
> Radim Kolar commented on YARN-160:
> ----------------------------------
>
> there is a interface from obtaining these values from OS. Its called resourcecalculator plugin. It might be good to do autodetection but i need ability to override these values in config.
>
>> nodemanagers should obtain cpu/memory values from underlying OS
>> ---------------------------------------------------------------
>>
>> Key: YARN-160
>> URL: https://issues.apache.org/jira/browse/YARN-160
>> Project: Hadoop YARN
>> Issue Type: Improvement
>> Components: nodemanager
>> Affects Versions: 2.0.3-alpha
>> Reporter: Alejandro Abdelnur
>> Assignee: Alejandro Abdelnur
>> Fix For: 2.0.3-alpha
>>
>>
>> As mentioned in YARN-2
>> *NM memory and CPU configs*
>> Currently these values are coming from the config of the NM, we should be able to obtain those values from the OS (ie, in the case of Linux from /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have an interface that obtains this information. In addition implementations of this interface should be able to specify a mem/cpu offset (amount of mem/cpu not to be avail as YARN resource), this would allow to reserve mem/cpu for the OS and other services outside of YARN containers.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-160) nodemanagers should obtain cpu/memory
values from underlying OS
Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476288#comment-13476288 ]
Radim Kolar commented on YARN-160:
----------------------------------
there is a interface from obtaining these values from OS. Its called resourcecalculator plugin. It might be good to do autodetection but i need ability to override these values in config.
> nodemanagers should obtain cpu/memory values from underlying OS
> ---------------------------------------------------------------
>
> Key: YARN-160
> URL: https://issues.apache.org/jira/browse/YARN-160
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.0.3-alpha
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 2.0.3-alpha
>
>
> As mentioned in YARN-2
> *NM memory and CPU configs*
> Currently these values are coming from the config of the NM, we should be able to obtain those values from the OS (ie, in the case of Linux from /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have an interface that obtains this information. In addition implementations of this interface should be able to specify a mem/cpu offset (amount of mem/cpu not to be avail as YARN resource), this would allow to reserve mem/cpu for the OS and other services outside of YARN containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira