You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2012/03/01 02:19:58 UTC
[jira] [Created] (MAPREDUCE-3946) If a resource requirement is
higher than available on any node, job should fail early
If a resource requirement is higher than available on any node, job should fail early
-------------------------------------------------------------------------------------
Key: MAPREDUCE-3946
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3946
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.24.0, 0.23.2
Reporter: Todd Lipcon
If you configure the NMs to have 1GB of RAM each, and then try to submit a job which has an AM resource requirement of 1.5GB, the job will neither run nor fail. Instead, it will slowly sop of all of the resources in the cluster as "reservations" despite the fact that it will never be able to schedule something. Instead, it should fail early indicating that the required memory allocation is infeasible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3946) If a resource requirement is
higher than available on any node, job should fail early
Posted by "Hitesh Shah (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219745#comment-13219745 ]
Hitesh Shah commented on MAPREDUCE-3946:
----------------------------------------
This probably could probably be marked as a duplicate for MAPREDUCE-3796.
MR-3796 is meant to be more yarn-specific to ensure that the application submission should fail if the AM memory request exceeds max allowed. However this jira may still be needed if the job client does not handle app submission failures properly.
> If a resource requirement is higher than available on any node, job should fail early
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3946
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3946
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.24.0, 0.23.2
> Reporter: Todd Lipcon
>
> If you configure the NMs to have 1GB of RAM each, and then try to submit a job which has an AM resource requirement of 1.5GB, the job will neither run nor fail. Instead, it will slowly sop of all of the resources in the cluster as "reservations" despite the fact that it will never be able to schedule something. Instead, it should fail early indicating that the required memory allocation is infeasible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3946) If a resource requirement is
higher than available on any node, job should fail early
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219767#comment-13219767 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3946:
----------------------------------------------------
Ah, I forgot about that. So, yes I think this is a duplicate.
Todd, essentially, we now have a cluster level configuration for max-resource-size which is supposed to be set by cluster admins such that all nodes have greater than that. RM will reject apps if the AM resource requirement is more than this. I think that should work, please close this as duplicate if you agree too. Thanks.
> If a resource requirement is higher than available on any node, job should fail early
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3946
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3946
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.24.0, 0.23.2
> Reporter: Todd Lipcon
>
> If you configure the NMs to have 1GB of RAM each, and then try to submit a job which has an AM resource requirement of 1.5GB, the job will neither run nor fail. Instead, it will slowly sop of all of the resources in the cluster as "reservations" despite the fact that it will never be able to schedule something. Instead, it should fail early indicating that the required memory allocation is infeasible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3946) If a resource requirement is
higher than available on any node, job should fail early
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219842#comment-13219842 ]
Todd Lipcon commented on MAPREDUCE-3946:
----------------------------------------
Can we automatically bound the max-resource-size configuration by max(available resources) across the registered NMs? It seems the most common reason one would hit this is if a newbie admin set the NM resource limit to something like 1G. In that case, the admin wouldn't know to also set any max-resource-size parameter.
Or perhaps we can default the max-resource-size parameter to be equal to the advertised NM resource size, for the same effect?
> If a resource requirement is higher than available on any node, job should fail early
> -------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3946
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3946
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.24.0, 0.23.2
> Reporter: Todd Lipcon
>
> If you configure the NMs to have 1GB of RAM each, and then try to submit a job which has an AM resource requirement of 1.5GB, the job will neither run nor fail. Instead, it will slowly sop of all of the resources in the cluster as "reservations" despite the fact that it will never be able to schedule something. Instead, it should fail early indicating that the required memory allocation is infeasible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira