You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Herman Chen (JIRA)" <ji...@apache.org> on 2012/06/01 23:33:24 UTC
[jira] [Created] (MAPREDUCE-4304) Deadlock where all containers are
held by ApplicationMasters should be prevented
Herman Chen created MAPREDUCE-4304:
--------------------------------------
Summary: Deadlock where all containers are held by ApplicationMasters should be prevented
Key: MAPREDUCE-4304
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4304
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2, resourcemanager
Affects Versions: 0.23.1
Reporter: Herman Chen
In my test cluster with 4 NodeManagers, each with only ~1.6G container memory, when a burst of jobs, e.g. >10, are concurrently submitted, it is likely that 4 jobs are accepted, with 4 ApplicationMasters allocated, but then the jobs block each other indefinitely because they're all waiting to allocate more containers.
Note that the problem is not limited to tiny cluster like this. As long as the number of jobs being submitted is greater than the rate jobs finish, it may run into a vicious cycle where more and more containers are locked up by ApplicationMasters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4304) Deadlock where all containers
are held by ApplicationMasters should be prevented
Posted by "Herman Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289680#comment-13289680 ]
Herman Chen commented on MAPREDUCE-4304:
----------------------------------------
I tried CapacityScheduler but I'm still running into the problem. My setup this time: there are 4 NM each with 1.5GB container memory configured. I'm using the default value of 0.1 (10%) for yarn.scheduler.capacity.maximum-am-resource-percent, and that ends up giving me a "Max Active Applications" of 5 for the default queue (initially I thought 10% would give me just 1 Application). And again when I submit 11 jobs simultaneously, one AM is allocated on each of the NM, taking up all resources (default AM size is 1.5GB). To make this work for my setup, I have to configure the property to at most 0.06, which gives me "Max Active Applications" of 3. So it looks like yes it can be done, but definitely not intuitively.
The discrepancy lies in how YARN plans resource usage versus how submitted jobs actually allocate resource. The way the maximum-am-resource-percent is used is total memory (6G) / minimum allocation (128MB) * max-am-percent (0.1) =~ 5. But this uses minimum allocation, does that make sense? For my jobs with default AM size of 1.5G, it certainly cannot support 5 applications.
It feels to me, the max-am-percent is more intuitively enforced in terms of memory, instead of translating it into number of applications. Alternatively a mechanism to preempt idle AMs that cannot allocate further resources seems valuable.
> Deadlock where all containers are held by ApplicationMasters should be prevented
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Herman Chen
>
> In my test cluster with 4 NodeManagers, each with only ~1.6G container memory, when a burst of jobs, e.g. >10, are concurrently submitted, it is likely that 4 jobs are accepted, with 4 ApplicationMasters allocated, but then the jobs block each other indefinitely because they're all waiting to allocate more containers.
> Note that the problem is not limited to tiny cluster like this. As long as the number of jobs being submitted is greater than the rate jobs finish, it may run into a vicious cycle where more and more containers are locked up by ApplicationMasters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4304) Deadlock where all containers
are held by ApplicationMasters should be prevented
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287700#comment-13287700 ]
Jason Lowe commented on MAPREDUCE-4304:
---------------------------------------
Is this with the FifoScheduler or the CapacityScheduler? If the latter then I believe the yarn.scheduler.capacity.maximum-am-resource-percent property aims to prevent this sort of problem.
> Deadlock where all containers are held by ApplicationMasters should be prevented
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Herman Chen
>
> In my test cluster with 4 NodeManagers, each with only ~1.6G container memory, when a burst of jobs, e.g. >10, are concurrently submitted, it is likely that 4 jobs are accepted, with 4 ApplicationMasters allocated, but then the jobs block each other indefinitely because they're all waiting to allocate more containers.
> Note that the problem is not limited to tiny cluster like this. As long as the number of jobs being submitted is greater than the rate jobs finish, it may run into a vicious cycle where more and more containers are locked up by ApplicationMasters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4304) Deadlock where all containers
are held by ApplicationMasters should be prevented
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287762#comment-13287762 ]
Ahmed Radwan commented on MAPREDUCE-4304:
-----------------------------------------
But, even for the capacity scheduler, what is a good strategy for tuning this property especially when there are variability in the sizes of jobs, and the resources they require? This can be really hard, and can endup in either an underutilization of the cluster or the described deadlock.
A possible approach will be preemption of AM containers if their resource requirements cannot be fulfilled within a period of time, they can go back to the queue and only restarted when there are resources. The scheduler can keep track of required resources to avoid recomputation of resources when the AM is restarted. Thoughts?
> Deadlock where all containers are held by ApplicationMasters should be prevented
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Herman Chen
>
> In my test cluster with 4 NodeManagers, each with only ~1.6G container memory, when a burst of jobs, e.g. >10, are concurrently submitted, it is likely that 4 jobs are accepted, with 4 ApplicationMasters allocated, but then the jobs block each other indefinitely because they're all waiting to allocate more containers.
> Note that the problem is not limited to tiny cluster like this. As long as the number of jobs being submitted is greater than the rate jobs finish, it may run into a vicious cycle where more and more containers are locked up by ApplicationMasters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4304) Deadlock where all containers
are held by ApplicationMasters should be prevented
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475367#comment-13475367 ]
Alejandro Abdelnur commented on MAPREDUCE-4304:
-----------------------------------------------
A way to avoid deadlocks is to use a dedicated queue for the oozie action launcher jobs, different that the queue used for the oozie action jobs themselves.
Also, launcher jobs could be configured to use little memory for the case of map-reduce actions (the launchers only submit the real job). For other action types, ie pig & hive, the launcher runs the actual client invocation (pig or hive), so launcher memory may have to higher than for map-reduce launcher action.
A more longer term approach (in combination with the above) would be to use a OozieLauncherAM thus reducing the number of containers for each launchers from 2 to 1.
> Deadlock where all containers are held by ApplicationMasters should be prevented
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Herman Chen
>
> In my test cluster with 4 NodeManagers, each with only ~1.6G container memory, when a burst of jobs, e.g. >10, are concurrently submitted, it is likely that 4 jobs are accepted, with 4 ApplicationMasters allocated, but then the jobs block each other indefinitely because they're all waiting to allocate more containers.
> Note that the problem is not limited to tiny cluster like this. As long as the number of jobs being submitted is greater than the rate jobs finish, it may run into a vicious cycle where more and more containers are locked up by ApplicationMasters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4304) Deadlock where all containers
are held by ApplicationMasters should be prevented
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287714#comment-13287714 ]
Arun C Murthy commented on MAPREDUCE-4304:
------------------------------------------
Agreed with Jason.
> Deadlock where all containers are held by ApplicationMasters should be prevented
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Herman Chen
>
> In my test cluster with 4 NodeManagers, each with only ~1.6G container memory, when a burst of jobs, e.g. >10, are concurrently submitted, it is likely that 4 jobs are accepted, with 4 ApplicationMasters allocated, but then the jobs block each other indefinitely because they're all waiting to allocate more containers.
> Note that the problem is not limited to tiny cluster like this. As long as the number of jobs being submitted is greater than the rate jobs finish, it may run into a vicious cycle where more and more containers are locked up by ApplicationMasters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4304) Deadlock where all containers
are held by ApplicationMasters should be prevented
Posted by "Herman Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287733#comment-13287733 ]
Herman Chen commented on MAPREDUCE-4304:
----------------------------------------
I am using FifoScheduler. Thanks for pointing out the CapacityScheduler property -- I will give that a try. Ideally though such state should be preventable with FifoScheduler, or any type of scheduler for that matter.
> Deadlock where all containers are held by ApplicationMasters should be prevented
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Herman Chen
>
> In my test cluster with 4 NodeManagers, each with only ~1.6G container memory, when a burst of jobs, e.g. >10, are concurrently submitted, it is likely that 4 jobs are accepted, with 4 ApplicationMasters allocated, but then the jobs block each other indefinitely because they're all waiting to allocate more containers.
> Note that the problem is not limited to tiny cluster like this. As long as the number of jobs being submitted is greater than the rate jobs finish, it may run into a vicious cycle where more and more containers are locked up by ApplicationMasters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4304) Deadlock where all containers
are held by ApplicationMasters should be prevented
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287699#comment-13287699 ]
Robert Joseph Evans commented on MAPREDUCE-4304:
------------------------------------------------
+1 this is not just for tiny clusters, it also happens on tiny queues.
> Deadlock where all containers are held by ApplicationMasters should be prevented
> --------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4304
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4304
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Herman Chen
>
> In my test cluster with 4 NodeManagers, each with only ~1.6G container memory, when a burst of jobs, e.g. >10, are concurrently submitted, it is likely that 4 jobs are accepted, with 4 ApplicationMasters allocated, but then the jobs block each other indefinitely because they're all waiting to allocate more containers.
> Note that the problem is not limited to tiny cluster like this. As long as the number of jobs being submitted is greater than the rate jobs finish, it may run into a vicious cycle where more and more containers are locked up by ApplicationMasters.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira