You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2017/02/22 14:54:44 UTC
[jira] [Updated] (MAPREDUCE-6849) Preemption by container priority
can be problematic for MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated MAPREDUCE-6849:
----------------------------------
Summary: Preemption by container priority can be problematic for MapReduce (was: CapacityScheduler preemption by container priority can be problematic for MapReduce)
I agree the first step to fixing this is getting the MR AM to react to the preemption messages. We cannot in a general sense solve these kinds of problems strictly from the YARN side, as YARN does not know the true costs and dependency graph of application containers. I still think there are issues with fragmentation that will be aggravated as container sizes approach node sizes, but those are secondary issues that can be cleaned up in separate JIRAs.
> Preemption by container priority can be problematic for MapReduce
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-6849
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6849
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Reporter: Jason Lowe
>
> A MapReduce job with thousands of reducers and just a couple of maps left to go was running in a preemptable queue. Periodically other queues would get busy and the RM would preempt some resources from the job, but it _always_ picked the job's map tasks first because they use the lowest priority containers. Even though the reducers had a shorter running time, most were spared but the maps were always shot. Since the map tasks ran for a longer time than the preemption period, the job was in a perpetual preemption loop.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org