You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Arun Suresh (JIRA)" <ji...@apache.org> on 2016/05/18 13:31:13 UTC
[jira] [Updated] (MAPREDUCE-6703) Add flag to allow MapReduce AM to
request for OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun Suresh updated MAPREDUCE-6703:
-----------------------------------
Attachment: MAPREDUCE-6703.001.patch
Uploading initial patch
* Changes to allow the MRAppMaster to request a mix of *OPPORTUNISTIC* and *GUARANTEED* containers for it first attempt map tasks (reduce tasks and maps to be retried are always GUARANTEED)
* User specifies number of *OPPORTUNISTIC* Containers via the *mapreduce.job.num-opportunistic-maps-per-100* job conf : Default value is 0, which implies all maps will be guaranteed. A value of 100 means all maps will be requested as opportunistic. For any other value say _x_, the *FIRST* _x_ maps requested by the AM will be opportunistic. If the total number of maps for the job < _x_, then *ALL* maps will be opportunistic
* The patch also contains a test case {{TestMROpportunisticMaps}} that uses the *MiniYARNCluster* to run a simple external sorting MR job using the above flag.
*PS* : This feature currently requires Distributed Scheduling (YARN-2885), NM Queuing (YARN-2883) and the AMRMProxyService (YARN-2884) to be turned on in the yarn cluster.
> Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-6703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Arun Suresh
> Assignee: Arun Suresh
> Attachments: MAPREDUCE-6703.001.patch
>
>
> YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes and specifically OPPORTUNISTIC containers.
> The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to provide hints via config to the MR framework as to the number of containers it would like to schedule as OPPORTUNISTIC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org