[jira] [Commented] (MAPREDUCE-6514) Update ask to indicate to RM that it need not allocate for ramped down reducers

["Rohith Sharma K S (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961789#comment-14961789 ] 

Rohith Sharma K S commented on MAPREDUCE-6514:
----------------------------------------------

Thanks [~varun_saxena] for creating issue. 

I too have seen many times this particular problem because of not updating ResourceRequest when Ramping down the reducers. As a result, number of containers count in the ask is increased as explained in the below

Consider MR job is running with 3 mapper and 100 reducers. All mappers were finished, and 3 reducers are running in the cluster.
Kill the task attempt of mapper. So Ramping down will happen.
{noformat}
2015-10-17 12:09:10,836 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping down all scheduled reduces:97
{noformat}

Since resourcerequest is not decremented from remoteRequestsTable. Next time when ask for 97 reducers i.e *pendingReducers --> scheduledReducers*, which starts adding number of containers to existing RR. So {{97(previous outstanding reducers)+ 100(total reducers)=197}}
{noformat}
2015-10-17 12:10:49,617 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps assigned. Ramping up all remaining reduces:100
2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=98 #asks=2
2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=99 #asks=2
2015-10-17 12:10:49,617 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=100 #asks=2
......
......
2015-10-17 12:10:49,647 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=195 #asks=2
2015-10-17 12:10:49,648 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=196 #asks=2
2015-10-17 12:10:49,648 DEBUG [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: addResourceRequest: applicationId=1 priority=10 resourceName=* numContainers=197 #asks=2
{noformat}

In ResourceManager.log
In the log, observe that *Containers: 100* before ramping down all the reducers. Afterwords, MR ask for reducers requests, *Containers: 197*
{noformat}
2015-10-17 12:07:40,613 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt: showRequests: application=application_1445063712158_0001 request={Priority: 10, Capability: <memory:2048, vCores:1>, # Containers: 100, Location: *, Relax Locality: true, Node Label Expression: }
{noformat}

After ramping down all the reducers and request for ask. 

{noformat}
2015-10-17 12:10:50,651 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt: showRequests: application=application_1445063712158_0001 request={Priority: 10, Capability: <memory:2048, vCores:1>, # Containers: 197, Location: *, Relax Locality: true, Node Label Expression: }
{noformat}




Basically it causes unnecessary overhead on ResourceManager that for assignment and also effect other applications container assignment. These allocated containers are rejected by MR.

> Update ask to indicate to RM that it need not allocate for ramped down reducers
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6514
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6514
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 2.7.1
>            Reporter: Varun Saxena
>            Assignee: Varun Saxena
>
> In RMContainerAllocator#preemptReducesIfNeeded, we simply clear the scheduled reduces map and put these reducers to pending. This is not updated in ask. So RM keeps on assigning and AM is not able to assign as no reducer is scheduled(check logs below the code).
> If this is updated immediately, RM will be able to schedule mappers immediately which anyways is the intention when we ramp down reducers.
> {code}
>  LOG.info("Ramping down all scheduled reduces:"
>             + scheduledRequests.reduces.size());
>         for (ContainerRequest req : scheduledRequests.reduces.values()) {
>           pendingReduces.add(req);
>         }
>         scheduledRequests.reduces.clear();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)