[jira] [Commented] (MAPREDUCE-7180) Relaunching Failed Containers

["Daniel Templeton (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782164#comment-16782164 ] 

Daniel Templeton commented on MAPREDUCE-7180:
---------------------------------------------

This sounds like a fun discussion.  Let me throw in my 2 cents!

The 80/20 should cover most cases, except for blatant misconfiguration.  That makes this feature very niche.  In the cases where there is a misconfiguration, we want the user to fix it, not limp along on the good will of a self-healing system.

The 80/20 probably isn't going to cover 100% of the cases, and there are definitely cases where the configuration is out of the user's hands, e.g. Hive.  In those cases, which can be enumerated by the cluster owner, it would make sense to let the admin turn on the "idiot switch" to hide failures because of memory overruns behind a smart retry policy.

On the flip side, if we see an MR container fail due to memory overruns, we could save some resources by not retrying the task.  If it fails once on a memory overrun, it will fail on all the retries as well.  Just kill the job and be done.  Since YARN can't know anything about MR, it would have to be the MR AM that notices and kills itself.

I like the idea of using the power of YARN for good (for once), to hide details and reduce end-user pain.  Yeah, we should be careful to make sure it's not going to unexpectedly hide problems in a bad way, but I like that's pretty easy to do.

> Relaunching Failed Containers
> -----------------------------
>
>                 Key: MAPREDUCE-7180
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7180
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv1, mrv2
>            Reporter: BELUGA BEHR
>            Priority: Major
>
> In my experience, it is very common that a MR job completely fails because a single Mapper/Reducer container is using more memory than has been reserved in YARN.  The following message is logging the the MapReduce ApplicationMaster:
> {code}
> Container [pid=46028,containerID=container_e54_1435155934213_16721_01_003666] is running beyond physical memory limits. 
> Current usage: 1.0 GB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used. Killing container.
> {code}
> In this case, the container is re-launched on another node, and of course, it is killed again for the same reason.  This process happens three (maybe four?) times before the entire MapReduce job fails.  It's often said that the definition of insanity is doing the same thing over and over and expecting different results.
> For all intents and purposes, the amount of resources requested by Mappers and Reducers is a fixed amount; based on the default configuration values.  Users can set the memory on a per-job basis, but it's a pain, not exact, and requires intimate knowledge of the MapReduce framework and its memory usage patterns.
> I propose that if the MR ApplicationMaster detects that a container is killed because of this specific memory resource constraint, that it requests a larger container for the subsequent task attempt.
> For example, increase the requested memory size by 50% each time the container fails and the task is retried.  This will prevent many Job failures and allow for additional memory tuning, per-Job, after the fact, to get better performance (v.s. fail/succeed).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org