[jira] [Comment Edited] (MAPREDUCE-6867) ApplicationMaster hung on OOM Error

["Brahma Reddy Battula (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405770#comment-16405770 ] 

Brahma Reddy Battula edited comment on MAPREDUCE-6867 at 3/20/18 3:52 AM:
--------------------------------------------------------------------------

Looks YARN-679 covered this one part of this issue.
{quote}{{LOG.fatal("Halt called", ee)}} possibility of one more OOM exception
{quote}
Yes, as you also mentioned, it's logged only for test.so I feel, might not require here.
{quote}i think option of {{XX:OnOutOfMemoryError=\"kill -9}} is dependent on the OS. so not sure its the right approach and not aware any option is there to set based on OS type. given that we have fix in {{ExitUtil}} is it required to have this modification too?
{quote}
{{ExitUtil}} just can through the error, but JVM will not be killed right..?

Maybe we can add based on OS..?


was (Author: brahmareddy):
Looks YARN-679 covered this one part of this issue.

bq.{{LOG.fatal("Halt called", ee)}} possibility of one more OOM exception

Yes, as you also mentioned, it's logged only for test.

bq.i think option of {{XX:OnOutOfMemoryError=\"kill -9}} is dependent on the OS. so not sure its the right approach and not aware any option is there to set based on OS type. given that we have fix in {{ExitUtil}} is it required to have this modification too?

{{ExitUtil}} just can through the error, but JVM will not be killed right..? Maybe we can add based on OS..?

> ApplicationMaster hung on OOM Error
> -----------------------------------
>
>                 Key: MAPREDUCE-6867
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6867
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>            Reporter: Bilwa S T
>            Assignee: Bilwa S T
>            Priority: Major
>         Attachments: MAPREDUCE-6867.patch
>
>
> Whenever OOM Error is thown, {{YarnUncaughtExceptionHandler}} will call {{ExitUtil.halt(-1)}}.But while halting, OOM might occur which is not handled. 
> We came across a scenario where in when we submit mapreduce application ,OOM error occured in {{committerEventProcessor}} and then AM did not halt and did not log the following.Finally AM got hang since it's not thrown to main thread.
> {code}LOG.info("Halt with status " + status + " Message: " + msg);{code}
>  *org.apache.hadoop.util.ExitUtil.halt(int, String)* 
> {code}
> public static void halt(int status, String msg) throws HaltException {
>     LOG.info("Halt with status " + status + " Message: " + msg);
>     if (systemHaltDisabled) {
>       HaltException ee = new HaltException(status, msg);
>       LOG.fatal("Halt called", ee);
>       if (null == firstHaltException) {
>         firstHaltException = ee;
>       }
>       throw ee;
>     }
>     Runtime.getRuntime().halt(status);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org