[jira] [Comment Edited] (OOZIE-3307) oozie workflow gets failed throwing error virtual memory limits

["Peter Bacsko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16602847#comment-16602847 ] 

Peter Bacsko edited comment on OOZIE-3307 at 9/4/18 10:07 AM:
--------------------------------------------------------------

That's right, we do set {{-Xmx}} automatically in MR-based Oozie: [https://github.com/apache/oozie/blob/branch-4.3/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L382-L415] 

It's missing from the Oozie-on-YARN based implementation. Perhaps, based on the resource request, we could limit {{-Xmx}} as  80-90 % of the requested memory to make sure that OOME happens before the container is killed by YARN.


was (Author: pbacsko):
That's right, we do set {{-Xmx}} automatically in MR-based Oozie: [https://github.com/apache/oozie/blob/branch-4.3/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L382-L415]

 

It's missing from the Oozie-on-YARN based implementation. Perhaps, based on the resource request, we could limit {{-Xmx}} as  80-90 % of the requested memory to make sure that OOME happens before the container is killyed by YARN.

> oozie workflow gets failed throwing error virtual memory limits
> ---------------------------------------------------------------
>
>                 Key: OOZIE-3307
>                 URL: https://issues.apache.org/jira/browse/OOZIE-3307
>             Project: Oozie
>          Issue Type: Bug
>    Affects Versions: 5.0.0
>            Reporter: Sabir Naikwadi
>            Priority: Critical
>
> Application application_1531909575787_0039 failed 2 times due to AM Container for appattempt_1531909575787_0039_000002 exited with exitCode: -103
>  Failing this attempt.Diagnostics: Container [pid=11516,containerID=container_1531909575787_0039_02_000001] is running beyond virtual memory limits. Current usage: 469.8 MB of 2 GB physical memory used; 10.0 GB of 10 GB virtual memory used. Killing container.
>  Dump of the process-tree for container_1531909575787_0039_02_000001 :
> | - PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|
> | - 11516 11514 11516 11516 (bash) 1 3 115863552 682 /bin/bash -c /usr/lib/jvm/java-openjdk/bin/java -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_000001 -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev org.apache.oozie.action.hadoop.LauncherAM 1>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_000001/stdout 2>/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_000001/stderr|
> | - 11755 11516 11516 11516 (java) 1142 71 10658242560 119576 /usr/lib/jvm/java-openjdk/bin/java -Dlog4j.configuration=container-log4j.properties -Dlog4j.debug=true -Dyarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1531909575787_0039/container_1531909575787_0039_02_000001 -Dyarn.app.container.log.filesize=1048576 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dsubmitter.user=dev org.apache.oozie.action.hadoop.LauncherAM
>  Container killed on request. Exit code is 143
>  Container exited with a non-zero exit code 143
>  For more detailed output, check the application tracking page: [http://ip-10-20-201-36.us-gov-west-1.compute.internal:8088/cluster/app/application_1531909575787_0039] Then click on links to logs of each attempt.
>  . Failing the application.|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)