You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Bikas Saha (JIRA)" <ji...@apache.org> on 2013/11/15 07:37:26 UTC
[jira] [Created] (TEZ-618) out of memory diagnostics not showing up
in final app diagnostics
Bikas Saha created TEZ-618:
------------------------------
Summary: out of memory diagnostics not showing up in final app diagnostics
Key: TEZ-618
URL: https://issues.apache.org/jira/browse/TEZ-618
Project: Apache Tez
Issue Type: Bug
Reporter: Bikas Saha
These diagnostics can be seen in the AM logs
2013-11-14 03:50:38,211 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.rm.container.AMContainerImpl: Container container_1383190789764_1762_01_000028 exited with diagnostics set to Container [pid=3528, containerID=container_1383190789764_1762_01_000028] is running beyond physical memory limits. Current usage: 4.1 GB of 4 GB physical memory used; 4.9 GB of 40 GB virtual memory used. Killing container.
Dump of the process-tree for container_1383190789764_1762_01_000028 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 3583 3528 3528 3528 (java) 20507 1138 5132771328 1068425 /grid/0/jdk/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics. log.level=WARN -server -Xmx4096m -Djava.net.preferIPv4Stack=true -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app. container.log.dir=/grid/2/cluster/yarn/logs/application_1383190789764_1762/container_1383190789764_1762_01_000028 -Dtez.root. logger=INFO,CLA -Djava.io.tmpdir=/grid/2/cluster/yarn/local/usercache/jenkins/appcache/application_1383190789764_1762/ container_1383190789764_1762_01_000028/tmp org.apache.hadoop.mapred.YarnTezDagChild 172.19.0.50 60163 container_1383190789764_1762_01_000028 application_1383190789764_1762 1
But overall diagnostics reported are
Status: Failed
Vertex failed, vertexName=Reducer 5, vertexId=vertex_1383190789764_1762_1_03, diagnostics=[Task failed, taskId=task_1383190789764_1762_1_03_000000, diagnostics=[AttemptID:attempt_1383190789764_1762_1_03_000000_0 Info: Containercontainer_1383190789764_1762_01_000153 received a STOP_REQUEST
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
, AttemptID:attempt_1383190789764_1762_1_03_000000_1 Info:Container container_1383190789764_1762_01_000228 received a STOP_REQUEST Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
, AttemptID:attempt_1383190789764_1762_1_03_000000_2 Info:Container container_1383190789764_1762_01_000248 received a STOP_REQUEST
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
, AttemptID:attempt_1383190789764_1762_1_03_000000_3 Info:Container container_1383190789764_1762_01_000258 received a STOP_REQUEST], Vertex failed as one or more tasks failed. failedTasks:1]
Vertex killed, vertexName=Map 3, vertexId=vertex_1383190789764_1762_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex killed as other vertex failed. failedTasks:0]
Vertex killed, vertexName=Reducer 4, vertexId=vertex_1383190789764_1762_1_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex killed as other vertex failed. failedTasks:0]
DAG failed due to vertex failure. failedVertices:1 killedVertices:2
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask
--
This message was sent by Atlassian JIRA
(v6.1#6144)