You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Arpit Gupta (JIRA)" <ji...@apache.org> on 2014/03/28 21:59:15 UTC
[jira] [Commented] (TEZ-997) Internal Errror in am logs during dag
shutdown
[ https://issues.apache.org/jira/browse/TEZ-997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951375#comment-13951375 ]
Arpit Gupta commented on TEZ-997:
---------------------------------
Here is the stack trace
{code}
014-03-28 09:47:20,545 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.dag.impl.VertexImpl: Num completed Tasks for vertex_1395999590378_0001_1_02 [Reducer 2] : 37
2014-03-28 09:47:20,545 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.dag.impl.VertexImpl: Num completed Tasks for vertex_1395999590378_0001_1_02 [Reducer 2] : 38
2014-03-28 09:47:20,545 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.dag.impl.VertexImpl: Num completed Tasks for vertex_1395999590378_0001_1_02 [Reducer 2] : 39
2014-03-28 09:47:20,545 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.dag.impl.VertexImpl: Num completed Tasks for vertex_1395999590378_0001_1_02 [Reducer 2] : 40
2014-03-28 09:47:20,545 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.dag.impl.VertexImpl: Num completed Tasks for vertex_1395999590378_0001_1_02 [Reducer 2] : 41
2014-03-28 09:47:20,545 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.dag.impl.VertexImpl: Num completed Tasks for vertex_1395999590378_0001_1_02 [Reducer 2] : 42
2014-03-28 09:47:20,545 INFO [AsyncDispatcher event handler] org.apache.tez.dag.app.dag.impl.VertexImpl: Num completed Tasks for vertex_1395999590378_0001_1_02 [Reducer 2] : 43
2014-03-28 09:47:20,546 FATAL [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
org.apache.tez.dag.api.TezUncheckedException: All tasks complete, but cannot determine final state of vertex, failedTaskCount=0, killedTaskCount=43, successfulTaskCount=0, completedTaskCount=43, terminationCause=INTERNAL_ERROR
at org.apache.tez.dag.app.dag.impl.VertexImpl.checkVertexForCompletion(VertexImpl.java:1405)
at org.apache.tez.dag.app.dag.impl.VertexImpl$TaskCompletedTransition.transition(VertexImpl.java:2707)
at org.apache.tez.dag.app.dag.impl.VertexImpl$TaskCompletedTransition.transition(VertexImpl.java:2684)
at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1203)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:155)
at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1581)
at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1567)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
at java.lang.Thread.run(Thread.java:722)
2014-03-28 09:47:20,546 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
2014-03-28 09:47:20,548 INFO [Thread-2] org.apache.tez.dag.app.DAGAppMaster: DAGAppMaster received a signal. Signaling TaskScheduler
2014-03-28 09:47:20,548 INFO [Thread-2] org.apache.tez.dag.app.rm.TaskSchedulerEventHandler: TaskScheduler notified that iSignalled was : true
2014-03-28 09:47:20,549 INFO [Thread-2] org.apache.tez.dag.history.HistoryEventHandler: Stopping HistoryEventHandler
2014-03-28 09:47:20,549 INFO [Thread-2] org.apache.tez.dag.history.recovery.RecoveryService: Stopping RecoveryService
2014-03-28 09:47:20,549 INFO [Thread-2] org.apache.tez.dag.history.recovery.RecoveryService: Closing Summary Stream
2014-03-28 09:47:20,549 INFO [RecoveryEventHandlingThread] org.apache.tez.dag.history.recovery.RecoveryService: EventQueue take interrupted. Returning
2014-03-28 09:47:20,574 INFO [Thread-2] org.apache.tez.dag.app.launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1395999590378_0001_02_000048
2014-03-28 09:47:20,585 INFO [Thread-2] org.apache.tez.dag.app.launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1395999590378_0001_02_000052
2014-03-28 09:47:20,589 INFO [Thread-2] org.apache.tez.dag.app.launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1395999590378_0001_02_000032
2014-03-28 09:47:20,591 INFO [Thread-2] org.apache.tez.dag.app.launcher.ContainerLauncherImpl: Sending a stop request to the NM for ContainerId: container_1395999590378_0001_02_000006
{code}
> Internal Errror in am logs during dag shutdown
> ----------------------------------------------
>
> Key: TEZ-997
> URL: https://issues.apache.org/jira/browse/TEZ-997
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Arpit Gupta
>
> During a hive on tez job which failed we saw an exception trace during dag shutdown.
--
This message was sent by Atlassian JIRA
(v6.2#6252)