[jira] [Commented] (TEZ-1486) TezUncheckedException when using dynamic partition pruning

["Gunther Hagleitner (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14108754#comment-14108754 ] 

Gunther Hagleitner commented on TEZ-1486:
-----------------------------------------

Actually I just found this is independent of dynamic partition pruning. Fails also when static pruning generates a 0-task vertex.

> TezUncheckedException when using dynamic partition pruning
> ----------------------------------------------------------
>
>                 Key: TEZ-1486
>                 URL: https://issues.apache.org/jira/browse/TEZ-1486
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.5.0
>            Reporter: Gunther Hagleitner
>            Assignee: Siddharth Seth
>
> I'm working on using the AM event mechanism to dynamically prune partitions at DAG runtime for certain queries. The query is:
> select count(*) from srcpart join srcpart_double_hour on (srcpart.hr*2 = srcpart_double_hour.hr) where srcpart_double_hour.hour = 11;
> This will result in two vertices connected through a broadcast edge. The vertex prepares two things: The list of partition keys (hr) that are being sent to the AM for dynamic pruning and the records to be used in the hash join.
> The second vertex will block until all events are received (initializer) then it will load and process the hash join.
> It's possible for queries like this to result in zero splits on the second vertex (i.e.: no matching rows for the join)
> The exception I get when this is run is:
> org.apache.tez.dag.api.TezUncheckedException: Event must be routed. sourceVertex=vertex_1408686217936_0003_3_00 srcIndex = 0 destAttemptId=vertex_1408686217936_0003_3_01 edgeManager=org.apache.tez.dag.app.dag.impl.BroadcastEdgeManager Ev\
> ent type=DATA_MOVEMENT_EVENT
>   at org.apache.tez.dag.app.dag.impl.Edge.sendTezEventToDestinationTasks(Edge.java:371)
>   at org.apache.tez.dag.app.dag.impl.VertexImpl$RouteEventTransition.transition(VertexImpl.java:3372)
>   at org.apache.tez.dag.app.dag.impl.VertexImpl.scheduleTasks(VertexImpl.java:1088)
>   at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerPluginContextImpl.scheduleVertexTasks(VertexManager.java:111)
>   at org.apache.tez.dag.app.dag.impl.ImmediateStartVertexManager.onVertexStarted(ImmediateStartVertexManager.java:49)
>   at org.apache.tez.dag.app.dag.impl.VertexManager.onVertexStarted(VertexManager.java:244)
>   at org.apache.tez.dag.app.dag.impl.VertexImpl.startVertex(VertexImpl.java:2923)
>   at org.apache.tez.dag.app.dag.impl.VertexImpl.access$5900(VertexImpl.java:169)
>   at org.apache.tez.dag.app.dag.impl.VertexImpl$StartTransition.transition(VertexImpl.java:2914)
>   at org.apache.tez.dag.app.dag.impl.VertexImpl$StartTransition.transition(VertexImpl.java:2906)
>   at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
>   at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>   at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
>   at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
>   at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1355)
>   at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:168)
>   at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1650)
>   at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1636)
>   at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
>   at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
>   at java.lang.Thread.run(Thread.java:695)



--
This message was sent by Atlassian JIRA
(v6.2#6252)