You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Bikas Saha (JIRA)" <ji...@apache.org> on 2013/05/31 01:23:19 UTC
[jira] [Commented] (TEZ-170) DAGSchedulerMRR does not schedule
tasks with a vertex distance > 1
[ https://issues.apache.org/jira/browse/TEZ-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670950#comment-13670950 ]
Bikas Saha commented on TEZ-170:
--------------------------------
For MRR, the third stage cannot start until the first stage the done. Until the first stage is complete, no output can be generated by any task of the second stage. This is by definition of MR. Until some output is generated by the second stage it makes no sense to start tasks from the third stage. Thus for MRR one can only have 2 concurrent stages in theory. That is what is being enforced. If tasks from the third stage are being started then the bug lies somewhere else in the start logic.
> DAGSchedulerMRR does not schedule tasks with a vertex distance > 1
> ------------------------------------------------------------------
>
> Key: TEZ-170
> URL: https://issues.apache.org/jira/browse/TEZ-170
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Labels: TEZ-0.2.0
>
> If cluster capacity is available, the scheduler should just allow multiple stages to be scheduled.
> With
> tez.slowstart-vertex-scheduler.min-src-fraction and tez.slowstart-vertex-scheduler.max-src-fraction set to 0, it ends up generating the following exception.
> {code}
> org.apache.tez.dag.api.TezException: vertex_1369944891811_12_1_000002 is neither the current partitioner: vertex_1369944891811_12_1_000000 nor the current shuffler: vertex_1369944891811_12_1_000001
> at org.apache.tez.dag.app.dag.impl.DAGSchedulerMRR.scheduleTask(DAGSchedulerMRR.java:102)
> at org.apache.tez.dag.app.dag.impl.DAGImpl$DAGSchedulerUpdateTransition.transition(DAGImpl.java:1203)
> at org.apache.tez.dag.app.dag.impl.DAGImpl$DAGSchedulerUpdateTransition.transition(DAGImpl.java:1196)
> at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:359)
> at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:299)
> at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
> at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:591)
> at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:99)
> at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:922)
> at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:918)
> at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
> at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
> at java.lang.Thread.run(Thread.java:680)
> {code}
> Bikas, could you please take a look ?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira