[jira] [Comment Edited] (TEZ-1750) Add a DAGScheduler which schedules tasks only when sources have been scheduled

["Siddharth Seth (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203590#comment-14203590 ] 

Siddharth Seth edited comment on TEZ-1750 at 11/8/14 8:52 PM:
--------------------------------------------------------------

Updated with expert level setting.

There's comments in their on some future enhancements - considering cluster capacity to allow scheduling downstream even if upstream is not scheduled, generic slow start.

Not enabling this by default in 0.5.3, because I think it's not a good change to have in place on a minor version. On 0.6.0, based on how this performs, it could be enabled by default.


was (Author: sseth):
Updated with experimental.

There's comments in their on some future enhancements - considering cluster capacity to allow scheduling downstream even if upstream is not scheduled, generic slow start.

Not enabling this by default in 0.5.3, because I think it's not a good change to have in place on a minor version. On 0.6.0, based on how this performs, it could be enabled by default.

> Add a DAGScheduler which schedules tasks only when sources have been scheduled
> ------------------------------------------------------------------------------
>
>                 Key: TEZ-1750
>                 URL: https://issues.apache.org/jira/browse/TEZ-1750
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>            Priority: Critical
>         Attachments: TEZ-1750.1.txt, TEZ-1750.2.txt, TEZ-1750.3.txt
>
>
> Splitting out the patch on TEZ-1522 into a separate jira.
> There's several scenarios in which we end up scheduling downstream tasks before their sources have been scheduled - and then get into a situation where the sources are starved. Currently, anywhere a ShuffleVertexManager is used can cause such behaviour - since it starts scheduling it's tasks after a certain number of sources are complete, but subsequen non-shuffle VertexManagers will scheduled immediately.
> Disabling slow-start is one option to achieve this (or setting slow start on all vertices), but it doesn't work for the situation where dynamic reducer parallelism kicks in - since it has to wait for source tasks to complete.
> The intent here is to add a DAGScheduler, which affectively negates the slow start, and in case of dynamic parallelism determination, waits for upstream tasks to be scheduled before scheduling downstream tasks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)