[jira] [Commented] (TEZ-3269) Provide basic fair routing and scheduling functionality via custom VertexManager and EdgeManager

["Zhiyuan Yang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542944#comment-15542944 ] 

Zhiyuan Yang commented on TEZ-3269:
-----------------------------------

I'm sorry for the late review...
Generally code looks good as core parts are mostly same wtih TEZ-3209 patch. Just some minor things:
In FairShuffleVertexManager:
{code:java}
if (pendingTasks.size() == 0) {
      // don't change routing when number of tasks is set to zero.
      return null;
}
{code}
Given initial parallelism of reducer determines the number of partitions of source vertices output, why pendingTasks.size() would be zero?  

In Test FairShuffleVertexManager:
{code:java}
// The 2nd destination task fetches one partition from the first source
// task.
// The 3rd destination task fetches one partition from the 2nd and 3rd
// source task.
{code}
The comments doesn't match code.

{code:java}
Assert.assertTrue(manager.config.isAutoParallelismEnabled() == true);
{code}
'== true' can be removed.

> Provide basic fair routing and scheduling functionality via custom VertexManager and EdgeManager
> ------------------------------------------------------------------------------------------------
>
>                 Key: TEZ-3269
>                 URL: https://issues.apache.org/jira/browse/TEZ-3269
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Ming Ma
>         Attachments: TEZ-3269-2.patch, TEZ-3269.patch
>
>
> With TEZ-3206 and TEZ-3216, we can build a custom VertexManager and EdgeManager that uses partition stats to do fair routing as well as the scheduling based on destination tasks’ dependency on source tasks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)