[jira] [Commented] (TEZ-3274) Vertex with MRInput and shuffle input does not respect slow start

["Eric Badger (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939286#comment-15939286 ] 

Eric Badger commented on TEZ-3274:
----------------------------------

[~sseth], I think I have a pretty good understanding of the slow start library and changing that should be fairly trivial. However, I'm struggling a little bit with the best way to wield the library when both MRInput and shuffle input are present for a vertex. Currently they will default to the RootInputVertexManager. It's basically a contradiction to call slow start here since it derives from the ImmediateStartVertexManager. So do you think it would be easier to extend the ShuffleVertexManager to include both shuffle input *and* vertices with both shuffle input and MRInput or to create a new vertex manager specifically for this case?

It seems that the vertex managers are all very similar and reuse a bunch of code. So to me there could definitely be some additional library/refactoring work here (or only have a single vertex manager handle all cases, but that seems to be going in the wrong direction). 

cc [~jeagles]

> Vertex with MRInput and shuffle input does not respect slow start
> -----------------------------------------------------------------
>
>                 Key: TEZ-3274
>                 URL: https://issues.apache.org/jira/browse/TEZ-3274
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Eric Badger
>         Attachments: TEZ-3274.001.patch, TEZ-3274.002.patch
>
>
> Vertices with shuffle input and MRInput choose RootInputVertexManager (and not ShuffleVertexManager) and start containers and tasks immediately. In this scenario, resources can be wasted since they do not respect tez.shuffle-vertex-manager.min-src-fraction tez.shuffle-vertex-manager.max-src-fraction. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)