You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Zhiyuan Yang (JIRA)" <ji...@apache.org> on 2017/04/11 01:27:41 UTC
[jira] [Commented] (TEZ-3274) Vertex with MRInput and shuffle input
does not respect slow start
[ https://issues.apache.org/jira/browse/TEZ-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15963717#comment-15963717 ]
Zhiyuan Yang commented on TEZ-3274:
-----------------------------------
[~ebadger] Although I don't know what algorithm requires this dag shape, I assume there exists one since you guys tried to make it work. It would help if the job details can be shown. I don't think it's right to make existing ShuffleVertexManager work with this scenario because data exchange pattern from MRInput is apparently not shuffle. I'm +1 for using custom vertex manager (and possibly reuse ShuffleVertexManager code).
> Vertex with MRInput and shuffle input does not respect slow start
> -----------------------------------------------------------------
>
> Key: TEZ-3274
> URL: https://issues.apache.org/jira/browse/TEZ-3274
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Eric Badger
> Attachments: TEZ-3274.001.patch, TEZ-3274.002.patch
>
>
> Vertices with shuffle input and MRInput choose RootInputVertexManager (and not ShuffleVertexManager) and start containers and tasks immediately. In this scenario, resources can be wasted since they do not respect tez.shuffle-vertex-manager.min-src-fraction tez.shuffle-vertex-manager.max-src-fraction.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)