You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Yingda Chen (JIRA)" <ji...@apache.org> on 2018/10/05 19:39:00 UTC
[jira] [Updated] (TEZ-3999) Extend VertexManagerPlugin interface to
allow for relevant events notification
[ https://issues.apache.org/jira/browse/TEZ-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yingda Chen updated TEZ-3999:
-----------------------------
External issue ID: TEZ-3997
> Extend VertexManagerPlugin interface to allow for relevant events notification
> ------------------------------------------------------------------------------
>
> Key: TEZ-3999
> URL: https://issues.apache.org/jira/browse/TEZ-3999
> Project: Apache Tez
> Issue Type: Task
> Reporter: Yingda Chen
> Priority: Major
>
> This is an umbrella task of TEZ-3997
> **For concurrent connection, the downstream and upstream vertices would be running concurrently, and in some cases, they would be scheduled at the same time as well, such as (sub-graph) gang scheduling. However, *this is not always true*. In the example in Fig. 1, tasks in PS vertex should be running before tasks in W vertex should be scheduled. Since otherwise if the resource requests for PS cannot be fulfilled first, W will be spinning in vain. In other examples, as long as part of tasks in upstream vertex are running, we can start scheduling downstream tasks.
>
> In other words, if we put this into the context of existing interface/implementation of VertexMangerPlugin, we can see strong duality of “OnSourceTaskRunning” for concurrent connection vs the “OnSourceTaskCompleted” for (existing) sequential connection. Therefore, we propose an addition of “_onConcurrentSourceTaskRunning(TaskAttemptIdentifer attempt)_” interface to the VertexManager Plugin, with default implementation being not supported.
> This change will also include the logic to add source task running event and to send such events to downstream vertices. To reduce unnecessary event traffic, we will limit the sending of such events to CONCURRENT edge, and when the ConcurrentSchedulingType is specified to be SOURCE_TASK_STARTED .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)