[jira] [Commented] (TEZ-2914) Ability to limit vertex concurrency

["Bikas Saha (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034325#comment-15034325 ] 

Bikas Saha commented on TEZ-2914:
---------------------------------

At least 2 ways to do this.
1) Make the change in the yarn task scheduler - now that we have external facing id's we could buffer requests by vertexId and add them to the actual scheduling based on allowed concurrency. Cons would be that this would make this a user land feature. The buffering has to be done before the request reaches AMRMCient as otherwise, the combination of YARN requests and internal concurrency limit would be difficult to coordinate.
2) Make the change in the DAGScheduler. The intent of the dag scheduler was supposed to be the central choke point where resource sharing and priority between vertices overall could be applied. Requests that go to it from attempts could be buffered after reaching the max desired concurrency. This would allow vertex concurrency limit, overall dag concurrency limit and other combinations.

> Ability to limit vertex concurrency
> -----------------------------------
>
>                 Key: TEZ-2914
>                 URL: https://issues.apache.org/jira/browse/TEZ-2914
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Bikas Saha
>
> Add equivalent functionality of *MAPREDUCE-5583. Ability to limit running map and reduce tasks* in tez



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)