[jira] [Comment Edited] (TEZ-2914) Ability to limit vertex concurrency

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034642#comment-15034642 ] 

Jason Lowe edited comment on TEZ-2914 at 12/1/15 9:44 PM:
----------------------------------------------------------

One advantage to doing it at the YARN level is that we can tell the RM the full ask of the job then limit the ANY (\*) request level to just the limit we need.  Then the RM will give us good locality based on what the cluster has available.  If we try to limit the tasks before sending the request then we may get poor locality when better was available since the Tez AM doesn't know what's free right now.  For example, we might have great locality for tasks 7, 8, and 9, but if we only ask the RM for tasks 0, 1, and 2 we could end up with bad locality for those tasks and others in the job (with container reuse).

I'm not sure we'd have to buffer the requests, rather we'd have to make sure the ANY request level acts as the rate-limiter.  We could send all the rack-level and host-level asks to the RM but then have to clamp down on the maximum ANY containers to keep the total number of containers assigned to the desired concurrency.  That's how MAPREDUCE-5583 works to get better locality than would be possible by trying to selectively avoid which tasks are requested to the RM.

I'll defer to your judgement which is better to go with long-term, but I wanted to point out that we would be sacrificing some locality by having the Tez AM avoid making the full host- and rack-local requests up front.


was (Author: jlowe):
One advantage to doing it at the YARN level is that we can tell the RM the full ask of the job then limit the ANY (*) request level to just the limit we need.  Then the RM will give us good locality based on what the cluster has available.  If we try to limit the tasks before sending the request then we may get poor locality when better was available since the Tez AM doesn't know what's free right now.  For example, we might have great locality for tasks 7, 8, and 9, but if we only ask the RM for tasks 0, 1, and 2 we could end up with bad locality for those tasks and others in the job (with container reuse).

I'm not sure we'd have to buffer the requests, rather we'd have to make sure the ANY request level acts as the rate-limiter.  We could send all the rack-level and host-level asks to the RM but then have to clamp down on the maximum ANY containers to keep the total number of containers assigned to the desired concurrency.  That's how MAPREDUCE-5583 works to get better locality than would be possible by trying to selectively avoid which tasks are requested to the RM.

I'll defer to your judgement which is better to go with long-term, but I wanted to point out that we would be sacrificing some locality by having the Tez AM avoid making the full host- and rack-local requests up front.

> Ability to limit vertex concurrency
> -----------------------------------
>
>                 Key: TEZ-2914
>                 URL: https://issues.apache.org/jira/browse/TEZ-2914
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Bikas Saha
>
> Add equivalent functionality of *MAPREDUCE-5583. Ability to limit running map and reduce tasks* in tez



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)