You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Carlo Curino (JIRA)" <ji...@apache.org> on 2016/05/28 00:13:12 UTC
[jira] [Comment Edited] (YARN-5164) CapacityOvertimePolicy does not
take advantaged of plan RLE
[ https://issues.apache.org/jira/browse/YARN-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305014#comment-15305014 ]
Carlo Curino edited comment on YARN-5164 at 5/28/16 12:12 AM:
--------------------------------------------------------------
Given some convo with [~chris.douglas] there might be a workaround if we avoid to report the exact point of violation. The intuition is to compute separately the entire set of integral "additions" and "removals" separately, and then use our usual {{RLESparseResourceAllocation.merge()}} to compute the correct integral and compare it to the avg constraint. This could work (passes all the tests), but we should dig deeper in corner-cases, as it is tricky. Posting an initial patch (fresh our of hacking it) to see if folks can help spot issues. I will circle back on this.
There is however a risk of false-positive/false-negatives during the transitions (as we are not capturing the "slope" of the integral, per our previous comment).
was (Author: curino):
Given some convo with [~chris.douglas] there might be a workaround if we avoid to report the exact point of violation. The intuition is to compute separately the entire set of integral "additions" and "removals" separately, and then use our usual {{RLESparseResourceAllocation.merge()}} to compute the correct integral and compare it to the avg constraint. This could work (passes all the tests), but we should dig deeper in corner-cases, as it is tricky. Posting an initial patch (fresh our of hacking it) to see if folks can help spot issues. I will circle back on this.
> CapacityOvertimePolicy does not take advantaged of plan RLE
> -----------------------------------------------------------
>
> Key: YARN-5164
> URL: https://issues.apache.org/jira/browse/YARN-5164
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler, fairscheduler, resourcemanager
> Reporter: Carlo Curino
> Assignee: Carlo Curino
> Attachments: YARN-5164-example.pdf, YARN-5164.1.patch
>
>
> As a consequence small time granularities (e.g., 1 sec) and long time horizon for a reservation (e.g., months) run rather slow (10 sec).
> Proposed resolution is to switch to interval math in checking, similar to how YARN-4359 does for agents.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org