You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Carlo Curino (JIRA)" <ji...@apache.org> on 2016/05/27 21:40:12 UTC
[jira] [Commented] (YARN-5164) CapacityOvertimePolicy does not take
advantaged of plan RLE
[ https://issues.apache.org/jira/browse/YARN-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304849#comment-15304849 ]
Carlo Curino commented on YARN-5164:
------------------------------------
Thinking more about this, it is non-trivial to RLE-ify this policy. The policy computes for every sliding window the integral of used resources and make sure it does not exceed an integral quota. The issue with this is that even for a constant value of the user reservation (or some of reservations) the computed integral over a moving window changes.
In the attached *YARN-5164-example.pdf* we visualize this for a simple 5-sample window. The orange line changes even for constant values of the blue line.
We could in principle leverage higher-order representations (where we store RLE-encoded vectors for non-horizontal lines as starting-point + slope), but this is far from trivial, and we need to make sure the code complexity is justified. Postponing this for now.
> CapacityOvertimePolicy does not take advantaged of plan RLE
> -----------------------------------------------------------
>
> Key: YARN-5164
> URL: https://issues.apache.org/jira/browse/YARN-5164
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler, fairscheduler, resourcemanager
> Reporter: Carlo Curino
> Assignee: Carlo Curino
> Attachments: YARN-5164-example.pdf
>
>
> As a consequence small time granularities (e.g., 1 sec) and long time horizon for a reservation (e.g., months) run rather slow (10 sec).
> Proposed resolution is to switch to interval math in checking, similar to how YARN-4359 does for agents.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org