You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Wilfred Spiegelenburg (JIRA)" <ji...@apache.org> on 2018/10/02 11:19:00 UTC
[jira] [Commented] (YARN-8059) Resource type is ignored when FS
decide to preempt
[ https://issues.apache.org/jira/browse/YARN-8059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16635312#comment-16635312 ]
Wilfred Spiegelenburg commented on YARN-8059:
---------------------------------------------
Thank you for adding the resource types into the pre-emption. When I look at the patch there is a lot of code in there that has nothing to do with the pre-emption about 3/4 of the change is adding the metrics for the custom resource types. I don't think it is a good idea to mix the two things. We should split this issue in two steps: first add the metrics change and then on top of that fix this.
It will also allow us to first refactor the metrics test code, YARN-8750, then add the metrics for resource types including tests (new jira) on top of that. These changes are independent of the pre-emption changes. As a third step we can then add the pre-emption code.
I also have comments on the QueueMetrics class changes and the new QueueMetricsForCustomResources class. I will keep them for the new jira.
> Resource type is ignored when FS decide to preempt
> --------------------------------------------------
>
> Key: YARN-8059
> URL: https://issues.apache.org/jira/browse/YARN-8059
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
> Affects Versions: 3.0.0
> Reporter: Yufei Gu
> Assignee: Szilard Nemeth
> Priority: Major
> Attachments: YARN-8059.001.patch
>
>
> Method Fairscheduler#shouldAttemptPreemption doesn't consider resources other than vcore and memory. We may need to rethink it in the resource type scenario. cc [~miklos.szegedi@cloudera.com], [~wilfreds] and [~snemeth].
> {code}
> if (context.isPreemptionEnabled()) {
> return (context.getPreemptionUtilizationThreshold() < Math.max(
> (float) rootMetrics.getAllocatedMB() /
> getClusterResource().getMemorySize(),
> (float) rootMetrics.getAllocatedVirtualCores() /
> getClusterResource().getVirtualCores()));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org