You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by doom <43...@qq.com> on 2020/09/08 02:22:56 UTC
Question about FSLeafQueue.updateStarvedAppsMinshare
Hi all:
i find the FSLeafQueue.updateStarvedAppsMinshare is responsible for `Distribute queue's minshare starvation to a set of apps`,
1.
so when appMinShare<=pending [ pending is queue's minshare starvation]:
pending = pending - appMinShare // allocat part of pending to appMinShare
app's MinshareStarvation = appMinShare
2.
when appMinShare>pending:
2.1 (the src code)
appMinShare = appMinShare - pending
pending = 0
app's MinshareStarvation = appMinShare
2.2(what i thought)
what confuse me is `appMinShare = appMinShare - pending`, when pending is not enough, i thought it should allocat all the rest of pending to appMinShare:
appMinShare = pending
pending = 0
app's MinshareStarvation = appMinShare
So 2.1 and 2.2 which one is right? if 2.1 is right, what dose `appMinShare = appMinShare - pending` mean?
Thanx!!!
https://github.com/apache/hadoop/blob/34fe74da0e9c68173e1de196c496b9cfca029618/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java#L250
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue
```java
/**
* Distribute minshare starvation to a set of apps
* @param appsWithDemand set of apps
* @param minShareStarvation minshare starvation to distribute
*/
private void updateStarvedAppsMinshare(
final TreeSet<FSAppAttempt> appsWithDemand,
final Resource minShareStarvation) {
Resource pending = Resources.clone(minShareStarvation);
// Keep adding apps to the starved list until the unmet demand goes over
// the remaining minshare
for (FSAppAttempt app : appsWithDemand) {
if (!Resources.isNone(pending)) {
Resource appMinShare = app.getPendingDemand();
Resources.subtractFromNonNegative(
appMinShare, app.getFairshareStarvation());
if (Resources.greaterThan(policy.getResourceCalculator(),
scheduler.getClusterResource(), appMinShare, pending)) {
Resources.subtractFromNonNegative(appMinShare, pending);
pending = none();
} else {
Resources.subtractFromNonNegative(pending, appMinShare);
}
app.setMinshareStarvation(appMinShare);
context.getStarvedApps().addStarvedApp(app);
} else {
// Reset minshare starvation in case we had set it in a previous
// iteration
app.resetMinshareStarvation();
}
}
}
```