You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/10/12 11:10:00 UTC
[jira] [Commented] (YARN-12022) CapacityOverTimePolicy rejects reservations within the quota when a user allocation is in progress at the end of the checked window
[ https://issues.apache.org/jira/browse/YARN-12022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18126273#comment-18126273 ]
ASF GitHub Bot commented on YARN-12022:
---------------------------------------
joseluisll opened a new pull request, #8815:
URL: https://github.com/apache/hadoop/pull/8815
### Description of PR
`CapacityOverTimePolicy#validate` rejects reservations that are well within the user's average and instantaneous limits. There are two causes:
1. **Integral overflow when an allocation is open at the end of the window.** The user's consumption is read with `getRangeOverlapping(checkStart, checkEnd)`. This keeps the start of an allocation that is still in progress at `checkEnd`, but not its end. The merged view went up to `Long.MAX_VALUE`, so that allocation stayed open until `Long.MAX_VALUE` and the integral computation overflowed. This happens when the user has an allocation in progress at the end of the checked window (reservation end + validity window), or a periodic allocation that crosses the period boundary.
Fix: end the merge at `checkEnd + validWindow` instead of `Long.MAX_VALUE`. Only the integral over `[checkStart, checkEnd]` is checked, and it depends only on points up to the time it is evaluated at. So the check is the same as before in that range, without the overflow. Ending the merge at `checkEnd` would not work: it adds a point inside the checked range, where the approximated integral overestimates the window and could still reject reservations within the quota. A code comment explains this.
2. **Old version counted twice on update.** `InMemoryPlan#updateReservation` validates the new version while the old one is still in the plan, so the user's consumption already includes it. `validate` then merged the old version in again with `RLEOperator.add` instead of removing it, and updates well within the average limit were rejected. Fix: use `RLEOperator.subtract`. This matches the existing comment above that code and what `availableResources()` already does.
### How was this patch tested?
Three new tests in `TestCapacityOverTimePolicy`, each of which fails without the fix:
- `testUserAllocationInProgressAtEndOfCheckedWindow`: a reservation within the quota is accepted while the user has an allocation in progress past the checked window.
- `testPeriodicAllocationCrossingPeriodBoundary`: same, with a periodic allocation that crosses the period boundary.
- `testUpdateReservationWithinQuota`: an update within the quota is accepted, and an update over the limit is still rejected with `PlanningQuotaException`.
The existing `TestCapacityOverTimePolicy` and reservation tests pass.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation? *(N/A)*
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? *(N/A, no new dependencies)*
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? *(N/A)*
### AI Tooling
- [x] The PR includes the phrase "Contains content generated by Claude Code"
- [x] My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Code.
> CapacityOverTimePolicy rejects reservations within the quota when a user allocation is in progress at the end of the checked window
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-12022
> URL: https://issues.apache.org/jira/browse/YARN-12022
> Project: Hadoop YARN
> Issue Type: Bug
> Components: reservation system, resourcemanager
> Reporter: Jose Luis López
> Priority: Major
>
> {{CapacityOverTimePolicy#validate}} rejects reservations that are far within the average and instantaneous limits when the user has an allocation in progress at the end of the checked window ({{reservation end + validWindow}}), or a periodic allocation that crosses the period boundary.
> h3. Root cause
> * The user's consumption is obtained with {{getRangeOverlapping(checkStart, checkEnd)}}. Both {{RLESparseResourceAllocation}} ({{headMap(end, true)}}) and {{PeriodicRLESparseResourceAllocation}} (the {{<= end}} unrolling condition) keep the step-up of an allocation in progress at {{checkEnd}} but drop its closing step, so the result is open-ended.
> * {{validate}} then merges consumption and reservation over {{[Long.MIN_VALUE, Long.MAX_VALUE]}}, and {{merge}} appends a key at {{Long.MAX_VALUE}}. The open-ended allocation therefore extends to {{Long.MAX_VALUE}}.
> * The integral loop computes {{value * (Long.MAX_VALUE - t)}} and {{integralDown.put(t + validWindow)}} at {{t = Long.MAX_VALUE}}; both overflow, and the integral check fails on garbage values.
> {noformat}
> PlanningQuotaException: Integral (avg over time) quota capacity 0.25 over a window of 86400 seconds, would be exceeded by accepting reservation: ...
> Caused by: PlanningException: RLESparseResourceAllocation: merge failed as the resulting RLESparseResourceAllocation would be negative, when testing: (-9223372036768375809=<memory:2123748267, vCores:2073973>) > (1791743400000=<memory:256000, vCores:250>)
> at RLESparseResourceAllocation.combineValue(RLESparseResourceAllocation.java:462)
> at RLESparseResourceAllocation.merge(RLESparseResourceAllocation.java:353)
> at CapacityOverTimePolicy.validate(CapacityOverTimePolicy.java:197)
> {noformat}
> ({{-9223372036768375809}} is {{Long.MAX_VALUE + 86400000}} after overflow.)
> h3. Reproduction
> Policy: 24h window, 25% average, 75% instantaneous; cluster of 1000 containers.
> * Non-periodic, real timestamps: user u1 reserves 10% for [t0+30h, t0+32h), then 10% for [t0+6h, t0+7h). The second reservation's checked window ends at t0+31h, while the first is in progress, and it is rejected.
> * Periodic (1 day): three reservations of 5% from 23:30 to 00:30 each day; the third is rejected.
> * Controls: with the first reservation at [t0+32h, t0+33h), or with the daily slot not crossing midnight (22:00-23:00), all reservations are accepted.
> Reproducer tests: {{TestCapacityOverTimePolicy#testUserAllocationInProgressAtEndOfCheckedWindow}} and {{#testPeriodicAllocationCrossingPeriodBoundary}}.
> h3. Fix
> End the merge that builds the integral view at {{checkEnd + validWindow}} instead of {{Long.MAX_VALUE}}:
> {code:java}
> RLESparseResourceAllocation toCheck = RLESparseResourceAllocation
> .merge(plan.getResourceCalculator(), plan.getTotalCapacity(),
> consumptionForUserOverTime, resRLE, RLEOperator.add, Long.MIN_VALUE,
> - Long.MAX_VALUE);
> + checkEnd + validWindow);
> {code}
> The integral is only checked over {{[checkStart, checkEnd]}}, and its value at a time T only depends on the points up to T, so the check is unchanged in that range; only the overflow at {{Long.MAX_VALUE}} goes away.
> Ending the merge at {{checkEnd}} instead is not enough: {{merge}} then adds a point at {{checkEnd}}, where the integral approximation (which subtracts the running total at the latest point at or before {{T - validWindow}}) overestimates the window and rejects reservations within the quota. For example, 4 periodic submissions averaging 20.8% of the cluster against a 25% limit are rejected that way.
> Verification:
> * The two reproducer tests pass, and the {{reservation}} test package (265 tests), {{TestClientRMService}}, {{TestRMWebServicesReservation}} and {{TestReservationSystemWithRMHA}} pass.
> * Sweeping the start time of {{TestCapacityOverTimePolicy}} over every minute of the day: the only failures left are the ones from the test generator itself (YARN-12023), and none with the YARN-12023 change applied.
> Patch (reproducer tests + fix): https://github.com/joseluisll/hadoop/tree/YARN-12022
> This issue also makes {{TestCapacityOverTimePolicy}} cases fail or be rejected for the wrong reason at specific times of the day (see the linked test issue).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org