[jira] [Assigned] (YARN-12022) CapacityOverTimePolicy rejects reservations within the quota when a user allocation is in progress at the end of the checked window

["Jose Luis López (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-12022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jose Luis López reassigned YARN-12022:
--------------------------------------

    Assignee: Jose Luis López

> CapacityOverTimePolicy rejects reservations within the quota when a user allocation is in progress at the end of the checked window
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-12022
>                 URL: https://issues.apache.org/jira/browse/YARN-12022
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: reservation system, resourcemanager
>            Reporter: Jose Luis López
>            Assignee: Jose Luis López
>            Priority: Major
>              Labels: pull-request-available
>
> {{CapacityOverTimePolicy#validate}} rejects reservations that are far within the average and instantaneous limits when the user has an allocation in progress at the end of the checked window ({{reservation end + validWindow}}), or a periodic allocation that crosses the period boundary.
> h3. Root cause
> * The user's consumption is obtained with {{getRangeOverlapping(checkStart, checkEnd)}}. Both {{RLESparseResourceAllocation}} ({{headMap(end, true)}}) and {{PeriodicRLESparseResourceAllocation}} (the {{<= end}} unrolling condition) keep the step-up of an allocation in progress at {{checkEnd}} but drop its closing step, so the result is open-ended.
> * {{validate}} then merges consumption and reservation over {{[Long.MIN_VALUE, Long.MAX_VALUE]}}, and {{merge}} appends a key at {{Long.MAX_VALUE}}. The open-ended allocation therefore extends to {{Long.MAX_VALUE}}.
> * The integral loop computes {{value * (Long.MAX_VALUE - t)}} and {{integralDown.put(t + validWindow)}} at {{t = Long.MAX_VALUE}}; both overflow, and the integral check fails on garbage values.
> {noformat}
> PlanningQuotaException: Integral (avg over time) quota capacity 0.25 over a window of 86400 seconds,  would be exceeded by accepting reservation: ...
> Caused by: PlanningException: RLESparseResourceAllocation: merge failed as the resulting RLESparseResourceAllocation would be negative, when testing: (-9223372036768375809=<memory:2123748267, vCores:2073973>) > (1791743400000=<memory:256000, vCores:250>)
>   at RLESparseResourceAllocation.combineValue(RLESparseResourceAllocation.java:462)
>   at RLESparseResourceAllocation.merge(RLESparseResourceAllocation.java:353)
>   at CapacityOverTimePolicy.validate(CapacityOverTimePolicy.java:197)
> {noformat}
> ({{-9223372036768375809}} is {{Long.MAX_VALUE + 86400000}} after overflow.)
> h3. Reproduction
> Policy: 24h window, 25% average, 75% instantaneous; cluster of 1000 containers.
> * Non-periodic, real timestamps: user u1 reserves 10% for [t0+30h, t0+32h), then 10% for [t0+6h, t0+7h). The second reservation's checked window ends at t0+31h, while the first is in progress, and it is rejected.
> * Periodic (1 day): three reservations of 5% from 23:30 to 00:30 each day; the third is rejected.
> * Controls: with the first reservation at [t0+32h, t0+33h), or with the daily slot not crossing midnight (22:00-23:00), all reservations are accepted.
> Reproducer tests: {{TestCapacityOverTimePolicy#testUserAllocationInProgressAtEndOfCheckedWindow}} and {{#testPeriodicAllocationCrossingPeriodBoundary}}.
> h3. Fix
> End the merge that builds the integral view at {{checkEnd + validWindow}} instead of {{Long.MAX_VALUE}}:
> {code:java}
>      RLESparseResourceAllocation toCheck = RLESparseResourceAllocation
>          .merge(plan.getResourceCalculator(), plan.getTotalCapacity(),
>              consumptionForUserOverTime, resRLE, RLEOperator.add, Long.MIN_VALUE,
> -            Long.MAX_VALUE);
> +            checkEnd + validWindow);
> {code}
> The integral is only checked over {{[checkStart, checkEnd]}}, and its value at a time T only depends on the points up to T, so the check is unchanged in that range; only the overflow at {{Long.MAX_VALUE}} goes away.
> Ending the merge at {{checkEnd}} instead is not enough: {{merge}} then adds a point at {{checkEnd}}, where the integral approximation (which subtracts the running total at the latest point at or before {{T - validWindow}}) overestimates the window and rejects reservations within the quota. For example, 4 periodic submissions averaging 20.8% of the cluster against a 25% limit are rejected that way.
> Verification:
> * The two reproducer tests pass, and the {{reservation}} test package (265 tests), {{TestClientRMService}}, {{TestRMWebServicesReservation}} and {{TestReservationSystemWithRMHA}} pass.
> * Sweeping the start time of {{TestCapacityOverTimePolicy}} over every minute of the day: the only failures left are the ones from the test generator itself (YARN-12023), and none with the YARN-12023 change applied.
> Patch (reproducer tests + fix): https://github.com/joseluisll/hadoop/tree/YARN-12022
> This issue also makes {{TestCapacityOverTimePolicy}} cases fail or be rejected for the wrong reason at specific times of the day (see the linked test issue).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org