[jira] [Created] (FLINK-19715) Optimize re-assignment of excess resources

["Chesnay Schepler (Jira)" <ji...@apache.org>]

Chesnay Schepler created FLINK-19715:
----------------------------------------

             Summary: Optimize re-assignment of excess resources
                 Key: FLINK-19715
                 URL: https://issues.apache.org/jira/browse/FLINK-19715
             Project: Flink
          Issue Type: Sub-task
          Components: Runtime / Coordination
            Reporter: Chesnay Schepler
             Fix For: 1.13.0


The {{JobScopedResourceTracker}} tracks acquired resources that exceed the jobs requirements as {{excess}} resources.

Whenever the requirements increase, or a (non-excess) resource was lost, we try to assign any excess resources we have to fill either fulfill the requirements of fill in the lost resource.

This re-assignment is currently implemented by doing a full copy of map containing the excess resources, and going through the usual code path for acquired resources.

This is fine in terms of correctness (although it can cause misleading log messages), but in the worst case, where we cannot re-assign any excess resources, we not only rebuild the original map in the process, but also potentially iterate over every outstanding requirement for every excess slot.

We should optimize this step by iterating over the excess resource map once, removing items on-demand and aborting early for a given excess resource profile if no matching requirement could be found.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)