[GitHub] [flink] HuangZhenQiu opened a new pull request #8306: [FLINK-12342] Use total requested pending containers to bound the maximum containers to…

[GitBox <gi...@apache.org>]

HuangZhenQiu opened a new pull request #8306: [FLINK-12342] Use total requested pending containers to bound the maximum containers to…
URL: https://github.com/apache/flink/pull/8306
 
 
   ## What is the purpose of the change
   
   Fix the issue of requesting too many containers issue in FlinkResourceManager. Original the new request issue is bounded by the number of pending containers in AMRMClientAsync. As every time AMRMClientAsync will issue request of pendingContainers  + 1 containers, it always ask more than we need. Especially when an Flink application needs 100+ containers, the number total containers will be in thousands. The issue will be enlarged when we do cluster maintenance and restart 1000+ applications as soon as possible. Too many request will overload yarn resource manager and introduce extra delay for restarting applications.
   
   ## Brief change log
   
     - record the total number of containers are requested and still in pending status
     - use the totalRequestedPendingContainers as upper bound fore new container acquirement
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
     - The serializers: (no)
     - The runtime per-record code paths (performance sensitive): (no)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / no)
     - If yes, how is the feature documented? (not applicable)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services