[GitHub] [flink] zhijiangW opened a new pull request #9132: [FLINK-13245][network] Fix the bug of file resource leak while canceling partition request

[GitBox <gi...@apache.org>]

zhijiangW opened a new pull request #9132: [FLINK-13245][network] Fix the bug of file resource leak while canceling partition request
URL: https://github.com/apache/flink/pull/9132
 
 
   ## What is the purpose of the change
   
   *On producer side the netty handler receives the `CancelPartitionRequest` for releasing the `ResultSubpartitionView` resource. In previous implementation we try to find the corresponding view via available queue in `PartitionRequestQueue`. But in reality the view is not always available to stay in this queue, then the view would never be released.*
   
   *Furthermore the release of `ResultPartition/ResultSubpartitions` is based on the reference counter in `ReleaseOnConsumptionResultPartition`, but while handling the `CancelPartitionRequest` in `PartitionRequestQueue`, the `ReleaseOnConsum#ptionResultPartition` is never notified of consumed subpartition. That means the reference counter would never decrease to 0 to trigger partition release, which would bring file resource leak in the case of `BoundedBlockingSubpartition`.*
       
   *In order to fix above two issues, the corresponding view is released via all reader queue instead, and then it would call `ReleaseOnConsumptionResultPartition#onConsumedSubpartition` meanwhile to solve this bug.*
   
   ## Brief change log
   
     - *Fix the logic in handling `CancelPartitionRequest` in `PartitionRequestQueue`*
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
     - *Added unit tests for verifying the partition and view are both released while receiving `CancelPartitionRequest` for the case of available/unavailable view.*
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / **no**)
     - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services