[GitHub] [flink] azagrebin opened a new pull request #9272: [FLINK-13435] Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per partition type

[GitBox <gi...@apache.org>]

azagrebin opened a new pull request #9272: [FLINK-13435] Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per partition type
URL: https://github.com/apache/flink/pull/9272
 
 
   ## What is the purpose of the change
   
   In a long term we do not need auto-release semantics for blocking (persistent) partition. We expect them always to be released externally by JM and assume they can be consumed multiple times.
   
   The pipelined partitions have always only one consumer and one consumption attempt. Afterwards they can be always released automatically.
   
   `ShuffleDescriptor#ReleaseType` was introduced to make release semantics more flexible but it is not needed in a long term.
   
   `FORCE_PARTITION_RELEASE_ON_CONSUMPTION` was introduced as a safety net to be able to fallback to 1.8 behaviour without the partition tracker and JM taking care about blocking partition release. We can make this option specific for `NettyShuffleEnvironment` which was the only existing shuffle service before. If it is activated then the blocking partition is also auto-released on a consumption attempt as it was before. The fine-grained recovery will just not find the partition after the job restart in this case and will restart the producer.
   
   ## Brief change log
   
     - Remove `ShuffleDescriptor#ReleaseType` and assume predefined release semantics for certain partition type
     - Adjust test and other involved code
     - Move `FORCE_PARTITION_RELEASE_ON_CONSUMPTION` from `JobManagerOptions` to `NettyShuffleEnvironmentOptions`
     - codestyle fixes
   
   
   ## Verifying this change
   
   existing tests
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
     - The serializers: (no)
     - The runtime per-record code paths (performance sensitive): (no)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (no)
     - If yes, how is the feature documented? (not applicable)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services