[GitHub] [flink] zhijiangW commented on issue #7186: [FLINK-10941] Keep slots which contain unconsumed result partitions

[GitBox <gi...@apache.org>]

zhijiangW commented on issue #7186: [FLINK-10941] Keep slots which contain unconsumed result partitions
URL: https://github.com/apache/flink/pull/7186#issuecomment-469132652
 
 
   @azagrebin , the most basic problem currently is when TM is released by RM, but the partition data has not transported completed yet, then it would cause unnecessary failover which exist in both stream and batch modes.
   
   As we know, the rule for releasing TM  by RM is based on whether all the slots are idle or not. I think we can still stick to this rule. Then the root problem is that when task finishes, that does not mean the corresponding slot should be released. Both task and its input/output occupy the slot resource, and we can regard the network buffer as one part of  resource in `ResourceProfile` of a slot. When task finishes, if partition data does not finish transporting, then the network buffer is not released yet. So the slot should be regarded as used to maintain active TM on RM side.
   
   From the aspect of decoupling task resource with partition/shuffle resource in slot, we can solve the issue of TM release in natural way and do not destroy anything.
   
   Considering delay release the partition for inactive scenarios, it is involved in another basic mechanism for releasing partition, apart from the above basic mechanisms of releasing TM and decoupling partition resource. I would give two points:
   
   - Closed connection should only release the network reader views (sequence view, sub partition view), but not release the sub partition from `ResultPartitionManager`. The view could be re-created to consume the partition repeatedly via re-establishing the connection. 
   
   - When to release partition should be determined by `ShuffleService` on TM or `ShuffleMaster` on JM. And there would be different strategies based on different `ShuffleManager` implementations, such as based on transport complete on producer side, or consumption complete on consumer side, or all kinds of TTL, etc.
   
   In priority I think we should solve the issue of TM release in first step as I mentioned in the beginning, and take the simple way for releasing partition based on transport finished. Further we could improve the failover case by supporting delay release partition or multiple consumptions  of one partition.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services