You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "zhijiang (JIRA)" <ji...@apache.org> on 2019/07/30 14:07:00 UTC
[jira] [Updated] (FLINK-13478) Decouple two different release
strategies in BoundedBlockingSubpartition
[ https://issues.apache.org/jira/browse/FLINK-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhijiang updated FLINK-13478:
-----------------------------
Description:
We have two basic release strategies atm. One is based on consumption via network notification from consumer. The other is based on notification via RPC from JM/scheduler.
But in current implementation of {{BoundedBlockingSubpartition}}, these two ways are a bit coupled with each other. If the JM decides to release partition and send the release RPC call, it has to wait all the reader views really released before finally closing the data file. So the JM-RPC-based release strategy still relies on the consumption confirmation via network to some extent.
In order to make these two release strategies independent, if the release call is from JM/scheduler RPC, we could immediately release all the view readers and then close the data file as well. If the release is based on consumption confirmation, after all the view readers for one subpartition are released, the subpartition could further notify the parent {{ResultPartition}} which decides whether to release the whole partition or not.
was:
We have two basic release strategies atm. One is based on consumption via network notification from consumer. The other is based on notification via RPC from JM/scheduler.
But in current implementation of {{BoundedBlockingSubpartition}}, these two ways are a bit coupled with each other. If the JM decides to release partition and send the release RPC call, it has to wait all the reader views really released before finally closing the data file. So the JM-RPC-based release strategy still relies on the consumption confirmation via network to some extent.
In order to make these two release strategies independent, if the release call is from JM/scheduler RPC, we could immediately release all the view readers and then close the data file as well. If the release is based on consumption notification, after all the view readers for one subpartition are released, the subpartition could further notify the parent {{ResultPartition}} which decides whether to release the whole partition or not.
> Decouple two different release strategies in BoundedBlockingSubpartition
> ------------------------------------------------------------------------
>
> Key: FLINK-13478
> URL: https://issues.apache.org/jira/browse/FLINK-13478
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Network
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Minor
>
> We have two basic release strategies atm. One is based on consumption via network notification from consumer. The other is based on notification via RPC from JM/scheduler.
> But in current implementation of {{BoundedBlockingSubpartition}}, these two ways are a bit coupled with each other. If the JM decides to release partition and send the release RPC call, it has to wait all the reader views really released before finally closing the data file. So the JM-RPC-based release strategy still relies on the consumption confirmation via network to some extent.
> In order to make these two release strategies independent, if the release call is from JM/scheduler RPC, we could immediately release all the view readers and then close the data file as well. If the release is based on consumption confirmation, after all the view readers for one subpartition are released, the subpartition could further notify the parent {{ResultPartition}} which decides whether to release the whole partition or not.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)