You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Andrey Zagrebin (JIRA)" <ji...@apache.org> on 2019/07/30 07:59:00 UTC
[jira] [Commented] (FLINK-13435) Remove
ShuffleDescriptor.ReleaseType and make release semantics fixed per
partition type
[ https://issues.apache.org/jira/browse/FLINK-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16895881#comment-16895881 ]
Andrey Zagrebin commented on FLINK-13435:
-----------------------------------------
[~zjwang] [~Zentol] Could you review?
> Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per partition type
> ----------------------------------------------------------------------------------------
>
> Key: FLINK-13435
> URL: https://issues.apache.org/jira/browse/FLINK-13435
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination, Runtime / Network
> Affects Versions: 1.9.0
> Reporter: Andrey Zagrebin
> Assignee: Andrey Zagrebin
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.9.0, 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In a long term we do not need auto-release semantics for blocking (persistent) partition. We expect them always to be released externally by JM and assume they can be consumed multiple times.
> The pipelined partitions have always only one consumer and one consumption attempt. Afterwards they can be always released automatically.
> ShuffleDescriptor.ReleaseType was introduced to make release semantics more flexible but it is not needed in a long term.
> FORCE_PARTITION_RELEASE_ON_CONSUMPTION was introduced as a safety net to be able to fallback to 1.8 behaviour without the partition tracker and JM taking care about blocking partition release. We can make this option specific for NettyShuffleEnvironment which was the only existing shuffle service before. If it is activated then the blocking partition is also auto-released on a consumption attempt as it was before. The fine-grained recovery will just not find the partition after the job restart in this case and will restart the producer.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)