You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Xuannan Su (Jira)" <ji...@apache.org> on 2020/12/01 09:24:00 UTC
[jira] [Commented] (FLINK-19343) FLIP-36: Support Interactive
Programming in Flink
[ https://issues.apache.org/jira/browse/FLINK-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241382#comment-17241382 ]
Xuannan Su commented on FLINK-19343:
------------------------------------
Hi, Jin Xing. Thanks for your comments.
1. I agree with you that it is indeed weird for the "Interactive" relies on the clusterPartitionTracker for managing the lifecycle of the cluster partition, which is also a form of shuffle data. And to support cache across jobs, we need to have a component whose lifecycle outlive the job to manage the shuffle data.
2. TBH, also, I wouldn't say I like the idea to pass the ShuffleDescriptor back to the client-side. But at that time being, as you say, we do not have a separate component to manage the lifecycle of shuffle data across jobs in the runtime. Therefore, the decision is made to support share shuffle data across jobs. To support caching in DataStream, it is a more clear design to have a component at runtime scope to manage the shuffle data across jobs.
3. I don't think the remote shuffle data should be managed by the PartitionTracker as well. Instead, I think the ClusterPartition is just a kind of shuffle data and therefore should be managed by the ShuffleService.
I am pulling in [~chesnay]. He may have more insight from the ClusterPartition perspective.
> FLIP-36: Support Interactive Programming in Flink
> -------------------------------------------------
>
> Key: FLINK-19343
> URL: https://issues.apache.org/jira/browse/FLINK-19343
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
> Reporter: Xuannan Su
> Priority: Major
>
> Please refer to the FLIP for any details: https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
--
This message was sent by Atlassian Jira
(v8.3.4#803005)