[jira] [Commented] (FLINK-19761) Add lookup method for registered ShuffleDescriptor in ShuffleMaster

["Xuannan Su (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-19761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17218826#comment-17218826 ] 

Xuannan Su commented on FLINK-19761:
------------------------------------

cc [~trohrmann] [~azagrebin] [~zjwang] 
I'd love to hear your thought on this.

> Add lookup method for registered ShuffleDescriptor in ShuffleMaster
> -------------------------------------------------------------------
>
>                 Key: FLINK-19761
>                 URL: https://issues.apache.org/jira/browse/FLINK-19761
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Network
>            Reporter: Xuannan Su
>            Priority: Major
>
> Currently, the ShuffleMaster can register a partition and get the shuffle descriptor. However, it lacks the ability to look up the registered ShuffleDescriptors belongs to an IntermediateResult by the IntermediateDataSetID.
> Adding the lookup method to the ShuffleMaster can make reusing the cluster partition more easily. For example, we don't have to return the ShuffleDescriptor to the client just so that the other job can somehow encode the ShuffleDescriptor in the JobGraph to consume the cluster partition. Instead, we only need to return the IntermediateDatSetID and use it to lookup the ShuffleDescriptor by another job.
> By adding the lookup method in ShuffleMaster, if we have an external shuffle service and the lifecycle of the IntermediateResult is not bounded to the cluster, we can look up the ShuffleDescriptor and reuse the IntermediateResult by a job running on another cluster even if the cluster that produced the IntermediateResult is shutdown.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)