[jira] [Created] (FLINK-19761) Add lookup method for registered ShuffleDescriptor in ShuffleMaster

["Xuannan Su (Jira)" <ji...@apache.org>]

Xuannan Su created FLINK-19761:
----------------------------------

             Summary: Add lookup method for registered ShuffleDescriptor in ShuffleMaster
                 Key: FLINK-19761
                 URL: https://issues.apache.org/jira/browse/FLINK-19761
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Network
            Reporter: Xuannan Su


Currently, the ShuffleMaster can register a partition and get the shuffle descriptor. However, it lacks the ability to look up the registered ShuffleDescriptors belongs to an IntermediateResult by the IntermediateDataSetID.
Adding the lookup method to the ShuffleMaster can make reusing the cluster partition more easily. For example, we don't have to return the ShuffleDescriptor to the client just so that the other job can somehow encode the ShuffleDescriptor in the JobGraph to consume the cluster partition. Instead, we only need to return the IntermediateDatSetID and use it to lookup the ShuffleDescriptor by another job.
By adding the lookup method in ShuffleMaster, if we have an external shuffle service and the lifecycle of the IntermediateResult is not bounded to the cluster, we can look up the ShuffleDescriptor and reuse the IntermediateResult by a job running on another cluster even if the cluster that produced the IntermediateResult is shutdown.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)