[GitHub] zhijiangW opened a new pull request #7631: [FLINK-11391][shuffle] Introduce PartitionShuffleDescriptor and ShuffleDeploymentDescriptor

[GitBox <gi...@apache.org>]

zhijiangW opened a new pull request #7631: [FLINK-11391][shuffle] Introduce PartitionShuffleDescriptor and ShuffleDeploymentDescriptor
URL: https://github.com/apache/flink/pull/7631
 
 
   ## What is the purpose of the change
   
   *This is a sub task for introducing `ShuffleMaster` component on JM side based on pluggable `ShuffleManager` architecture.*
   
   *In the first step, we try to refactor the related information structures during deployment. So we introduce the `PartitionShuffleDescriptor (PSD)` to cover all the necessary info which might come from `ExecutionGraph` directly or registration from `TM/ShuffleService`. 
   
   *The `ShuffleDeploymentDescriptor (SDD)` is also introduced for covering only shuffle specific info and SDD is created by `ShuffleMaster` during `registerPartitionProducer`. *
   
   *PSD and SDD would be cached and used for generating `ResultPartitionDeploymentDescriptor (RPDD)`, `InputGateDeploymentDescriptor (IGDD)`, `InputChannelDeploymentDescriptor (ICDD)', etc during producer/consumer task deployments. The relationship between them seems PSD+SDD = RPDD/IGDD/ICDD. *
   
   *In addition, we remove the `ResultPartitionLocation` structure to separate the `ConnectionID` and `LocationType` info. The `ConnectionID` can be regarded as shuffle specific info which would be covered in PSD, SDD, ICDD. And `LocationType` is covered only in ICDD when both producer and consumer are deployed.*
   
   Notes:
   1. *The `DefaultShuffleMaster` here is only for interacting with the related logics, and the formal implementation would be done in a separate pr. *
   
   2. *We might not confirm the deployment sequence of scheduler, that means it might exist the scenario of deploying consumer before producer. So we can not rely on the producer's PSD/SDD to generate IGDD/ICDD of consumer, and this part is still relying on the `ExecutionEdge`.*
   
   ## Brief change log
   
     - *Introduce the structure of `PSD`*
     - *Introduce the structure of `SDD`*
     - *Remove the `ResultPartitionLocation`*
     - *Refactor the related process of task deployment*
     - *Refactor the related process of `scheduleOrUpdateConsumers`*
   
   ## Verifying this change
   
   *The related tests would be added after reviewing to confirm current refactoring make sense. *
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (**yes** / no)
     - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services