[GitHub] [flink] zentol opened a new pull request #8778: [FLINK-12615][coordination] Track partitions on JM

[GitBox <gi...@apache.org>]

zentol opened a new pull request #8778: [FLINK-12615][coordination] Track partitions on JM
URL: https://github.com/apache/flink/pull/8778
 
 
   Based on #8687 . When reviewing this PR, please ignore all commits preceding 
   "[FLINK-12615][coordination] Support generic key in PartitionTable".
   
   ## What is the purpose of the change
   
   With this PR we track on the JobMaster side which partitions are still on task executors, maintain the connection to a task executor until all partitions on it have been released, and issue a release call for all partitions when a job terminates.
   
   **Attention:** This PR only serves as a MVP. It ignores the shuffle master (and thus will not work with other shuffle service implementation) and the release logic is not viable for high-volume jobs (because it requires all blocking partitions to be persisted on disk until the job is finished).
   
   ## Brief changelog
   
   6009f00 modifies the PartitionTable to support arbitrary keys; so far it was hard-wired to use JobIDs as keys for use on the TE side, but now we want to organize them by ResourceID (== TE ID) instead.
   
   c7193dc introduces a PartitionTable into the JM, Scheduler and EG. This commit only modifies constructor-related methods and does not contain any actual logic. This commit only exists to make reviewing easier.
   
   ea418da contains the tracking logic. Partitions are added to the table, grouped by task executor, by an Execution when it reaches the state FINISHED, and removed again by the JM when disassociating from a TE.
   
   3087792 introduces a call to TE#releasePartitions for when a job is finished.
   
   d5a9f3c finally modifies the check for empty taskmanagers to take partitions into account.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services