[GitHub] jgrier opened a new pull request #7099: [FLINK-10887] [jobmaster] Add source watermark tracking to the JobMaster

[GitBox <gi...@apache.org>]

jgrier opened a new pull request #7099: [FLINK-10887] [jobmaster] Add source watermark tracking to the JobMaster
URL: https://github.com/apache/flink/pull/7099
 
 
   ## What is the purpose of the change
   
   This commit adds a JobMaster RPC endpoint that is used to for global information sharing.  One use case will be event time source synchronization where it will be used to share watermarks but there are others.  It takes the form of a set of named aggregates that can be updated by a client-supplied AggregateFunction.
   
   Note that the RPC endpoint accepts a serialized AggregateFunction in the form of a byte array.  We need to do this so that we can deserialize this using the UserCodeClassLoader.  The normal RpcService path does not use the UserCodeClassLoader nor is there any easy way to make it do so.
   
   This PR also includes the code/wiring neccessary to expose this functionality to user functions via the `StreamingRuntimeEnvironment`.
   
   The PR seems large but it is mostly wiring.  To quickly assess the changes I suggest looking at the following classes:
   - `GlobalAggregateManager` (to understand the API)
   - `RpcGlobalAggregateManager` (to see the client-side RPC with the JobMaster)
   - `JobMaster / JobMasterGateway` (server side implementation of the above)
   - `GlobalAggregateManagerITCase` (for typical usage from user code)
   
   Most of the rest of the PR is just wiring it all up.
   
   ## Brief change log
     - New RPC endpoint on JobMaster to create, update, and retrieve named aggregates.
     - Updated JobMaster Tests
     - Client side exposure of above endpoint via the StreamingRuntimeEnvironment and GlobalAggregateManager classes.
     - Integration test exercising typical usage from user code.
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
     - JobMasterTest
     - GlobalAggregateManagerITCase
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): No
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: Yes
     - The serializers: No
     - The runtime per-record code paths (performance sensitive): No
     - Anything that affects deployment or recovery: No
     - The S3 file system connector: No
   
   ## Documentation
   
     - Does this pull request introduce a new feature? No
     - If yes, how is the feature documented? not applicable
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services