[jira] [Commented] (FLINK-10887) Add source watermark tracking to the JobMaster

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687311#comment-16687311 ] 

ASF GitHub Bot commented on FLINK-10887:
----------------------------------------

jgrier opened a new pull request #7099: [FLINK-10887] [jobmaster] Add source watermark tracking to the JobMaster
URL: https://github.com/apache/flink/pull/7099
 
 
   ## What is the purpose of the change
   
   This commit adds a JobMaster RPC endpoint that is used to share information across source subtasks regarding event time progress.
   
   This will be used implement event time source synchronization across sources.
   
   ## Brief change log
     - New RPC endpoint on JobMaster to track event time progress across source sub-tasks
     - Updated JobMaster Tests
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
     - Added testCase to ensure watermark stats are computed correctly to JobMasterTest
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): No
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: No
     - The serializers: No
     - The runtime per-record code paths (performance sensitive): No
     - Anything that affects deployment or recovery: No
     - The S3 file system connector: No
   
   ## Documentation
   
     - Does this pull request introduce a new feature? No
     - If yes, how is the feature documented? not applicable
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Add source watermark tracking to the JobMaster
> ----------------------------------------------
>
>                 Key: FLINK-10887
>                 URL: https://issues.apache.org/jira/browse/FLINK-10887
>             Project: Flink
>          Issue Type: Sub-task
>          Components: JobManager
>            Reporter: Jamie Grier
>            Assignee: Jamie Grier
>            Priority: Major
>              Labels: pull-request-available
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> We need to add a new RPC to the JobMaster such that the current watermark for every source sub-task can be reported and the current global minimum/maximum watermark can be retrieved so that each source can adjust their partition read rates in an attempt to keep sources roughly aligned in event time.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)