You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Thomas Weise (JIRA)" <ji...@apache.org> on 2019/02/17 01:53:00 UTC
[jira] [Commented] (FLINK-10887) Add source watermark tracking to
the JobMaster
[ https://issues.apache.org/jira/browse/FLINK-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16770265#comment-16770265 ]
Thomas Weise commented on FLINK-10887:
--------------------------------------
For verification, I used a synthetic source with a timer that would update an aggregate every second, with parallelism 128. The overhead is negligible.
Note that while aggregators can be custom classes available only through the user code class loader, aggregand and results currently cannot. If they are custom classes, then (de)serialization needs to be performed in user code and byte arrays used with updateGlobalAggregate.
> Add source watermark tracking to the JobMaster
> ----------------------------------------------
>
> Key: FLINK-10887
> URL: https://issues.apache.org/jira/browse/FLINK-10887
> Project: Flink
> Issue Type: Sub-task
> Components: JobManager
> Reporter: Jamie Grier
> Assignee: Jamie Grier
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
> Original Estimate: 24h
> Time Spent: 50m
> Remaining Estimate: 23h 10m
>
> We need to add a new RPC to the JobMaster such that the current watermark for every source sub-task can be reported and the current global minimum/maximum watermark can be retrieved so that each source can adjust their partition read rates in an attempt to keep sources roughly aligned in event time.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)