You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "chen yong (Jira)" <ji...@apache.org> on 2020/07/15 06:44:00 UTC
[jira] [Comment Edited] (FLINK-18523) Advance watermark if there is
no data in all of the partitions after some time
[ https://issues.apache.org/jira/browse/FLINK-18523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17157933#comment-17157933 ]
chen yong edited comment on FLINK-18523 at 7/15/20, 6:43 AM:
-------------------------------------------------------------
Hi. In DataStream API module , in version 1.11 by implementing WatermarkGenerator interface can easily solve the problem, of course in the remaining version can also be used by inheritance AssignerWithPeriodicWatermarks to realize many logic to solve this problem, but it is not easy。
In the Table API/SQL Module, because the DataStream API and the Table API can be converted to each other, according to the above mentioned can solve this problem.But it can be difficult for users who are not very familiar with it.
was (Author: chenyong):
In DataStream API module , in version 1.11 by implementing WatermarkGenerator interface can easily solve the problem, of course in the remaining version can also be used by inheritance AssignerWithPeriodicWatermarks to realize many logic to solve this problem, but it is not easy。
In the Table API/SQL Module, because the DataStream API and the Table API can be converted to each other, according to the above mentioned can solve this problem.But it can be difficult for users who are not very familiar with it.
> Advance watermark if there is no data in all of the partitions after some time
> ------------------------------------------------------------------------------
>
> Key: FLINK-18523
> URL: https://issues.apache.org/jira/browse/FLINK-18523
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
> Affects Versions: 1.11.0
> Reporter: chen yong
> Priority: Major
>
> In the case of window calculations and eventTime scenarios, watermark cannot update because the source does not have data for some reason, and the last Windows cannot trigger the calculations.
> One parameter, table.exec.source. Idle -timeout, can only solve the problem of ignoring parallelism of watermark alignment that does not occur.But when there is no watermark in each parallel degree, you still cannot update the watermark.
> Is it possible to add a lock-timeout parameter (which should be larger than maxOutOfOrderness with default of "-1 ms") and if the watermark is not updated beyond this time (i.e., there is no data), then the current time is taken and sent downstream as the watermark.
>
> thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)