You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org> on 2019/01/08 13:59:00 UTC
[jira] [Comment Edited] (FLINK-11286) Support to send
StreamStatus.IDLE for non-source operators
[ https://issues.apache.org/jira/browse/FLINK-11286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737126#comment-16737126 ]
Tzu-Li (Gordon) Tai edited comment on FLINK-11286 at 1/8/19 1:58 PM:
---------------------------------------------------------------------
Yes, the watermark assigner would not be able to generate watermarks once upstream operators are idle.
And after a specific timeout (like what is currently done to determine idleness in sources), it should emit the idleness stream status downstream (from the watermark assigner).
was (Author: tzulitai):
Yes, the watermark assigner would not be able to generate watermarks once upstream operators are idle.
And after a specific timeout (like what is currently done to determine idleness in sources), in should emit the idleness stream status downstream (from the watermark assigner).
> Support to send StreamStatus.IDLE for non-source operators
> -----------------------------------------------------------
>
> Key: FLINK-11286
> URL: https://issues.apache.org/jira/browse/FLINK-11286
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
> Reporter: vinoyang
> Assignee: vinoyang
> Priority: Major
>
> Currently, only stream source tasks can be marked as temporary idle. But many times, this approach has limitations.
> Considering such a scenario, there is a DAG as follows: {{source->map->filter->flatmap->keyBy->window}}, with a degree of parallelism of 10. Among them, the watermark is not sent by the source operator, but is downstream, such as flatmap. Every source subtask will not be idle. However, after the filter, some pipelines generate "idle". For example, there are 3 pipelines that will no longer have data sent downstream. At this time, we can't call the {{markAsTemporarilyIdle}} method to mark the current pipeline in the idle state. This will affect the downstream window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)