[jira] [Comment Edited] (FLINK-26098) TableAPI does not forward idleness configuration from DataStream

["Marios Trivyzas (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-26098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518956#comment-17518956 ] 

Marios Trivyzas edited comment on FLINK-26098 at 4/7/22 3:35 PM:
-----------------------------------------------------------------

For {{DataStream}} API, with [FLIP-27|[https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface]] , there is {{WatermarkStrategy}} interface

and {{WatermarkStrategyWithIdleness}} to configure the idleness, which is only in the context of {{DataStream}} API.

 

For TableApi/SQL we have {{PushWatermarkIntoTableSourceScanRuleBase}} which uses the {{table.exec.source.idle-timeout and in turn}} create a {{WatermarkStrategyWithIdleness}} to be used.

 

So I think that this issue can now be closed, since it has been addressed by the work in [[FLINK-16835]|https://issues.apache.org/jira/browse/FLINK-16835] and is already available for {{{}Flink 1.15{}}}.

 


was (Author: matriv):
For {{DataStream}} API, with [FLIP-27|[https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface]] , there is {{WatermarkStrategy}} interface

and {{WatermarkStrategyWithIdleness}} to configure the idleness, which is only in the context of {{DataStream}} API.

 

For TableApi/SQL we have {{PushWatermarkIntoTableSourceScanRuleBase}} which uses the {{table.exec.source.idle-timeout and in turn}} create a {{WatermarkStrategyWithIdleness}} to be used.

 

So I think that this issue can now be closed, since it has been addressed by the work in https://issues.apache.org/jira/browse/FLINK-16835.

 

> TableAPI does not forward idleness configuration from DataStream
> ----------------------------------------------------------------
>
>                 Key: FLINK-26098
>                 URL: https://issues.apache.org/jira/browse/FLINK-26098
>             Project: Flink
>          Issue Type: Bug
>          Components: Table SQL / API
>    Affects Versions: 1.15.0, 1.14.3
>            Reporter: Till Rohrmann
>            Assignee: Marios Trivyzas
>            Priority: Major
>         Attachments: Screenshot_20220407_150020.png, Screenshot_20220407_151012.png
>
>
> The TableAPI does not forward the idleness configuration from a DataStream source. That can lead to the halt of processing if all sources are idle because {{WatermarkAssignerOperator}} [1] will never set a channel to active again. The only way to mitigate the problem is to explicitly configure the idleness for table sources via {{table.exec.source.idle-timeout}}. Configuring this value is actually not easy because creating a {{StreamExecutionEnvironment}} via {{create(StreamExecutionEnvironment, TableConfig)}} is deprecated.
> [1] https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/wmassigners/WatermarkAssignerOperator.java#L103



--
This message was sent by Atlassian Jira
(v8.20.1#820001)