[jira] [Closed] (FLINK-20573) Add a configuration item to flinksql's 'with' configuration syntax to filter on exception data

["Jark Wu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-20573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jark Wu closed FLINK-20573.
---------------------------
    Resolution: Not A Problem

As discussed offline, this can be done by using computed column to clean the rowtime value or using user-defined watermark. 

> Add a configuration item to flinksql's 'with' configuration syntax to filter on exception data
> ----------------------------------------------------------------------------------------------
>
>                 Key: FLINK-20573
>                 URL: https://issues.apache.org/jira/browse/FLINK-20573
>             Project: Flink
>          Issue Type: Improvement
>          Components: Table SQL / API
>    Affects Versions: 1.11.2
>            Reporter: Xiao Cai
>            Priority: Major
>
> Currently, after defining the event time and watermark in flink sql, if abnormal data is received, such as test data, and it happens that the event time of that test data is very large, it will cause the watermark of the whole stream to become large, so that any normal data coming in afterwards will not be processed by the stream.
> Is it possible to set a configuration item in 'with' to identify what kind of data is abnormal data that can be automatically filtered, such as event time more than half an hour or more than 2 hours from the current time, will be automatically filtered by the stream; or the event time of an event with a certain value range in the event stream id will not affect the stream's watermark generation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)