[jira] [Updated] (FLINK-29225) Delete message incorrectly ignored in SinkUpsertMaterializer

["Fabian Paul (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-29225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fabian Paul updated FLINK-29225:
--------------------------------
    Affects Version/s: 1.15.3
                           (was: 1.15.2)

> Delete message incorrectly ignored in SinkUpsertMaterializer
> ------------------------------------------------------------
>
>                 Key: FLINK-29225
>                 URL: https://issues.apache.org/jira/browse/FLINK-29225
>             Project: Flink
>          Issue Type: Bug
>    Affects Versions: 1.14.5, 1.15.3
>            Reporter: lincoln lee
>            Priority: Major
>
> Currently if the interval between the arrival of the delete message and the insert/update message exceeds state ttl, the delete message was ignored incorrectly in `SinkUpsertMaterializer`. This will cause wrong result in corresponding sink table(dirty data left).
>  
> 1. if state ttl is set to '10 hour', then the following delete message will be ignored (the '+I (1, a1)' will be left in sink table forever)
> {code:java}
> 00:00:01 +I (1, a1) 
> 10:00:02 -D (1, a1)
> {code}
>  
> 2. but another contrast case which will wrongly delete data in sink table if we send delete message when state staled
> {code:java}
> 00:00:01 +I (1, a1)
> 00:00:02 +I (1, a2) 
> 10:00:03 -D (1, a1)
> {code}
>  
> compare the two choice of current implementation and eager deletion, the former will cause dirty data left, but the later will cause some data lost(seems the former is less harmful..)
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)