[jira] [Commented] (FLINK-29678) Data may loss when sink bounded stream into filesystem with auto compact enabled in streaming mode

["Martijn Visser (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-29678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619423#comment-17619423 ] 

Martijn Visser commented on FLINK-29678:
----------------------------------------

[~luoyuxia] Could you elaborate a bit on this? Is this a mistake in the file system implementation, in the Sink interfaces, etc? 

> Data may loss when sink bounded stream into filesystem with auto compact enabled in streaming mode 
> ---------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-29678
>                 URL: https://issues.apache.org/jira/browse/FLINK-29678
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / FileSystem
>    Affects Versions: 1.15.0
>            Reporter: luoyuxia
>            Priority: Major
>
> In stream mode, when writing bounded data stream into filesystem with auto compactation enabel, the data may loss.
> We can reproduce it by adding one code line `'auto-compaction'='true' ` in `FileSystemITCaseBase#open` to enable auto compact.
> {code:java}
> tableEnv.executeSql(
>   s"""
>      |create table partitionedTable (
>      |  x string,
>      |  y int,
>      |  a int,
>      |  b bigint,
>      |  c as b + 1
>      |) partitioned by (a, b) with (
>      |  'connector' = 'filesystem',
>      |  'auto-compaction'='true', // added line to enable auto compaction.
>      |  'path' = '$getScheme://$resultPath',
>      |  ${formatProperties().mkString(",\n")}
>      |)
>    """.stripMargin
> ) {code}
> Then the test `StreamFileSystemTestCsvITCase#testPartialDynamicPartition` will fail with the assert failure:
> {code:java}
> java.lang.AssertionError: 
> Expected :List(x18,18)
> Actual   :List() {code}
> There is no data has been written into the table.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)