You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "luoyuxia (Jira)" <ji...@apache.org> on 2022/10/18 10:12:00 UTC
[jira] [Updated] (FLINK-29678) Data may loss when sink bounded stream into filesystem with auto compact enabled in streaming mode
[ https://issues.apache.org/jira/browse/FLINK-29678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
luoyuxia updated FLINK-29678:
-----------------------------
Description:
In stream mode, when writing bounded data stream into filesystem with auto compactation enabel, the data may loss.
We can reproduce it by adding one code line `'auto-compaction'='true' ` in `FileSystemITCaseBase#open` to enable auto compact.
{code:java}
tableEnv.executeSql(
s"""
|create table partitionedTable (
| x string,
| y int,
| a int,
| b bigint,
| c as b + 1
|) partitioned by (a, b) with (
| 'connector' = 'filesystem',
| 'auto-compaction'='true', // added line to enable auto compaction.
| 'path' = '$getScheme://$resultPath',
| ${formatProperties().mkString(",\n")}
|)
""".stripMargin
) {code}
Then the test `StreamFileSystemTestCsvITCase#testPartialDynamicPartition` will fail with the assert failure:
{code:java}
java.lang.AssertionError:
Expected :List(x18,18)
Actual :List() {code}
There is no data has been written into the table.
> Data may loss when sink bounded stream into filesystem with auto compact enabled in streaming mode
> ---------------------------------------------------------------------------------------------------
>
> Key: FLINK-29678
> URL: https://issues.apache.org/jira/browse/FLINK-29678
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.15.0
> Reporter: luoyuxia
> Priority: Major
>
> In stream mode, when writing bounded data stream into filesystem with auto compactation enabel, the data may loss.
> We can reproduce it by adding one code line `'auto-compaction'='true' ` in `FileSystemITCaseBase#open` to enable auto compact.
> {code:java}
> tableEnv.executeSql(
> s"""
> |create table partitionedTable (
> | x string,
> | y int,
> | a int,
> | b bigint,
> | c as b + 1
> |) partitioned by (a, b) with (
> | 'connector' = 'filesystem',
> | 'auto-compaction'='true', // added line to enable auto compaction.
> | 'path' = '$getScheme://$resultPath',
> | ${formatProperties().mkString(",\n")}
> |)
> """.stripMargin
> ) {code}
> Then the test `StreamFileSystemTestCsvITCase#testPartialDynamicPartition` will fail with the assert failure:
> {code:java}
> java.lang.AssertionError:
> Expected :List(x18,18)
> Actual :List() {code}
> There is no data has been written into the table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)