You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Aljoscha Krettek (JIRA)" <ji...@apache.org> on 2019/05/21 11:29:00 UTC

[jira] [Commented] (FLINK-12574) using sink StreamingFileSink files are overwritten when resuming application causing data loss

    [ https://issues.apache.org/jira/browse/FLINK-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844740#comment-16844740 ] 

Aljoscha Krettek commented on FLINK-12574:
------------------------------------------

What do you mean by "restarting or resuming". If everything works correctly, the {{StreamingFileSink}} should checkpoint the counter that it uses for part files and new part files should have a higher part index than files that were already written. Of course, this only works when restoring from the latest checkpoint/savepoint, otherwise that part counter state will be outdated.

> using sink StreamingFileSink files are overwritten when resuming application causing data loss
> ----------------------------------------------------------------------------------------------
>
>                 Key: FLINK-12574
>                 URL: https://issues.apache.org/jira/browse/FLINK-12574
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / FileSystem
>    Affects Versions: 1.8.0
>            Reporter: yitzchak lieberman
>            Priority: Critical
>
> when part files are saved to s3 bucket (with bucket assigner) with simple names such as:
> part-0-0 and part-1-2
> restarting or resuming application causes checkpoint id to start from 0 and old files will be replaced by new part files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)