[jira] [Commented] (FLINK-7784) Don't fail TwoPhaseCommitSinkFunction when failing to commit

["Piotr Nowojski (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16215088#comment-16215088 ] 

Piotr Nowojski commented on FLINK-7784:
---------------------------------------

Are we sure that we want to ignore such failures by default? The same issue that's described here for {{TwoPhaseCommitSinkFunction}} applies also to {{BucketingSink}} and in both places, if we ignore those failures in case of intermittent errors, there will be a data loss :(

> Don't fail TwoPhaseCommitSinkFunction when failing to commit
> ------------------------------------------------------------
>
>                 Key: FLINK-7784
>                 URL: https://issues.apache.org/jira/browse/FLINK-7784
>             Project: Flink
>          Issue Type: Bug
>          Components: DataStream API
>    Affects Versions: 1.4.0
>            Reporter: Aljoscha Krettek
>            Assignee: Gary Yao
>            Priority: Blocker
>             Fix For: 1.4.0
>
>
> Currently, {{TwoPhaseCommitSinkFunction}} will fail if committing fails (either when doing it via the completed checkpoint notification or when trying to commit after restoring after failure). This means that the job will go into an infinite recovery loop because we will always keep failing.
> In some cases it might be better to ignore those failures and keep on processing and this should be the default. We can provide an option that allows failing the sink on failing commits.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)