You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Tzu-Li (Gordon) Tai (JIRA)" <ji...@apache.org> on 2017/01/13 21:12:26 UTC
[jira] [Updated] (FLINK-5487) Proper at-least-once support for
ElasticsearchSink
[ https://issues.apache.org/jira/browse/FLINK-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tzu-Li (Gordon) Tai updated FLINK-5487:
---------------------------------------
Description:
Discussion in ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html
Currently, the Elasticsearch Sink actually doesn't offer any guarantees for message delivery.
For proper support of at-least-once, the sink will need to participate in Flink's checkpointing: when snapshotting is triggered at the {{ElasticsearchSink}}, we need to synchronize on the pending {{ActionRequest}}s by flushing the {{BulkProcessor}}. For temporary ES failures (see FLINK-5122) that may happen on the flush, we should retry them before returning from snapshotting and acking the checkpoint. If there are non-temporary ES failures on the flush, the current snapshot should fail.
was:
Discussion in ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html
Currently, the Elasticsearch Sink actually doesn't offer any guarantees for message delivery.
For proper support of at-least-once, the sink will need to participate in Flink's checkpointing: when snapshotting is triggered at the `ElasticsearchSink`, we need to synchronize on the pending `ActionRequest`s by flushing the `BulkProcessor`. For temporary ES failures (see FLINK-5122) that may happen on the flush, we should retry them before returning from snapshotting and acking the checkpoint. If there are non-temporary ES failures on the flush, the current snapshot should fail.
> Proper at-least-once support for ElasticsearchSink
> --------------------------------------------------
>
> Key: FLINK-5487
> URL: https://issues.apache.org/jira/browse/FLINK-5487
> Project: Flink
> Issue Type: Bug
> Components: Streaming Connectors
> Reporter: Tzu-Li (Gordon) Tai
> Priority: Critical
>
> Discussion in ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fault-tolerance-guarantees-of-Elasticsearch-sink-in-flink-elasticsearch2-td10982.html
> Currently, the Elasticsearch Sink actually doesn't offer any guarantees for message delivery.
> For proper support of at-least-once, the sink will need to participate in Flink's checkpointing: when snapshotting is triggered at the {{ElasticsearchSink}}, we need to synchronize on the pending {{ActionRequest}}s by flushing the {{BulkProcessor}}. For temporary ES failures (see FLINK-5122) that may happen on the flush, we should retry them before returning from snapshotting and acking the checkpoint. If there are non-temporary ES failures on the flush, the current snapshot should fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)