You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by al...@apache.org on 2017/09/06 14:14:52 UTC
flink git commit: [hotfix][kafka][docs] Add warning regarding data
losses when writing to Kafka
Repository: flink
Updated Branches:
refs/heads/master 8119baaf1 -> d0636c8ce
[hotfix][kafka][docs] Add warning regarding data losses when writing to Kafka
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/d0636c8c
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/d0636c8c
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/d0636c8c
Branch: refs/heads/master
Commit: d0636c8ce01d3d0cf032235647887c3560c4833a
Parents: 8119baa
Author: Piotr Nowojski <pi...@gmail.com>
Authored: Thu Aug 31 17:10:36 2017 +0200
Committer: Aljoscha Krettek <al...@gmail.com>
Committed: Wed Sep 6 16:13:51 2017 +0200
----------------------------------------------------------------------
docs/dev/connectors/kafka.md | 23 +++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/d0636c8c/docs/dev/connectors/kafka.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/kafka.md b/docs/dev/connectors/kafka.md
index 042ad11..f95c8c0 100644
--- a/docs/dev/connectors/kafka.md
+++ b/docs/dev/connectors/kafka.md
@@ -475,8 +475,14 @@ are other constructor variants that allow providing the following:
### Kafka Producers and Fault Tolerance
-With Flink's checkpointing enabled, the Flink Kafka Producer can provide
-at-least-once delivery guarantees.
+#### Kafka 0.8
+
+Before 0.9 Kafka did not provide any mechanisms to guarantee at-least-once or exactly-once semantics.
+
+#### Kafka 0.9 and 0.10
+
+With Flink's checkpointing enabled, the `FlinkKafkaProducer09` and `FlinkKafkaProducer010`
+can provide at-least-once delivery guarantees.
Besides enabling Flink's checkpointing, you should also configure the setter
methods `setLogFailuresOnly(boolean)` and `setFlushOnCheckpoint(boolean)` appropriately,
@@ -499,6 +505,19 @@ we recommend setting the number of retries to a higher value.
**Note**: There is currently no transactional producer for Kafka, so Flink can not guarantee exactly-once delivery
into a Kafka topic.
+<div class="alert alert-warning">
+ <strong>Attention:</strong> Depending on your Kafka configuration, even after Kafka acknowledges
+ writes you can still experience data loss. In particular keep in mind the following Kafka settings:
+ <ul>
+ <li><tt>acks</tt></li>
+ <li><tt>log.flush.interval.messages</tt></li>
+ <li><tt>log.flush.interval.ms</tt></li>
+ <li><tt>log.flush.*</tt></li>
+ </ul>
+ Default values for the above options can easily lead to data loss. Please refer to Kafka documentation
+ for more explanation.
+</div>
+
## Using Kafka timestamps and Flink event time in Kafka 0.10
Since Apache Kafka 0.10+, Kafka's messages can carry [timestamps](https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message), indicating