You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by fh...@apache.org on 2018/04/16 19:43:52 UTC

[2/2] flink git commit: [FLINK-9183] [docs] Add warning about idle partitions to Kafka connector docs.

[FLINK-9183] [docs] Add warning about idle partitions to Kafka connector docs.

This closes #5858.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/fc58f987
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/fc58f987
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/fc58f987

Branch: refs/heads/release-1.5
Commit: fc58f987144a307e7208de0bdfe439922bed4b55
Parents: 4c3d018
Author: juhoautio <au...@gmail.com>
Authored: Mon Apr 16 17:40:23 2018 +0300
Committer: Fabian Hueske <fh...@apache.org>
Committed: Mon Apr 16 21:42:21 2018 +0200

----------------------------------------------------------------------
 docs/dev/connectors/kafka.md | 5 +++++
 1 file changed, 5 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/fc58f987/docs/dev/connectors/kafka.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/kafka.md b/docs/dev/connectors/kafka.md
index 27fca7a..47a6651 100644
--- a/docs/dev/connectors/kafka.md
+++ b/docs/dev/connectors/kafka.md
@@ -451,6 +451,11 @@ the `Watermark getCurrentWatermark()` (for periodic) or the
 `Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp)` (for punctuated) is called to determine
 if a new watermark should be emitted and with which timestamp.
 
+**Note**: If a watermark assigner depends on records read from Kafka to advance its watermarks (which is commonly the case), all topics and partitions need to have a continuous stream of records. Otherwise, the watermarks of the whole application cannot advance and all time-based operations, such as time windows or functions with timers, cannot make progress. A single idle Kafka partition causes this behavior. 
+A Flink improvement is planned to prevent this from happening 
+(see [FLINK-5479: Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions](
+https://issues.apache.org/jira/browse/FLINK-5479)).
+In the meanwhile, a possible workaround is to send *heartbeat messages* to all consumed partitions that advance the watermarks of idle partitions.
 
 ## Kafka Producer