You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by fh...@apache.org on 2018/04/16 19:44:23 UTC

flink git commit: [FLINK-9183] [docs] Add warning about idle partitions to Kafka connector docs.

Repository: flink
Updated Branches:
  refs/heads/release-1.4 e76b10d07 -> a9b497749


[FLINK-9183] [docs] Add warning about idle partitions to Kafka connector docs.

This closes #5858.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/a9b49774
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/a9b49774
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/a9b49774

Branch: refs/heads/release-1.4
Commit: a9b497749710c708d076fc45688fff7b72416af1
Parents: e76b10d
Author: juhoautio <au...@gmail.com>
Authored: Mon Apr 16 17:40:23 2018 +0300
Committer: Fabian Hueske <fh...@apache.org>
Committed: Mon Apr 16 21:44:12 2018 +0200

----------------------------------------------------------------------
 docs/dev/connectors/kafka.md | 5 +++++
 1 file changed, 5 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/a9b49774/docs/dev/connectors/kafka.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/kafka.md b/docs/dev/connectors/kafka.md
index 8e38146..75c5e23 100644
--- a/docs/dev/connectors/kafka.md
+++ b/docs/dev/connectors/kafka.md
@@ -444,6 +444,11 @@ the `Watermark getCurrentWatermark()` (for periodic) or the
 `Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp)` (for punctuated) is called to determine
 if a new watermark should be emitted and with which timestamp.
 
+**Note**: If a watermark assigner depends on records read from Kafka to advance its watermarks (which is commonly the case), all topics and partitions need to have a continuous stream of records. Otherwise, the watermarks of the whole application cannot advance and all time-based operations, such as time windows or functions with timers, cannot make progress. A single idle Kafka partition causes this behavior. 
+A Flink improvement is planned to prevent this from happening 
+(see [FLINK-5479: Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions](
+https://issues.apache.org/jira/browse/FLINK-5479)).
+In the meanwhile, a possible workaround is to send *heartbeat messages* to all consumed partitions that advance the watermarks of idle partitions.
 
 ## Kafka Producer