[flink] branch master updated: [FLINK-26159][doc] add description for MAX_FETCH_RECORD related question

[fp...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

fpaul pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git


The following commit(s) were added to refs/heads/master by this push:
     new 52df489  [FLINK-26159][doc] add description for MAX_FETCH_RECORD related question
52df489 is described below

commit 52df489063ae90af73c663dc6abbe42e3794f1ac
Author: Yufei Zhang <af...@gmail.com>
AuthorDate: Tue Feb 15 21:49:44 2022 +0800

    [FLINK-26159][doc] add description for MAX_FETCH_RECORD related question
    
    Update docs/content/docs/connectors/datastream/pulsar.md
    
    Co-authored-by: MartijnVisser <ma...@2symbols.com>
    
    Update docs/content/docs/connectors/datastream/pulsar.md
    
    Co-authored-by: MartijnVisser <ma...@2symbols.com>
    
    add Chinese documentation
---
 docs/content.zh/docs/connectors/datastream/pubsub.md | 17 +++++++++++++++++
 docs/content/docs/connectors/datastream/pulsar.md    |  9 +++++++++
 2 files changed, 26 insertions(+)

diff --git a/docs/content.zh/docs/connectors/datastream/pubsub.md b/docs/content.zh/docs/connectors/datastream/pubsub.md
index ebe7e3c..18225ca 100644
--- a/docs/content.zh/docs/connectors/datastream/pubsub.md
+++ b/docs/content.zh/docs/connectors/datastream/pubsub.md
@@ -150,4 +150,21 @@ env.addSource(pubsubSource)
 
 Sink function 会把准备发到 PubSub 的信息短暂地缓存以提高性能。每次 checkpoint 前，它会刷新缓冲区，并且只有当所有信息成功发送到 PubSub 之后，checkpoint 才会成功完成。
 
+## 常见问题
+
+由于Pulsar Connector大量依赖
+[PulsarClient](https://pulsar.apache.org/docs/en/client-libraries-java/) 和
+[PulsarAdmin](https://pulsar.apache.org/docs/en/admin-api-overview/) 来实现相关功能，有时Flink Job出现
+可能是由于Pulsar broker版本过低或者配置不合适导致的。调试配置或升级pulsar可能会解决部分问题。
+
+
+### 当数据量很小时，source读数据出现大约10s延迟
+
+当Pulsar Source从一个数据量很小的topic读取数据时，用户可能会观测到消息间出现10秒钟的间隔。这是由于Pulsar Source
+会将从broker读到的消息暂存至等待队列中，只有当队列中消息数量达到了 `PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS`
+才会向下游算子发送数据。如果数据量一直很小，那么就会在等待直到等待时长超过`PULSAR_MAX_FETCH_TIME`规定的值 (默认10秒钟)时
+也向下游算子发送数据。
+
+为了避免这种情况，用户需要改变两个配置项的值。
+
 {{< top >}}
diff --git a/docs/content/docs/connectors/datastream/pulsar.md b/docs/content/docs/connectors/datastream/pulsar.md
index cbf1b15..2432227 100644
--- a/docs/content/docs/connectors/datastream/pulsar.md
+++ b/docs/content/docs/connectors/datastream/pulsar.md
@@ -417,4 +417,13 @@ If you have a problem with Pulsar when using Flink, keep in mind that Flink only
 and your problem might be independent of Flink and sometimes can be solved by upgrading Pulsar brokers,
 reconfiguring Pulsar brokers or reconfiguring Pulsar connector in Flink.
 
+### Messages can be delayed on low volume topics
+
+When the Pulsar source connector reads from a low volume topic, users might observe a 10 seconds delay between messages. Pulsar buffers messages from topics by default. Before emitting to downstream
+operators, the number of buffered records must be equal or larger than `PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS`. If the data volume is low, it could be that filling up the number of buffered records takes longer than `PULSAR_MAX_FETCH_TIME` (default to 10 seconds). If that's the case, it means that only after this time has passed the messages will be emitted. 
+
+To avoid this behaviour, you need to change either the buffered records or the waiting time. 
+
+
+
 {{< top >}}