You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by ar...@apache.org on 2021/09/01 07:14:03 UTC
[flink] branch release-1.14 updated (ad44b55 -> ff8f33a)
This is an automated email from the ASF dual-hosted git repository.
arvid pushed a change to branch release-1.14
in repository https://gitbox.apache.org/repos/asf/flink.git.
from ad44b55 [FLINK-24029][docs] Fix "dept_id" typo in SQL "Getting Started" page
new 7eba94b [hotfix][docs] Soft-deprecating Streaming File Sink
new 4062e1b [FLINK-23817][docs] Add metric description to Kafka source.
new ff8f33a [hotfix][docs] Remove general operator task/operator metrics in Kafka Sink docs.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/content/docs/connectors/datastream/kafka.md | 113 +++++++++++++--------
.../content/docs/connectors/datastream/overview.md | 1 -
.../docs/connectors/datastream/streamfile_sink.md | 5 +
docs/content/docs/ops/metrics.md | 47 +--------
4 files changed, 78 insertions(+), 88 deletions(-)
[flink] 02/03: [FLINK-23817][docs] Add metric description to Kafka
source.
Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
arvid pushed a commit to branch release-1.14
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 4062e1b24e8becf398fe76bd40a407421c21e1de
Author: Arvid Heise <ar...@ververica.com>
AuthorDate: Tue Aug 31 08:58:56 2021 +0200
[FLINK-23817][docs] Add metric description to Kafka source.
---
docs/content/docs/connectors/datastream/kafka.md | 86 +++++++++++++++++++-----
docs/content/docs/ops/metrics.md | 47 +------------
2 files changed, 72 insertions(+), 61 deletions(-)
diff --git a/docs/content/docs/connectors/datastream/kafka.md b/docs/content/docs/connectors/datastream/kafka.md
index 8f2c56e..d49303a 100644
--- a/docs/content/docs/connectors/datastream/kafka.md
+++ b/docs/content/docs/connectors/datastream/kafka.md
@@ -212,26 +212,82 @@ Note that Kafka source does **NOT** rely on committed offsets for fault toleranc
is only for exposing the progress of consumer and consuming group for monitoring.
### Monitoring
-Kafka source exposes metrics in Flink's metric group for monitoring and diagnosing.
+
+Kafka source exposes the following metrics in the respective [scope]({{< ref "docs/ops/metrics" >}}/#scope).
+
#### Scope of Metric
-All metrics of Kafka source reader are registered under group ```KafkaSourceReader```, which is a
-child group of operator metric group. Metrics related to a specific topic partition will be registered
-in the group ```KafkaSourceReader.topic.<topic_name>.partition.<partition_id>```.
-For example, current consuming offset of topic "my-topic" and partition 1 will be reported in metric:
-```<some_parent_groups>.operator.KafkaSourceReader.topic.my-topic.partition.1.currentOffset``` ,
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 15%">Scope</th>
+ <th class="text-left" style="width: 18%">Metrics</th>
+ <th class="text-left" style="width: 18%">User Variables</th>
+ <th class="text-left" style="width: 39%">Description</th>
+ <th class="text-left" style="width: 10%">Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <th rowspan="8">Operator</th>
+ <td>currentEmitEventTimeLag</td>
+ <td>n/a</td>
+ <td>The time span from the record event timestamp to the time the record is emitted by the source connector¹: <code>currentEmitEventTimeLag = EmitTime - EventTime.</code></td>
+ <td>Gauge</td>
+ </tr>
+ <tr>
+ <td>watermarkLag</td>
+ <td>n/a</td>
+ <td>The time span that the watermark lags behind the wall clock time: <code>watermarkLag = CurrentTime - Watermark</code></td>
+ <td>Gauge</td>
+ </tr>
+ <tr>
+ <td>sourceIdleTime</td>
+ <td>n/a</td>
+ <td>The time span that the source has not processed any record: <code>sourceIdleTime = CurrentTime - LastRecordProcessTime</code></td>
+ <td>Gauge</td>
+ </tr>
+ <tr>
+ <td>pendingRecords</td>
+ <td>n/a</td>
+ <td>The number of records that have not been fetched by the source. e.g. the available records after the consumer offset in a Kafka partition.</td>
+ <td>Gauge</td>
+ </tr>
+ <tr>
+ <td>KafkaSourceReader.commitsSucceeded</td>
+ <td>n/a</td>
+ <td>The total number of successful offset commits to Kafka, if offset committing is turned on and checkpointing is enabled.</td>
+ <td>Counter</td>
+ </tr>
+ <tr>
+ <td>KafkaSourceReader.commitsFailed</td>
+ <td>n/a</td>
+ <td>The total number of offset commit failures to Kafka, if offset committing is
+ turned on and checkpointing is enabled. Note that committing offsets back to Kafka
+ is only a means to expose consumer progress, so a commit failure does not affect
+ the integrity of Flink's checkpointed partition offsets.</td>
+ <td>Counter</td>
+ </tr>
+ <tr>
+ <td>KafkaSourceReader.committedOffsets</td>
+ <td>topic, partition</td>
+ <td>The last successfully committed offsets to Kafka, for each partition.
+ A particular partition's metric can be specified by topic name and partition id.</td>
+ <td>Gauge</td>
+ </tr>
+ <tr>
+ <td>KafkaSourceReader.currentOffsets</td>
+ <td>topic, partition</td>
+ <td>The consumer's current read offset, for each partition. A particular
+ partition's metric can be specified by topic name and partition id.</td>
+ <td>Gauge</td>
+ </tr>
+ </tbody>
+</table>
-and number of successful commits will be reported in metric:
-```<some_parent_groups>.operator.KafkaSourceReader.commitsSucceeded``` .
-#### List of Metrics
+¹ This metric is an instantaneous value recorded for the last processed record. This metric is provided because latency histogram could be expensive. The instantaneous latency value is usually a good enough indication of the latency.
-| Metric Name | Description | Scope |
-|:----------------:|:-----------------------------------------------:|:-----------------:|
-| currentOffset | Current consuming offset of the topic partition | TopicPartition |
-| committedOffset | Committed offset of the topic partition | TopicPartition |
-| commitsSucceeded | Number of successful commits | KafkaSourceReader |
-| commitsFailed | Number of failed commits | KafkaSourceReader |
#### Kafka Consumer Metrics
All metrics of Kafka consumer are also registered under group ```KafkaSourceReader.KafkaConsumer```.
diff --git a/docs/content/docs/ops/metrics.md b/docs/content/docs/ops/metrics.md
index 53b59d9..1580136 100644
--- a/docs/content/docs/ops/metrics.md
+++ b/docs/content/docs/ops/metrics.md
@@ -1345,52 +1345,7 @@ Certain RocksDB native metrics are available but disabled by default, you can fi
### Connectors
#### Kafka Connectors
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 15%">Scope</th>
- <th class="text-left" style="width: 18%">Metrics</th>
- <th class="text-left" style="width: 18%">User Variables</th>
- <th class="text-left" style="width: 39%">Description</th>
- <th class="text-left" style="width: 10%">Type</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <th rowspan="1">Operator</th>
- <td>commitsSucceeded</td>
- <td>n/a</td>
- <td>The total number of successful offset commits to Kafka, if offset committing is turned on and checkpointing is enabled.</td>
- <td>Counter</td>
- </tr>
- <tr>
- <th rowspan="1">Operator</th>
- <td>commitsFailed</td>
- <td>n/a</td>
- <td>The total number of offset commit failures to Kafka, if offset committing is
- turned on and checkpointing is enabled. Note that committing offsets back to Kafka
- is only a means to expose consumer progress, so a commit failure does not affect
- the integrity of Flink's checkpointed partition offsets.</td>
- <td>Counter</td>
- </tr>
- <tr>
- <th rowspan="1">Operator</th>
- <td>committedOffsets</td>
- <td>topic, partition</td>
- <td>The last successfully committed offsets to Kafka, for each partition.
- A particular partition's metric can be specified by topic name and partition id.</td>
- <td>Gauge</td>
- </tr>
- <tr>
- <th rowspan="1">Operator</th>
- <td>currentOffsets</td>
- <td>topic, partition</td>
- <td>The consumer's current read offset, for each partition. A particular
- partition's metric can be specified by topic name and partition id.</td>
- <td>Gauge</td>
- </tr>
- </tbody>
-</table>
+Please refer to [Kafka monitoring]({{< ref "docs/connectors/datastream/kafka" >}}/#monitoring).
#### Kinesis Connectors
<table class="table table-bordered">
[flink] 03/03: [hotfix][docs] Remove general operator task/operator
metrics in Kafka Sink docs.
Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
arvid pushed a commit to branch release-1.14
in repository https://gitbox.apache.org/repos/asf/flink.git
commit ff8f33afb0383be5cbdeb0eee53933074fe80fa3
Author: Arvid Heise <ar...@ververica.com>
AuthorDate: Tue Aug 31 22:42:32 2021 +0200
[hotfix][docs] Remove general operator task/operator metrics in Kafka Sink docs.
---
docs/content/docs/connectors/datastream/kafka.md | 27 +-----------------------
1 file changed, 1 insertion(+), 26 deletions(-)
diff --git a/docs/content/docs/connectors/datastream/kafka.md b/docs/content/docs/connectors/datastream/kafka.md
index d49303a..22bfa05 100644
--- a/docs/content/docs/connectors/datastream/kafka.md
+++ b/docs/content/docs/connectors/datastream/kafka.md
@@ -437,32 +437,7 @@ Kafka sink exposes the following metrics in the respective [scope]({{< ref "docs
</thead>
<tbody>
<tr>
- <th rowspan="2">Task</th>
- <td>numBytesOut</td>
- <td>n/a</td>
- <td>The total number of output bytes since the sink started.</td>
- <td>Counter</td>
- </tr>
- <tr>
- <td>numBytesOutPerSecond</td>
- <td>n/a</td>
- <td>The output bytes per second.</td>
- <td>Meter</td>
- </tr>
- <tr>
- <th rowspan="3">Operator</th>
- <td>numRecordsOut</td>
- <td>n/a</td>
- <td>The total number of output records since the sink started.</td>
- <td>Counter</td>
- </tr>
- <tr>
- <td>numRecordsOutPerSecond</td>
- <td>n/a</td>
- <td>The output records per seconds.</td>
- <td>Meter</td>
- </tr>
- <tr>
+ <th rowspan="1">Operator</th>
<td>currentSendTime</td>
<td>n/a</td>
<td>The time it takes to send the last record. This metric is an instantaneous value recorded for the last processed record.</td>
[flink] 01/03: [hotfix][docs] Soft-deprecating Streaming File Sink
Posted by ar...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
arvid pushed a commit to branch release-1.14
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 7eba94b61005a8cf70d7f3afc3122bc3c8453116
Author: Arvid Heise <ar...@ververica.com>
AuthorDate: Tue Aug 31 08:06:24 2021 +0200
[hotfix][docs] Soft-deprecating Streaming File Sink
---
docs/content/docs/connectors/datastream/overview.md | 1 -
docs/content/docs/connectors/datastream/streamfile_sink.md | 5 +++++
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/docs/content/docs/connectors/datastream/overview.md b/docs/content/docs/connectors/datastream/overview.md
index b6b6f24..a558d76 100644
--- a/docs/content/docs/connectors/datastream/overview.md
+++ b/docs/content/docs/connectors/datastream/overview.md
@@ -42,7 +42,6 @@ Connectors provide code for interfacing with various third-party systems. Curren
* [Apache Cassandra]({{< ref "docs/connectors/datastream/cassandra" >}}) (sink)
* [Amazon Kinesis Streams]({{< ref "docs/connectors/datastream/kinesis" >}}) (source/sink)
* [Elasticsearch]({{< ref "docs/connectors/datastream/elasticsearch" >}}) (sink)
- * [FileSystem (Hadoop included) - Streaming only]({{< ref "docs/connectors/datastream/streamfile_sink" >}}) (sink)
* [FileSystem (Hadoop included) - Streaming and Batch]({{< ref "docs/connectors/datastream/file_sink" >}}) (sink)
* [RabbitMQ]({{< ref "docs/connectors/datastream/rabbitmq" >}}) (source/sink)
* [Apache NiFi]({{< ref "docs/connectors/datastream/nifi" >}}) (source/sink)
diff --git a/docs/content/docs/connectors/datastream/streamfile_sink.md b/docs/content/docs/connectors/datastream/streamfile_sink.md
index 9d277f0..c008898 100644
--- a/docs/content/docs/connectors/datastream/streamfile_sink.md
+++ b/docs/content/docs/connectors/datastream/streamfile_sink.md
@@ -4,6 +4,7 @@ weight: 6
type: docs
aliases:
- /dev/connectors/streamfile_sink.html
+bookHidden: true
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -29,6 +30,10 @@ under the License.
This connector provides a Sink that writes partitioned files to filesystems
supported by the [Flink `FileSystem` abstraction]({{< ref "docs/deployment/filesystems/overview" >}}).
+{{< hint warning >}}
+This Streaming File Sink is in the process of being phased out. Please use the unified [File Sink]({{< ref "docs/connectors/datastream/file_sink" >}}) as a drop-in replacement.
+{{< /hint >}}
+
The streaming file sink writes incoming data into buckets. Given that the incoming streams can be unbounded,
data in each bucket are organized into part files of finite size. The bucketing behaviour is fully configurable
with a default time-based bucketing where we start writing a new bucket every hour. This means that each resulting