You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by ki...@apache.org on 2022/09/30 09:29:26 UTC
[incubator-seatunnel] branch dev updated: [Doc] [Connector-V2] Add connector-v2 kafka source doc (#2953)
This is an automated email from the ASF dual-hosted git repository.
kirs pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new ed7a75cfe [Doc] [Connector-V2] Add connector-v2 kafka source doc (#2953)
ed7a75cfe is described below
commit ed7a75cfe54b0af367a5fffda996318399a8b2af
Author: TaoZex <45...@users.noreply.github.com>
AuthorDate: Fri Sep 30 17:29:20 2022 +0800
[Doc] [Connector-V2] Add connector-v2 kafka source doc (#2953)
* kafka source doc
---
docs/en/connector-v2/source/kafka.md | 89 ++++++++++++++++++++++++++++++++++++
1 file changed, 89 insertions(+)
diff --git a/docs/en/connector-v2/source/kafka.md b/docs/en/connector-v2/source/kafka.md
new file mode 100644
index 000000000..40b1ae5ab
--- /dev/null
+++ b/docs/en/connector-v2/source/kafka.md
@@ -0,0 +1,89 @@
+# Kafka
+
+> Kafka source connector
+
+## Description
+
+Source connector for Apache Kafka.
+
+## Key features
+
+- [x] [batch](../../concept/connector-v2-features.md)
+- [x] [stream](../../concept/connector-v2-features.md)
+- [x] [exactly-once](../../concept/connector-v2-features.md)
+- [ ] [schema projection](../../concept/connector-v2-features.md)
+- [x] [parallelism](../../concept/connector-v2-features.md)
+- [ ] [support user-defined split](../../concept/connector-v2-features.md)
+
+## Options
+
+| name | type | required | default value |
+| -------------------- | ------- | -------- | ------------------------ |
+| topic | String | yes | - |
+| bootstrap.servers | String | yes | - |
+| pattern | Boolean | no | false |
+| consumer.group | String | no | SeaTunnel-Consumer-Group |
+| commit_on_checkpoint | Boolean | no | true |
+| kafka.* | String | no | - |
+| common-options | | no | - |
+
+### topic [string]
+
+`Kafka topic` name. If there are multiple `topics`, use `,` to split, for example: `"tpc1,tpc2"`.
+
+### bootstrap.servers [string]
+
+`Kafka` cluster address, separated by `","`.
+
+### pattern [boolean]
+
+If `pattern` is set to `true`,the regular expression for a pattern of topic names to read from. All topics in clients with names that match the specified regular expression will be subscribed by the consumer.
+
+### consumer.group [string]
+
+`Kafka consumer group id`, used to distinguish different consumer groups.
+
+### commit_on_checkpoint [boolean]
+
+If true the consumer's offset will be periodically committed in the background.
+
+### kafka.* [string]
+
+In addition to the above necessary parameters that must be specified by the `Kafka consumer` client, users can also specify multiple `consumer` client non-mandatory parameters, covering [all consumer parameters specified in the official Kafka document](https://kafka.apache.org/documentation.html#consumerconfigs).
+
+The way to specify parameters is to add the prefix `kafka.` to the original parameter name. For example, the way to specify `auto.offset.reset` is: `kafka.auto.offset.reset = latest` . If these non-essential parameters are not specified, they will use the default values given in the official Kafka documentation.
+
+### common-options
+
+Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details.
+
+## Example
+
+### Simple
+
+```hocon
+source {
+
+ Kafka {
+ topic = "seatunnel"
+ bootstrap.servers = "localhost:9092"
+ consumer.group = "seatunnel_group"
+ }
+
+}
+```
+
+### Regex Topic
+
+```hocon
+source {
+
+ Kafka {
+ topic = ".*seatunnel*."
+ pattern = "true"
+ bootstrap.servers = "localhost:9092"
+ consumer.group = "seatunnel_group"
+ }
+
+}
+```
\ No newline at end of file