You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by ki...@apache.org on 2022/09/30 09:29:26 UTC

[incubator-seatunnel] branch dev updated: [Doc] [Connector-V2] Add connector-v2 kafka source doc (#2953)

This is an automated email from the ASF dual-hosted git repository.

kirs pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-seatunnel.git


The following commit(s) were added to refs/heads/dev by this push:
     new ed7a75cfe [Doc] [Connector-V2] Add connector-v2 kafka source doc (#2953)
ed7a75cfe is described below

commit ed7a75cfe54b0af367a5fffda996318399a8b2af
Author: TaoZex <45...@users.noreply.github.com>
AuthorDate: Fri Sep 30 17:29:20 2022 +0800

    [Doc] [Connector-V2] Add connector-v2 kafka source doc (#2953)
    
    * kafka source doc
---
 docs/en/connector-v2/source/kafka.md | 89 ++++++++++++++++++++++++++++++++++++
 1 file changed, 89 insertions(+)

diff --git a/docs/en/connector-v2/source/kafka.md b/docs/en/connector-v2/source/kafka.md
new file mode 100644
index 000000000..40b1ae5ab
--- /dev/null
+++ b/docs/en/connector-v2/source/kafka.md
@@ -0,0 +1,89 @@
+# Kafka
+
+> Kafka source connector
+
+## Description
+
+Source connector for Apache Kafka.
+
+## Key features
+
+- [x] [batch](../../concept/connector-v2-features.md)
+- [x] [stream](../../concept/connector-v2-features.md)
+- [x] [exactly-once](../../concept/connector-v2-features.md)
+- [ ] [schema projection](../../concept/connector-v2-features.md)
+- [x] [parallelism](../../concept/connector-v2-features.md)
+- [ ] [support user-defined split](../../concept/connector-v2-features.md)
+
+## Options
+
+| name                 | type    | required | default value            |
+| -------------------- | ------- | -------- | ------------------------ |
+| topic                | String  | yes      | -                        |
+| bootstrap.servers    | String  | yes      | -                        |
+| pattern              | Boolean | no       | false                    |
+| consumer.group       | String  | no       | SeaTunnel-Consumer-Group |
+| commit_on_checkpoint | Boolean | no       | true                     |
+| kafka.*              | String  | no       | -                        |
+| common-options       |         | no       | -                        |
+
+### topic [string]
+
+`Kafka topic` name. If there are multiple `topics`, use `,` to split, for example: `"tpc1,tpc2"`.
+
+### bootstrap.servers [string]
+
+`Kafka` cluster address, separated by `","`.
+
+### pattern [boolean]
+
+If `pattern` is set to `true`,the regular expression for a pattern of topic names to read from. All topics in clients with names that match the specified regular expression will be subscribed by the consumer.
+
+### consumer.group [string]
+
+`Kafka consumer group id`, used to distinguish different consumer groups.
+
+### commit_on_checkpoint [boolean]
+
+If true the consumer's offset will be periodically committed in the background.
+
+### kafka.* [string]
+
+In addition to the above necessary parameters that must be specified by the `Kafka consumer` client, users can also specify multiple `consumer` client non-mandatory parameters, covering [all consumer parameters specified in the official Kafka document](https://kafka.apache.org/documentation.html#consumerconfigs).
+
+The way to specify parameters is to add the prefix `kafka.` to the original parameter name. For example, the way to specify `auto.offset.reset` is: `kafka.auto.offset.reset = latest` . If these non-essential parameters are not specified, they will use the default values given in the official Kafka documentation.
+
+### common-options
+
+Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details.
+
+## Example
+
+###  Simple
+
+```hocon
+source {
+
+    Kafka {
+          topic = "seatunnel"
+          bootstrap.servers = "localhost:9092"
+          consumer.group = "seatunnel_group"
+    }
+
+}
+```
+
+### Regex Topic
+
+```hocon
+source {
+
+    Kafka {
+          topic = ".*seatunnel*."
+          pattern = "true" 
+          bootstrap.servers = "localhost:9092"
+          consumer.group = "seatunnel_group"
+    }
+
+}
+```
\ No newline at end of file