You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@rocketmq.apache.org by zh...@apache.org on 2023/02/02 07:05:44 UTC
[rocketmq-connect] branch master updated: Update README.md (#417)
This is an automated email from the ASF dual-hosted git repository.
zhoubo pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/rocketmq-connect.git
The following commit(s) were added to refs/heads/master by this push:
new c6051929 Update README.md (#417)
c6051929 is described below
commit c60519291f16fbb7f4a26a0bd00d233ce2636326
Author: Ao Qiao <qi...@foxmail.com>
AuthorDate: Thu Feb 2 15:05:38 2023 +0800
Update README.md (#417)
---
connectors/rocketmq-connect-jdbc/README.md | 97 ++++++++++++++++++------------
1 file changed, 59 insertions(+), 38 deletions(-)
diff --git a/connectors/rocketmq-connect-jdbc/README.md b/connectors/rocketmq-connect-jdbc/README.md
index d98fdb1b..003c91e6 100644
--- a/connectors/rocketmq-connect-jdbc/README.md
+++ b/connectors/rocketmq-connect-jdbc/README.md
@@ -1,16 +1,39 @@
-# rocketmq-connect-doris
+# rocketmq-connect-jdbc
+
+为方便扩展，rocketmq-connect-jdbc目前采用模块化插件的形式进行组织，内部的connector需要用户手动编译成jar包来进行使用。目前支持Mysql、OpenMLDB等数据库
+
+## rocketmq-connect-jdbc使用方法
+
+1. 进入想要使用的connector
+ 目录下（以rocketmq-connect-jdbc-mysql目录为例），使用以下指令将此connector打包为jar文件
+
+ ```shell
+ mvn clean package -Dmaven.test.skip=true
+ ```
+2. 打包好的jar文件将出现在`rocketmq-connect-jdbc-mysql/target/`目录下
+3. 在`distribution/conf`目录下找的对应的配置文件进行更新，对于standalone的启动方式，更新`connect-standalone.conf`文件中的`pluginPaths`变量
+
+ ```lombok.config
+ pluginPaths=rocketmq-connect-sample/target/rocketmq-connect-jdbc-mysql-0.0.1-SNAPSHOT-jar-with-dependencies.jar
+ ```
+
+ 相应的，使用distributed启动方式，则更新`connect-distributed.conf`中的变量
+4. 在源数据库和目的数据库准备好需要使用的表，可参考[RocketMQ Connect实战1](https://rocketmq.apache.org/zh/docs/%e6%95%b0%e6%8d%ae%e9%9b%86%e6%88%90/28RocketMQ%20Connect%20In%20Action1/)
+5. 创建并启动对应的`SourceConnector`以及`SinkConnector`
+
+
+
+## 示例：jdbc -> Rocketmq -> doirs
-```
注: 目前支持将根据mysql ,openMLDB对应的source connect以流式的方式导入到rocketmq中的数据导入到doris
-```
-## rocketmq-connect-doirs 打包
+### 打包connectors并配置conf文件
```
mvn clean package -Dmaven.test.skip=true
```
-## rocketmq-connect-doris 启动
+### 启动connector
* **jdbc-source-connector** 启动
@@ -29,7 +52,6 @@ POST http://${runtime-ip}:${runtime-port}/connectors/${rocketmq-jdbc-source-con
"key.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter",
"value.converter":"org.apache.rocketmq.connect.runtime.converter.record.json.JsonConverter"
}
-
```
* **doris-sink-connector** 启动
@@ -59,48 +81,47 @@ POST http://${runtime-ip}:${runtime-port}/connectors/${rocketmq-jdbc-sink-conne
> **注：** `rocketmq-jdbc-connect` 的启动依赖于`rocketmq-connect-runtime`项目的启动，需将打好的所有`jar`包放置到`runtime`项目中`pluginPaths`配置的路径后再执行上面的启动请求,该值配置在`runtime`项目下的`connect.conf`文件中
-## rocketmq-connect-jdbc 停止
+### rocketmq-connect-jdbc 停止
```
http://${runtime-ip}:${runtime-port}/connectors/${rocketmq-jdbc-connector-name}/stop
```
-## rocketmq-connect-jdbc 参数说明
+### rocketmq-connect-jdbc 参数说明
* **jdbc-source-connector 参数说明**
-| KEY | TYPE | Must be filled | Description| Example
-|------------------------|----|---------|---------------|------------------|
-|connection.url | String | YES | source端 jdbc连接 | jdbc:mysql://XXXXXXXXX:3306|
-|connection.user | String | YES | source端 DB 用户名 | root |
-|connection.password | String | YES | source端 DB 密码 | root |
-|connection.attempts | String | YES | source端 DB连接重试次数 | 3 |
-|connection.backoff.ms | Long | YES | |
-|poll.interval.ms | Long | YES |拉取间隔时间 | 3000ms |
-|batch.max.rows | Integer | NO |每次拉取数量 | 300 |
-|mode | Integer | NO |拉取模式 | bulk、timestamp、incrementing、timestamp+incrementing |
-|incrementing.column.name | Integer | NO |增量字段，常用ID | id |
-|timestamp.column.name | String | YES |时间增量字段 | modified_time |
-|table.whitelist | String | YES |需要扫描的表 | db.table,db.table01 |
-|max-task | Integer | YES |任务数量，最大不能大于表的数量 | 2 |
-|source-record-converter | Integer | YES |data转换器 | org.apache.rocketmq.connect.doris.converter.JsonConverter |
-
-```
+| KEY | TYPE | Must be filled | Description | Example |
+| ------------------------ | ------- | -------------- | ---------------- | --------------------------------------------------------- |
+| connection.url | String | YES | source端 jdbc连接 | jdbc:mysql://XXXXXXXXX:3306 |
+| connection.user | String | YES | source端 DB 用户名 | root |
+| connection.password | String | YES | source端 DB 密码 | root |
+| connection.attempts | String | YES | source端 DB连接重试次数 | 3 |
+| connection.backoff.ms | Long | YES | | |
+| poll.interval.ms | Long | YES | 拉取间隔时间 | 3000ms |
+| batch.max.rows | Integer | NO | 每次拉取数量 | 300 |
+| mode | Integer | NO | 拉取模式 | bulk、timestamp、incrementing、timestamp+incrementing |
+| incrementing.column.name | Integer | NO | 增量字段，常用ID | id |
+| timestamp.column.name | String | YES | 时间增量字段 | modified_time |
+| table.whitelist | String | YES | 需要扫描的表 | db.table,db.table01 |
+| max-task | Integer | YES | 任务数量，最大不能大于表的数量 | 2 |
+| source-record-converter | Integer | YES | data转换器 | org.apache.rocketmq.connect.doris.converter.JsonConverter |
+
+```
注：1.source拉取的数据写入到以表名自动创建的topic中，如果需要写入特定的topic中则需要指定"connect-topicname" 参数
2.topic.prefix参数可以为自动创建的topic增加前缀，用来进行逻辑的隔离
-```
+```
* **jdbc-sink-connector 参数说明**
-| KEY | TYPE | Must be filled | Description| Example
-|------------------------|----|---------|---------------|------------------|
-|connection.url | String | YES | sink端 jdbc连接 | jdbc:mysql://XXXXXXXXX:3306|
-|connection.user | String | YES | sink端 DB 用户名 | root |
-|connection.password | String | YES | sink端 DB 密码 | root |
-|host | String | YES |doris host | 192.168.0.1 |
-|port | String | YES |doris http port | 8030 |
-|user | String | YES |监听的topic | root |
-|passwd | String | YES |监听的topic | passwd |
-|max-task | Integer | NO |任务数量 | 2 |
-|source-record-converter | Integer | YES |data转换器 | org.apache.rocketmq.connect.doris.converter.JsonConverter |
-```
+| KEY | TYPE | Must be filled | Description | Example |
+| ----------------------- | ------- | -------------- | --------------- | --------------------------------------------------------- |
+| connection.url | String | YES | sink端 jdbc连接 | jdbc:mysql://XXXXXXXXX:3306 |
+| connection.user | String | YES | sink端 DB 用户名 | root |
+| connection.password | String | YES | sink端 DB 密码 | root |
+| host | String | YES | doris host | 192.168.0.1 |
+| port | String | YES | doris http port | 8030 |
+| user | String | YES | 监听的topic | root |
+| passwd | String | YES | 监听的topic | passwd |
+| max-task | Integer | NO | 任务数量 | 2 |
+| source-record-converter | Integer | YES | data转换器 | org.apache.rocketmq.connect.doris.converter.JsonConverter |