You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/06/17 08:31:50 UTC
[GitHub] [rocketmq-connect] duhenglucky commented on a diff in pull request #175: [ISSUES #174]optimize quick start
duhenglucky commented on code in PR #175:
URL: https://github.com/apache/rocketmq-connect/pull/175#discussion_r899894234
##########
README.md:
##########
@@ -1,114 +1,70 @@
# rocketmq-connect
[](https://www.apache.org/licenses/LICENSE-2.0.html)
-# GitBook文档
-
-[快速开始](https://rocketmq-1.gitbook.io/rocketmq-connector/quick-start/qian-qi-zhun-bei)
-
[Runtime](https://rocketmq-1.gitbook.io/rocketmq-connector/quick-start/runtime-qs)
# 快速开始
-文档以rocketmq-connect-sample作为demo
+以rocketmq-connect-sample作为demo
+
+rocketmq-connect-sample的主要作用是从文件中读取数据发送到RocketMQ集群
+然后从Topic中读取消息，写入到目标文件
## 1.准备
1. 64bit JDK 1.8+;
2. Maven 3.2.x或以上版本;
-3. Start the [RocketMQ](https://rocketmq.apache.org/docs/quick-start/) Service;
-
-4. 创建 connectors 存放目录:
-```
-mkdir ~/connectors
-```
+3. 启动RocketMQ， [RocketMQ](https://rocketmq.apache.org/docs/quick-start/);
-5. 创建 connector runtime 配置存放目录:
-```
-mkdir ~/storeRoot
-```
-## 2.构建
+## 2.构建Connect
```
-mvn clean install -Dmaven.test.skip=true
-mv rocketmq-connect-sample/target/rocketmq-connect-sample-0.0.1-SNAPSHOT.jar ~/connectors
-```
-
-## 3.配置
+git clone git@github.com:apache/rocketmq-connect.git
+cd rocketmq-connect
-cd rocketmq-connect-runtime/target/distribution/conf
-1. 修改配置文件 vi connect-distributed.conf
+mvn -Prelease-connect -DskipTests clean install -U
```
-#current cluster node uniquely identifies
-workerId=DEFAULT_WORKER_1
-
-# Http prot for user to access REST API
-httpPort=8081
-
-# Local file dir for config store
-storePathRootDir=~/storeRoot
-
-#需要修改为自己的rocketmq nameserver 接入点
-# RocketMQ namesrvAddr
-namesrvAddr=127.0.0.1:9876
-
-#修改为 1.5 小节中创建的connectors 加载目录，加载 sample 中 source/sink connector
-# Source or sink connector jar file dir
-pluginPaths=~/connectors
-```
-## 4.运行
+## 3.运行Worker
-返回 rocketmq-connect-runtime 目录执行：
-
-```
-sh ./connect-distributed.sh
```
+cd distribution/target/rocketmq-connect-0.0.1-SNAPSHOT/rocketmq-connect-0.0.1-SNAPSHOT
-查看日志文件${user.home}/logs/rocketmqconnect/connect_runtime.log
-以下日志表示runtime启动成功：
+sh bin/connect-standalone.sh -c conf/connect-standalone.conf &
-The worker [DEFAULT_WORKER_1] boot success.
-
-```
-注：启动之前 RocketMQ 创建以下topic
-connector-cluster-topic 集群信息
-connector-config-topic 配置信息
-connector-offset-topic sink消费进度
-connector-position-topic source数据处理进度
-并且为了保证消息有序，每个 topic 可以只建一个 queue
```
-## 5.日志目录
+查看日志文件
- ${user.home}/logs/rocketmqconnect
+tail -100f ~/logs/rocketmqconnect/connect_runtime.log
-## 6.配置文件
+以下日志表示runtime启动成功：
-持久化配置文件默认目录 ～/storeRoot
+The standalone worker boot success.
Review Comment:
需要在最后面介绍standalone 跟distributed 的区别，以及增加 distribute 的部署方式
##########
README.md:
##########
@@ -1,114 +1,70 @@
# rocketmq-connect
[](https://www.apache.org/licenses/LICENSE-2.0.html)
-# GitBook文档
-
-[快速开始](https://rocketmq-1.gitbook.io/rocketmq-connector/quick-start/qian-qi-zhun-bei)
-
[Runtime](https://rocketmq-1.gitbook.io/rocketmq-connector/quick-start/runtime-qs)
# 快速开始
-文档以rocketmq-connect-sample作为demo
+以rocketmq-connect-sample作为demo
+
+rocketmq-connect-sample的主要作用是从文件中读取数据发送到RocketMQ集群
+然后从Topic中读取消息，写入到目标文件
## 1.准备
1. 64bit JDK 1.8+;
2. Maven 3.2.x或以上版本;
-3. Start the [RocketMQ](https://rocketmq.apache.org/docs/quick-start/) Service;
-
-4. 创建 connectors 存放目录:
-```
-mkdir ~/connectors
-```
+3. 启动RocketMQ， [RocketMQ](https://rocketmq.apache.org/docs/quick-start/);
-5. 创建 connector runtime 配置存放目录:
-```
-mkdir ~/storeRoot
-```
-## 2.构建
+## 2.构建Connect
```
-mvn clean install -Dmaven.test.skip=true
-mv rocketmq-connect-sample/target/rocketmq-connect-sample-0.0.1-SNAPSHOT.jar ~/connectors
-```
-
-## 3.配置
+git clone git@github.com:apache/rocketmq-connect.git
+cd rocketmq-connect
-cd rocketmq-connect-runtime/target/distribution/conf
-1. 修改配置文件 vi connect-distributed.conf
+mvn -Prelease-connect -DskipTests clean install -U
```
-#current cluster node uniquely identifies
-workerId=DEFAULT_WORKER_1
-
-# Http prot for user to access REST API
-httpPort=8081
-
-# Local file dir for config store
-storePathRootDir=~/storeRoot
-
-#需要修改为自己的rocketmq nameserver 接入点
-# RocketMQ namesrvAddr
-namesrvAddr=127.0.0.1:9876
-
-#修改为 1.5 小节中创建的connectors 加载目录，加载 sample 中 source/sink connector
-# Source or sink connector jar file dir
-pluginPaths=~/connectors
-```
-## 4.运行
+## 3.运行Worker
-返回 rocketmq-connect-runtime 目录执行：
-
-```
-sh ./connect-distributed.sh
```
+cd distribution/target/rocketmq-connect-0.0.1-SNAPSHOT/rocketmq-connect-0.0.1-SNAPSHOT
-查看日志文件${user.home}/logs/rocketmqconnect/connect_runtime.log
-以下日志表示runtime启动成功：
+sh bin/connect-standalone.sh -c conf/connect-standalone.conf &
-The worker [DEFAULT_WORKER_1] boot success.
-
-```
-注：启动之前 RocketMQ 创建以下topic
-connector-cluster-topic 集群信息
-connector-config-topic 配置信息
-connector-offset-topic sink消费进度
-connector-position-topic source数据处理进度
-并且为了保证消息有序，每个 topic 可以只建一个 queue
```
-## 5.日志目录
+查看日志文件
- ${user.home}/logs/rocketmqconnect
+tail -100f ~/logs/rocketmqconnect/connect_runtime.log
-## 6.配置文件
+以下日志表示runtime启动成功：
-持久化配置文件默认目录 ～/storeRoot
+The standalone worker boot success.
-1. connectorConfig.json connector配置持久化文件
-2. position.json source connect数据处理进度持久化文件
-3. taskConfig.json task配置持久化文件
-4. offset.json sink connect数据消费进度持久化文件
+ctrl + c 退出日志
-## 7.启动source connector
-
-提前创建测试文件~/test/test-source-file.txt，随意写一些内容在里面
+## 4.启动source connector
+当前目录创建测试文件test-source-file.txt
```
-curl -X POST -H "Content-Type: application/json" http://127.0.0.1:8082/connectors/fileSourceConnector -d '{"connector-class":"org.apache.rocketmq.connect.file.FileSourceConnector","filename":"~/test/test-source-file.txt","connect-topicname":"fileTopic"}'
+touch test-source-file.txt
+
+echo "Hello \r\nRocketMQ\r\n Connect" > test-source-file.txt
+
+curl -X POST -H "Content-Type: application/json" http://127.0.0.1:8082/connectors/fileSourceConnector -d '{"connector-class":"org.apache.rocketmq.connect.file.FileSourceConnector","filename":"test-source-file.txt","connect-topicname":"fileTopic"}'
```
- 看到一下日志说明file source connector启动成功了
-
- 2019-07-16 11:18:39 INFO pool-7-thread-1 - Source task start, config:{"properties":{"source-record-...
+看到一下日志说明file source connector启动成功了
Review Comment:
一下 改为 以下
##########
README.md:
##########
@@ -119,49 +75,94 @@ curl -X POST -H "Content-Type: application/json" http://127.0.0.1:8082/connector
| filename | false | | 数据源文件名称 |
| task-class | false | | 实现SourceTask类名称（包含包名） |
| connect-topicname | false | | 同步文件数据所需topic |
-| update-timestamp | false | | 配置更新时间戳 |
-| source-record-converter | false | | Full class name of the impl of the converter used to convert SourceDataEntry to byte[] |
-## 8.启动sink connector
+## 5.启动sink connector
```
-curl -X POST -H "Content-Type: application/json" http://127.0.0.1:8082/connectors/fileSinkConnector -d '{"connector-class":"org.apache.rocketmq.connect.file.FileSinkConnector","filename":"~/test/test-sink-file.txt","connect-topicname":"fileTopic"}'
+curl -X POST -H "Content-Type: application/json" http://127.0.0.1:8082/connectors/fileSinkConnector -d '{"connector-class":"org.apache.rocketmq.connect.file.FileSinkConnector","filename":"test-sink-file.txt","connect-topicname":"fileTopic"}'
+
+cat test-sink-file.txt
```
看到一下日志说明file sink connector启动成功了
2019-07-16 11:24:58 INFO pool-7-thread-2 - Sink task start, config:{"properties":{"source-record-...
-查看配置中"filename":"/home/connect/rocketmq-externals/rocketmq-connect-runtime/sink-file.txt"配置文件
-如果sink-file.txt生成并且与source-file.txt内容一样，说明整个流程已经跑通
+test-sink-file.txt文件即内容
+如果test-sink-file.txt生成并且与source-file.txt内容一样，说明整个流程正常运行
#### sink connector配置说明
| key | nullable | default | description |
| ----------------------- | -------- | ------- | -------------------------------------------------------------------------------------- |
| connector-class | false | | 实现Connector接口的类名称（包含包名） |
| connect-topicname | false | | sink需要处理数据消息topics |
-| task-class | false | | 实现SourceTask类名称（包含包名） |
| filename | false | | sink拉去的数据保存到文件 |
-| update-timestamp | false | | 配置更新时间戳 |
-| source-record-converter | false | | Full class name of the impl of the converter used to convert SourceDataEntry to byte[] |
```
注：source/sink配置文件说明是以rocketmq-connect-sample为demo，不同source/sink connector配置有差异，请以具体sourc/sink connector为准
```
-## 9.停止connector
+## 6.停止connector
```
GET请求
http://(your worker ip):(port)/connectors/(connector name)/stop
+
+ 停止demo中的两个connector
+ curl http://127.0.0.1:8082/connectors/fileSinkConnector/stop
+ curl http://127.0.0.1:8082/connectors/fileSourceConnector/stop
+
```
看到一下日志说明connector停止成功了
Source task stop, config:{"properties":{"source-record-converter":"org.apache.rocketmq.connect.runtime.converter.JsonConverter","filename":"/home/zhoubo/IdeaProjects/my-new3-rocketmq-externals/rocketmq-connect/rocketmq-connect-runtime/source-file.txt","task-class":"org.apache.rocketmq.connect.file.FileSourceTask","topic":"fileTopic","connector-class":"org.apache.rocketmq.connect.file.FileSourceConnector","update-timestamp":"1564765189322"}}
+## 7.停止Worker进程
+
+```
+sh bin/connectshutdown.sh
+
+```
+
+
+## 8.日志目录
+
+ ${user.home}/logs/rocketmqconnect
+
+## 9.配置文件
+
+持久化配置文件默认目录 ～/storeRoot
+
+1. connectorConfig.json connector配置持久化文件
+2. position.json source connect数据处理进度持久化文件
+3. taskConfig.json task配置持久化文件
+4. offset.json sink connect数据消费进度持久化文件
+
+## 10.配置说明
+
+可根据使用情况修改端口，目录，RocketMQ链接等信息
+```
+#current cluster node uniquely identifies
+workerId=DEFAULT_WORKER_1
+
+# Http prot for user to access REST API
+httpPort=8081
+
+# Local file dir for config store
+storePathRootDir=/home/connect/storeRoot
+
+#需要修改为自己的rocketmq nameserver 接入点
+# RocketMQ namesrvAddr
+namesrvAddr=127.0.0.1:9876
+
+#用于加载Connector插件，类似于jvm启动加载jar包或者class类，这里目录目录用于放Connector相关的实现插件，
+支持文件和目录
+# Source or sink connector jar file dir
+pluginPaths=rocketmq-connect-sample/target/rocketmq-connect-sample-0.0.1-SNAPSHOT.jar
+```
-## 10.其它restful接口
+## 11.其它restful接口
Review Comment:
增加rest 接口中每个变量的含义
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org