You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by do...@apache.org on 2021/07/08 14:24:41 UTC
[incubator-inlong-website] branch master updated: [INLONG-690]
optimize readme.md in agent tubemanager proxy (#94)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new b5791c5 [INLONG-690] optimize readme.md in agent tubemanager proxy (#94)
b5791c5 is described below
commit b5791c53df5403e2ea5c20f174d0143a781c501d
Author: ziruipeng <zp...@connect.ust.hk>
AuthorDate: Thu Jul 8 22:23:15 2021 +0800
[INLONG-690] optimize readme.md in agent tubemanager proxy (#94)
Co-authored-by: stingpeng <st...@tencent.com>
---
docs/en-us/modules/agent/quick_start.md | 13 +++++++--
docs/en-us/modules/dataproxy/quick_start.md | 4 +--
.../modules/tubemq/tubemq-manager/quick_start.md | 8 +++---
docs/zh-cn/modules/agent/quick_start.md | 31 +++++++++++++++-------
docs/zh-cn/modules/dataproxy/quick_start.md | 5 ++--
.../modules/tubemq/tubemq-manager/quick_start.md | 7 ++---
6 files changed, 45 insertions(+), 23 deletions(-)
diff --git a/docs/en-us/modules/agent/quick_start.md b/docs/en-us/modules/agent/quick_start.md
index 667d7bb..667d342 100644
--- a/docs/en-us/modules/agent/quick_start.md
+++ b/docs/en-us/modules/agent/quick_start.md
@@ -32,6 +32,7 @@ Create a new folder named .inlong in the agent directory, and create a new bid+.
write:
```ini
{"cluster_id":1,"isInterVisit":1,"size":1,"address": [{"port":write proxy port,"host":"write proxy ip"}], "switch":0}
+Among them, cluster_id, isInterVisit, and switch are reserved fields, please fill in the default values
```
### 2.3 local job configuration
@@ -110,7 +111,7 @@ agent.http.port=Available ports
},
"name": "fileAgentTest",
"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
- "sink": "org.apache.inlong.agent.plugin.sinks.TdBusSink",
+ "sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
},
"bus": {
@@ -128,4 +129,12 @@ After decompression, run the following command
```bash
sh agent.sh start
-```
\ No newline at end of file
+```
+
+## 4、eg for directory config
+
+ E.g:
+ /data/inlong-agent/test.log //Represents reading the new file test.log in the inlong-agent folder
+ /data/inlong-agent/test[0-9]{1} // means to read the new file test in the inlong-agent folder followed by a number at the end
+ /data/inlong-agent/test //If test is a directory, it means to read all new files under test
+ /data/inlong-agent/^\\d+(\\.\\d+)? // Start with one or more digits, followed by. or end with one. or more digits (? stands for optional, can match Examples: "5", "1.5" and "2.21"
\ No newline at end of file
diff --git a/docs/en-us/modules/dataproxy/quick_start.md b/docs/en-us/modules/dataproxy/quick_start.md
index d556fbf..da17b2c 100644
--- a/docs/en-us/modules/dataproxy/quick_start.md
+++ b/docs/en-us/modules/dataproxy/quick_start.md
@@ -34,8 +34,8 @@ notice that conf/flume.conf FLUME_HOME is proxy the directory for proxy inner da
### config manager
vim conf/common.properties
write：
- cluster_id=1
- manager_hosts=ip:port
+ cluster_id=1 // cluter id is for future use please write 1
+ manager_hosts=ip:port // manager open api
## 3、run
sh bin/start.sh
diff --git a/docs/en-us/modules/tubemq/tubemq-manager/quick_start.md b/docs/en-us/modules/tubemq/tubemq-manager/quick_start.md
index 3b4ea81..f965707 100644
--- a/docs/en-us/modules/tubemq/tubemq-manager/quick_start.md
+++ b/docs/en-us/modules/tubemq/tubemq-manager/quick_start.md
@@ -6,8 +6,10 @@
environment requirements：
-mysql
-java(1.8+)
+ mysql
+ java(1.8+)
+
+
In the InLong/inlong-tubemq/tubemq-manager/dist/ directory, you can find the file apache-inlong-tubemanager-0.9.0-incubating-SNAPSHOT-bin.tar.gz
tar -zxvf apache-inlong-tubemanager-0.9.0-incubating-SNAPSHOT-bin.tar.gz
@@ -126,7 +128,7 @@ POST
return json
{ "result":true, "errCode":0, "errMsg":"OK", }
- { "result":false, "errCode": 100, "errMsg":"topic thirty5is not writable"}
+ { "result":false, "errCode": 100, "errMsg":"topic test is not writable"}
{ "result":false, "errCode": 101, "errMsg":"no such topic in master"}
result is false as not writable
diff --git a/docs/zh-cn/modules/agent/quick_start.md b/docs/zh-cn/modules/agent/quick_start.md
index 073ce2c..a6baa2b 100644
--- a/docs/zh-cn/modules/agent/quick_start.md
+++ b/docs/zh-cn/modules/agent/quick_start.md
@@ -1,4 +1,4 @@
-## ~~~~一、安装
+## 一、安装
环境要求：
java JDK 1.8
@@ -25,18 +25,18 @@ agent.manager.vip.http.host=manager open api host
agent.manager.vip.http.port=manager open api port
```
-### 2.2 Proxy 相关设置
+### 2.2 DataProxy 相关设置
在agent目录下新建.inlong文件夹，内部新建bid+.local文件，例如设置的发送bid为a, 则新建a.local文件
内部将写入：
```ini
{"cluster_id":1,"isInterVisit":1,"size":1,"address": [{"port":写入proxy port,"host":"写入proxy ip"}], "switch":0}
+其中cluster_id, isInterVisit，switch为预留字段，请填写默认值
```
### 2.3 agent本地运行job配置
如果不使用线上模式，可以使用本地文件新建读取任务
-在conf中新建目录jobs，
-vim job1.json：
+在conf中新建目录jobs，jobs内部vim job1.json：
写入
```json
@@ -78,7 +78,7 @@ vim job1.json：
- job.dir.pattern: 配置读取的文件路径，可包含正则表达式
- job.trigger: 触发器名称，默认为DirectoryTrigger，功能为监听文件夹下的文件产生事件，任务运行时已有的文件不会读取
- job.source: 使用的数据源类型，默认为TextFileSource，读取文本文件
-- job.sink：使用的写入器类型，默认为ProxySink，发送消息到proxy中
+- job.sink：使用的写入器类型，默认为ProxySink，发送消息到dataproxy中
- proxy.bid: 写入proxy时使用的bid类型
### 2.4 实时添加job配置
@@ -93,7 +93,7 @@ agent.http.port=可用端口
#### 2.4.2 执行如下命令：
```bash
-curl --location --request POST 'http://localhost:8129/config/job' \
+curl --location --request POST 'http://localhost:8018/config/job' \
--header 'Content-Type: application/json' \
--data '{
"job": {
@@ -110,7 +110,7 @@ curl --location --request POST 'http://localhost:8129/config/job' \
},
"name": "fileAgentTest",
"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
-"sink": "org.apache.inlong.agent.plugin.sinks.TdBusSink",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
},
"proxy": {
@@ -119,13 +119,24 @@ curl --location --request POST 'http://localhost:8129/config/job' \
"op": "add"
}'
```
-
+
其中参数分别为：
pattern: 代表读取/data/inlong-agent/test.log文件，需要新建任务后再创建文件
- proxy.bid: 代表发送到proxy使用的bid
+ proxy.bid: 代表发送到dataproxy使用的bid
## 三、运行
解压后如下命令运行
```bash
sh agent.sh start
-```
\ No newline at end of file
+```
+
+
+## 四、可支持的路径配置方案
+
+ 例如：
+ /data/inlong-agent/test.log //代表读取inlong-agent文件夹下的的新增文件test.log
+ /data/inlong-agent/test[0-9]{1} //代表读取inlong-agent文件夹下的新增文件test后接一个数字结尾
+ /data/inlong-agent/test //如果test为目录，则代表读取test下的所有新增文件
+ /data/inlong-agent/^\\d+(\\.\\d+)? // 以一个或多个数字开头,之后可以是.或者一个.或多个数字结尾，?代表可选,可以匹配的实例："5", "1.5" 和 "2.21"
+
+
diff --git a/docs/zh-cn/modules/dataproxy/quick_start.md b/docs/zh-cn/modules/dataproxy/quick_start.md
index 07b0882..c4d031c 100644
--- a/docs/zh-cn/modules/dataproxy/quick_start.md
+++ b/docs/zh-cn/modules/dataproxy/quick_start.md
@@ -20,7 +20,6 @@
sed -i 's/TUBE_LIST/ip1:port,ip2:port/g' conf/flume.conf
-
修改以下内容为proxy端口号
agent1.sources.tcp-source.port = 46801
@@ -35,8 +34,8 @@ sed -i 's/TUBE_LIST/ip1:port,ip2:port/g' conf/flume.conf
vim conf/common.properties
写入以下配置：
- cluster_id=1
- manager_hosts=ip:port // 这里写入的是manager_api地址
+ cluster_id=1 // 预留字段，请填写默认值1
+ manager_hosts=ip:port // 这里写入的是manager open api地址
## 三、启动
sh bin/start.sh
diff --git a/docs/zh-cn/modules/tubemq/tubemq-manager/quick_start.md b/docs/zh-cn/modules/tubemq/tubemq-manager/quick_start.md
index 560c75a..c83473e 100644
--- a/docs/zh-cn/modules/tubemq/tubemq-manager/quick_start.md
+++ b/docs/zh-cn/modules/tubemq/tubemq-manager/quick_start.md
@@ -6,8 +6,9 @@
环境要求：
-mysql
-java(1.8+)
+ mysql
+ java(1.8+)
+
在InLong/inlong-tubemq/tubemq-manager/dist/目录当中，能够找到文件为apache-inlong-tubemanager-0.9.0-incubating-SNAPSHOT-bin.tar.gz
tar -zxvf apache-inlong-tubemanager-0.9.0-incubating-SNAPSHOT-bin.tar.gz
@@ -129,7 +130,7 @@ POST
返回json格式样例
{ "result":true, "errCode":0, "errMsg":"OK", }
- { "result":false, "errCode": 100, "errMsg":"topic thirty5is not writable"}
+ { "result":false, "errCode": 100, "errMsg":"topic test is not writable"}
{ "result":false, "errCode": 101, "errMsg":"no such topic in master"}
result为false为不可写