You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by do...@apache.org on 2022/04/18 11:44:44 UTC
[incubator-inlong-website] branch fix-355 created (now e8901ad31)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a change to branch fix-355
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
at e8901ad31 [INLONG-355][Doc] Refactor the File collector guide
This branch includes the following new commits:
new e8901ad31 [INLONG-355][Doc] Refactor the File collector guide
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[incubator-inlong-website] 01/01: [INLONG-355][Doc] Refactor the File collector guide
Posted by do...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch fix-355
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
commit e8901ad310735b3f54f94031b175bc96297bab8a
Author: dockerzhang <do...@apache.org>
AuthorDate: Mon Apr 18 19:44:29 2022 +0800
[INLONG-355][Doc] Refactor the File collector guide
---
docs/administration/_category_.json | 2 +-
docs/contact.md | 2 +-
docs/data_node/_category_.json | 4 ++
docs/data_node/extract_node/_category_.json | 4 ++
.../agent => data_node/extract_node}/file.md | 19 ++++---
docs/data_node/extract_node/img/file_param.png | Bin 0 -> 17445 bytes
docs/data_node/load_node/_category_.json | 4 ++
docs/development/_category_.json | 2 +-
docs/modules/agent/sql.md | 60 ---------------------
docs/sdk/_category_.json | 2 +-
docs/user_guide/_category_.json | 2 +-
.../current/how-to-release.md | 2 +-
.../docusaurus-plugin-content-docs/current.json | 4 ++
.../agent => data_node/extract_node}/file.md | 16 ++++--
.../data_node/extract_node/img/file_param.png | Bin 0 -> 12487 bytes
.../current/modules/agent/sql.md | 60 ---------------------
16 files changed, 45 insertions(+), 138 deletions(-)
diff --git a/docs/administration/_category_.json b/docs/administration/_category_.json
index cc9b71eac..56431da33 100644
--- a/docs/administration/_category_.json
+++ b/docs/administration/_category_.json
@@ -1,4 +1,4 @@
{
"label": "Administration",
- "position": 9
+ "position": 10
}
\ No newline at end of file
diff --git a/docs/contact.md b/docs/contact.md
index 867e82d26..918702dae 100644
--- a/docs/contact.md
+++ b/docs/contact.md
@@ -1,6 +1,6 @@
---
title: Contact Us
-sidebar_position: 10
+sidebar_position: 11
---
Contact information
diff --git a/docs/data_node/_category_.json b/docs/data_node/_category_.json
new file mode 100644
index 000000000..c44226d96
--- /dev/null
+++ b/docs/data_node/_category_.json
@@ -0,0 +1,4 @@
+{
+ "label": "Data Nodes",
+ "position": 6
+}
\ No newline at end of file
diff --git a/docs/data_node/extract_node/_category_.json b/docs/data_node/extract_node/_category_.json
new file mode 100644
index 000000000..c61e62a3b
--- /dev/null
+++ b/docs/data_node/extract_node/_category_.json
@@ -0,0 +1,4 @@
+{
+ "label": "Extract Nodes",
+ "position": 1
+}
\ No newline at end of file
diff --git a/docs/modules/agent/file.md b/docs/data_node/extract_node/file.md
similarity index 70%
rename from docs/modules/agent/file.md
rename to docs/data_node/extract_node/file.md
index a85201c76..f02d249be 100644
--- a/docs/modules/agent/file.md
+++ b/docs/data_node/extract_node/file.md
@@ -1,9 +1,16 @@
---
title: File
-sidebar_position: 3
+sidebar_position: 1
---
-## File Agent Configuration
+## Parameters
+
+- DataSource Name
+- Data source IP: Collect Node Agent IP.
+- File path: Must be an absolute path and support regular expressions.
+- Time offset: The file will be collected from a certain time,' 1m' means 1 minute later,' -1m' means 1 minute before, and m(minute), h(hour), d(day) are supported. If it is empty, the file will be collected from the current time.
+
+## Path Configuration
```
/data/inlong-agent/test.log //Represents reading the new file test.log in the inlong-agent folder
/data/inlong-agent/test[0-9]{1} // means to read the new file test in the inlong-agent folder followed by a number at the end
@@ -11,19 +18,17 @@ sidebar_position: 3
/data/inlong-agent/^\\d+(\\.\\d+)? // Start with one or more digits, followed by. or end with one. or more digits (? stands for optional, can match Examples: "5", "1.5" and "2.21"
```
-## Get data time from file name
-
+## Data Time
Agent supports obtaining the time from the file name as the production time of the data. The configuration instructions are as follows:
```
/data/inlong-agent/***YYYYMMDDHH***
```
+
Where YYYYDDMMHH represents the data time, YYYY represents the year, MM represents the month, DD represents the day, and HH represents the hour
Where *** is any character
At the same time, you need to add the current data cycle to the job conf, the current support day cycle and hour cycle,
-When adding a task, add the property job.cycleUnit
-
-job.cycleUnit contains the following two types:
+When adding a task, add the property job.cycleUnit. job.cycleUnit contains the following two types:
- D: Represents the data time and day dimension
- H: Represents the data time and hour dimension
diff --git a/docs/data_node/extract_node/img/file_param.png b/docs/data_node/extract_node/img/file_param.png
new file mode 100644
index 000000000..c3cbcdaea
Binary files /dev/null and b/docs/data_node/extract_node/img/file_param.png differ
diff --git a/docs/data_node/load_node/_category_.json b/docs/data_node/load_node/_category_.json
new file mode 100644
index 000000000..429218e09
--- /dev/null
+++ b/docs/data_node/load_node/_category_.json
@@ -0,0 +1,4 @@
+{
+ "label": "Load Nodes",
+ "position": 2
+}
\ No newline at end of file
diff --git a/docs/development/_category_.json b/docs/development/_category_.json
index 6244591cb..dc8cb43dc 100644
--- a/docs/development/_category_.json
+++ b/docs/development/_category_.json
@@ -1,4 +1,4 @@
{
"label": "Development",
- "position": 8
+ "position": 9
}
\ No newline at end of file
diff --git a/docs/modules/agent/sql.md b/docs/modules/agent/sql.md
deleted file mode 100644
index c4256c05f..000000000
--- a/docs/modules/agent/sql.md
+++ /dev/null
@@ -1,60 +0,0 @@
----
-title: MySQL SQL
-sidebar_position: 3
----
-
-## Overview
-Currently, Agent supports MYSQL version 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x
-Currently, the Agent only supports the curl request to create a Job to submit collection tasks, and temporarily does not support the manager front-end to create SQL collection
-
-## Create A MySQL Job
-
-1. Apply for access on the manager, when filling in the data information, select the message source as "Independent Push"
-2. Select the source data field separator
-3. Fill in the source data fields, and the field order should be consistent with the field order in the sql query result
-4. Create a SQL read task using curl request
-
-## Parameter Description
-
-````
-Each parameter used in SQL Agent Job is described as
-1. job.sql.command: The actual executed sql statement, for example: select * from apache_inlong_manager.user
-2. job.sql.user: the user used when connecting to the database, for example: abc
-3. job.sql.password: The password used when connecting to the database, for example: 123456
-4. job.sql.hostname: The IP address of the connected database, for example: 127.0.0.1
-5. job.sql.port: the connected database port, for example: 3306
-6. job.sql.separator: The separator used to separate multiple fields needs to be used with the manager front-end
-````
-
-## Example
-
-```bash
-curl --location --request POST 'http://localhost:8008/config/job' \--header 'Content-Type: application/json' \--data '{
- "job": {
- "sql": {
- "command": "select * from apache_inlong_manager.user",
- "user": "root",
- "password": "inlong",
- "hostname": "10.0.0.6",
- "port": "3306",
- "separator": "|"
- },
- "id": 1,
- "thread": {
- "running": {
- "core": "4"
- }
- },
- "name": "test",
- "source": "org.apache.inlong.agent.plugin.sources.DataBaseSource",
- "sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
- "channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
- },
- "proxy": {
- "inlongGroupId": "b_test_tube_hive_20211221_01",
- "inlongStreamId": "test_data_stream_20211221_01_01"
- },
- "op": "add"
-}
-'
-```
\ No newline at end of file
diff --git a/docs/sdk/_category_.json b/docs/sdk/_category_.json
index 6b355ca6b..9c997dee9 100644
--- a/docs/sdk/_category_.json
+++ b/docs/sdk/_category_.json
@@ -1,4 +1,4 @@
{
"label": "SDK",
- "position": 6
+ "position": 7
}
\ No newline at end of file
diff --git a/docs/user_guide/_category_.json b/docs/user_guide/_category_.json
index 891650af2..514ec933b 100644
--- a/docs/user_guide/_category_.json
+++ b/docs/user_guide/_category_.json
@@ -1,4 +1,4 @@
{
"label": "User Guide",
- "position": 7
+ "position": 8
}
\ No newline at end of file
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/how-to-release.md b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/how-to-release.md
index 63a1cd790..cc3b5d3bb 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/how-to-release.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs-community/current/how-to-release.md
@@ -232,7 +232,7 @@ cd /tmp/apache-inlong-${release_version}-${rc_version} # 进入源码包目录
tar xzvf apache-inlong-${release_version}-src.tar.gz #解压源码包
cd apache-inlong-${release_version} # 进入源码目录
mvn compile clean install package -DskipTests # 编译
-cp ./inlong-distribution/target/apache-inlong-${release_version}-bin.tar.gz /tmp/apache-inlong-${release_version}-${rc_version}/ # 拷贝二进制包拷到源码包目录下，方面下一步对包进行签名
+cp ./inlong-distribution/target/apache-inlong-${release_version}-bin.tar.gz /tmp/apache-inlong-${release_version}-${rc_version}/ # 拷贝二进制包拷到源码包目录下，方便下一步对包进行签名
```
### 对源码包/二进制包进行签名/sha512
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current.json b/i18n/zh-CN/docusaurus-plugin-content-docs/current.json
index c5ca17905..f0e315eb8 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current.json
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current.json
@@ -58,5 +58,9 @@
"sidebar.tutorialSidebar.category.User Guide": {
"message": "用户指引",
"description": "The label for category User Guide in sidebar tutorialSidebar"
+ },
+ "sidebar.tutorialSidebar.category.Data Nodes": {
+ "message": "数据节点",
+ "description": "The label for category Data Nodes in sidebar tutorialSidebar"
}
}
\ No newline at end of file
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/file.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/file.md
similarity index 70%
rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/file.md
rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/file.md
index 0744cce32..1bb2d91ae 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/file.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/file.md
@@ -3,7 +3,14 @@ title: 文件
sidebar_position: 3
---
-## 文件Agent配置
+## 参数说明
+
+- 数据源名称
+- 数据源IP：采集节点 Agent IP
+- ⽂件路径：必须是绝对路径，支持正则表达式，多个时以逗号分隔
+- 时间偏移量：从文件的某个时间开始采集，'1m'表示1分钟之后，'-1m'表示1分钟之前，支持m(分钟)，h(小时)，d(天)，空则从当前时间开始采集
+
+## 路径配置
```
/data/inlong-agent/test.log //代表读取inlong-agent文件夹下的的新增文件test.log
/data/inlong-agent/test[0-9]{1} //代表读取inlong-agent文件夹下的新增文件test后接一个数字结尾
@@ -11,18 +18,17 @@ sidebar_position: 3
/data/inlong-agent/^\\d+(\\.\\d+)? // 以一个或多个数字开头,之后可以是.或者一个.或多个数字结尾，?代表可选,可以匹配的实例："5", "1.5" 和 "2.21"
```
-## 从文件名称中获取数据时间
+## 数据时间
Agent支持从文件名称中获取时间当作数据的生产时间，配置说明如下：
```
/data/inlong-agent/***YYYYMMDDHH***
```
+
其中YYYYDDMMHH代表数据时间，YYYY表示年，MM表示月份，DD表示天，HH表示小时
其中***为任意字符
同时需要在job conf中加入当前数据的周期，当前支持天周期以及小时周期，
-在添加任务时，加入属性job.cycleUnit
-
-job.cycleUnit 包含如下两种类型：
+在添加任务时，加入属性 job.cycleUnit。job.cycleUnit 包含如下两种类型：
- D : 代表数据时间天维度
- H : 代表数据时间小时维度
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/img/file_param.png b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/img/file_param.png
new file mode 100644
index 000000000..d4226b267
Binary files /dev/null and b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/extract_node/img/file_param.png differ
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/sql.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/sql.md
deleted file mode 100644
index 8299313f3..000000000
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/agent/sql.md
+++ /dev/null
@@ -1,60 +0,0 @@
----
-title: MySQL SQL
-sidebar_position: 3
----
-
-## 总览
-目前 Agent 支持 MySQL 版本为5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x
-目前 Agent 只支持 curl 请求创建 Job 方式提交采集任务，暂时不支持 manager 前端创建 SQL 采集
-
-## MySQL Job创建步骤
-
-1、在 manager 上申请接入，填写数据信息时，选择消息来源为"自主推送"
-2、选择源数据字段分隔符
-3、填写源数据字段，字段顺序与 sql 查询结果中的字段顺序保持一致
-4、使用 curl 请求创建一个 SQL 读取任务
-
-## 参数说明
-
-```
-SQL Agent Job 中各个使用参数说明为
-1、job.sql.command: 实际执行的 sql 语句，举例: select * from apache_inlong_manager.user
-2、job.sql.user: 连接数据库时使用的 user，举例: abc
-3、job.sql.password: 连接数据库时使用的 password, 举例: 123456
-4、job.sql.hostname: 连接的数据库 ip 地址，举例：127.0.0.1
-5、job.sql.port：连接的数据库端口，举例：3306
-6、job.sql.separator: 使用的分割符来分割多个字段，需要与 manager 前端
-```
-
-## 举例
-
-```bash
-curl --location --request POST 'http://localhost:8008/config/job' \--header 'Content-Type: application/json' \--data '{
- "job": {
- "sql": {
- "command": "select * from apache_inlong_manager.user",
- "user": "root",
- "password": "inlong",
- "hostname": "10.0.0.6",
- "port": "3306",
- "separator": "|"
- },
- "id": 1,
- "thread": {
- "running": {
- "core": "4"
- }
- },
- "name": "test",
- "source": "org.apache.inlong.agent.plugin.sources.DataBaseSource",
- "sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
- "channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
- },
- "proxy": {
- "inlongGroupId": "b_test_tube_hive_20211221_01",
- "inlongStreamId": "test_data_stream_20211221_01_01"
- },
- "op": "add"
-}
-'
-```
\ No newline at end of file