You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by gi...@apache.org on 2021/08/27 11:27:41 UTC
[incubator-inlong-website] branch asf-site updated: Automated
deployment: Fri Aug 27 11:27:30 UTC 2021
639a47b58beb95d91a4187aff61d439b373bbd04
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new b3f9058 Automated deployment: Fri Aug 27 11:27:30 UTC 2021 639a47b58beb95d91a4187aff61d439b373bbd04
b3f9058 is described below
commit b3f90585b4c4d5c3fe661b83e480b679cdf61ff3
Author: dockerzhang <do...@users.noreply.github.com>
AuthorDate: Fri Aug 27 11:27:31 2021 +0000
Automated deployment: Fri Aug 27 11:27:30 UTC 2021 639a47b58beb95d91a4187aff61d439b373bbd04
---
docs/en-us/modules/agent/quick_start.md | 111 +++++++++++++++++++++++++++---
docs/zh-cn/modules/agent/quick_start.md | 107 +++++++++++++++++++++++++---
en-us/docs/modules/agent/quick_start.html | 103 ++++++++++++++++++++++++---
en-us/docs/modules/agent/quick_start.json | 2 +-
en-us/docs/modules/agent/quick_start.md | 111 +++++++++++++++++++++++++++---
zh-cn/docs/modules/agent/quick_start.html | 101 ++++++++++++++++++++++++---
zh-cn/docs/modules/agent/quick_start.json | 2 +-
zh-cn/docs/modules/agent/quick_start.md | 107 +++++++++++++++++++++++++---
8 files changed, 584 insertions(+), 60 deletions(-)
diff --git a/docs/en-us/modules/agent/quick_start.md b/docs/en-us/modules/agent/quick_start.md
index e015889..b6283c6 100644
--- a/docs/en-us/modules/agent/quick_start.md
+++ b/docs/en-us/modules/agent/quick_start.md
@@ -6,7 +6,7 @@ cd inlong-agent
The agent supports two modes of operation: local operation and online operation
-### 1.1 Agent configuration
+### Agent configuration
Online operation needs to pull the configuration from inlong-manager, the configuration conf/agent.properties is as follows:
```ini
@@ -16,15 +16,6 @@ agent.manager.vip.http.host=manager web host
agent.manager.vip.http.port=manager web port
```
-### 1.2 Proxy configuration
-Create a new folder named .inlong in the agent directory, and create a new bid+.local file inside. For example, if the sending bid is set to a, then create a new file a.local
-
-write:
-```ini
-{"cluster_id":1,"isInterVisit":1,"size":1,"address": [{"port":write proxy port,"host":"write proxy ip"}], "switch":0}
-Among them, cluster_id, isInterVisit, and switch are reserved fields, please fill in the default values
-```
-
## 2、run
After decompression, run the following command
@@ -89,4 +80,102 @@ agent.http.port=Available ports
/data/inlong-agent/test.log //Represents reading the new file test.log in the inlong-agent folder
/data/inlong-agent/test[0-9]{1} // means to read the new file test in the inlong-agent folder followed by a number at the end
/data/inlong-agent/test //If test is a directory, it means to read all new files under test
- /data/inlong-agent/^\\d+(\\.\\d+)? // Start with one or more digits, followed by. or end with one. or more digits (? stands for optional, can match Examples: "5", "1.5" and "2.21"
\ No newline at end of file
+ /data/inlong-agent/^\\d+(\\.\\d+)? // Start with one or more digits, followed by. or end with one. or more digits (? stands for optional, can match Examples: "5", "1.5" and "2.21"
+
+
+## 5. Support to get data time from file name
+
+ Agent supports obtaining the time from the file name as the production time of the data. The configuration instructions are as follows:
+ /data/inlong-agent/***YYYYMMDDHH***
+ Where YYYYDDMMHH represents the data time, YYYY represents the year, MM represents the month, DD represents the day, and HH represents the hour
+ Where *** is any character
+
+ At the same time, you need to add the current data cycle to the job conf, the current support day cycle and hour cycle,
+ When adding a task, add the property job.cycleUnit
+
+ job.cycleUnit contains the following two types:
+ 1. D: Represents the data time and day dimension
+ 2. H: Represents the data time and hour dimension
+
+ E.g:
+ The configuration data source is
+ /data/inlong-agent/YYYYMMDDHH.log
+ Write data to 2021020211.log
+ Configure job.cycleUnit as D
+ Then the agent will try the 202020211.log file at the time of 202020211. When reading the data in the file, it will write all the data to the backend proxy at the time of 20210202.
+ If job.cycleUnit is configured as H
+ When collecting data in the 2021020211.log file, all data will be written to the backend proxy at the time of 2021020211
+
+
+ Examples of job submission
+
+```bash
+curl --location --request POST'http://localhost:8008/config/job' \
+--header'Content-Type: application/json' \
+--data'{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'
+```
+
+## 6. Support time offset reading
+
+ After the configuration is read by time, if you want to read data at other times than the current time, you can configure the time offset to complete
+ Configure the job attribute name as job.timeOffset, the value is number + time dimension, time dimension includes day and hour
+ For example, the following settings are supported
+ 1. 1d Read the data one day after the current time
+ 2. -1h read the data one hour before the current time
+
+
+ Examples of job submission
+```bash
+curl --location --request POST'http://localhost:8008/config/job' \
+--header'Content-Type: application/json' \
+--data'{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"timeOffset": "-1d",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'
+```
\ No newline at end of file
diff --git a/docs/zh-cn/modules/agent/quick_start.md b/docs/zh-cn/modules/agent/quick_start.md
index 8fb71fe..21314d6 100644
--- a/docs/zh-cn/modules/agent/quick_start.md
+++ b/docs/zh-cn/modules/agent/quick_start.md
@@ -5,7 +5,7 @@ cd inlong-agent
agent 支持本地运行以及线上运行，其中线上运行从inlong manager拉取任务，本地运行可使用http请求提交任务
-### 1.1 Agent 线上运行相关设置
+### Agent 线上运行相关设置
线上运行需要从inlong-manager拉取配置，配置conf/agent.properties如下：
```ini
@@ -15,15 +15,6 @@ agent.manager.vip.http.host=manager web host
agent.manager.vip.http.port=manager web port
```
-### 1.2 DataProxy 相关设置
-在agent目录下新建.inlong文件夹，内部新建bid+.local文件，例如设置的发送bid为a, 则新建a.local文件
-bid是指manager界面中，数据接入中业务信息的业务ID，此处不是创建的tube topic名称
-内部将写入：
-```ini
-{"cluster_id":1,"isInterVisit":1,"size":1,"address": [{"port":写入proxy port,"host":"写入proxy ip"}], "switch":0}
-其中cluster_id, isInterVisit，switch为预留字段，请填写默认值
-```
-
## 2、运行
解压后如下命令运行
```bash
@@ -87,3 +78,99 @@ curl --location --request POST 'http://localhost:8008/config/job' \
/data/inlong-agent/^\\d+(\\.\\d+)? // 以一个或多个数字开头,之后可以是.或者一个.或多个数字结尾，?代表可选,可以匹配的实例："5", "1.5" 和 "2.21"
+## 5、支持从文件名称中获取数据时间
+
+ Agent支持从文件名称中获取时间当作数据的生产时间，配置说明如下：
+ /data/inlong-agent/***YYYYMMDDHH***
+ 其中YYYYDDMMHH代表数据时间，YYYY表示年，MM表示月份，DD表示天，HH表示小时
+ 其中***为任意字符
+
+ 同时需要在job conf中加入当前数据的周期，当前支持天周期以及小时周期，
+ 在添加任务时，加入属性job.cycleUnit
+
+ job.cycleUnit 包含如下两种类型：
+ 1、D : 代表数据时间天维度
+ 2、H : 代表数据时间小时维度
+
+ 例如：
+ 配置数据源为
+ /data/inlong-agent/YYYYMMDDHH.log
+ 写入数据到 2021020211.log
+ 配置 job.cycleUnit 为 D
+ 则agent会在2021020211时间尝试2021020211.log文件，读取文件中的数据时，会将所有数据以20210202这个时间写入到后端proxy
+ 如果配置 job.cycleUnit 为 H
+ 则采集2021020211.log文件中的数据时，会将所有数据以2021020211这个时间写入到后端proxy
+
+
+ 提交job举例
+```bash
+curl --location --request POST 'http://localhost:8008/config/job' \
+--header 'Content-Type: application/json' \
+--data '{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'
+```
+
+
+## 6、支持时间偏移量offset读取
+
+ 在配置按照时间读取之后，如果想要读取当前时间之外的其他时间的数据，可以通过配置时间偏移量完成
+ 配置job属性名称为job.timeOffset，值为数字 + 时间维度，时间维度包括天和小时
+ 例如支持如下设置
+ 1、 1d 读取当前时间后一天的数据
+ 2、 -1h 读取当前时间前一个小时的数据
+
+
+ 提交job举例
+```bash
+curl --location --request POST 'http://localhost:8008/config/job' \
+--header 'Content-Type: application/json' \
+--data '{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"timeOffset": "-1d",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'
+```
\ No newline at end of file
diff --git a/en-us/docs/modules/agent/quick_start.html b/en-us/docs/modules/agent/quick_start.html
index 5506a4b..f002948 100644
--- a/en-us/docs/modules/agent/quick_start.html
+++ b/en-us/docs/modules/agent/quick_start.html
@@ -16,19 +16,13 @@
<pre><code>cd inlong-agent
</code></pre>
<p>The agent supports two modes of operation: local operation and online operation</p>
-<h3>1.1 Agent configuration</h3>
+<h3>Agent configuration</h3>
<p>Online operation needs to pull the configuration from inlong-manager, the configuration conf/agent.properties is as follows:</p>
<pre><code class="language-ini"><span class="hljs-attr">agent.fetcher.classname</span>=org.apache.inlong.agent.plugin.fetcher.ManagerFetcher (the class name for fetch tasks, default ManagerFetcher）
<span class="hljs-attr">agent.local.ip</span>=Write local ip
<span class="hljs-attr">agent.manager.vip.http.host</span>=manager web host
<span class="hljs-attr">agent.manager.vip.http.port</span>=manager web port
</code></pre>
-<h3>1.2 Proxy configuration</h3>
-<p>Create a new folder named .inlong in the agent directory, and create a new bid+.local file inside. For example, if the sending bid is set to a, then create a new file a.local</p>
-<p>write:</p>
-<pre><code class="language-ini">{"cluster_id":1,"isInterVisit":1,"size":1,"address": [{"port":write proxy port,"host":"write proxy ip"}], "switch":0}
-Among them, cluster_id, isInterVisit, and switch are reserved fields, please fill in the default values
-</code></pre>
<h2>2、run</h2>
<p>After decompression, run the following command</p>
<pre><code class="language-bash">sh agent.sh start
@@ -82,7 +76,100 @@ Among them, cluster_id, isInterVisit, and switch are reserved fields, please fil
/data/inlong-agent/test.log //Represents reading the new file test.log in the inlong-agent folder
/data/inlong-agent/test[0-9]{1} // means to read the new file test in the inlong-agent folder followed by a number at the end
/data/inlong-agent/test //If test is a directory, it means to read all new files under test
-/data/inlong-agent/^\\d+(\\.\\d+)? // Start with one or more digits, followed by. or end with one. or more digits (? stands for optional, can match Examples: "5", "1.5" and "2.21"</code></pre>
+/data/inlong-agent/^\\d+(\\.\\d+)? // Start with one or more digits, followed by. or end with one. or more digits (? stands for optional, can match Examples: "5", "1.5" and "2.21"
+</code></pre>
+<h2>5. Support to get data time from file name</h2>
+<pre><code>Agent supports obtaining the time from the file name as the production time of the data. The configuration instructions are as follows:
+/data/inlong-agent/***YYYYMMDDHH***
+Where YYYYDDMMHH represents the data time, YYYY represents the year, MM represents the month, DD represents the day, and HH represents the hour
+Where *** is any character
+
+At the same time, you need to add the current data cycle to the job conf, the current support day cycle and hour cycle,
+When adding a task, add the property job.cycleUnit
+
+job.cycleUnit contains the following two types:
+1. D: Represents the data time and day dimension
+2. H: Represents the data time and hour dimension
+
+E.g:
+The configuration data source is
+/data/inlong-agent/YYYYMMDDHH.log
+Write data to 2021020211.log
+Configure job.cycleUnit as D
+Then the agent will try the 202020211.log file at the time of 202020211. When reading the data in the file, it will write all the data to the backend proxy at the time of 20210202.
+If job.cycleUnit is configured as H
+When collecting data in the 2021020211.log file, all data will be written to the backend proxy at the time of 2021020211
+
+
+Examples of job submission
+</code></pre>
+<pre><code class="language-bash">curl --location --request POST<span class="hljs-string">'http://localhost:8008/config/job'</span> \
+--header<span class="hljs-string">'Content-Type: application/json'</span> \
+--data<span class="hljs-string">'{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'</span>
+</code></pre>
+<h2>6. Support time offset reading</h2>
+<pre><code>After the configuration is read by time, if you want to read data at other times than the current time, you can configure the time offset to complete
+Configure the job attribute name as job.timeOffset, the value is number + time dimension, time dimension includes day and hour
+For example, the following settings are supported
+1. 1d Read the data one day after the current time
+2. -1h read the data one hour before the current time
+
+
+Examples of job submission
+</code></pre>
+<pre><code class="language-bash">curl --location --request POST<span class="hljs-string">'http://localhost:8008/config/job'</span> \
+--header<span class="hljs-string">'Content-Type: application/json'</span> \
+--data<span class="hljs-string">'{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"timeOffset": "-1d",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'</span>
+</code></pre>
</div></section><footer class="footer-container"><div class="footer-body"><img src="/img/incubator-logo.svg"/><div class="cols-container"><div class="col col-24"><p>Apache InLong (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with [...]
<script src="https://f.alicdn.com/react/15.4.1/react-with-addons.min.js"></script>
<script src="https://f.alicdn.com/react/15.4.1/react-dom.min.js"></script>
diff --git a/en-us/docs/modules/agent/quick_start.json b/en-us/docs/modules/agent/quick_start.json
index 1733c8f..c1b2c43 100644
--- a/en-us/docs/modules/agent/quick_start.json
+++ b/en-us/docs/modules/agent/quick_start.json
@@ -1,6 +1,6 @@
{
"filename": "quick_start.md",
- "__html": "<h2>1、Configuration</h2>\n<pre><code>cd inlong-agent\n</code></pre>\n<p>The agent supports two modes of operation: local operation and online operation</p>\n<h3>1.1 Agent configuration</h3>\n<p>Online operation needs to pull the configuration from inlong-manager, the configuration conf/agent.properties is as follows:</p>\n<pre><code class=\"language-ini\"><span class=\"hljs-attr\">agent.fetcher.classname</span>=org.apache.inlong.agent.plugin.fetcher.ManagerFetcher (the class [...]
+ "__html": "<h2>1、Configuration</h2>\n<pre><code>cd inlong-agent\n</code></pre>\n<p>The agent supports two modes of operation: local operation and online operation</p>\n<h3>Agent configuration</h3>\n<p>Online operation needs to pull the configuration from inlong-manager, the configuration conf/agent.properties is as follows:</p>\n<pre><code class=\"language-ini\"><span class=\"hljs-attr\">agent.fetcher.classname</span>=org.apache.inlong.agent.plugin.fetcher.ManagerFetcher (the class nam [...]
"link": "/en-us/docs/modules/agent/quick_start.html",
"meta": {}
}
\ No newline at end of file
diff --git a/en-us/docs/modules/agent/quick_start.md b/en-us/docs/modules/agent/quick_start.md
index e015889..b6283c6 100644
--- a/en-us/docs/modules/agent/quick_start.md
+++ b/en-us/docs/modules/agent/quick_start.md
@@ -6,7 +6,7 @@ cd inlong-agent
The agent supports two modes of operation: local operation and online operation
-### 1.1 Agent configuration
+### Agent configuration
Online operation needs to pull the configuration from inlong-manager, the configuration conf/agent.properties is as follows:
```ini
@@ -16,15 +16,6 @@ agent.manager.vip.http.host=manager web host
agent.manager.vip.http.port=manager web port
```
-### 1.2 Proxy configuration
-Create a new folder named .inlong in the agent directory, and create a new bid+.local file inside. For example, if the sending bid is set to a, then create a new file a.local
-
-write:
-```ini
-{"cluster_id":1,"isInterVisit":1,"size":1,"address": [{"port":write proxy port,"host":"write proxy ip"}], "switch":0}
-Among them, cluster_id, isInterVisit, and switch are reserved fields, please fill in the default values
-```
-
## 2、run
After decompression, run the following command
@@ -89,4 +80,102 @@ agent.http.port=Available ports
/data/inlong-agent/test.log //Represents reading the new file test.log in the inlong-agent folder
/data/inlong-agent/test[0-9]{1} // means to read the new file test in the inlong-agent folder followed by a number at the end
/data/inlong-agent/test //If test is a directory, it means to read all new files under test
- /data/inlong-agent/^\\d+(\\.\\d+)? // Start with one or more digits, followed by. or end with one. or more digits (? stands for optional, can match Examples: "5", "1.5" and "2.21"
\ No newline at end of file
+ /data/inlong-agent/^\\d+(\\.\\d+)? // Start with one or more digits, followed by. or end with one. or more digits (? stands for optional, can match Examples: "5", "1.5" and "2.21"
+
+
+## 5. Support to get data time from file name
+
+ Agent supports obtaining the time from the file name as the production time of the data. The configuration instructions are as follows:
+ /data/inlong-agent/***YYYYMMDDHH***
+ Where YYYYDDMMHH represents the data time, YYYY represents the year, MM represents the month, DD represents the day, and HH represents the hour
+ Where *** is any character
+
+ At the same time, you need to add the current data cycle to the job conf, the current support day cycle and hour cycle,
+ When adding a task, add the property job.cycleUnit
+
+ job.cycleUnit contains the following two types:
+ 1. D: Represents the data time and day dimension
+ 2. H: Represents the data time and hour dimension
+
+ E.g:
+ The configuration data source is
+ /data/inlong-agent/YYYYMMDDHH.log
+ Write data to 2021020211.log
+ Configure job.cycleUnit as D
+ Then the agent will try the 202020211.log file at the time of 202020211. When reading the data in the file, it will write all the data to the backend proxy at the time of 20210202.
+ If job.cycleUnit is configured as H
+ When collecting data in the 2021020211.log file, all data will be written to the backend proxy at the time of 2021020211
+
+
+ Examples of job submission
+
+```bash
+curl --location --request POST'http://localhost:8008/config/job' \
+--header'Content-Type: application/json' \
+--data'{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'
+```
+
+## 6. Support time offset reading
+
+ After the configuration is read by time, if you want to read data at other times than the current time, you can configure the time offset to complete
+ Configure the job attribute name as job.timeOffset, the value is number + time dimension, time dimension includes day and hour
+ For example, the following settings are supported
+ 1. 1d Read the data one day after the current time
+ 2. -1h read the data one hour before the current time
+
+
+ Examples of job submission
+```bash
+curl --location --request POST'http://localhost:8008/config/job' \
+--header'Content-Type: application/json' \
+--data'{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"timeOffset": "-1d",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'
+```
\ No newline at end of file
diff --git a/zh-cn/docs/modules/agent/quick_start.html b/zh-cn/docs/modules/agent/quick_start.html
index 00dd1d6..4246554 100644
--- a/zh-cn/docs/modules/agent/quick_start.html
+++ b/zh-cn/docs/modules/agent/quick_start.html
@@ -16,20 +16,13 @@
<pre><code>cd inlong-agent
</code></pre>
<p>agent 支持本地运行以及线上运行，其中线上运行从inlong manager拉取任务，本地运行可使用http请求提交任务</p>
-<h3>1.1 Agent 线上运行相关设置</h3>
+<h3>Agent 线上运行相关设置</h3>
<p>线上运行需要从inlong-manager拉取配置，配置conf/agent.properties如下：</p>
<pre><code class="language-ini"><span class="hljs-attr">agent.fetcher.classname</span>=org.apache.inlong.agent.plugin.fetcher.ManagerFetcher (设置任务获取的类名，默认为ManagerFetcher）
<span class="hljs-attr">agent.local.ip</span>=写入本机ip
<span class="hljs-attr">agent.manager.vip.http.host</span>=manager web host
<span class="hljs-attr">agent.manager.vip.http.port</span>=manager web port
</code></pre>
-<h3>1.2 DataProxy 相关设置</h3>
-<p>在agent目录下新建.inlong文件夹，内部新建bid+.local文件，例如设置的发送bid为a, 则新建a.local文件
-bid是指manager界面中，数据接入中业务信息的业务ID，此处不是创建的tube topic名称
-内部将写入：</p>
-<pre><code class="language-ini">{"cluster_id":1,"isInterVisit":1,"size":1,"address": [{"port":写入proxy port,"host":"写入proxy ip"}], "switch":0}
-其中cluster_id, isInterVisit，switch为预留字段，请填写默认值
-</code></pre>
<h2>2、运行</h2>
<p>解压后如下命令运行</p>
<pre><code class="language-bash">sh agent.sh start
@@ -84,6 +77,98 @@ bid是指manager界面中，数据接入中业务信息的业务ID，此处不
/data/inlong-agent/test //如果test为目录，则代表读取test下的所有新增文件
/data/inlong-agent/^\\d+(\\.\\d+)? // 以一个或多个数字开头,之后可以是.或者一个.或多个数字结尾，?代表可选,可以匹配的实例："5", "1.5" 和 "2.21"
</code></pre>
+<h2>5、支持从文件名称中获取数据时间</h2>
+<pre><code>Agent支持从文件名称中获取时间当作数据的生产时间，配置说明如下：
+/data/inlong-agent/***YYYYMMDDHH***
+其中YYYYDDMMHH代表数据时间，YYYY表示年，MM表示月份，DD表示天，HH表示小时
+其中***为任意字符
+
+同时需要在job conf中加入当前数据的周期，当前支持天周期以及小时周期，
+在添加任务时，加入属性job.cycleUnit
+
+job.cycleUnit 包含如下两种类型：
+1、D : 代表数据时间天维度
+2、H : 代表数据时间小时维度
+
+例如：
+配置数据源为
+/data/inlong-agent/YYYYMMDDHH.log
+写入数据到 2021020211.log
+配置 job.cycleUnit 为 D
+则agent会在2021020211时间尝试2021020211.log文件，读取文件中的数据时，会将所有数据以20210202这个时间写入到后端proxy
+如果配置 job.cycleUnit 为 H
+则采集2021020211.log文件中的数据时，会将所有数据以2021020211这个时间写入到后端proxy
+
+
+提交job举例
+</code></pre>
+<pre><code class="language-bash">curl --location --request POST <span class="hljs-string">'http://localhost:8008/config/job'</span> \
+--header <span class="hljs-string">'Content-Type: application/json'</span> \
+--data <span class="hljs-string">'{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'</span>
+</code></pre>
+<h2>6、支持时间偏移量offset读取</h2>
+<pre><code>在配置按照时间读取之后，如果想要读取当前时间之外的其他时间的数据，可以通过配置时间偏移量完成
+配置job属性名称为job.timeOffset，值为数字 + 时间维度，时间维度包括天和小时
+例如支持如下设置
+1、 1d 读取当前时间后一天的数据
+2、 -1h 读取当前时间前一个小时的数据
+
+
+提交job举例
+</code></pre>
+<pre><code class="language-bash">curl --location --request POST <span class="hljs-string">'http://localhost:8008/config/job'</span> \
+--header <span class="hljs-string">'Content-Type: application/json'</span> \
+--data <span class="hljs-string">'{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"timeOffset": "-1d",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'</span>
+</code></pre>
</div></section><footer class="footer-container"><div class="footer-body"><img src="/img/incubator-logo.svg"/><div class="cols-container"><div class="col col-24"><p>Apache InLong (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with [...]
<script src="https://f.alicdn.com/react/15.4.1/react-with-addons.min.js"></script>
<script src="https://f.alicdn.com/react/15.4.1/react-dom.min.js"></script>
diff --git a/zh-cn/docs/modules/agent/quick_start.json b/zh-cn/docs/modules/agent/quick_start.json
index 0c57210..76cdcaa 100644
--- a/zh-cn/docs/modules/agent/quick_start.json
+++ b/zh-cn/docs/modules/agent/quick_start.json
@@ -1,6 +1,6 @@
{
"filename": "quick_start.md",
- "__html": "<h2>1、配置</h2>\n<pre><code>cd inlong-agent\n</code></pre>\n<p>agent 支持本地运行以及线上运行，其中线上运行从inlong manager拉取任务，本地运行可使用http请求提交任务</p>\n<h3>1.1 Agent 线上运行相关设置</h3>\n<p>线上运行需要从inlong-manager拉取配置，配置conf/agent.properties如下：</p>\n<pre><code class=\"language-ini\"><span class=\"hljs-attr\">agent.fetcher.classname</span>=org.apache.inlong.agent.plugin.fetcher.ManagerFetcher (设置任务获取的类名，默认为ManagerFetcher）\n<span class=\"hljs-attr\">agent.local.ip</span>=写入本机ip\n<span class=\"hljs-attr\">ag [...]
+ "__html": "<h2>1、配置</h2>\n<pre><code>cd inlong-agent\n</code></pre>\n<p>agent 支持本地运行以及线上运行，其中线上运行从inlong manager拉取任务，本地运行可使用http请求提交任务</p>\n<h3>Agent 线上运行相关设置</h3>\n<p>线上运行需要从inlong-manager拉取配置，配置conf/agent.properties如下：</p>\n<pre><code class=\"language-ini\"><span class=\"hljs-attr\">agent.fetcher.classname</span>=org.apache.inlong.agent.plugin.fetcher.ManagerFetcher (设置任务获取的类名，默认为ManagerFetcher）\n<span class=\"hljs-attr\">agent.local.ip</span>=写入本机ip\n<span class=\"hljs-attr\">agent. [...]
"link": "/zh-cn/docs/modules/agent/quick_start.html",
"meta": {}
}
\ No newline at end of file
diff --git a/zh-cn/docs/modules/agent/quick_start.md b/zh-cn/docs/modules/agent/quick_start.md
index 8fb71fe..21314d6 100644
--- a/zh-cn/docs/modules/agent/quick_start.md
+++ b/zh-cn/docs/modules/agent/quick_start.md
@@ -5,7 +5,7 @@ cd inlong-agent
agent 支持本地运行以及线上运行，其中线上运行从inlong manager拉取任务，本地运行可使用http请求提交任务
-### 1.1 Agent 线上运行相关设置
+### Agent 线上运行相关设置
线上运行需要从inlong-manager拉取配置，配置conf/agent.properties如下：
```ini
@@ -15,15 +15,6 @@ agent.manager.vip.http.host=manager web host
agent.manager.vip.http.port=manager web port
```
-### 1.2 DataProxy 相关设置
-在agent目录下新建.inlong文件夹，内部新建bid+.local文件，例如设置的发送bid为a, 则新建a.local文件
-bid是指manager界面中，数据接入中业务信息的业务ID，此处不是创建的tube topic名称
-内部将写入：
-```ini
-{"cluster_id":1,"isInterVisit":1,"size":1,"address": [{"port":写入proxy port,"host":"写入proxy ip"}], "switch":0}
-其中cluster_id, isInterVisit，switch为预留字段，请填写默认值
-```
-
## 2、运行
解压后如下命令运行
```bash
@@ -87,3 +78,99 @@ curl --location --request POST 'http://localhost:8008/config/job' \
/data/inlong-agent/^\\d+(\\.\\d+)? // 以一个或多个数字开头,之后可以是.或者一个.或多个数字结尾，?代表可选,可以匹配的实例："5", "1.5" 和 "2.21"
+## 5、支持从文件名称中获取数据时间
+
+ Agent支持从文件名称中获取时间当作数据的生产时间，配置说明如下：
+ /data/inlong-agent/***YYYYMMDDHH***
+ 其中YYYYDDMMHH代表数据时间，YYYY表示年，MM表示月份，DD表示天，HH表示小时
+ 其中***为任意字符
+
+ 同时需要在job conf中加入当前数据的周期，当前支持天周期以及小时周期，
+ 在添加任务时，加入属性job.cycleUnit
+
+ job.cycleUnit 包含如下两种类型：
+ 1、D : 代表数据时间天维度
+ 2、H : 代表数据时间小时维度
+
+ 例如：
+ 配置数据源为
+ /data/inlong-agent/YYYYMMDDHH.log
+ 写入数据到 2021020211.log
+ 配置 job.cycleUnit 为 D
+ 则agent会在2021020211时间尝试2021020211.log文件，读取文件中的数据时，会将所有数据以20210202这个时间写入到后端proxy
+ 如果配置 job.cycleUnit 为 H
+ 则采集2021020211.log文件中的数据时，会将所有数据以2021020211这个时间写入到后端proxy
+
+
+ 提交job举例
+```bash
+curl --location --request POST 'http://localhost:8008/config/job' \
+--header 'Content-Type: application/json' \
+--data '{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'
+```
+
+
+## 6、支持时间偏移量offset读取
+
+ 在配置按照时间读取之后，如果想要读取当前时间之外的其他时间的数据，可以通过配置时间偏移量完成
+ 配置job属性名称为job.timeOffset，值为数字 + 时间维度，时间维度包括天和小时
+ 例如支持如下设置
+ 1、 1d 读取当前时间后一天的数据
+ 2、 -1h 读取当前时间前一个小时的数据
+
+
+ 提交job举例
+```bash
+curl --location --request POST 'http://localhost:8008/config/job' \
+--header 'Content-Type: application/json' \
+--data '{
+"job": {
+"dir": {
+"path": "",
+"pattern": "/data/inlong-agent/test.log"
+},
+"trigger": "org.apache.inlong.agent.plugin.trigger.DirectoryTrigger",
+"id": 1,
+"thread": {
+"running": {
+"core": "4"
+}
+},
+"name": "fileAgentTest",
+"cycleUnit": "D",
+"timeOffset": "-1d",
+"source": "org.apache.inlong.agent.plugin.sources.TextFileSource",
+"sink": "org.apache.inlong.agent.plugin.sinks.ProxySink",
+"channel": "org.apache.inlong.agent.plugin.channel.MemoryChannel"
+},
+"proxy": {
+"bid": "bid10",
+"tid": "bid10"
+},
+"op": "add"
+}'
+```
\ No newline at end of file