You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "justinwwhuang (via GitHub)" <gi...@apache.org> on 2024/09/24 07:35:52 UTC
[PR] [INLONG-980][Agent] Add Agent inner data format doc [inlong-website]
justinwwhuang opened a new pull request, #981:
URL: https://github.com/apache/inlong-website/pull/981
Fixes #980
### Motivation
In order to deepen everyone's understanding of the internal data flow principle of agent
### Modifications
Add Agent inner data format doc
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
- [ ] This change is a trivial rework/code cleanup without any test coverage.
- [ ] This change is already covered by existing tests, such as:
*(please describe tests)*
- [ ] This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Documentation
No doc needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-980][Agent] Add Agent data format doc [inlong-website]
Posted by "aloyszhang (via GitHub)" <gi...@apache.org>.
aloyszhang commented on code in PR #981:
URL: https://github.com/apache/inlong-website/pull/981#discussion_r1772906743
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/development/Internal_protocol_format/agent.md:
##########
@@ -0,0 +1,154 @@
+---
+title: Agent 数据组织
+sidebar_position: 2
+---
+## 概述
+本文主要介绍 Agent 数据在子模块间流转以及对应的数据结构。
+## 数据流转
+
+
+在 Agent 模块介绍时我们知道 Agent 中有 Source 和 Sink 模块，其中 Source 负责从数据源采集数据；Sink 负责将数据发送到下游，当
+前我们认为下游只有 DataProxy。Instance 则是负责将数据从 Source 搬运到 Sink。
+## 数据格式
+### Source
+
+
+Source 主要有 3 个功能：
+
+1 从数据源采集数据，每条数据填充到一个新的 SourceData 对象。
+
+2 将填充完成的 SourceData 对象放入 Source 模块的缓存队列中。
+
+3 外部调用 Source 模块 Read 方法时从缓存队列中取出一个 SourceData 组装成 Message 并返回。
+#### SourceData
+```
+public class SourceData {
+ private byte[] data;
+ private String offset;
+}
+```
+从数据源采集的数据可能会有各种各样的格式，因此我们用 byte[] 来存放（原始数据在后续各个缓解流转时均保持 byte[]）。同时每一条数据都会有对应的位点信息，
Review Comment:
```suggestion
从数据源采集的数据可能会有各种各样的格式，因此我们用 byte[] 来存放（原始数据在后续各个环节流转时均保持 byte[]）。同时每一条数据都会有对应的位点信息，
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-980][Agent] Add Agent data format doc [inlong-website]
Posted by "justinwwhuang (via GitHub)" <gi...@apache.org>.
justinwwhuang merged PR #981:
URL: https://github.com/apache/inlong-website/pull/981
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-980][Agent] Add Agent data format doc [inlong-website]
Posted by "doleyzi (via GitHub)" <gi...@apache.org>.
doleyzi commented on code in PR #981:
URL: https://github.com/apache/inlong-website/pull/981#discussion_r1772804838
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/development/Internal_protocol_format/agent.md:
##########
@@ -0,0 +1,154 @@
+---
+title: Agent 数据组织
+sidebar_position: 2
+---
+## 概述
+本文主要介绍 Agent 数据在子模块间流转以及对应的数据结构。
+## 数据流转
+
+
+在 Agent 模块介绍时我们知道 Agent 中有 Source 和 Sink 模块，其中 Source 负责从数据源采集数据；Sink 负责将数据发送到下游，当
+前我们认为下游只有 DataProxy。Instance 则是负责将数据从 Source 搬运到 Sink。
Review Comment:
It is recommended not to wrap here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [INLONG-980][Agent] Add Agent data format doc [inlong-website]
Posted by "aloyszhang (via GitHub)" <gi...@apache.org>.
aloyszhang commented on code in PR #981:
URL: https://github.com/apache/inlong-website/pull/981#discussion_r1772910002
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/development/Internal_protocol_format/agent.md:
##########
@@ -0,0 +1,154 @@
+---
+title: Agent 数据组织
+sidebar_position: 2
+---
+## 概述
+本文主要介绍 Agent 数据在子模块间流转以及对应的数据结构。
+## 数据流转
+
+
+在 Agent 模块介绍时我们知道 Agent 中有 Source 和 Sink 模块，其中 Source 负责从数据源采集数据；Sink 负责将数据发送到下游，当
+前我们认为下游只有 DataProxy。Instance 则是负责将数据从 Source 搬运到 Sink。
+## 数据格式
+### Source
+
+
+Source 主要有 3 个功能：
+
+1 从数据源采集数据，每条数据填充到一个新的 SourceData 对象。
+
+2 将填充完成的 SourceData 对象放入 Source 模块的缓存队列中。
+
+3 外部调用 Source 模块 Read 方法时从缓存队列中取出一个 SourceData 组装成 Message 并返回。
+#### SourceData
+```
+public class SourceData {
+ private byte[] data;
+ private String offset;
+}
+```
+从数据源采集的数据可能会有各种各样的格式，因此我们用 byte[] 来存放（原始数据在后续各个缓解流转时均保持 byte[]）。同时每一条数据都会有对应的位点信息，
+并且由于位点信息多种多样，我们用 String 来存放位点信息。
+#### 缓存队列
+顾名思义，我们将 SourceData 放到 queue 中以达到缓存的目的，这样可以解耦数据源和 Agent 内部处理速度不匹配的问题。queue 类
Review Comment:
```suggestion
顾名思义，我们将 SourceData 放到 queue 中以达到缓存的目的，这样可以解耦数据源和 Agent 内部处理速度不匹配的问题。Queue 类
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org