You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by gi...@apache.org on 2021/07/28 09:14:57 UTC
[incubator-inlong-website] branch asf-site updated: Automated
deployment: Wed Jul 28 09:14:49 UTC 2021
1035ff61c743b3f0ca9c1178423017a9685f1a12
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7c4c8e1 Automated deployment: Wed Jul 28 09:14:49 UTC 2021 1035ff61c743b3f0ca9c1178423017a9685f1a12
7c4c8e1 is described below
commit 7c4c8e1ed6f5ef3694f65d510eb14276efe96934
Author: yuanboliu <yu...@users.noreply.github.com>
AuthorDate: Wed Jul 28 09:14:49 2021 +0000
Automated deployment: Wed Jul 28 09:14:49 UTC 2021 1035ff61c743b3f0ca9c1178423017a9685f1a12
---
docs/en-us/modules/dataproxy/architecture.md | 11 +++++------
docs/zh-cn/modules/dataproxy/architecture.md | 12 ++++++------
en-us/docs/modules/dataproxy/architecture.html | 9 ++++-----
en-us/docs/modules/dataproxy/architecture.json | 2 +-
en-us/docs/modules/dataproxy/architecture.md | 11 +++++------
zh-cn/docs/modules/dataproxy/architecture.html | 10 +++++-----
zh-cn/docs/modules/dataproxy/architecture.json | 2 +-
zh-cn/docs/modules/dataproxy/architecture.md | 12 ++++++------
8 files changed, 33 insertions(+), 36 deletions(-)
diff --git a/docs/en-us/modules/dataproxy/architecture.md b/docs/en-us/modules/dataproxy/architecture.md
index 8e6d2ef..cd30eeb 100644
--- a/docs/en-us/modules/dataproxy/architecture.md
+++ b/docs/en-us/modules/dataproxy/architecture.md
@@ -26,8 +26,7 @@ Source configuration example and corresponding notes:
tcp resolution type definition, here provide the class name for instantiation, SimpleTcpSource is mainly to initialize the configuration and start port monitoring
agent1.sources.tcp-source.msg-factory-name = org.apache.flume.source.ServerMessageFactory
- Handler used for message structure analysis, and set read stream handler and write stream handler, netty concept, please refer to: https://blog.csdn.net/u013252773/article/details/21195593
-
+ Handler used for message structure analysis, and set read stream handler and write stream handler
agent1.sources.tcp-source.host = 0.0.0.0
tcp ip binding monitoring, binding all network cards by default
@@ -57,10 +56,10 @@ Source configuration example and corresponding notes:
Netty thread pool work thread upper limit, generally recommended to choose twice the cpu
agent1.sources.tcp-source.receiveBufferSize = 524288
- Netty server tcp tuning parameters, please refer to: https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
-
+ Netty server tcp tuning parameters
+
agent1.sources.tcp-source.sendBufferSize = 524288
- Netty server tcp tuning parameters, please refer to: https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
+ Netty server tcp tuning parameters
agent1.sources.tcp-source.custom-cp = true
Whether to use the self-developed channel process, the self-developed channel process can select the alternate channel to send when the main channel is blocked
@@ -128,7 +127,7 @@ Sink configuration example and corresponding notes
agent1.sinks.meta-sink-more1.type = org.apache.flume.sink.MetaSink
The sink class is implemented, where the message is implemented to push data to the tube cluster
- agent1.sinks.meta-sink-more1.master-host-port-list = sk-tegspider-tube-master-1:8609,sk-tegspider-tube-master-2:8609
+ agent1.sinks.meta-sink-more1.master-host-port-list =
Tube cluster master node list
agent1.sinks.meta-sink-more1.send_timeout = 30000
diff --git a/docs/zh-cn/modules/dataproxy/architecture.md b/docs/zh-cn/modules/dataproxy/architecture.md
index 7cf7770..97f89fd 100644
--- a/docs/zh-cn/modules/dataproxy/architecture.md
+++ b/docs/zh-cn/modules/dataproxy/architecture.md
@@ -28,7 +28,7 @@ Source配置示例以及对应的注解：
tcp解析类型定义，这里提供类名用于实例化，SimpleTcpSource主要是初始化配置并启动端口监听
agent1.sources.tcp-source.msg-factory-name = org.apache.flume.source.ServerMessageFactory
- 用于构造消息解析的handler，并设置read stream handler和write stream handler，netty概念，具体可参考：https://blog.csdn.net/u013252773/article/details/21195593
+ 用于构造消息解析的handler，并设置read stream handler和write stream handler
agent1.sources.tcp-source.host = 0.0.0.0
tcp ip绑定监听，默认绑定所有网卡
@@ -37,7 +37,7 @@ Source配置示例以及对应的注解：
tcp 端口绑定，默认绑定46801端口
agent1.sources.tcp-source.highWaterMark=2621440
- netty概念，设置netty高水位值，高水位详细解答可参考：https://www.jianshu.com/p/c9e9a5a11943
+ netty概念，设置netty高水位值
agent1.sources.tcp-source.max-msg-length = 524288
限制单个包大小，这里如果传输的是压缩包，则是压缩包大小，限制512KB
@@ -55,10 +55,10 @@ Source配置示例以及对应的注解：
netty线程池工作线程上限，一般推荐选择cpu的两倍
agent1.sources.tcp-source.receiveBufferSize = 524288
- netty server tcp调优参数，具体解释可参考：https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
-
+ netty server tcp调优参数
+
agent1.sources.tcp-source.sendBufferSize = 524288
- netty server tcp调优参数，具体解释可参考：https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
+ netty server tcp调优参数
agent1.sources.tcp-source.custom-cp = true
是否使用自研的channel process，自研channel process可在主channel阻塞时，选择备用channel发送
@@ -125,7 +125,7 @@ Sink配置示例以及对应的注解
agent1.sinks.meta-sink-more1.type = org.apache.flume.sink.MetaSink
sink类实现，此处实现消息向tube集群推送数据
- agent1.sinks.meta-sink-more1.master-host-port-list = sk-tegspider-tube-master-1:8609,sk-tegspider-tube-master-2:8609
+ agent1.sinks.meta-sink-more1.master-host-port-list =
tube集群master节点列表
agent1.sinks.meta-sink-more1.send_timeout = 30000
diff --git a/en-us/docs/modules/dataproxy/architecture.html b/en-us/docs/modules/dataproxy/architecture.html
index fd8a7ea..55adf39 100644
--- a/en-us/docs/modules/dataproxy/architecture.html
+++ b/en-us/docs/modules/dataproxy/architecture.html
@@ -33,8 +33,7 @@ agent1.sources.tcp-source.type = org.apache.flume.source.SimpleTcpSource
tcp resolution type definition, here provide the class name for instantiation, SimpleTcpSource is mainly to initialize the configuration and start port monitoring
agent1.sources.tcp-source.msg-factory-name = org.apache.flume.source.ServerMessageFactory
-Handler used for message structure analysis, and set read stream handler and write stream handler, netty concept, please refer to: https://blog.csdn.net/u013252773/article/details/21195593
-
+Handler used for message structure analysis, and set read stream handler and write stream handler
agent1.sources.tcp-source.host = 0.0.0.0
tcp ip binding monitoring, binding all network cards by default
@@ -64,10 +63,10 @@ agent1.sources.tcp-source.max-threads = 64
Netty thread pool work thread upper limit, generally recommended to choose twice the cpu
agent1.sources.tcp-source.receiveBufferSize = 524288
-Netty server tcp tuning parameters, please refer to: https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
+Netty server tcp tuning parameters
agent1.sources.tcp-source.sendBufferSize = 524288
-Netty server tcp tuning parameters, please refer to: https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
+Netty server tcp tuning parameters
agent1.sources.tcp-source.custom-cp = true
Whether to use the self-developed channel process, the self-developed channel process can select the alternate channel to send when the main channel is blocked
@@ -130,7 +129,7 @@ The upstream channel name of the sink
agent1.sinks.meta-sink-more1.type = org.apache.flume.sink.MetaSink
The sink class is implemented, where the message is implemented to push data to the tube cluster
-agent1.sinks.meta-sink-more1.master-host-port-list = sk-tegspider-tube-master-1:8609,sk-tegspider-tube-master-2:8609
+agent1.sinks.meta-sink-more1.master-host-port-list =
Tube cluster master node list
agent1.sinks.meta-sink-more1.send_timeout = 30000
diff --git a/en-us/docs/modules/dataproxy/architecture.json b/en-us/docs/modules/dataproxy/architecture.json
index 78c155f..5b2e422 100644
--- a/en-us/docs/modules/dataproxy/architecture.json
+++ b/en-us/docs/modules/dataproxy/architecture.json
@@ -1,6 +1,6 @@
{
"filename": "architecture.md",
- "__html": "<h1>1、intro</h1>\n<pre><code>Inlong-dataProxy belongs to the inlong proxy layer and is used for data collection, reception and forwarding. Through format conversion, the data is converted into TDMsg1 format that can be cached and processed by the cache layer\nInLong-dataProxy acts as a bridge from the InLong collection end to the InLong buffer end. Dataproxy pulls the relationship between the business id and the corresponding topic name from the manager module, and internall [...]
+ "__html": "<h1>1、intro</h1>\n<pre><code>Inlong-dataProxy belongs to the inlong proxy layer and is used for data collection, reception and forwarding. Through format conversion, the data is converted into TDMsg1 format that can be cached and processed by the cache layer\nInLong-dataProxy acts as a bridge from the InLong collection end to the InLong buffer end. Dataproxy pulls the relationship between the business id and the corresponding topic name from the manager module, and internall [...]
"link": "/en-us/docs/modules/dataproxy/architecture.html",
"meta": {}
}
\ No newline at end of file
diff --git a/en-us/docs/modules/dataproxy/architecture.md b/en-us/docs/modules/dataproxy/architecture.md
index 8e6d2ef..cd30eeb 100644
--- a/en-us/docs/modules/dataproxy/architecture.md
+++ b/en-us/docs/modules/dataproxy/architecture.md
@@ -26,8 +26,7 @@ Source configuration example and corresponding notes:
tcp resolution type definition, here provide the class name for instantiation, SimpleTcpSource is mainly to initialize the configuration and start port monitoring
agent1.sources.tcp-source.msg-factory-name = org.apache.flume.source.ServerMessageFactory
- Handler used for message structure analysis, and set read stream handler and write stream handler, netty concept, please refer to: https://blog.csdn.net/u013252773/article/details/21195593
-
+ Handler used for message structure analysis, and set read stream handler and write stream handler
agent1.sources.tcp-source.host = 0.0.0.0
tcp ip binding monitoring, binding all network cards by default
@@ -57,10 +56,10 @@ Source configuration example and corresponding notes:
Netty thread pool work thread upper limit, generally recommended to choose twice the cpu
agent1.sources.tcp-source.receiveBufferSize = 524288
- Netty server tcp tuning parameters, please refer to: https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
-
+ Netty server tcp tuning parameters
+
agent1.sources.tcp-source.sendBufferSize = 524288
- Netty server tcp tuning parameters, please refer to: https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
+ Netty server tcp tuning parameters
agent1.sources.tcp-source.custom-cp = true
Whether to use the self-developed channel process, the self-developed channel process can select the alternate channel to send when the main channel is blocked
@@ -128,7 +127,7 @@ Sink configuration example and corresponding notes
agent1.sinks.meta-sink-more1.type = org.apache.flume.sink.MetaSink
The sink class is implemented, where the message is implemented to push data to the tube cluster
- agent1.sinks.meta-sink-more1.master-host-port-list = sk-tegspider-tube-master-1:8609,sk-tegspider-tube-master-2:8609
+ agent1.sinks.meta-sink-more1.master-host-port-list =
Tube cluster master node list
agent1.sinks.meta-sink-more1.send_timeout = 30000
diff --git a/zh-cn/docs/modules/dataproxy/architecture.html b/zh-cn/docs/modules/dataproxy/architecture.html
index 42d268d..57dab69 100644
--- a/zh-cn/docs/modules/dataproxy/architecture.html
+++ b/zh-cn/docs/modules/dataproxy/architecture.html
@@ -34,7 +34,7 @@ agent1.sources.tcp-source.type = org.apache.flume.source.SimpleTcpSource
tcp解析类型定义，这里提供类名用于实例化，SimpleTcpSource主要是初始化配置并启动端口监听
agent1.sources.tcp-source.msg-factory-name = org.apache.flume.source.ServerMessageFactory
-用于构造消息解析的handler，并设置read stream handler和write stream handler，netty概念，具体可参考：https://blog.csdn.net/u013252773/article/details/21195593
+用于构造消息解析的handler，并设置read stream handler和write stream handler
agent1.sources.tcp-source.host = 0.0.0.0
tcp ip绑定监听，默认绑定所有网卡
@@ -43,7 +43,7 @@ agent1.sources.tcp-source.port = 46801
tcp 端口绑定，默认绑定46801端口
agent1.sources.tcp-source.highWaterMark=2621440
-netty概念，设置netty高水位值，高水位详细解答可参考：https://www.jianshu.com/p/c9e9a5a11943
+netty概念，设置netty高水位值
agent1.sources.tcp-source.max-msg-length = 524288
限制单个包大小，这里如果传输的是压缩包，则是压缩包大小，限制512KB
@@ -61,10 +61,10 @@ agent1.sources.tcp-source.max-threads = 64
netty线程池工作线程上限，一般推荐选择cpu的两倍
agent1.sources.tcp-source.receiveBufferSize = 524288
-netty server tcp调优参数，具体解释可参考：https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
+netty server tcp调优参数
agent1.sources.tcp-source.sendBufferSize = 524288
-netty server tcp调优参数，具体解释可参考：https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
+netty server tcp调优参数
agent1.sources.tcp-source.custom-cp = true
是否使用自研的channel process，自研channel process可在主channel阻塞时，选择备用channel发送
@@ -127,7 +127,7 @@ sink的上游channel名称
agent1.sinks.meta-sink-more1.type = org.apache.flume.sink.MetaSink
sink类实现，此处实现消息向tube集群推送数据
-agent1.sinks.meta-sink-more1.master-host-port-list = sk-tegspider-tube-master-1:8609,sk-tegspider-tube-master-2:8609
+agent1.sinks.meta-sink-more1.master-host-port-list =
tube集群master节点列表
agent1.sinks.meta-sink-more1.send_timeout = 30000
diff --git a/zh-cn/docs/modules/dataproxy/architecture.json b/zh-cn/docs/modules/dataproxy/architecture.json
index ebb7463..7bc2409 100644
--- a/zh-cn/docs/modules/dataproxy/architecture.json
+++ b/zh-cn/docs/modules/dataproxy/architecture.json
@@ -1,6 +1,6 @@
{
"filename": "architecture.md",
- "__html": "<h1>一、说明</h1>\n<pre><code>InLong-dataProxy属于inlong proxy层，用于数据的汇集接收以及转发。通过格式转换，将数据转为cache层可以缓存处理的TDMsg1格式\nInLong-dataProxy充当了InLong采集端到InLong缓冲端的桥梁，dataproxy从manager模块拉取业务id与对应topic名称的关系，内部管理多个topic的生产者\n当dataproxy收到消息时，会首先缓存到本地的Channel中，并使用本地的producer往后端即cache层发送数据\nInLong-dataProxy整体架构基于Apache Flume。inlong-dataproxy在该项目的基础上，扩展了source层和sink层，并对容灾转发做了优化处理，提升了系统的稳定性。\n</code></pre>\n<h1>二、架构</h1>\n<p><img src=\"img/architecture.png\" alt=\"\"></p>\n<pre><code>1.Source层开启端口监听 [...]
+ "__html": "<h1>一、说明</h1>\n<pre><code>InLong-dataProxy属于inlong proxy层，用于数据的汇集接收以及转发。通过格式转换，将数据转为cache层可以缓存处理的TDMsg1格式\nInLong-dataProxy充当了InLong采集端到InLong缓冲端的桥梁，dataproxy从manager模块拉取业务id与对应topic名称的关系，内部管理多个topic的生产者\n当dataproxy收到消息时，会首先缓存到本地的Channel中，并使用本地的producer往后端即cache层发送数据\nInLong-dataProxy整体架构基于Apache Flume。inlong-dataproxy在该项目的基础上，扩展了source层和sink层，并对容灾转发做了优化处理，提升了系统的稳定性。\n</code></pre>\n<h1>二、架构</h1>\n<p><img src=\"img/architecture.png\" alt=\"\"></p>\n<pre><code>1.Source层开启端口监听 [...]
"link": "/zh-cn/docs/modules/dataproxy/architecture.html",
"meta": {}
}
\ No newline at end of file
diff --git a/zh-cn/docs/modules/dataproxy/architecture.md b/zh-cn/docs/modules/dataproxy/architecture.md
index 7cf7770..97f89fd 100644
--- a/zh-cn/docs/modules/dataproxy/architecture.md
+++ b/zh-cn/docs/modules/dataproxy/architecture.md
@@ -28,7 +28,7 @@ Source配置示例以及对应的注解：
tcp解析类型定义，这里提供类名用于实例化，SimpleTcpSource主要是初始化配置并启动端口监听
agent1.sources.tcp-source.msg-factory-name = org.apache.flume.source.ServerMessageFactory
- 用于构造消息解析的handler，并设置read stream handler和write stream handler，netty概念，具体可参考：https://blog.csdn.net/u013252773/article/details/21195593
+ 用于构造消息解析的handler，并设置read stream handler和write stream handler
agent1.sources.tcp-source.host = 0.0.0.0
tcp ip绑定监听，默认绑定所有网卡
@@ -37,7 +37,7 @@ Source配置示例以及对应的注解：
tcp 端口绑定，默认绑定46801端口
agent1.sources.tcp-source.highWaterMark=2621440
- netty概念，设置netty高水位值，高水位详细解答可参考：https://www.jianshu.com/p/c9e9a5a11943
+ netty概念，设置netty高水位值
agent1.sources.tcp-source.max-msg-length = 524288
限制单个包大小，这里如果传输的是压缩包，则是压缩包大小，限制512KB
@@ -55,10 +55,10 @@ Source配置示例以及对应的注解：
netty线程池工作线程上限，一般推荐选择cpu的两倍
agent1.sources.tcp-source.receiveBufferSize = 524288
- netty server tcp调优参数，具体解释可参考：https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
-
+ netty server tcp调优参数
+
agent1.sources.tcp-source.sendBufferSize = 524288
- netty server tcp调优参数，具体解释可参考：https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-rcvbuf
+ netty server tcp调优参数
agent1.sources.tcp-source.custom-cp = true
是否使用自研的channel process，自研channel process可在主channel阻塞时，选择备用channel发送
@@ -125,7 +125,7 @@ Sink配置示例以及对应的注解
agent1.sinks.meta-sink-more1.type = org.apache.flume.sink.MetaSink
sink类实现，此处实现消息向tube集群推送数据
- agent1.sinks.meta-sink-more1.master-host-port-list = sk-tegspider-tube-master-1:8609,sk-tegspider-tube-master-2:8609
+ agent1.sinks.meta-sink-more1.master-host-port-list =
tube集群master节点列表
agent1.sinks.meta-sink-more1.send_timeout = 30000