You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by do...@apache.org on 2022/03/22 07:37:24 UTC
[incubator-inlong-website] branch master updated: [INLONG-3288][Audit] Support TubMQ for website (#314)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new b01e345 [INLONG-3288][Audit] Support TubMQ for website (#314)
b01e345 is described below
commit b01e345c81b7525c520362fe6b4f622d734fd1fd
Author: xueyingzhang <86...@users.noreply.github.com>
AuthorDate: Tue Mar 22 15:37:20 2022 +0800
[INLONG-3288][Audit] Support TubMQ for website (#314)
---
docs/modules/audit/overview.md | 2 +-
docs/modules/audit/quick_start.md | 39 ++++++++++++++++++----
.../current/modules/audit/overview.md | 2 +-
.../current/modules/audit/quick_start.md | 37 ++++++++++++++++----
4 files changed, 65 insertions(+), 15 deletions(-)
diff --git a/docs/modules/audit/overview.md b/docs/modules/audit/overview.md
index 591d812..56c39f7 100644
--- a/docs/modules/audit/overview.md
+++ b/docs/modules/audit/overview.md
@@ -12,7 +12,7 @@ The transmission status of each module, and whether the data stream is lost or r
## Architecture
1. The audit SDK is nested in the service that needs to be audited, audits the service, and sends the audit result to the audit access layer
-2. The audit proxy writes audit data to MQ (kafak or pulsar)
+2. The audit proxy writes audit data to MQ (Pulsar or TubeMQ)
3. The distribution service consumes the audit data of MQ, and writes the audit data to MySQL and Elasticsearch
4. The interface layer encapsulates the data of MySQL and Elasticsearch
5. Application scenarios mainly include report display, audit reconciliation, etc.
diff --git a/docs/modules/audit/quick_start.md b/docs/modules/audit/quick_start.md
index 979d1bb..3bb3670 100644
--- a/docs/modules/audit/quick_start.md
+++ b/docs/modules/audit/quick_start.md
@@ -5,20 +5,35 @@ title: Deployment
All deploying files at `inlong-audit` directory, if you use MySQL to store audit data, you need to first create the database through `sql/apache_inlong_audit.sql`.
## Audit Proxy
-### Configure Pulsar
-The configuration file is `conf/audit.conf`.
+### Configure MessageQueue
+You can choose Apache Pulsar or InLong TubeMQ as your MessageQueue service:
+
+- If using Pulsar, the configuration file is `conf/audit-proxy-pulsar.conf`. Replace **`PULSAR_BROKER_LIST`** with the service url of your Pulsar cluster.
+
```Shell
agent1.sources.tcp-source.host = 0.0.0.0
-agent1.sources.tcp-source.port = 46801
+agent1.sources.tcp-source.port = 10081
agent1.sinks.pulsar-sink-msg1.pulsar_server_url= pulsar://PULSAR_BROKER_LIST
agent1.sinks.pulsar-sink-msg1.topic = persistent://public/default/inlong-audit
agent1.sinks.pulsar-sink-msg2.pulsar_server_url = pulsar://PULSAR_BROKER_LIST
agent1.sinks.pulsar-sink-msg2.topic = persistent://public/default/inlong-audit
```
+- If using TubeMQ, the configuration file is `conf/audit-proxy-tube.conf`. Replace **`TUBE_LIST`** with the master address of your TubeMQ cluster.
+
+```Shell
+agent1.sources.tcp-source.host = 0.0.0.0
+agent1.sources.tcp-source.port = 10081
+agent1.sinks.tube-sink-msg1.master-host-port-list = TUBE_LIST
+agent1.sinks.tube-sink-msg1.topic = inlong-audit
+agent1.sinks.tube-sink-msg2.master-host-port-list = TUBE_LIST
+agent1.sinks.tube-sink-msg2.topic = inlong-audit
+```
+
### Start
```Shell
-sh ./bin/proxy-start.sh
+#By default, pulsar is used as the MessageQueue, and the audit-proxy-pulsar.conf configuration file is loaded.
+sh ./bin/proxy-start.sh [pulsar｜tube]
```
## Audit Store
@@ -26,12 +41,22 @@ sh ./bin/proxy-start.sh
The configuration file is `conf/application.properties`.
```Shell
+# proxy.type: pulsar / tube
+audit.config.proxy.type=pulsar
+
# store.server: mysql / elasticsearch
audit.config.store.mode=mysql
-# audit pulsar topic
-audit.pulsar.server.url=pulsar://127.0.0.1:6650
+
+# audit pulsar config (optional), replace PULSAR_BROKER_LIST with your Pulsar service url
+audit.pulsar.server.url=pulsar://PULSAR_BROKER_LIST
audit.pulsar.topic=persistent://public/default/inlong-audit
audit.pulsar.consumer.sub.name=sub-audit
+
+# audit tube config (optional), replace TUBE_LIST with your TubeMQ master address
+audit.tube.masterlist=TUBE_LIST
+audit.tube.topic=inlong-audit
+audit.tube.consumer.group.name=inlong-audit-consumer
+
# mysql
spring.datasource.druid.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
spring.datasource.druid.username=root
@@ -49,4 +74,4 @@ The startup script file `./bin/store-start.sh`
sh ./bin/store-start.sh
```
-The default listen port is `46801`.
\ No newline at end of file
+The default listen port is `10081`.
\ No newline at end of file
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md
index dc52b83..51a5181 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md
@@ -12,7 +12,7 @@ InLong审计是独立于InLong的一个子系统，对InLong系统的Agent、Dat
## 架构
1. 审计SDK嵌套在需要审计的服务，对服务进行审计，将审计结果发送到审计接入层。
-2. 审计接入层将审计数据写到MQ(kafak或者pulsar)。
+2. 审计接入层将审计数据写到MQ(Pulsar或者TubeMQ)。
3. 分发服务消费MQ的审计数据，将审计数据写到MySQL、Elasticsearch。
4. 接口层将MySQL、Elasticsearch的数据进行封装。
5. 应用场景主要包括报表展示、审计对账等等。
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md
index 84fec0a..fff004c 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md
@@ -5,8 +5,11 @@ title: 安装部署
所有的安装文件都在 `inlong-audit` 目录下，如果使用 MySQL 存储审计数据，需要先通过`sql/apache_inlong_audit.sql`出生数据库。
## Audit Proxy
-### 配置 Pulsar
-配置文件 `conf/audit-proxy.conf`.
+### 配置消息队列
+消息队列服务目前可以使用Apache Pulsar或者InLong TubeMQ：
+
+- 若使用Pulsar，配置文件 `conf/audit-proxy-pulsar.conf`，将**`PULSAR_BROKER_LIST`**替换为Pulsar集群的服务地址。
+
```Shell
agent1.sources.tcp-source.host = 0.0.0.0
agent1.sources.tcp-source.port = 10081
@@ -16,22 +19,44 @@ agent1.sinks.pulsar-sink-msg2.pulsar_server_url = pulsar://PULSAR_BROKER_LIST
agent1.sinks.pulsar-sink-msg2.topic = persistent://public/default/inlong-audit
```
+- 若使用TubeMQ，配置文件 `conf/audit-proxy-tube.conf`，将**`TUBE_LIST`**替换为TubeMQ集群的master地址。
+```Shell
+agent1.sources.tcp-source.host = 0.0.0.0
+agent1.sources.tcp-source.port = 10081
+agent1.sinks.tube-sink-msg1.master-host-port-list = TUBE_LIST
+agent1.sinks.tube-sink-msg1.topic = inlong-audit
+agent1.sinks.tube-sink-msg2.master-host-port-list = TUBE_LIST
+agent1.sinks.tube-sink-msg2.topic = inlong-audit
+```
+
### 启动
+
```Shell
-sh ./bin/proxy-start.sh
+#默认使用 pulsar 作为消息队列，加载 audit-proxy-pulsar.conf 配置文件
+sh ./bin/proxy-start.sh [pulsar｜tube]
```
## Audit Store
### 配置
-配置文件 `conf/application.properties`.
+配置文件 `conf/application.properties`
```Shell
+# proxy.type: pulsar / tube
+audit.config.proxy.type=pulsar
+
# store.server: mysql / elasticsearch
audit.config.store.mode=mysql
-# audit pulsar topic
-audit.pulsar.server.url=pulsar://127.0.0.1:6650
+
+# audit pulsar config (optional)，将PULSAR_BROKER_LIST替换为Pulsar集群的服务地址
+audit.pulsar.server.url=pulsar://PULSAR_BROKER_LIST
audit.pulsar.topic=persistent://public/default/inlong-audit
audit.pulsar.consumer.sub.name=sub-audit
+
+# audit tube config (optional)，将TUBE_LIST替换为TubeMQ集群的master地址
+audit.tube.masterlist=TUBE_LIST
+audit.tube.topic=inlong-audit
+audit.tube.consumer.group.name=inlong-audit-consumer
+
# mysql
spring.datasource.druid.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
spring.datasource.druid.username=root