You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by do...@apache.org on 2023/03/08 06:56:04 UTC
[inlong-website] branch master updated: [INLONG-704][Doc] Add Kafka document for Audit (#705)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 43fb921c78 [INLONG-704][Doc] Add Kafka document for Audit (#705)
43fb921c78 is described below
commit 43fb921c785826bbbb8487e12764454d47f4707e
Author: haifxu <xh...@gmail.com>
AuthorDate: Wed Mar 8 14:55:59 2023 +0800
[INLONG-704][Doc] Add Kafka document for Audit (#705)
---
docs/modules/audit/configure.md | 4 +--
docs/modules/audit/overview.md | 2 +-
docs/modules/audit/quick_start.md | 22 +++++++++++++---
.../current/modules/audit/configure.md | 8 +++---
.../current/modules/audit/overview.md | 6 ++---
.../current/modules/audit/quick_start.md | 30 +++++++++++++++++-----
6 files changed, 52 insertions(+), 20 deletions(-)
diff --git a/docs/modules/audit/configure.md b/docs/modules/audit/configure.md
index 2240d113a0..b53ff068f7 100644
--- a/docs/modules/audit/configure.md
+++ b/docs/modules/audit/configure.md
@@ -5,10 +5,10 @@ sidebar_position: 3
## Overview
-Audit-proxy source-channel-sink pipeline configuration (dataproxy-{tube|pulsar}.conf).Audit-store storage service
+Audit-proxy source-channel-sink pipeline configuration (audit-proxy-{tube|pulsar|kafka}.conf).Audit-store storage service
configuration (application.properties)
-## Audit-proxy source-channel-sink pipeline configuration (`dataproxy-{tube|pulsar}.conf`)
+## Audit-proxy source-channel-sink pipeline configuration (`audit-proxy-{tube|pulsar|kafka}.conf`)
### Basic Configuration
diff --git a/docs/modules/audit/overview.md b/docs/modules/audit/overview.md
index 7c3601688c..46178d62c2 100644
--- a/docs/modules/audit/overview.md
+++ b/docs/modules/audit/overview.md
@@ -12,7 +12,7 @@ The transmission status of each module, and whether the data stream is lost or r
## Architecture
1. The audit SDK is nested in the service that needs to be audited, audits the service, and sends the audit result to the audit access layer
-2. The audit proxy writes audit data to MQ (Pulsar or TubeMQ)
+2. The audit proxy writes audit data to MQ (Pulsar, Kafka or TubeMQ)
3. The distribution service consumes the audit data of MQ, and writes the audit data to MySQL, Elasticsearch and ClickHouse.
4. The interface layer encapsulates the data of MySQL, Elasticsearch and ClickHouse.
5. Application scenarios mainly include report display, audit reconciliation, etc.
diff --git a/docs/modules/audit/quick_start.md b/docs/modules/audit/quick_start.md
index fbd4103d49..f59224f8a4 100644
--- a/docs/modules/audit/quick_start.md
+++ b/docs/modules/audit/quick_start.md
@@ -15,7 +15,7 @@ If you use ClickHouse to store audit data, you need to first create the database
## Audit Proxy
### Configure MessageQueue
-You can choose Apache Pulsar or InLong TubeMQ as your MessageQueue service:
+You can choose Apache Pulsar, Apache Kafka or InLong TubeMQ as your MessageQueue service:
- If using Pulsar, the configuration file is `conf/audit-proxy-pulsar.conf`. Change the Pulsar service url for next configuration.
@@ -37,10 +37,20 @@ agent1.sinks.tube-sink-msg2.master-host-port-list = localhost:8715
agent1.sinks.tube-sink-msg2.topic = inlong-audit
```
+- If using Kafka, the configuration file is `conf/audit-proxy-kafka.conf`. Change the Kafka service address for next configuration.
+
+```Shell
+agent1.sources.tcp-source.port = 10081
+agent1.sinks.kafka-sink-msg1.bootstrap_servers = localhost:9092
+agent1.sinks.kafka-sink-msg1.topic = inlong-audit
+agent1.sinks.kafka-sink-msg2.bootstrap_servers = localhost:9092
+agent1.sinks.kafka-sink-msg2.topic = inlong-audit
+```
+
### Start
```Shell
# By default, pulsar is used as the MessageQueue, and the audit-proxy-pulsar.conf configuration file is loaded.
-bash +x ./bin/proxy-start.sh [pulsar｜tube]
+bash +x ./bin/proxy-start.sh [pulsar｜tube｜kafka]
```
## Audit Store
@@ -48,7 +58,7 @@ bash +x ./bin/proxy-start.sh [pulsar｜tube]
The configuration file is `conf/application.properties`.
```Shell
-# proxy.type: pulsar / tube
+# proxy.type: pulsar / tube / kafka
audit.config.proxy.type=pulsar
# store.server: mysql / elasticsearch / clickhouse
@@ -64,6 +74,12 @@ audit.tube.masterlist=TUBE_LIST
audit.tube.topic=inlong-audit
audit.tube.consumer.group.name=inlong-audit-consumer
+# kafka config (optional), replace KAFKA_LIST with your Kafka service url
+audit.kafka.server.url=KAFKA_LIST
+audit.kafka.topic=inlong-audit
+audit.kafka.consumer.name=inlong-audit-consumer
+audit.kafka.group.id=audit-consumer-group
+
# mysql config
spring.datasource.druid.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
spring.datasource.druid.username=root
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/configure.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/configure.md
index 2e7c602782..560580f4a2 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/configure.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/configure.md
@@ -5,9 +5,9 @@ sidebar_position: 3
## 概览
-审计代理服务audit-proxy `audit-proxy-pulsar.conf` 或者`audit-proxy-tube.conf`中设置。 审计存储服务audit-store `application.properties`中设置。
+审计代理服务 audit-proxy 在 `audit-proxy-{tube|pulsar|kafka}.conf`中设置。 审计存储服务 audit-store 在 `application.properties`中设置。
-## 审计代理层audit-proxy source-channel-sink 管道配置（dataproxy-{tube|pulsar}.conf）
+## 审计代理层 audit-proxy source-channel-sink 管道配置（audit-proxy-{tube|pulsar|kafka}.conf）
### 通用设置
@@ -44,7 +44,7 @@ sidebar_position: 3
| agent1.sinks.pulsar-sink-msg1.enable_token_auth | 是否需要安全认证 | false | |
| agent1.sinks.pulsar-sink-msg1.auth_token | pulsar认证token | 空 | |
-## 审计存储服务audit-store 配置 `application.properties`
+## 审计存储服务 audit-store 配置 `application.properties`
### MQ配置
@@ -66,7 +66,7 @@ sidebar_position: 3
| spring.datasource.druid.username | 账号名 | root | |
| spring.datasource.druid.password | 密码 | inlong | |
-### ElasticSearch配置
+### ElasticSearch 配置
| 参数 | 描述 | 默认值 | 备注 |
|--------------------------|----------------------------|------|-----|
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md
index 3863f4dbb6..312d2dbb11 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/overview.md
@@ -12,9 +12,9 @@ InLong审计是独立于InLong的一个子系统，对InLong系统的Agent、Dat
## 架构
1. 审计SDK嵌套在需要审计的服务，对服务进行审计，将审计结果发送到审计接入层。
-2. 审计接入层将审计数据写到MQ(Pulsar或者TubeMQ)。
-3. 分发服务消费MQ的审计数据，将审计数据写到MySQL、Elasticsearch、ClickHouse。
-4. 接口层将MySQL、Elasticsearch、ClickHouse的数据进行封装。
+2. 审计接入层将审计数据写到 MQ(Pulsar、Kafka 或者 TubeMQ)。
+3. 分发服务消费 MQ 的审计数据，将审计数据写到 MySQL、Elasticsearch、ClickHouse。
+4. 接口层将 MySQL、Elasticsearch、ClickHouse 的数据进行封装。
5. 应用场景主要包括报表展示、审计对账等等。
## 审计维度
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md
index 6ef1ee9e76..54ed36012d 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/audit/quick_start.md
@@ -19,9 +19,9 @@ title: 安装部署
## Audit Proxy
### 配置消息队列
-消息队列服务目前可以使用Apache Pulsar或者InLong TubeMQ：
+消息队列服务目前可以使用 Apache Pulsar、Apache Kafka 或者 InLong TubeMQ：
-- 若使用Pulsar，配置文件 `conf/audit-proxy-pulsar.conf`，修改下列配置中的 Pulsar service 地址。
+- 若使用 Pulsar，配置文件 `conf/audit-proxy-pulsar.conf`，修改下列配置中的 Pulsar service 地址。
```Shell
agent1.sources.tcp-source.port = 10081
@@ -31,7 +31,7 @@ agent1.sinks.pulsar-sink-msg2.pulsar_server_url = pulsar://localhost:6650
agent1.sinks.pulsar-sink-msg2.topic = persistent://public/default/inlong-audit
```
-- 若使用TubeMQ，配置文件 `conf/audit-proxy-tube.conf`，修改下列配置中的 TubeMQ master 地址。
+- 若使用 TubeMQ，配置文件 `conf/audit-proxy-tube.conf`，修改下列配置中的 TubeMQ master 地址。
```Shell
agent1.sources.tcp-source.port = 10081
agent1.sinks.tube-sink-msg1.master-host-port-list = localhost:8715
@@ -40,10 +40,20 @@ agent1.sinks.tube-sink-msg2.master-host-port-list = localhost:8715
agent1.sinks.tube-sink-msg2.topic = inlong-audit
```
+- 若使用 Kafka，配置文件 `conf/audit-proxy-kafka.conf`，修改下列配置中的 Kafka service 地址。
+
+```Shell
+agent1.sources.tcp-source.port = 10081
+agent1.sinks.kafka-sink-msg1.bootstrap_servers = localhost:9092
+agent1.sinks.kafka-sink-msg1.topic = inlong-audit
+agent1.sinks.kafka-sink-msg2.bootstrap_servers = localhost:9092
+agent1.sinks.kafka-sink-msg2.topic = inlong-audit
+```
+
### 启动
```Shell
# 默认使用 pulsar 作为消息队列，加载 audit-proxy-pulsar.conf 配置文件
-bash +x ./bin/proxy-start.sh [pulsar｜tube]
+bash +x ./bin/proxy-start.sh [pulsar｜tube｜kafka]
```
## Audit Store
@@ -51,22 +61,28 @@ bash +x ./bin/proxy-start.sh [pulsar｜tube]
配置文件 `conf/application.properties`
```Shell
-# proxy.type: pulsar / tube
+# proxy.type: pulsar / tube / kafka
audit.config.proxy.type=pulsar
# store.server: mysql / elasticsearch
audit.config.store.mode=mysql
-# audit pulsar config (optional)，将PULSAR_BROKER_LIST替换为Pulsar集群的服务地址
+# audit pulsar config (optional)，将 PULSAR_BROKER_LIST 替换为 Pulsar 集群的服务地址
audit.pulsar.server.url=pulsar://PULSAR_BROKER_LIST
audit.pulsar.topic=persistent://public/default/inlong-audit
audit.pulsar.consumer.sub.name=sub-audit
-# audit tube config (optional)，将TUBE_LIST替换为TubeMQ集群的master地址
+# audit tube config (optional)，将 TUBE_LIST 替换为 TubeMQ 集群的 master 地址
audit.tube.masterlist=TUBE_LIST
audit.tube.topic=inlong-audit
audit.tube.consumer.group.name=inlong-audit-consumer
+# kafka config (optional), 将 KAFKA_LIST 替换为 Kafka 集群的服务地址
+audit.kafka.server.url=KAFKA_LIST
+audit.kafka.topic=inlong-audit
+audit.kafka.consumer.name=inlong-audit-consumer
+audit.kafka.group.id=audit-consumer-group
+
# mysql config
spring.datasource.druid.url=jdbc:mysql://127.0.0.1:3306/apache_inlong_audit?characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2b8&rewriteBatchedStatements=true&allowMultiQueries=true&zeroDateTimeBehavior=CONVERT_TO_NULL
spring.datasource.druid.username=root