You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by he...@apache.org on 2022/02/26 10:22:58 UTC
[incubator-inlong-website] branch master updated: [INLONG-2677] Manager get MQ cluster Info from database, remove local file configuration dependencies (#287)
This is an automated email from the ASF dual-hosted git repository.
healchow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 96b3a16 [INLONG-2677] Manager get MQ cluster Info from database, remove local file configuration dependencies (#287)
96b3a16 is described below
commit 96b3a169474fde7a035e81ae2aa003b2e98e27f8
Author: xueyingzhang <86...@users.noreply.github.com>
AuthorDate: Sat Feb 26 18:22:54 2022 +0800
[INLONG-2677] Manager get MQ cluster Info from database, remove local file configuration dependencies (#287)
---
docs/modules/dataproxy/quick_start.md | 15 +----
docs/modules/manager/quick_start.md | 66 ++++++++++++++--------
.../current/modules/dataproxy/quick_start.md | 12 +---
.../current/modules/manager/quick_start.md | 58 +++++++++++--------
4 files changed, 80 insertions(+), 71 deletions(-)
diff --git a/docs/modules/dataproxy/quick_start.md b/docs/modules/dataproxy/quick_start.md
index ef313e3..bac1f10 100644
--- a/docs/modules/dataproxy/quick_start.md
+++ b/docs/modules/dataproxy/quick_start.md
@@ -5,17 +5,6 @@ title: Deployment
All deploying files at `inlong-dataproxy` directory.
## Configuration
-### Configure Message Queue
-
-- If you use InLong TubeMQ, `tubemq_master_list` is the rpc address of TubeMQ Master.
-```
-$ sed -i 's/TUBE_LIST/tubemq_master_list/g' conf/flume.conf
-```
-
-- If you use Apache Pulsar, replace `pulsar_server_url_list` in `conf/flume-mulit-pulsar-tcp-example.conf` and overwrite `conf/flume.conf`.
-
-notice that conf/flume.conf FLUME_HOME is proxy the directory for proxy inner data
-
### Configure InLong-Manager URL
configuration file: `conf/common.properties`:
@@ -40,12 +29,12 @@ telnet 127.0.0.1 46801
## Add DataProxy configuration to InLong-Manager
-After installing the DataProxy, you need to insert the IP and port of the DataProxy service is located into the backend database of InLong-Manager.
+After installing the DataProxy, you need to insert the IP address of the DataProxy service is located into the backend database of InLong-Manager.
The SQL statement is:
```sql
-- address is the IP of the DataProxy service is located
-UPDATE apache_inlong_manager.data_proxy_cluster SET address="replace_by_dataproxy_ip" WHERE name="default_dataproxy";
+UPDATE apache_inlong_manager.data_proxy_cluster SET address="replace_by_dataproxy_ip", mq_set_name="default_set_name" WHERE name="default_dataproxy";
```
diff --git a/docs/modules/manager/quick_start.md b/docs/modules/manager/quick_start.md
index 2b0a99d..c408bf8 100644
--- a/docs/modules/manager/quick_start.md
+++ b/docs/modules/manager/quick_start.md
@@ -38,30 +38,8 @@ The dev configuration is specified above, then modify the `conf/application-dev.
spring.datasource.druid.username=root
spring.datasource.druid.password=inlong
```
-
-2) Configure the Message Queue Service, you could choose InLong TubeMQ or Apache Pulsar:
-
-- Configuration TubeMQ cluster information if using TubeMQ
- ```properties
- # Manager address of TubeMQ cluster, used to create Topic
- cluster.tube.manager=http://127.0.0.1:8081
- # Broker used to manage TubeMQ
- cluster.tube.master=127.0.0.1:8000,127.0.0.1:8010
- # TubeMQ cluster ID
- cluster.tube.clusterId=1
- ```
-
-- Configuration Pulsar cluster information if using Pulsar
- ```properties
- # Pulsar admin URL
- pulsar.adminUrl=http://127.0.0.1:8080,127.0.0.2:8080,127.0.0.3:8080
- # Pulsar broker address
- pulsar.serviceUrl=pulsar://127.0.0.1:6650,127.0.0.1:6650,127.0.0.1:6650
- # Default tenant of Pulsar
- pulsar.defaultTenant=public
- ```
-
-3) Configure ZooKeeper clusters information:
+
+2) Configure ZooKeeper clusters information:
```properties
# ZK cluster, used to push the configuration of Sort
@@ -70,6 +48,46 @@ The dev configuration is specified above, then modify the `conf/application-dev.
# application name, that is the cluster-id parameter of InLong Sort
sort.appName=inlong_app
```
+
+### Add Message Queue configuration to InLong-Manager
+You can choose InLong TubeMQ or Apache Pulsar as Message Queue Service:
+- If using TubeMQ, the SQL statement is:
+
+```sql
+INSERT INTO apache_inlong_manager.third_party_cluster
+(name, type, url, ext_params, mq_set_name, in_charges, creator)
+VALUES
+('tube_cluster', 'TUBE', 'tubemq_master_list', 'tube_config', 'default_set_name', 'admin', 'admin');
+```
+
+- `tubemq_master_list`: the master address of your TubeMQ cluster. If there are multiple master nodes, addresses is separated by comma, such as `127.0.0.1:8000,127.0.0.1:8010`.
+- `tube_config`: the other info of your cluster. It is described in JSON format, for example:
+
+```json
+{
+ "cluster_tube_manager": "http://127.0.0.1:8081",
+ "cluster_tube_clusterId": "1"
+}
+```
+
+- If using Pulsar, the SQL statement is:
+
+```sql
+INSERT INTO apache_inlong_manager.third_party_cluster
+(name, type, url, token, ext_params, mq_set_name, in_charges, creator)
+VALUES
+('pulsar_cluster', 'PULSAR', 'puslar_service_url', 'pulsar_token', 'pulsar_config', 'default_set_name', 'admin', 'admin');
+```
+
+- `puslar_service_url`: the address of your Pulsar cluster, such as `pulsar://127.0.0.1:6650,127.0.0.1:6650,127.0.0.1:6650`.
+- `pulsar_config`: the other info of your cluster. It is described in JSON format, for example:
+
+```json
+{
+ "pulsar_adminUrl": "http://127.0.0.1:8080,127.0.0.2:8080,127.0.0.3:8080"
+}
+```
+
### Start the service
Enter the decompressed directory, execute `sh bin/startup.sh` to start the service, and check the
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/quick_start.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/quick_start.md
index 83718ec..029a28c 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/quick_start.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/quick_start.md
@@ -6,16 +6,6 @@ title: 安装部署
## 配置
-### 配置消息队列
-
-- 如果使用InLong TubeMQ, 按以下配置即可。`tubemq_master_list`是TubeMQ master rpc地址，多个逗号分隔。
-```
-$ sed -i 's/TUBE_LIST/tubemq_master_list/g' conf/flume.conf
-```
-- 如果使用Apache Pulsar, 替换`conf/flume-mulit-pulsar-tcp-example.conf`中`pulsar_server_url_list`并覆盖`conf/flume.conf`。
-
-注意conf/flume.conf中FLUME_HOME为proxy的中间数据文件存放地址
-
### 配置InLong-Manager 地址
配置文件：`conf/common.properties`:
@@ -46,6 +36,6 @@ SQL 语句为：
```sql
-- address 为 DataProxy 服务所在主机的 IP
-UPDATE apache_inlong_manager.data_proxy_cluster SET address="replace_by_dataproxy_ip" WHERE name="default_dataproxy";
+UPDATE apache_inlong_manager.data_proxy_cluster SET address="replace_by_dataproxy_ip", mq_set_name="default_set_name" WHERE name="default_dataproxy";
```
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md
index 13870c0..e97e372 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md
@@ -37,30 +37,8 @@ spring.profiles.active=dev
spring.datasource.druid.username=root
spring.datasource.druid.password=inlong
```
-
-2) 配置消息队列服务，可以使用InLong TubeMQ 或者 Apache Pulsar：
-
-- 若使用TubeMQ，配置TubeMQ 集群信息
- ```properties
- # TubeMQ 集群的 Manager 地址，用来创建 Topic
- cluster.tube.manager=http://127.0.0.1:8081
- # 用来管理 TubeMQ 的 Broker
- cluster.tube.master=127.0.0.1:8000,127.0.0.1:8010
- # TubeMQ 集群的 ID
- cluster.tube.clusterId=1
- ```
-- 若使用Pulsar，配置Pulsar 集群信息
- ```properties
- # Pulsar admin URL
- pulsar.adminUrl=http://127.0.0.1:8080,127.0.0.2:8080,127.0.0.3:8080
- # Pulsar broker address
- pulsar.serviceUrl=pulsar://127.0.0.1:6650,127.0.0.1:6650,127.0.0.1:6650
- # Default tenant of Pulsar
- pulsar.defaultTenant=public
- ```
-
-3) 配置ZooKeeper 集群信息：
+2) 配置ZooKeeper 集群信息：
```properties
# ZK 集群，用来推送 Sort 的配置
@@ -69,6 +47,40 @@ spring.profiles.active=dev
# 应用名称，即InLong Sort 的 cluster-id 参数
sort.appName=inlong_app
```
+### 将消息队列配置添加到InLong-Manager
+消息队列服务可以使用InLong TubeMQ 或者 Apache Pulsar：
+- 若使用TubeMQ，SQL语句为：
+
+```sql
+INSERT INTO apache_inlong_manager.third_party_cluster
+(name, type, url, ext_params, mq_set_name, in_charges, creator)
+VALUES
+('tube_cluster', 'TUBE', 'tubemq_master_list', 'tube_config', 'default_set_name', 'admin', 'admin');
+```
+其中，`tubemq_master_list`为TubeMQ集群的master地址，多个则逗号分隔，形如`127.0.0.1:8000,127.0.0.1:8010`；`tube_config`为集群配置信息，按照JSON格式设置：
+
+```json
+{
+ "cluster_tube_manager": "http://127.0.0.1:8081",
+ "cluster_tube_clusterId": "1"
+}
+```
+
+- 若使用Pulsar，SQL语句为：
+
+```sql
+INSERT INTO apache_inlong_manager.third_party_cluster
+(name, type, url, token, ext_params, mq_set_name, in_charges, creator)
+VALUES
+('pulsar_cluster', 'PULSAR', 'puslar_service_url', 'pulsar_token', 'pulsar_config', 'default_set_name', 'admin', 'admin');
+```
+其中，`puslar_service_url`为Pulsar集群的地址，形如`pulsar://127.0.0.1:6650,127.0.0.1:6650,127.0.0.1:6650`；`pulsar_config`为集群配置信息，按照JSON格式设置：
+
+```json
+{
+ "pulsar_adminUrl": "http://127.0.0.1:8080,127.0.0.2:8080,127.0.0.3:8080"
+}
+```
### 启动服务