You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by do...@apache.org on 2022/03/25 09:59:21 UTC
[incubator-inlong-website] branch master updated: [INLONG-3363][TubeMQ]Added how to use optional BDB components and documentation (#318)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new d29d9e5 [INLONG-3363][TubeMQ]Added how to use optional BDB components and documentation (#318)
d29d9e5 is described below
commit d29d9e59e85678b2eaad40acb2a992d3ddbdc28b
Author: gosonzhang <46...@qq.com>
AuthorDate: Fri Mar 25 17:59:15 2022 +0800
[INLONG-3363][TubeMQ]Added how to use optional BDB components and documentation (#318)
---
docs/modules/tubemq/configure_introduction.md | 5 +++++
docs/modules/tubemq/quick_start.md | 23 +++++++++++-----------
.../modules/tubemq/configure_introduction.md | 5 +++++
.../current/modules/tubemq/quick_start.md | 23 +++++++++++-----------
4 files changed, 34 insertions(+), 22 deletions(-)
diff --git a/docs/modules/tubemq/configure_introduction.md b/docs/modules/tubemq/configure_introduction.md
index 3e4e313..5976c1c 100644
--- a/docs/modules/tubemq/configure_introduction.md
+++ b/docs/modules/tubemq/configure_introduction.md
@@ -11,6 +11,11 @@ Master and Broker use the ini configuration file format, and the relevant config
Their configuration is defined by a set of configuration units. The Master configuration consists of four mandatory units: [master], required but optional in [meta_zookeeper], [meta_bdb], and optional [tlsSetting]. The Broker configuration is mandatory. Broker], [zookeeper] and optional [tlsSetting] consist of a total of 3 configuration units; in actual use, you can also combine the contents of the two configuration files into one ini file.
+**Note**:
+- Due to the LICENSE problem of the Apache dependency package, the package released by TubeMQ from version 1.1.0 no longer contains the BDB package;
+- Starting from version 1.1.0, the metadata is stored in ZooKeeper by default, and BDB is optionally supported. In version 1.1.0, the master.ini configuration file needs to be manually set, and the [meta_bdb] configuration unit can be added to support BDB storage.
+- If the business uses the BDB component, you need to download the com.sleepycat.je-7.3.7.jar package by yourself, otherwise the system will report the error "java.lang.ClassNotFoundException: com.sleepycat.je.ReplicaConsistencyPolicy";
+
In addition to the back-end system configuration file, the Master also stores the Web front-end page module in the resources. The root directory velocity.properties file of the resources is the Web front-end page configuration file of the Master.
diff --git a/docs/modules/tubemq/quick_start.md b/docs/modules/tubemq/quick_start.md
index 61e4030..b89f0f7 100644
--- a/docs/modules/tubemq/quick_start.md
+++ b/docs/modules/tubemq/quick_start.md
@@ -9,11 +9,11 @@ There're two components in the cluster: **Master** and **Broker**. Master and Br
can be deployed on the same server or different servers. In this example, we setup our cluster
like this, and all services run on the same node. Zookeeper should be setup in your environment also.
-| Role | TCP Port | TLS Port | Web Port | Comment |
-| ---- | -------- | -------- | -------- | ------- |
-| Master | 8099 | 8199 | 8080 | Meta data is stored at /stage/meta_data |
-| Broker | 8123 | 8124 | 8081 | Message is stored at /stage/msg_data |
-| Zookeeper | 2181 | | | Offset is stored at /tubemq |
+| Role | TCP Port | TLS Port | Web Port | Comment |
+| ---- | -------- | -------- | -------- |----------------------------------------------------------------------|
+| Master | 8099 | 8199 | 8080 | Meta data is stored in ZooKeeper /tubemq |
+| Broker | 8123 | 8124 | 8081 | Message is stored at /stage/msg_data |
+| Zookeeper | 2181 | | | Master metadata or Broker offset information are stored at /tubemq |
### 1.2 Prerequisites
- ZooKeeper Cluster
@@ -36,7 +36,6 @@ You can change configurations in `conf/master.ini` according to cluster informat
hostName=YOUR_SERVER_IP // replaced with your server IP
port=8099
webPort=8080
-metaDataPath=/stage/meta_data
```
- Access Authorization Token
@@ -44,18 +43,20 @@ metaDataPath=/stage/meta_data
confModAuthToken=abc // for configuring Web Resources\API etc
```
-- ZooKeeper Cluster
+- meta_zookeeper Cluster
```ini
-[zookeeper] // Master and Broker in the same cluster must use the same zookeeper environment and have the same configuration
+[meta_zookeeper] // Masters in the same cluster must use the same zookeeper environment and have the same configuration
zkNodeRoot=/tubemq
zkServerAddr=localhost:2181 // multi zookeeper addresses can separate with ","
```
-- Replication Strategy
+- meta_bdb Strategy(Optional)
+ **Note**: Due to the LICENSE problem of Apache dependency packages, the packages released by TubeMQ from version 1.1.0 no longer contain BDB packages. If you need BDB to store metadata, you need to download com.sleepycat.je-7.3.7 by yourself. .jar package, otherwise a "java.lang.ClassNotFoundException: com.sleepycat.je.ReplicaConsistencyPolicy" error will be reported when the system is running.
```ini
-[replication]
+[meta_bdb]
repGroupName=tubemqGroup1 // The Master of the same cluster must use the same group name, and the group names of different clusters must be different
repNodeName=tubemqGroupNode1 // The master node names of the same cluster must be different names
+metaDataPath=/stage/meta_data
repHelperHost=FIRST_MASTER_NODE_IP:9001 // helperHost is used for building HA master.
```
@@ -71,7 +72,7 @@ the introduction of availability level.
| Minimum | 1 master | After the master crashed, there's no affect on existing producer and consumer. |
**Notice**:
-- Based on the need of Docker containerization, the [replication] above 3 parameters in the master.ini file are all the default settings used, and the actual information of the Master node needs to be configured when used in actual networking.
+- Based on the need of Docker containerization, the [meta_zookeeper] or [meta_bdb] above 3 parameters in the master.ini file are all the default settings used, and the actual information of the Master node needs to be configured when used in actual networking.
- The IP information of all master nodes should be mapped to the hostName in the hosts configuration file, such as "192.168.0.1 192-168-0-1"
- It is necessary to ensure the clock synchronization between all master nodes
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/tubemq/configure_introduction.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/tubemq/configure_introduction.md
index eb7b2a3..3558ff9 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/tubemq/configure_introduction.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/tubemq/configure_introduction.md
@@ -12,6 +12,11 @@ Master与Broker采用ini配置文件格式，相关配置文件分别放置在tu
他们的配置是按照配置单元集合来定义的，Master配置由必选的[master]、可选择的必选[meta_zookeeper]、[meta_bdb]和可选的[tlsSetting]一共4个配置单元组成，Broker配置由必选的[broker]、[zookeeper]和可选的[tlsSetting]一共3个配置单元组成；实际使用时，大家也可将两个配置文件内容合并放置为一个ini文件。
+**注意**：
+- 由于Apache依赖包的LICENSE问题，从1.1.0版本开始TubeMQ发布的包不再包含BDB包；
+- 1.1.0版本开始，元数据缺省采用ZooKeeper存储，可选支持BDB，1.1.0版本需要手工设置master.ini配置文件，增加[meta_bdb]配置单元才能支持BDB存储。
+- 若业务使用BDB组件，需要自行下载com.sleepycat.je-7.3.7.jar包，要不会系统运行时会报“ java.lang.ClassNotFoundException: com.sleepycat.je.ReplicaConsistencyPolicy”错误；
+
Master除了后端系统配置文件外，还在resources里存放了Web前端页面模块，resources的根目录velocity.properties文件为Master的Web前端页面配置文件。
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/tubemq/quick_start.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/tubemq/quick_start.md
index ab6914c..4258b1e 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/tubemq/quick_start.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/tubemq/quick_start.md
@@ -7,11 +7,11 @@ sidebar_position: 1
### 1.1 配置示例
TubeMQ 集群包含有两个组件: **Master** 和 **Broker**. Master 和 Broker 可以部署在相同或者不同的节点上，依照业务对机器的规划进行处理。我们通过如下3台机器搭建有2台Master的生产、消费的集群进行配置示例：
-| 所属角色 | TCP端口 | TLS端口 | WEB端口 | 备注 |
-| --- | --- | --- | --- | --- |
-| Master | 8099 | 8199 | 8080 | 元数据存储在`/stage/meta_data` |
-| Broker | 8123 | 8124 | 8081 | 消息储存在`/stage/msg_data` |
-| ZooKeeper | 2181 | | | Offset储存在根目录`/tubemq` |
+| 所属角色 | TCP端口 | TLS端口 | WEB端口 | 备注 |
+| --- | --- | --- | --- |------------------------------------------|
+| Master | 8099 | 8199 | 8080 | 元数据存储在ZooKeeper的`/tubemq`目录 |
+| Broker | 8123 | 8124 | 8081 | 消息储存在`/stage/msg_data` |
+| ZooKeeper | 2181 | | | 存储Master元数据及Broker的Offset内容，根目录`/tubemq` |
### 1.2 准备工作
- ZooKeeper集群
@@ -35,7 +35,6 @@ TubeMQ 集群包含有两个组件: **Master** 和 **Broker**. Master 和 Broker
hostName=YOUR_SERVER_IP // 替换为当前主机IP
port=8099
webPort=8080
-metaDataPath=/stage/meta_data
```
- 访问授权Token
@@ -43,18 +42,20 @@ metaDataPath=/stage/meta_data
confModAuthToken=abc // 该token用于页面配置、API调用等
```
-- ZooKeeper集群地址
+- 配置meta_zookeeper策略
```ini
-[zookeeper] // 同一个集群里Master和Broker必须使用同一套zookeeper环境，且配置一致
+[meta_zookeeper] // 同一个集群里Master必须使用同一套zookeeper环境，且配置一致
zkNodeRoot=/tubemq
zkServerAddr=localhost:2181 // 指向zookeeper集群，多个地址逗号分开
```
-- 配置Replication策略
+- 配置meta_bdb策略（可选）
+ **注意**：由于Apache依赖包的LICENSE问题，从1.1.0版本开始TubeMQ发布的包不再包含BDB包，如果需要BDB存储元数据，业务需要自行下载com.sleepycat.je-7.3.7.jar包，要不系统运行时会报“ java.lang.ClassNotFoundException: com.sleepycat.je.ReplicaConsistencyPolicy”错误。
```ini
-[replication]
+[meta_bdb]
repGroupName=tubemqGroup1 // 同一个集群的Master必须要用同一个组名，且不同集群的组名必须不同
repNodeName=tubemqGroupNode1 // 同一个集群的master节点名必须是不同的名称
+metaDataPath=/stage/meta_data
repHelperHost=FIRST_MASTER_NODE_IP:9001 // helperHost用于创建master集群，一般配置第一个master节点ip
```
@@ -67,7 +68,7 @@ repHelperHost=FIRST_MASTER_NODE_IP:9001 // helperHost用于创建master集群
| 低 | 1 master | 主节点崩溃后，对现有的生产者和消费者没有影响。 |
**注意**：
-- 基于Docker容器化的需要，master.ini文件里对[replication]如上3个参数部分都是使用的缺省设置，在实际组网使用时需要以Master节点真实信息配置
+- 基于Docker容器化的需要，master.ini文件里对[meta_zookeeper] 或 [meta_bdb] 如上3个参数部分都是使用的缺省设置，在实际组网使用时需要以Master节点真实信息配置
- Master所有节点的IP信息要在hosts配置文件里构造IP与hostName映射关系，如类似“192.168.0.1 192-168-0-1”
- 需保证Master所有节点之间的时钟同步