You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by do...@apache.org on 2022/04/11 07:04:13 UTC
[incubator-inlong-website] branch master updated: [INLONG-3606] Add more guide for configure sort/manager/k8s (#341)
This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new 965273fac [INLONG-3606] Add more guide for configure sort/manager/k8s (#341)
965273fac is described below
commit 965273fac288b7ad5ea79bc8c108990e3f40ded3
Author: dockerzhang <do...@apache.org>
AuthorDate: Mon Apr 11 15:04:09 2022 +0800
[INLONG-3606] Add more guide for configure sort/manager/k8s (#341)
---
docs/deployment/k8s.md | 40 +++++++++++++++++++++
docs/modules/manager/quick_start.md | 13 ++++++-
docs/modules/sort/quick_start.md | 3 +-
.../current/deployment/k8s.md | 41 ++++++++++++++++++++++
.../current/modules/manager/quick_start.md | 12 ++++++-
.../current/modules/sort/quick_start.md | 3 +-
package.json | 8 ++---
7 files changed, 112 insertions(+), 8 deletions(-)
diff --git a/docs/deployment/k8s.md b/docs/deployment/k8s.md
index 4d0cda545..0e0d92d49 100644
--- a/docs/deployment/k8s.md
+++ b/docs/deployment/k8s.md
@@ -18,8 +18,48 @@ helm upgrade inlong --install -n inlong ./
```
## Configuration
+The configuration file is **values.yaml**, and the following tables lists the configurable parameters of InLong and their default values.
+| Parameter | Default | Description |
+|:--------------------------------------------------------------------------------:|:----------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| `timezone` | `Asia/Shanghai` | World time and date for cities in all time zones |
+| `images.pullPolicy` | `IfNotPresent` | Image pull policy. One of `Always`, `Never`, `IfNotPresent` |
+| `images.<component>.repository` | | Docker image repository for the component |
+| `images.<component>.tag` | `latest` | Docker image tag for the component |
+| `<component>.component` | | Component name |
+| `<component>.replicaCount` | `1` | Replicas is the desired number of replicas of a given Template |
+| `<component>.podManagementPolicy` | `OrderedReady` | PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down |
+| `<component>.annotations` | `{}` | The `annotations` field can be used to attach arbitrary non-identifying metadata to objects |
+| `<component>.tolerations` | `[]` | Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints |
+| `<component>.nodeSelector` | `{}` | You can add the `nodeSelector` field to your Pod specification and specify the node labels you want the target node to have |
+| `<component>.affinity` | `{}` | Node affinity is conceptually similar to nodeSelector, allowing you to constrain which nodes your Pod can be scheduled on based on node labels |
+| `<component>.terminationGracePeriodSeconds` | `30` | Optional duration in seconds the pod needs to terminate gracefully |
+| `<component>.resources` | `{}` | Optionally specify how much of each resource a container needs |
+| `<component>.port(s)` | | The port(s) for each component service |
+| `<component>.env` | `{}` | Environment variables for each component container |
+| <code>\<component\>.probe.\<liveness|readiness\>.enabled</code> | `true` | Turn on and off liveness or readiness probe |
+| <code>\<component\>.probe.\<liveness|readiness\>.failureThreshold</code> | `10` | Minimum consecutive successes for the probe |
+| <code>\<component\>.probe.\<liveness|readiness\>.initialDelaySeconds</code> | `10` | Delay before the probe is initiated |
+| <code>\<component\>.probe.\<liveness|readiness\>.periodSeconds</code> | `30` | How often to perform the probe |
+| `<component>.volumes.name` | | Volume name |
+| `<component>.volumes.size` | `10Gi` | Volume size |
+| `<component>.service.annotations` | `{}` | The `annotations` field may need to be set when service.type is `LoadBalancer` |
+| `<component>.service.type` | `ClusterIP` | The `type` field determines how the service is exposed. Valid options are `ClusterIP`, `NodePort`, `LoadBalancer` and `ExternalName` |
+| `<component>.service.clusterIP` | `nil` | ClusterIP is the IP address of the service and is usually assigned randomly by the master |
+| `<component>.service.nodePort` | `nil` | NodePort is the port on each node on which this service is exposed when service type is `NodePort` |
+| `<component>.service.loadBalancerIP` | `nil` | LoadBalancer will get created with the IP specified in this field when service type is `LoadBalancer` |
+| `<component>.service.externalName` | `nil` | ExternalName is the external reference that kubedns or equivalent will return as a CNAME record for this service, requires service type to be `ExternalName` |
+| `<component>.service.externalIPs` | `[]` | ExternalIPs is a list of IP addresses for which nodes in the cluster will also accept traffic for this service |
+| `external.mysql.enabled` | `false` | If not exists external MySQL, InLong will use the internal MySQL by default |
+| `external.mysql.hostname` | `localhost` | External MySQL hostname |
+| `external.mysql.port` | `3306` | External MySQL port |
+| `external.mysql.username` | `root` | External MySQL username |
+| `external.mysql.password` | `password` | External MySQL password |
+| `external.pulsar.enabled` | `false` | If not exists external Pulsar, InLong will use the internal TubeMQ by default |
+| `external.pulsar.serviceUrl` | `localhost:6650` | External Pulsar service URL |
+| `external.pulsar.adminUrl` | `localhost:8080` | External Pulsar admin URL |
+> The components include `agent`, `audit`, `dashboard`, `dataproxy`, `manager`, `tubemq-manager`, `tubemq-master`, `tubemq-broker`, `zookeeper` and `mysql`.
## Uninstall
diff --git a/docs/modules/manager/quick_start.md b/docs/modules/manager/quick_start.md
index 1c29965e3..edc932809 100644
--- a/docs/modules/manager/quick_start.md
+++ b/docs/modules/manager/quick_start.md
@@ -36,7 +36,18 @@ spring.datasource.druid.username=root
spring.datasource.druid.password=inlong
```
-## 启动
+## Flink Plugin
+InLong support to start a Sort task by Manager, you need to configure a Flink environment in the `plugins/flink-sort-plugin.properties`.
+```properties
+# Flink host split by coma if more than one host, such as 'host1,host2'
+flink.rest.address=127.0.0.1
+# Flink port
+flink.rest.port=8081
+# Flink jobmanager port
+flink.jobmanager.port=6123
+```
+
+## Start
```shell
bash +x bin/startup.sh
```
diff --git a/docs/modules/sort/quick_start.md b/docs/modules/sort/quick_start.md
index 32d5838c3..3ebe017e0 100644
--- a/docs/modules/sort/quick_start.md
+++ b/docs/modules/sort/quick_start.md
@@ -22,7 +22,7 @@ Example：
./bin/flink run -c org.apache.inlong.sort.flink.Entrance inlong-sort/sort-[version].jar \
--cluster-id debezium2hive --dataflow.info.file /YOUR_DATAFLOW_INFO_DIR/debezium-to-hive.json \
--source.type pulsar --sink.type hive --sink.hive.rolling-policy.rollover-interval 60000 \
---sink.hive.rolling-policy.check-interval 30000
+--metrics.audit.proxy.hosts 127.0.0.1:10081 --sink.hive.rolling-policy.check-interval 30000
```
Notice：
@@ -36,6 +36,7 @@ Notice：
- `--dataflow.info.file` dataflow configuration file path
- `--source.type` source of the application, currently "pulsar" is supported
- `--sink.type` sink of the application, currently "clickhouse", "hive", "iceberg", "kafka" are supported
+- `--metrics.audit.proxy.hosts` audit proxy host address for reporting audit metrics
**Example**
```
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/k8s.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/k8s.md
index f7c22de6f..fa2df15e2 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/k8s.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/k8s.md
@@ -18,7 +18,48 @@ helm upgrade inlong --install -n inlong ./
```
## 配置
+配置内容都在 **values.yaml** 文件中，以下为所有可配置项及其默认值，包括：
+| Parameter | Default | Description |
+|:--------------------------------------------------------------------------------:|:----------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+| `timezone` | `Asia/Shanghai` | World time and date for cities in all time zones |
+| `images.pullPolicy` | `IfNotPresent` | Image pull policy. One of `Always`, `Never`, `IfNotPresent` |
+| `images.<component>.repository` | | Docker image repository for the component |
+| `images.<component>.tag` | `latest` | Docker image tag for the component |
+| `<component>.component` | | Component name |
+| `<component>.replicaCount` | `1` | Replicas is the desired number of replicas of a given Template |
+| `<component>.podManagementPolicy` | `OrderedReady` | PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down |
+| `<component>.annotations` | `{}` | The `annotations` field can be used to attach arbitrary non-identifying metadata to objects |
+| `<component>.tolerations` | `[]` | Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints |
+| `<component>.nodeSelector` | `{}` | You can add the `nodeSelector` field to your Pod specification and specify the node labels you want the target node to have |
+| `<component>.affinity` | `{}` | Node affinity is conceptually similar to nodeSelector, allowing you to constrain which nodes your Pod can be scheduled on based on node labels |
+| `<component>.terminationGracePeriodSeconds` | `30` | Optional duration in seconds the pod needs to terminate gracefully |
+| `<component>.resources` | `{}` | Optionally specify how much of each resource a container needs |
+| `<component>.port(s)` | | The port(s) for each component service |
+| `<component>.env` | `{}` | Environment variables for each component container |
+| <code>\<component\>.probe.\<liveness|readiness\>.enabled</code> | `true` | Turn on and off liveness or readiness probe |
+| <code>\<component\>.probe.\<liveness|readiness\>.failureThreshold</code> | `10` | Minimum consecutive successes for the probe |
+| <code>\<component\>.probe.\<liveness|readiness\>.initialDelaySeconds</code> | `10` | Delay before the probe is initiated |
+| <code>\<component\>.probe.\<liveness|readiness\>.periodSeconds</code> | `30` | How often to perform the probe |
+| `<component>.volumes.name` | | Volume name |
+| `<component>.volumes.size` | `10Gi` | Volume size |
+| `<component>.service.annotations` | `{}` | The `annotations` field may need to be set when service.type is `LoadBalancer` |
+| `<component>.service.type` | `ClusterIP` | The `type` field determines how the service is exposed. Valid options are `ClusterIP`, `NodePort`, `LoadBalancer` and `ExternalName` |
+| `<component>.service.clusterIP` | `nil` | ClusterIP is the IP address of the service and is usually assigned randomly by the master |
+| `<component>.service.nodePort` | `nil` | NodePort is the port on each node on which this service is exposed when service type is `NodePort` |
+| `<component>.service.loadBalancerIP` | `nil` | LoadBalancer will get created with the IP specified in this field when service type is `LoadBalancer` |
+| `<component>.service.externalName` | `nil` | ExternalName is the external reference that kubedns or equivalent will return as a CNAME record for this service, requires service type to be `ExternalName` |
+| `<component>.service.externalIPs` | `[]` | ExternalIPs is a list of IP addresses for which nodes in the cluster will also accept traffic for this service |
+| `external.mysql.enabled` | `false` | If not exists external MySQL, InLong will use the internal MySQL by default |
+| `external.mysql.hostname` | `localhost` | External MySQL hostname |
+| `external.mysql.port` | `3306` | External MySQL port |
+| `external.mysql.username` | `root` | External MySQL username |
+| `external.mysql.password` | `password` | External MySQL password |
+| `external.pulsar.enabled` | `false` | If not exists external Pulsar, InLong will use the internal TubeMQ by default |
+| `external.pulsar.serviceUrl` | `localhost:6650` | External Pulsar service URL |
+| `external.pulsar.adminUrl` | `localhost:8080` | External Pulsar admin URL |
+
+> The components include `agent`, `audit`, `dashboard`, `dataproxy`, `manager`, `tubemq-manager`, `tubemq-master`, `tubemq-broker`, `zookeeper` and `mysql`.
## 卸载
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md
index b14d50c0f..cf2362f42 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/manager/quick_start.md
@@ -17,7 +17,6 @@ title: 安装部署
- 如果后端连接 PostgreSQL 数据库，不需要引入额外依赖。
## 配置
-
前往 `inlong-manager` 目录，修改 `conf/application.properties` 文件：
```properties
@@ -36,6 +35,17 @@ spring.datasource.druid.username=root
spring.datasource.druid.password=inlong
```
+## Flink 插件
+InLong 支持 Manager 发起 Sort 任务进行数据分拣，需要先配置 Flink 环境信息。配置文件为`plugins/flink-sort-plugin.properties`.
+```properties
+# Flink host split by coma if more than one host, such as 'host1,host2'
+flink.rest.address=127.0.0.1
+# Flink port
+flink.rest.port=8081
+# Flink jobmanager port
+flink.jobmanager.port=6123
+```
+
## 启动
```shell
bash +x bin/startup.sh
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/quick_start.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/quick_start.md
index 74ccf066e..0e501049c 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/quick_start.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/sort/quick_start.md
@@ -21,7 +21,7 @@ flink环境配置完成后，可以通过浏览器访问flink的web ui，对应
./bin/flink run -c org.apache.inlong.sort.flink.Entrance inlong-sort/sort-[version].jar \
--cluster-id debezium2hive --dataflow.info.file /YOUR_DATAFLOW_INFO_DIR/debezium-to-hive.json \
--source.type pulsar --sink.type hive --sink.hive.rolling-policy.rollover-interval 60000 \
---sink.hive.rolling-policy.check-interval 30000
+--metrics.audit.proxy.hosts 127.0.0.1:10081 --sink.hive.rolling-policy.check-interval 30000
```
注意：
@@ -35,6 +35,7 @@ flink环境配置完成后，可以通过浏览器访问flink的web ui，对应
- `--dataflow.info.file` 流配置文件路径
- `--source.type` 数据源的种类, 当前支持："pulsar"
- `--sink.type` 存储系统的种类，当前支持："clickhouse"、"hive"、"iceberg"、"kafka"
+- `--metrics.audit.proxy.hosts` audit proxy 地址用于上报审计指标数据
**启动参数配置示例**
```
diff --git a/package.json b/package.json
index cdf71d654..4368f0c35 100644
--- a/package.json
+++ b/package.json
@@ -15,10 +15,10 @@
"write-heading-ids": "docusaurus write-heading-ids"
},
"dependencies": {
- "@docusaurus/core": "^2.0.0-beta.17",
- "@docusaurus/plugin-content-blog": "^2.0.0-beta.17",
- "@docusaurus/plugin-content-docs": "^2.0.0-beta.17",
- "@docusaurus/preset-classic": "2.0.0-beta.17",
+ "@docusaurus/core": "^2.0.0-beta.18",
+ "@docusaurus/plugin-content-blog": "^2.0.0-beta.18",
+ "@docusaurus/plugin-content-docs": "^2.0.0-beta.18",
+ "@docusaurus/preset-classic": "2.0.0-beta.18",
"@mdx-js/react": "^1.6.21",
"@svgr/webpack": "^5.5.0",
"acorn": "^8.6.0",