You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/10/20 07:39:40 UTC
[GitHub] [inlong-website] gosonzhang opened a new pull request, #571: [INLONG-6238][Doc] Add configuration doc for DataProxy
gosonzhang opened a new pull request, #571:
URL: https://github.com/apache/inlong-website/pull/571
- Fixes : https://github.com/apache/inlong/issues/6238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong-website] healchow commented on a diff in pull request #571: [INLONG-6238][Doc] Add configuration doc for DataProxy
Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #571:
URL: https://github.com/apache/inlong-website/pull/571#discussion_r1000292138
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/configuration.md:
##########
@@ -0,0 +1,52 @@
+---
+title: 配置
+sidebar_position: 3
+---
+
+## 基本配置（common.properties）
+
+| 参数 | 描述 | 默认值 | 备注 |
+|---------------------------------------------------------|--------------------------|--------------------------------------|-----------------------------------------------------------------------------|
+| manager.hosts | Manager服务器IP和端口列表 | 127.0.0.1:8083 | 不允许为空，格式按照 {ip1:port1}[,{ip2:port2}][,{ip3:port3}] 格式进行配置 |
+| manager.auth.secretId | 登录所需帐号 | 无 | 允许为空不填写 |
+| manager.auth.secretKey | 登录所需密码 | 无 | 允许为空不填写 |
+| proxy.cluster.tag | dataproxy所处的集群Set名 | default_cluster | 一个集群set里可以包含多个dataproxy、MQ的集群 |
Review Comment:
`dataproxy所处的集群Set名` -> `DataProxy 所处的集群 Tag 名`.
`Set` is a new name, suggest unifying it to `Tag`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong-website] healchow commented on a diff in pull request #571: [INLONG-6238][Doc] Add configuration doc for DataProxy
Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #571:
URL: https://github.com/apache/inlong-website/pull/571#discussion_r1000263529
##########
docs/modules/dataproxy/configuration.md:
##########
@@ -0,0 +1,50 @@
+---
+title: Configuration
+sidebar_position: 3
+---
+## Basic Configuration (common.properties)
+
+| Parameter | Description | Default | Notes |
+|--------------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------------------|-------------------------------------------------------------------------------------------------------------------|
+| manager.hosts | InLong-Manager http host and port | 127.0.0.1:8083 | Empty is not allowed, the format is configured according to the format of {ip1:port1}[,{ip2:port2}][,{ip3:port3}] |
+| manager.auth.secretId | InLong-Manager authentic secretId | `blank` | If InLong-Manager doesn't open authentic service, this parameter can be empty. |
+| manager.auth.secretKey | InLong-Manager authentic secretKey | `blank` | |
+| proxy.cluster.tag | The cluster set name where the dataproxy is located | default_cluster | A cluster set can contain multiple dataproxy and MQ clusters |
+| proxy.cluster.name | The cluster name where dataproxy is located | default_dataproxy | used to distinguish different environments |
+| proxy.cluster.inCharges | The incharge of the cluster where dataproxy is located | admin | |
+| configCheckInterval | Configuration file checking and loading interval of the conf directory (unit: ms) | 10000 | |
+| metricDomains | JMX domain name | DataProxy | Obtain the following configuration items of "metricDomains.${metricDomains}.xxx" through this configuration value |
+| metricDomains.DataProxy.domainListeners | The class for indicator monitoring, and the service is started through reflection of the class name | org.apache.inlong.dataproxy.metrics.prometheus.PrometheusMetricListener | If there are multiple indicator monitoring class configurations, separate them by spaces, carriage returns, or line feeds |
+| metricDomains.DataProxy.snapshotInterval | Time interval for periodic reporting of indicators (unit: ms) | 60000 | |
+| prometheusHttpPort | The port when reporting using Prometheus | 9081 | |
+| audit.enable | Whether to enable data reporting InLong-Audit service | true | |
+| audit.proxys | The address of the InLong-Audit service | 127.0.0.1:10081 | 格式按照 {ip1:port1}[ {ip2:port2}][ {ip3:port3}]进行配置，不同ip:port间通过空格、回车，或换行符进行分隔 |
Review Comment:
Should change to English.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong-website] healchow commented on a diff in pull request #571: [INLONG-6238][Doc] Add configuration doc for DataProxy
Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #571:
URL: https://github.com/apache/inlong-website/pull/571#discussion_r1000294768
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/configuration.md:
##########
@@ -0,0 +1,52 @@
+---
+title: 配置
+sidebar_position: 3
+---
+
+## 基本配置（common.properties）
+
+| 参数 | 描述 | 默认值 | 备注 |
+|---------------------------------------------------------|--------------------------|--------------------------------------|-----------------------------------------------------------------------------|
+| manager.hosts | Manager服务器IP和端口列表 | 127.0.0.1:8083 | 不允许为空，格式按照 {ip1:port1}[,{ip2:port2}][,{ip3:port3}] 格式进行配置 |
+| manager.auth.secretId | 登录所需帐号 | 无 | 允许为空不填写 |
+| manager.auth.secretKey | 登录所需密码 | 无 | 允许为空不填写 |
+| proxy.cluster.tag | dataproxy所处的集群Set名 | default_cluster | 一个集群set里可以包含多个dataproxy、MQ的集群 |
+| proxy.cluster.name | dataproxy所处的集群名 | default_dataproxy | 用于区分不同的环境 |
+| proxy.cluster.inCharges | dataproxy所处的集群负责人 | admin | |
+| configCheckInterval | conf目录的配置文件检查及加载间隔（单位ms） | 10000 | |
+| metricDomains | JMX域名 | DataProxy | 通过该配置值获取如下“metricDomains.${metricDomains}.xxx”的配置项 |
+| metricDomains.DataProxy.domainListeners | 指标监听的类，通过该类名反射开启服务 | org.apache.inlong.dataproxy.metrics.prometheus.PrometheusMetricListener | 如果存在多个指标监听类配置，通过空格、回车，或换行符进行分隔 |
+| metricDomains.DataProxy.snapshotInterval | 周期性上报指标的时隙（单位ms） | 60000 | |
+| prometheusHttpPort | 使用 Prometheus 上报时设置的端口号 | 9081 | |
+| audit.enable | 是否开启数据上报 InLong-Audit 服务 | true | |
+| audit.proxys | InLong-Audit 服务的地址 | 127.0.0.1:10081 | 格式按照 {ip1:port1}[ {ip2:port2}][ {ip3:port3}]进行配置，不同ip:port间通过空格、回车，或换行符进行分隔 |
+
+
+## 日志输出配置（log4j2.xml）
+DataProxy采用的是Log4j2输出日志，相关配置基于Log4j2进行设置，本配置只提及常用设置项：
+
+| 参数 | 描述 | 默认值 | 备注 |
+|-------------------------|-----------|-------|----------------|
+| basePath | 日志文件的存储路径 | ${sys:dataproxy.log.path} | |
+| every_file_size | 每个日志文件的大小，单位字节 | 1G | |
+| output_log_level | 日志输出级别 | DEBUG | 线上使用时建议设置为INFO |
+| rolling_max | 同类型日志同一目录下可存放的数量 | 50 | |
+| debug_max | DEBUG类型日志同一目录下可存放的数量 | 7 | |
+| info_max | INFO类型日志同一目录下可存放的数量 | 7 | |
+| warn_max | WARN类型日志同一目录下可存放的数量 | 7 | |
+| error_max | ERROR类型日志同一目录下可存放的数量 | 7 | |
+
+
+## source-channel-sink管道配置（dataproxy-*.conf）
+
+DataProxy 支持配置化的 source-channel-sink，配置方式与 flume 的配置文件结构类似。配置文件放在 dataproxy-*.conf 文件中，目前支持 dataproxy-pulsar.conf 和 dataproxy-tube.conf 两种，用于区分不同的中间件类型，具体类型可以在启动时指定，默认（未指定时）使用 dataproxy-pulsar.conf 作为配置文件。
+
+
+## 其他自动更新配置
+DataProxy启动并成功链接到Manager后，会自动从Manager同步运行配置，并定期更新，如下部分配置不可修改
+
+| 配置文件名 | 描述 | 备注 |
+|------------------|-----------------------|-----------------|
+| mq_cluster.properties | MQ集群配置 | |
+| topics.properties | groupId到MQ的Topic的映射配置 | |
Review Comment:
`groupId到MQ的Topic的映射配置` -> `InlongGroupId 和不同类型 MQ 的 Topic 的映射配置`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong-website] gosonzhang commented on a diff in pull request #571: [INLONG-6238][Doc] Add configuration doc for DataProxy
Posted by GitBox <gi...@apache.org>.
gosonzhang commented on code in PR #571:
URL: https://github.com/apache/inlong-website/pull/571#discussion_r1000267545
##########
docs/modules/dataproxy/configuration.md:
##########
@@ -0,0 +1,50 @@
+---
+title: Configuration
+sidebar_position: 3
+---
+## Basic Configuration (common.properties)
+
+| Parameter | Description | Default | Notes |
+|--------------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------------------|-------------------------------------------------------------------------------------------------------------------|
+| manager.hosts | InLong-Manager http host and port | 127.0.0.1:8083 | Empty is not allowed, the format is configured according to the format of {ip1:port1}[,{ip2:port2}][,{ip3:port3}] |
+| manager.auth.secretId | InLong-Manager authentic secretId | `blank` | If InLong-Manager doesn't open authentic service, this parameter can be empty. |
+| manager.auth.secretKey | InLong-Manager authentic secretKey | `blank` | |
+| proxy.cluster.tag | The cluster set name where the dataproxy is located | default_cluster | A cluster set can contain multiple dataproxy and MQ clusters |
+| proxy.cluster.name | The cluster name where dataproxy is located | default_dataproxy | used to distinguish different environments |
+| proxy.cluster.inCharges | The incharge of the cluster where dataproxy is located | admin | |
+| configCheckInterval | Configuration file checking and loading interval of the conf directory (unit: ms) | 10000 | |
+| metricDomains | JMX domain name | DataProxy | Obtain the following configuration items of "metricDomains.${metricDomains}.xxx" through this configuration value |
+| metricDomains.DataProxy.domainListeners | The class for indicator monitoring, and the service is started through reflection of the class name | org.apache.inlong.dataproxy.metrics.prometheus.PrometheusMetricListener | If there are multiple indicator monitoring class configurations, separate them by spaces, carriage returns, or line feeds |
+| metricDomains.DataProxy.snapshotInterval | Time interval for periodic reporting of indicators (unit: ms) | 60000 | |
+| prometheusHttpPort | The port when reporting using Prometheus | 9081 | |
+| audit.enable | Whether to enable data reporting InLong-Audit service | true | |
+| audit.proxys | The address of the InLong-Audit service | 127.0.0.1:10081 | 格式按照 {ip1:port1}[ {ip2:port2}][ {ip3:port3}]进行配置，不同ip:port间通过空格、回车，或换行符进行分隔 |
Review Comment:
Ok!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong-website] healchow commented on a diff in pull request #571: [INLONG-6238][Doc] Add configuration doc for DataProxy
Posted by GitBox <gi...@apache.org>.
healchow commented on code in PR #571:
URL: https://github.com/apache/inlong-website/pull/571#discussion_r1000290536
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/configuration.md:
##########
@@ -0,0 +1,52 @@
+---
+title: 配置
+sidebar_position: 3
+---
+
+## 基本配置（common.properties）
+
+| 参数 | 描述 | 默认值 | 备注 |
+|---------------------------------------------------------|--------------------------|--------------------------------------|-----------------------------------------------------------------------------|
+| manager.hosts | Manager服务器IP和端口列表 | 127.0.0.1:8083 | 不允许为空，格式按照 {ip1:port1}[,{ip2:port2}][,{ip3:port3}] 格式进行配置 |
Review Comment:
Suggest adding one blank before or between English and Chinese.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong-website] dockerzhang commented on a diff in pull request #571: [INLONG-6238][Doc] Add configuration doc for DataProxy
Posted by GitBox <gi...@apache.org>.
dockerzhang commented on code in PR #571:
URL: https://github.com/apache/inlong-website/pull/571#discussion_r1000273876
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/modules/dataproxy/configuration.md:
##########
@@ -0,0 +1,52 @@
+---
+title: 配置
+sidebar_position: 3
+---
+
+## 基本配置（common.properties）
+
+| 参数 | 描述 | 默认值 | 备注 |
+|---------------------------------------------------------|--------------------------|--------------------------------------|-----------------------------------------------------------------------------|
+| manager.hosts | Manager服务器IP和端口列表 | 127.0.0.1:8083 | 不允许为空，格式按照 {ip1:port1}[,{ip2:port2}][,{ip3:port3}] 格式进行配置 |
+| manager.auth.secretId | 登录所需帐号 | 无 | 允许为空不填写 |
+| manager.auth.secretKey | 登录所需密码 | 无 | 允许为空不填写 |
+| proxy.cluster.tag | dataproxy所处的集群Set名 | default_cluster | 一个集群set里可以包含多个dataproxy、MQ的集群 |
+| proxy.cluster.name | dataproxy所处的集群名 | default_dataproxy | 用于区分不同的环境 |
+| proxy.cluster.inCharges | dataproxy所处的集群负责人 | admin | |
+| configCheckInterval | conf目录的配置文件检查及加载间隔（单位ms） | 10000 | |
+| metricDomains | JMX域名 | DataProxy | 通过该配置值获取如下“metricDomains.${metricDomains}.xxx”的配置项 |
+| metricDomains.DataProxy.domainListeners | 指标监听的类，通过该类名反射开启服务 | org.apache.inlong.dataproxy.metrics.prometheus.PrometheusMetricListener | 如果存在多个指标监听类配置，通过空格、回车，或换行符进行分隔 |
+| metricDomains.DataProxy.snapshotInterval | 周期性上报指标的时隙（单位ms） | 60000 | |
+| prometheusHttpPort | 使用 Prometheus 上报时设置的端口号 | 9081 | |
+| audit.enable | 是否开启数据上报 InLong-Audit 服务 | true | |
+| audit.proxys | InLong-Audit 服务的地址 | 127.0.0.1:10081 | 格式按照 {ip1:port1}[ {ip2:port2}][ {ip3:port3}]进行配置，不同ip:port间通过空格、回车，或换行符进行分隔 |
+
+
+## 日志输出配置（log4j2.xml）
+DataProxy采用的是Log4j2输出日志，相关配置基于Log4j2进行设置，本配置只提及常用设置项：
+
+| 参数 | 描述 | 默认值 | 备注 |
+|-------------------------|-----------|-------|----------------|
+| basePath | 日志文件的存储路径 | ${sys:dataproxy.log.path} | |
+| every_file_size | 每个日志文件的大小，单位字节 | 1G | |
+| output_log_level | 日志输出级别 | DEBUG | 线上使用时建议设置为INFO |
+| rolling_max | 同类型日志同一目录下可存放的数量 | 50 | |
+| debug_max | DEBUG类型日志同一目录下可存放的数量 | 7 | |
+| info_max | INFO类型日志同一目录下可存放的数量 | 7 | |
+| warn_max | WARN类型日志同一目录下可存放的数量 | 7 | |
+| error_max | ERROR类型日志同一目录下可存放的数量 | 7 | |
+
+
+## source-channel-sink管道配置（dataproxy-*.conf）
Review Comment:
dataproxy-*.conf
->
dataproxy-{tube|pulsar}.conf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong-website] dockerzhang merged pull request #571: [INLONG-6238][Doc] Add configuration doc for DataProxy
Posted by GitBox <gi...@apache.org>.
dockerzhang merged PR #571:
URL: https://github.com/apache/inlong-website/pull/571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org