You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/10/24 10:24:51 UTC
[GitHub] [inlong-website] doleyzi opened a new pull request, #575: [INLONG-574][Audit]Update configuration item description
doleyzi opened a new pull request, #575:
URL: https://github.com/apache/inlong-website/pull/575
### Prepare a Pull Request
- [INLONG-574][Audit]Update configuration item description
- Fixes #574
### Motivation
Update configuration item description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong-website] gosonzhang commented on a diff in pull request #575: [INLONG-574][Audit] Update configuration item description
Posted by GitBox <gi...@apache.org>.
gosonzhang commented on code in PR #575:
URL: https://github.com/apache/inlong-website/pull/575#discussion_r1004177378
##########
docs/modules/audit/configure.md:
##########
@@ -0,0 +1,274 @@
+---
+title: Configuration
+sidebar_position: 3
+---
+
+## audit-proxy source-channel-sink pipeline configuration (dataproxy-{tube|pulsar}.con)
+
+audit-proxy supports configured source-channel-sink, and the configuration method is consistent with the configuration
+file structure of flume. When modifying the configuration file, it should be modified according to the configuration
+file definition of Apache flume. config file in audit-proxy-{tube|pulsar}.conf file, currently supports
+audit-proxy-pulsar.conf and audit-proxy-tube.conf Two types are used to distinguish different middleware types. The
+specific type can be specified at startup. By default (when not specified), audit-proxy-pulsar.conf is used as the
+configuration file. The following is an example for this configuration file:
+
+- Source Configuration example：
+
+```shell
+agent1.sources.tcp-source.channels = ch-msg1
+Define the channel used in the source. Note that if the configuration below this source uses the channel, it needs to be commented here
+
+agent1.sources.tcp-source.type = org.apache.inlong.audit.source.SimpleTcpSource
+tcp parsing type definition, class name is provided here for instantiation, SimpleTcpSource is mainly to initialize configuration and start port listening
+
+agent1.sources.tcp-source.msg-factory-name = org.apache.inlong.audit.source.ServerMessageFactory
+Handler for constructing message parsing, and setting read stream handler and write stream handler
+
+agent1.sources.tcp-source.host = 0.0.0.0
+tcp ip binding and listening, all network cards are bound by default
+
+agent1.sources.tcp-source.port = 10081
+tcp port binding, default binding to port 46801
+
+agent1.sources.tcp-source.max-msg-length = 524288
+Limit the size of a single package, here if the compressed package is transmitted, it is the compressed package size, and the limit is 512KB
+
+agent1.sources.tcp-source.connections = 30000
+Concurrent connections are online, and new connections will be disconnected when the upper limit is exceeded
+
+agent1.sources.tcp-source.max-threads = 64
+Netty thread pool worker thread upper limit, it is generally recommended to choose twice the cpu
+
+agent1.sources.tcp-source.receiveBufferSize = 1048576
+Netty server tcp tuning parameters, the upper limit of the capacity of the TCP accept buffer
+
+agent1.sources.tcp-source.sendBufferSize = 1048576
+Netty server tcp tuning parameters, the upper limit of the capacity of the TCP send buffer
+
+agent1.sources.tcp-source.custom-cp = true
+Whether to use the self-developed channel process, the self-developed channel process can choose the backup channel to send when the main channel is blocked
+
+agent1.sources.tcp-source.selector.type = org.apache.inlong.audit.channel.FailoverChannelSelector
+This channel selector is a self-developed channel selector, which is not much different from the official website, mainly because of the channel master-slave selection logic
+
+agent1.sources.tcp-source.selector.master = ch-msg1
+Specify master channels, which will be preferred for data push. Those channels that are not in the master, transfer, fileMetric, and slaMetric configuration items, but are
+There are defined channels in channels, which are all classified as slave channels. When the master channel is full, it will choose to use the slave channel.
+```
+
+- Channel Configuration example，memory channel：
+
+```shell
+agent1.channels.ch-msg1.type = memory
+memory channel type
+
+agent1.channels.ch-msg1.capacity = 10000
+The size of the memory channel queue, the maximum number of messages that can be cached
+
+agent1.channels.ch-msg1.transactionCapacity = 200
+The maximum number of batches is processed in atomic operations, and the memory channel needs to be locked when using it, so there will be a batch process to increase efficiency
+```
+
+- Channel configuration example, file channel:
+
+```shell
+agent1.channels.ch-msg2.type = file
+file channel type
+
+agent1.channels.ch-msg2.capacity = 100000000
+The maximum number of messages that can be cached by file channel
+
+agent1.channels.ch-msg2.maxFileSize = 1073741824
+file channel file maximum upper limit, the number of bytes
+
+agent1.channels.ch-msg2.minimumRequiredSpace = 1073741824
+The minimum free space on the disk where the file channel is located. Setting this value can prevent the disk from being full.
+
+agent1.channels.ch-msg2.checkpointDir =./data/file/ch-msg5/check
+file channel checkpoint path
+
+agent1.channels.ch-msg2.dataDirs =./data/file/ch-msg5/data
+file channel data path
+
+agent1.channels.ch-msg2.fsyncPerTransaction = false
+Whether to synchronize the disk for each atomic operation, it is recommended to change to false, otherwise it will affect the performance
+
+agent1.channels.ch-msg2.fsyncInterval = 10
+The time interval between flushing data from memory to disk, in seconds
+```
+
+- Sink 配置示例：
Review Comment:
English is required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong-website] doleyzi commented on a diff in pull request #575: [INLONG-574][Audit] Update configuration item description
Posted by GitBox <gi...@apache.org>.
doleyzi commented on code in PR #575:
URL: https://github.com/apache/inlong-website/pull/575#discussion_r1004226475
##########
docs/modules/audit/configure.md:
##########
@@ -0,0 +1,274 @@
+---
+title: Configuration
+sidebar_position: 3
+---
+
+## audit-proxy source-channel-sink pipeline configuration (dataproxy-{tube|pulsar}.con)
+
+audit-proxy supports configured source-channel-sink, and the configuration method is consistent with the configuration
+file structure of flume. When modifying the configuration file, it should be modified according to the configuration
+file definition of Apache flume. config file in audit-proxy-{tube|pulsar}.conf file, currently supports
+audit-proxy-pulsar.conf and audit-proxy-tube.conf Two types are used to distinguish different middleware types. The
+specific type can be specified at startup. By default (when not specified), audit-proxy-pulsar.conf is used as the
+configuration file. The following is an example for this configuration file:
+
+- Source Configuration example：
+
+```shell
+agent1.sources.tcp-source.channels = ch-msg1
+Define the channel used in the source. Note that if the configuration below this source uses the channel, it needs to be commented here
+
+agent1.sources.tcp-source.type = org.apache.inlong.audit.source.SimpleTcpSource
+tcp parsing type definition, class name is provided here for instantiation, SimpleTcpSource is mainly to initialize configuration and start port listening
+
+agent1.sources.tcp-source.msg-factory-name = org.apache.inlong.audit.source.ServerMessageFactory
+Handler for constructing message parsing, and setting read stream handler and write stream handler
+
+agent1.sources.tcp-source.host = 0.0.0.0
+tcp ip binding and listening, all network cards are bound by default
+
+agent1.sources.tcp-source.port = 10081
+tcp port binding, default binding to port 46801
+
+agent1.sources.tcp-source.max-msg-length = 524288
+Limit the size of a single package, here if the compressed package is transmitted, it is the compressed package size, and the limit is 512KB
+
+agent1.sources.tcp-source.connections = 30000
+Concurrent connections are online, and new connections will be disconnected when the upper limit is exceeded
+
+agent1.sources.tcp-source.max-threads = 64
+Netty thread pool worker thread upper limit, it is generally recommended to choose twice the cpu
+
+agent1.sources.tcp-source.receiveBufferSize = 1048576
+Netty server tcp tuning parameters, the upper limit of the capacity of the TCP accept buffer
+
+agent1.sources.tcp-source.sendBufferSize = 1048576
+Netty server tcp tuning parameters, the upper limit of the capacity of the TCP send buffer
+
+agent1.sources.tcp-source.custom-cp = true
+Whether to use the self-developed channel process, the self-developed channel process can choose the backup channel to send when the main channel is blocked
+
+agent1.sources.tcp-source.selector.type = org.apache.inlong.audit.channel.FailoverChannelSelector
+This channel selector is a self-developed channel selector, which is not much different from the official website, mainly because of the channel master-slave selection logic
+
+agent1.sources.tcp-source.selector.master = ch-msg1
+Specify master channels, which will be preferred for data push. Those channels that are not in the master, transfer, fileMetric, and slaMetric configuration items, but are
+There are defined channels in channels, which are all classified as slave channels. When the master channel is full, it will choose to use the slave channel.
+```
+
+- Channel Configuration example，memory channel：
+
+```shell
+agent1.channels.ch-msg1.type = memory
+memory channel type
+
+agent1.channels.ch-msg1.capacity = 10000
+The size of the memory channel queue, the maximum number of messages that can be cached
+
+agent1.channels.ch-msg1.transactionCapacity = 200
+The maximum number of batches is processed in atomic operations, and the memory channel needs to be locked when using it, so there will be a batch process to increase efficiency
+```
+
+- Channel configuration example, file channel:
+
+```shell
+agent1.channels.ch-msg2.type = file
+file channel type
+
+agent1.channels.ch-msg2.capacity = 100000000
+The maximum number of messages that can be cached by file channel
+
+agent1.channels.ch-msg2.maxFileSize = 1073741824
+file channel file maximum upper limit, the number of bytes
+
+agent1.channels.ch-msg2.minimumRequiredSpace = 1073741824
+The minimum free space on the disk where the file channel is located. Setting this value can prevent the disk from being full.
+
+agent1.channels.ch-msg2.checkpointDir =./data/file/ch-msg5/check
+file channel checkpoint path
+
+agent1.channels.ch-msg2.dataDirs =./data/file/ch-msg5/data
+file channel data path
+
+agent1.channels.ch-msg2.fsyncPerTransaction = false
+Whether to synchronize the disk for each atomic operation, it is recommended to change to false, otherwise it will affect the performance
+
+agent1.channels.ch-msg2.fsyncInterval = 10
+The time interval between flushing data from memory to disk, in seconds
+```
+
+- Sink 配置示例：
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong-website] dockerzhang merged pull request #575: [INLONG-574][Audit] Update configuration item description
Posted by GitBox <gi...@apache.org>.
dockerzhang merged PR #575:
URL: https://github.com/apache/inlong-website/pull/575
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org