You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by ji...@apache.org on 2022/12/09 08:05:22 UTC
[doris] branch master updated: [typo](docs)optimize the BE config document (#14957)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0aa32ec1af [typo](docs)optimize the BE config document (#14957)
0aa32ec1af is described below
commit 0aa32ec1af63fb25eefd661b6ad413869c3d9ff4
Author: Liqf <10...@users.noreply.github.com>
AuthorDate: Fri Dec 9 16:05:15 2022 +0800
[typo](docs)optimize the BE config document (#14957)
---
docs/en/docs/admin-manual/config/be-config.md | 1773 ++++++++++-----------
docs/zh-CN/docs/admin-manual/config/be-config.md | 1775 ++++++++++------------
2 files changed, 1529 insertions(+), 2019 deletions(-)
diff --git a/docs/en/docs/admin-manual/config/be-config.md b/docs/en/docs/admin-manual/config/be-config.md
index 80de6b7d49..9612c45617 100644
--- a/docs/en/docs/admin-manual/config/be-config.md
+++ b/docs/en/docs/admin-manual/config/be-config.md
@@ -1,7 +1,9 @@
---
{
"title": "BE Configuration",
- "language": "en"
+ "language": "en",
+ "toc_min_heading_level": 2,
+ "toc_max_heading_level": 4
}
---
@@ -48,17 +50,17 @@ There are two ways to configure BE configuration items:
1. Static configuration
- Add and set configuration items in the `conf/be.conf` file. The configuration items in `be.conf` will be read when BE starts. Configuration items not in `be.conf` will use default values.
+ Add and set configuration items in the `conf/be.conf` file. The configuration items in `be.conf` will be read when BE starts. Configuration items not in `be.conf` will use default values.
2. Dynamic configuration
- After BE starts, the configuration items can be dynamically set with the following commands.
+ After BE starts, the configuration items can be dynamically set with the following commands.
```
curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}'
```
- In version 0.13 and before, the configuration items modified in this way will become invalid after the BE process restarts. In 0.14 and later versions, the modified configuration can be persisted through the following command. The modified configuration items are stored in the `be_custom.conf` file.
+ In version 0.13 and before, the configuration items modified in this way will become invalid after the BE process restarts. In 0.14 and later versions, the modified configuration can be persisted through the following command. The modified configuration items are stored in the `be_custom.conf` file.
```
curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}&persis=true
@@ -68,21 +70,21 @@ There are two ways to configure BE configuration items:
1. Modify `max_base_compaction_threads` statically
- By adding in the `be.conf` file:
+ By adding in the `be.conf` file:
- ```max_base_compaction_threads=5```
+ ```max_base_compaction_threads=5```
- Then restart the BE process to take effect the configuration.
+ Then restart the BE process to take effect the configuration.
2. Modify `streaming_load_max_mb` dynamically
- After BE starts, the configuration item `streaming_load_max_mb` is dynamically set by the following command:
+ After BE starts, the configuration item `streaming_load_max_mb` is dynamically set by the following command:
```
curl -X POST http://{be_ip}:{be_http_port}/api/update_config?streaming_load_max_mb=1024
```
- The return value is as follows, indicating that the setting is successful.
+ The return value is as follows, indicating that the setting is successful.
```
{
@@ -91,7 +93,7 @@ There are two ways to configure BE configuration items:
}
```
- The configuration will become invalid after the BE restarts. If you want to persist the modified results, use the following command:
+ The configuration will become invalid after the BE restarts. If you want to persist the modified results, use the following command:
```
curl -X POST http://{be_ip}:{be_http_port}/api/update_config?streaming_load_max_mb=1024\&persist=true
@@ -99,1349 +101,1179 @@ There are two ways to configure BE configuration items:
## Configurations
-### `alter_tablet_worker_count`
+### Services
-Default: 3
-
-The number of threads making schema changes
-
-### `generate_compaction_tasks_interval_ms`
-
-Default: 10 (ms)
-
-Minimal interval (ms) to generate compaction tasks
-
-### `enable_vectorized_compaction`
-
-Default: true
-
-Whether to enable vectorized compaction
-
-### `base_compaction_min_rowset_num`
-
-Default: 5
-
-One of the triggering conditions of BaseCompaction: The limit of the number of Cumulative files to be reached. After reaching this limit, BaseCompaction will be triggered
-
-### base_compaction_trace_threshold
+#### `be_port`
* Type: int32
-* Description: Threshold to logging base compaction's trace information, in seconds
-* Default value: 10
-
-Base compaction is a long time cost background task, this configuration is the threshold to logging trace information. Trace information in log file looks like:
-
-```
-W0610 11:26:33.804431 56452 storage_engine.cpp:552] Trace:
-0610 11:23:03.727535 (+ 0us) storage_engine.cpp:554] start to perform base compaction
-0610 11:23:03.728961 (+ 1426us) storage_engine.cpp:560] found best tablet 546859
-0610 11:23:03.728963 (+ 2us) base_compaction.cpp:40] got base compaction lock
-0610 11:23:03.729029 (+ 66us) base_compaction.cpp:44] rowsets picked
-0610 11:24:51.784439 (+108055410us) compaction.cpp:46] got concurrency lock and start to do compaction
-0610 11:24:51.784818 (+ 379us) compaction.cpp:74] prepare finished
-0610 11:26:33.359265 (+101574447us) compaction.cpp:87] merge rowsets finished
-0610 11:26:33.484481 (+125216us) compaction.cpp:102] output rowset built
-0610 11:26:33.484482 (+ 1us) compaction.cpp:106] check correctness finished
-0610 11:26:33.513197 (+ 28715us) compaction.cpp:110] modify rowsets finished
-0610 11:26:33.513300 (+ 103us) base_compaction.cpp:49] compaction finished
-0610 11:26:33.513441 (+ 141us) base_compaction.cpp:56] unused rowsets have been moved to GC queue
-Metrics: {"filtered_rows":0,"input_row_num":3346807,"input_rowsets_count":42,"input_rowsets_data_size":1256413170,"input_segments_num":44,"merge_rowsets_latency_us":101574444,"merged_rows":0,"output_row_num":3346807,"output_rowset_data_size":1228439659,"output_segments_num":6}
-```
-
-### `base_compaction_write_mbytes_per_sec`
-
-Default: 5（MB）
+* Description: The port of the thrift server on BE which used to receive requests from FE
+* Default value: 9060
-Maximum disk write speed per second of BaseCompaction task
+#### `heartbeat_service_port`
-### `base_compaction_min_data_ratio`
+* Type: int32
+* Description: Heartbeat service port (thrift) on BE, used to receive heartbeat from FE
+* Default value: 9050
-Default: 0.3 （30%）
+#### `webserver_port`
-One of the trigger conditions of BaseCompaction: Cumulative file size reaches the proportion of Base file
+* Type: int32
+* Description: Service port of http server on BE
+* Default value: 8040
-### `be_port`
+#### `brpc_port`
* Type: int32
-* Description: The port of the thrift server on BE which used to receive requests from FE
-* Default value: 9060
+* Description: The port of BRPC on BE, used for communication between BEs
+* Default value: 8060
-### `be_service_threads`
+#### `single_replica_load_brpc_port`
* Type: int32
-* Description: The number of execution threads of the thrift server service on BE which represents the number of threads that can be used to execute FE requests.
-* Default value: 64
+* Description: The port of BRPC on BE, used for single replica load. There is a independent BRPC thread pool for the communication between the Master replica and Slave replica during single replica load, which prevents data synchronization between the replicas from preempt the thread resources for data distribution and query tasks when the load concurrency is large.
+* Default value: 9070
-### `brpc_max_body_size`
+#### `single_replica_load_download_port`
-This configuration is mainly used to modify the parameter `max_body_size` of brpc.
+* Type: int32
+* Description: The port of http for segment download on BE, used for single replica load. There is a independent HTTP thread pool for the Slave replica to download segments during single replica load, which prevents data synchronization between the replicas from preempt the thread resources for other http tasks when the load concurrency is large.
+* Default value: 8050
-Sometimes the query fails and an error message of `body_size is too large` will appear in the BE log. This may happen when the SQL mode is "multi distinct + no group by + more than 1T of data".
+#### `priority_networks`
-This error indicates that the packet size of brpc exceeds the configured value. At this time, you can avoid this error by increasing the configuration.
+* Description: Declare a selection strategy for those servers with many IPs. Note that at most one ip should match this list. This is a semicolon-separated list in CIDR notation, such as 10.10.10.0/24. If there is no IP matching this rule, one will be randomly selected
+* Default value: blank
-### `brpc_socket_max_unwritten_bytes`
+#### `storage_root_path`
-This configuration is mainly used to modify the parameter `socket_max_unwritten_bytes` of brpc.
+* Type: string
-Sometimes the query fails and an error message of `The server is overcrowded` will appear in the BE log. This means there are too many messages to buffer at the sender side, which may happen when the SQL needs to send large bitmap value. You can avoid this error by increasing the configuration.
+* Description: data root path, separate by ';'.you can specify the storage medium of each root path, HDD or SSD. you can add capacity limit at the end of each root path, separate by ','.If the user does not use a mix of SSD and HDD disks, they do not need to configure the configuration methods in Example 1 and Example 2 below, but only need to specify the storage directory; they also do not need to modify the default storage media configuration of FE.
-### `transfer_large_data_by_brpc`
+ eg.1: `storage_root_path=/home/disk1/doris.HDD;/home/disk2/doris.SSD;/home/disk2/doris`
-* Type: bool
-* Description: This configuration is used to control whether to serialize the protoBuf request and embed the Tuple/Block data into the controller attachment and send it through http brpc when the length of the Tuple/Block data is greater than 1.8G. To avoid errors when the length of the protoBuf request exceeds 2G: Bad request, error_text=[E1003]Fail to compress request. In the past version, after putting Tuple/Block data in the attachment, it was sent through the default baidu_std brpc, [...]
-* Default value: true
+ - 1./home/disk1/doris.HDD, indicates that the storage medium is HDD;
+ - 2./home/disk2/doris.SSD, indicates that the storage medium is SSD;
+ - 3./home/disk2/doris, indicates that the storage medium is HDD by default
-### `brpc_num_threads`
+ eg.2: `storage_root_path=/home/disk1/doris,medium:hdd;/home/disk2/doris,medium:ssd`
-This configuration is mainly used to modify the number of bthreads for brpc. The default value is set to -1, which means the number of bthreads is #cpu-cores.
+ - 1./home/disk1/doris,medium:hdd，indicates that the storage medium is HDD;
+ - 2./home/disk2/doris,medium:ssd，indicates that the storage medium is SSD;
-User can set this configuration to a larger value to get better QPS performance. For more information, please refer to `https://github.com/apache/incubator-brpc/blob/master/docs/cn/benchmark.md`
+* Default value: ${DORIS_HOME}/storage
-### `brpc_port`
+#### `heartbeat_service_thread_count`
* Type: int32
-* Description: The port of BRPC on BE, used for communication between BEs
-* Default value: 9060
-
-### `buffer_pool_clean_pages_limit`
-
-default: 50%
-
-Clean up pages that may be saved by the buffer pool
-
-### `buffer_pool_limit`
-
-* Type: string
-* Description: The largest allocatable memory of the buffer pool
-* Default value: 20%
-
-The maximum amount of memory available in the BE buffer pool. The buffer pool is a new memory management structure of BE, which manages the memory by the buffer page and enables spill data to disk. The memory for all concurrent queries will be allocated from the buffer pool. The current buffer pool only works on **AggregationNode** and **ExchangeNode**.
-
-### `check_consistency_worker_count`
-
-Default: 1
-
-The number of worker threads to calculate the checksum of the tablet
+* Description: The number of threads that execute the heartbeat service on BE. the default is 1, it is not recommended to modify
+* Default value: 1
-### `chunk_reserved_bytes_limit`
+#### `ignore_broken_disk`
-Default: 20%
+* Type: bool
+* Description: When BE starts, check ``storage_root_path`` All paths under configuration.
-The reserved bytes limit of Chunk Allocator, usually set as a percentage of mem_limit. defaults to bytes if no unit is given, the number of bytes must be a multiple of 2. must larger than 0. and if larger than physical memory size, it will be set to physical memory size. increase this variable can improve performance, but will acquire more free memory which can not be used by other modules.
+ - `ignore_broken_disk=true`
-### `clear_transaction_task_worker_count`
+ If the path does not exist or the file (bad disk) cannot be read or written under the path, the path will be ignored. If there are other available paths, the startup will not be interrupted.
-Default: 1
+ - `ignore_broken_disk=false`
-Number of threads used to clean up transactions
+ If the path does not exist or the file (bad disk) cannot be read or written under the path, the system will abort the startup failure and exit.
-### `clone_worker_count`
+* Default value: false
-Default: 3
+#### `mem_limit`
-Number of threads used to perform cloning tasks
+* Type: string
+* Description: Limit the percentage of the server's maximum memory used by the BE process. It is used to prevent BE memory from occupying to many the machine's memory. This parameter must be greater than 0. When the percentage is greater than 100%, the value will default to 100%.
+* Default value: 80%
-### `cluster_id`
+#### `cluster_id`
* Type: int32
* Description: Configure the cluster id to which the BE belongs.
-* Default value: -1
+ - This value is usually delivered by the FE to the BE by the heartbeat, no need to configure. When it is confirmed that a BE belongs to a certain Drois cluster, it can be configured. The cluster_id file under the data directory needs to be modified to make sure same as this parament.
+* Default value: - 1
-This value is usually delivered by the FE to the BE by the heartbeat, no need to configure. When it is confirmed that a BE belongs to a certain Drois cluster, it can be configured. The cluster_id file under the data directory needs to be modified to make sure same as this parament.
+#### `custom_config_dir`
-### `column_dictionary_key_ratio_threshold`
+* Description: Configure the location of the `be_custom.conf` file. The default is in the `conf/` directory.
+ - In some deployment environments, the `conf/` directory may be overwritten due to system upgrades. This will cause the user modified configuration items to be overwritten. At this time, we can store `be_custom.conf` in another specified directory to prevent the configuration file from being overwritten.
+* Default value: blank
-Default: 0
+#### `trash_file_expire_time_sec`
-The value ratio of string type, less than this ratio, using dictionary compression algorithm
+* Description: The interval for cleaning the recycle bin is 72 hours. When the disk space is insufficient, the file retention period under trash may not comply with this parameter
+* Default value: 259200
-### `column_dictionary_key_size_threshold`
+#### `es_http_timeout_ms`
-Default: 0
+* Description: The timeout period for connecting to ES via http.
+* Default value: 5000 (ms)
-Dictionary compression column size, less than this value using dictionary compression algorithm
+#### `es_scroll_keepalive`
-### `compaction_tablet_compaction_score_factor`
+* Description: es scroll Keeplive hold time
+* Default value: 5 (m)
-* Type: int32
-* Description: Coefficient for compaction score when calculating tablet score to find a tablet for compaction.
-* Default value: 1
-
-### `compaction_tablet_scan_frequency_factor`
+#### `external_table_connect_timeout_sec`
* Type: int32
-* Description: Coefficient for tablet scan frequency when calculating tablet score to find a tablet for compaction.
-* Default value: 0
+* Description: The timeout when establishing connection with external table such as ODBC table.
+* Default value: 5 seconds
-Tablet scan frequency can be taken into consideration when selecting an tablet for compaction and preferentially do compaction for those tablets which are scanned frequently during a latest period of time at the present.
-Tablet score can be calculated like this:
+#### `status_report_interval`
-tablet_score = compaction_tablet_scan_frequency_factor * tablet_scan_frequency + compaction_tablet_compaction_score_factor * compaction_score
+* Description: Interval between profile reports
+* Default value: 5
-### `compaction_task_num_per_disk`
+#### `brpc_max_body_size`
-* Type: int32
-* Description: The number of compaction tasks which execute in parallel for a disk(HDD).
-* Default value: 2
+* Description: This configuration is mainly used to modify the parameter `max_body_size` of brpc.
-### `compaction_task_num_per_fast_disk`
+ - Sometimes the query fails and an error message of `body_size is too large` will appear in the BE log. This may happen when the SQL mode is "multi distinct + no group by + more than 1T of data".This error indicates that the packet size of brpc exceeds the configured value. At this time, you can avoid this error by increasing the configuration.
-* Type: int32
-* Description: The number of compaction tasks which execute in parallel for a fast disk(SSD).
-* Default value: 4
+#### `brpc_socket_max_unwritten_bytes`
-### `compress_rowbatches`
+* Description: This configuration is mainly used to modify the parameter `socket_max_unwritten_bytes` of brpc.
+ - Sometimes the query fails and an error message of `The server is overcrowded` will appear in the BE log. This means there are too many messages to buffer at the sender side, which may happen when the SQL needs to send large bitmap value. You can avoid this error by increasing the configuration.
+
+#### `transfer_large_data_by_brpc`
* Type: bool
-* Description: enable to use Snappy compression algorithm for data compression when serializing RowBatch
+* Description: This configuration is used to control whether to serialize the protoBuf request and embed the Tuple/Block data into the controller attachment and send it through http brpc when the length of the Tuple/Block data is greater than 1.8G. To avoid errors when the length of the protoBuf request exceeds 2G: Bad request, error_text=[E1003]Fail to compress request. In the past version, after putting Tuple/Block data in the attachment, it was sent through the default baidu_std brpc, [...]
* Default value: true
-### `create_tablet_worker_count`
+#### `brpc_num_threads`
-Default: 3
-
-Number of worker threads for BE to create a tablet
-
-### `cumulative_compaction_rounds_for_each_base_compaction_round`
+* Description: This configuration is mainly used to modify the number of bthreads for brpc. The default value is set to -1, which means the number of bthreads is #cpu-cores.
+ - User can set this configuration to a larger value to get better QPS performance. For more information, please refer to `https://github.com/apache/incubator-brpc/blob/master/docs/cn/benchmark.md`
+* Default value: 1
-* Type: int32
-* Description: How many rounds of cumulative compaction for each round of base compaction when compaction tasks generation.
-* Default value: 9
+#### `thrift_rpc_timeout_ms`
-### `disable_auto_compaction`
+* Description: thrift default timeout time
+* Default value: 5000
-* Type: bool
-* Description: Whether disable automatic compaction task
-* Default value: false
+#### `thrift_client_retry_interval_ms`
-Generally it needs to be turned off. When you want to manually operate the compaction task in the debugging or test environment, you can turn on the configuration.
+* Type: int64
+* Description: Used to set retry interval for thrift client in be to avoid avalanche disaster in fe thrift server, the unit is ms.
+* Default value: 1000
-### `cumulative_compaction_budgeted_bytes`
+#### `thrift_connect_timeout_seconds`
-Default: 104857600
+* Description: The default thrift client connection timeout time
+* Default value: 3 (m)
-One of the trigger conditions of BaseCompaction: Singleton file size limit, 100MB
+#### `thrift_server_type_of_fe`
-### cumulative_compaction_trace_threshold
+* Type: string
+* Description:This configuration indicates the service model used by FE's Thrift service. The type is string and is case-insensitive. This parameter needs to be consistent with the setting of fe's thrift_server_type parameter. Currently there are two values for this parameter, `THREADED` and `THREAD_POOL`.
+
+ - If the parameter is `THREADED`, the model is a non-blocking I/O model.
-* Type: int32
-* Description: Threshold to logging cumulative compaction's trace information, in seconds
-* Default value: 10
+ - If the parameter is `THREAD_POOL`, the model is a blocking I/O model.
-Similar to `base_compaction_trace_threshold`.
+#### `txn_commit_rpc_timeout_ms`
-### disable_compaction_trace_log
+* Description:txn submit rpc timeout
+* Default value: 10,000 (ms)
-* Type: bool
-* Description: disable the trace log of compaction
-* Default value: true
+#### `txn_map_shard_size`
-If set to true, the `cumulative_compaction_trace_threshold` and `base_compaction_trace_threshold` won't work and log is disabled.
+* Description: txn_map_lock fragment size, the value is 2^n, n=0,1,2,3,4. This is an enhancement to improve the performance of managing txn
+* Default value: 128
-### `compaction_promotion_size_mbytes`
+#### `txn_shard_size`
-* Type: int64
-* Description: The total disk size of the output rowset of cumulative compaction exceeds this configuration size, and the rowset will be used for base compaction. The unit is m bytes.
+* Description: txn_lock shard size, the value is 2^n, n=0,1,2,3,4, this is an enhancement function that can improve the performance of submitting and publishing txn
* Default value: 1024
-In general, if the configuration is less than 2G, in order to prevent the cumulative compression time from being too long, resulting in the version backlog.
-
-### `compaction_promotion_ratio`
-
-* Type: double
-* Description: When the total disk size of the cumulative compaction output rowset exceeds the configuration ratio of the base version rowset, the rowset will be used for base compaction.
-* Default value: 0.05
-
-Generally, it is recommended that the configuration should not be higher than 0.1 and lower than 0.02.
-
-### `compaction_promotion_min_size_mbytes`
-
-* Type: int64
-* Description: If the total disk size of the output rowset of the cumulative compaction is lower than this configuration size, the rowset will not undergo base compaction and is still in the cumulative compaction process. The unit is m bytes.
-* Default value: 64
-
-Generally, the configuration is within 512m. If the configuration is too large, the size of the early base version is too small, and base compaction has not been performed.
-
-### `compaction_min_size_mbytes`
-
-* Type: int64
-* Description: When the cumulative compaction is merged, the selected rowsets to be merged have a larger disk size than this configuration, then they are divided and merged according to the level policy. When it is smaller than this configuration, merge directly. The unit is m bytes.
-* Default value: 64
+#### `unused_rowset_monitor_interval`
-Generally, the configuration is within 128m. Over configuration will cause more cumulative compaction write amplification.
+* Description: Time interval for clearing expired Rowset
+* Default value: 30 (s)
-### `custom_config_dir`
+#### `max_client_cache_size_per_host`
-Configure the location of the `be_custom.conf` file. The default is in the `conf/` directory.
-
-In some deployment environments, the `conf/` directory may be overwritten due to system upgrades. This will cause the user modified configuration items to be overwritten. At this time, we can store `be_custom.conf` in another specified directory to prevent the configuration file from being overwritten.
+* Description: The maximum number of client caches per host. There are multiple client caches in BE, but currently we use the same cache size configuration. If necessary, use different configurations to set up different client-side caches
+* Default value: 10
-### `default_num_rows_per_column_file_block`
+#### `string_type_length_soft_limit_bytes`
* Type: int32
-* Description: Configure how many rows of data are contained in a single RowBlock.
-* Default value: 1024
+* Description: The soft limit of the maximum length of String type.
+* Default value: 1,048,576
-### `default_rowset_type`
+#### `big_column_size_buffer`
-* Type: string
-* Description: Identifies the storage format selected by BE by default. The configurable parameters are: "**ALPHA**", "**BETA**". Mainly play the following two roles
-1. When the storage_format of the table is set to Default, select the storage format of BE through this configuration.
-2. Select the storage format of when BE performing Compaction
-* Default value: BETA
-
-### `delete_worker_count`
-
-Default: 3
-
-Number of threads performing data deletion tasks
-
-### `disable_mem_pools`
-
-Default: false
+* Type: int64
+* Description: When using the odbc external table, if a column type of the odbc source table is HLL, CHAR or VARCHAR, and the length of the column value exceeds this value, the query will report an error 'column value length longer than buffer length'. You can increase this value
+* Default value: 65535
-Whether to disable the memory cache pool, it is not disabled by default
+#### `small_column_size_buffer`
-### `disable_storage_page_cache`
+* Type: int64
+* Description: When using the odbc external table, if a column type of the odbc source table is not HLL, CHAR or VARCHAR, and the length of the column value exceeds this value, the query will report an error 'column value length longer than buffer length'. You can increase this value
+* Default value: 100
-* Type: bool
-* Description: Disable to use page cache for index caching, this configuration only takes effect in BETA storage format, usually it is recommended to false
-* Default value: false
+### Query
-### `disk_stat_monitor_interval`
+#### `fragment_pool_queue_size`
-Default: 5（s）
+* Description: The upper limit of query requests that can be processed on a single node
+* Default value: 2048
-Disk status check interval
+#### `fragment_pool_thread_num_min`
-### `doris_cgroups`
+* Description: Query the number of threads. By default, the minimum number of threads is 64.
+* Default value: 64
-Default: empty
+#### `fragment_pool_thread_num_max`
-Cgroups assigned to doris
+* Description: Follow up query requests create threads dynamically, with a maximum of 256 threads created.
+* Default value: 256
-### `doris_max_pushdown_conjuncts_return_rate`
+#### `doris_max_pushdown_conjuncts_return_rate`
* Type: int32
-* Description: When BE performs HashJoin, it will adopt a dynamic partitioning method to push the join condition to OlapScanner. When the data scanned by OlapScanner is larger than 32768 rows, BE will check the filter condition. If the filter rate of the filter condition is lower than this configuration, Doris will stop using the dynamic partition clipping condition for data filtering.
+* Description: When BE performs HashJoin, it will adopt a dynamic partitioning method to push the join condition to OlapScanner. When the data scanned by OlapScanner is larger than 32768 rows, BE will check the filter condition. If the filter rate of the filter condition is lower than this configuration, Doris will stop using the dynamic partition clipping condition for data filtering.
* Default value: 90
-### `doris_max_scan_key_num`
+#### `doris_max_scan_key_num`
* Type: int
* Description: Used to limit the maximum number of scan keys that a scan node can split in a query request. When a conditional query request reaches the scan node, the scan node will try to split the conditions related to the key column in the query condition into multiple scan key ranges. After that, these scan key ranges will be assigned to multiple scanner threads for data scanning. A larger value usually means that more scanner threads can be used to increase the parallelism of the s [...]
+ - When the concurrency cannot be improved in high concurrency scenarios, try to reduce this value and observe the impact.
* Default value: 1024
-When the concurrency cannot be improved in high concurrency scenarios, try to reduce this value and observe the impact.
-
-### `doris_scan_range_row_count`
+#### `doris_scan_range_row_count`
* Type: int32
* Description: When BE performs data scanning, it will split the same scanning range into multiple ScanRanges. This parameter represents the scan data range of each ScanRange. This parameter can limit the time that a single OlapScanner occupies the io thread.
* Default value: 524288
-### `doris_scanner_queue_size`
+#### `doris_scanner_queue_size`
* Type: int32
* Description: The length of the RowBatch buffer queue between TransferThread and OlapScanner. When Doris performs data scanning, it is performed asynchronously. The Rowbatch scanned by OlapScanner will be placed in the scanner buffer queue, waiting for the upper TransferThread to take it away.
* Default value: 1024
-### `doris_scanner_row_num`
+#### `doris_scanner_row_num`
-Default: 16384
+* Description: The maximum number of data rows returned by each scanning thread in a single execution
+* Default value: 16384
-The maximum number of data rows returned by each scanning thread in a single execution
-
-### `doris_scanner_thread_pool_queue_size`
+#### `doris_scanner_thread_pool_queue_size`
* Type: int32
-* Description: The queue length of the Scanner thread pool. In Doris' scanning tasks, each Scanner will be submitted as a thread task to the thread pool waiting to be scheduled, and after the number of submitted tasks exceeds the length of the thread pool queue, subsequent submitted tasks will be blocked until there is a empty slot in the queue.
+* Description: The queue length of the Scanner thread pool. In Doris' scanning tasks, each Scanner will be submitted as a thread task to the thread pool waiting to be scheduled, and after the number of submitted tasks exceeds the length of the thread pool queue, subsequent submitted tasks will be blocked until there is a empty slot in the queue.
* Default value: 102400
-### `doris_scanner_thread_pool_thread_num`
+#### `doris_scanner_thread_pool_thread_num`
* Type: int32
* Description: The number of threads in the Scanner thread pool. In Doris' scanning tasks, each Scanner will be submitted as a thread task to the thread pool to be scheduled. This parameter determines the size of the Scanner thread pool.
* Default value: 48
-### `download_low_speed_limit_kbps`
-
-Default: 50 (KB/s)
-
-Minimum download speed
-
-### `download_low_speed_time`
-
-Default: 300（s）
-
-Download time limit, 300 seconds by default
-
-### `download_worker_count`
-
-Default: 1
-
-The number of download threads, the default is 1
-
-### `drop_tablet_worker_count`
-
-Default: 3
-
-Number of threads to delete tablet
-
-### `enable_metric_calculator`
-
-Default: true
-
-If set to true, the metric calculator will run to collect BE-related indicator information, if set to false, it will not run
-
-### `enable_partitioned_aggregation`
+#### `enable_prefetch`
-* Type: bool
-* Description: Whether the BE node implements the aggregation operation by PartitionAggregateNode, if false, AggregateNode will be executed to complete the aggregation. It is not recommended to set it to false in non-special demand scenarios.
-* Default value: true
-
-### `enable_prefetch`
* Type: bool
* Description: When using PartitionedHashTable for aggregation and join calculations, whether to perform HashBuket prefetch. Recommended to be set to true
* Default value: true
-### `enable_quadratic_probing`
+#### `enable_quadratic_probing`
* Type: bool
* Description: When a Hash conflict occurs when using PartitionedHashTable, enable to use the square detection method to resolve the Hash conflict. If the value is false, linear detection is used to resolve the Hash conflict. For the square detection method, please refer to: [quadratic_probing](https://en.wikipedia.org/wiki/Quadratic_probing)
* Default value: true
-### `enable_system_metrics`
-
-Default: true
-
-User control to turn on and off system indicators.
-
-### `enable_token_check`
-
-Default: true
-
-Used for forward compatibility, will be removed later.
-
-### `es_http_timeout_ms`
-
-Default: 5000 （ms）
-
-The timeout period for connecting to ES via http, the default is 5 seconds.
-
-### `enable_stream_load_record`
-
-Default: false
-
-Whether to enable stream load record function, the default is false，Disable stream load record.
-
-### `es_scroll_keepalive`
-
-Default: 5m
-
-es scroll Keeplive hold time, the default is 5 minutes
-
-### `etl_thread_pool_queue_size`
-
-Default: 256
-
-The size of the ETL thread pool
-
-### `etl_thread_pool_size`
-
-### `exchg_node_buffer_size_bytes`
+#### `exchg_node_buffer_size_bytes`
* Type: int32
* Description: The size of the Buffer queue of the ExchangeNode node, in bytes. After the amount of data sent from the Sender side is larger than the Buffer size of ExchangeNode, subsequent data sent will block until the Buffer frees up space for writing.
* Default value: 10485760
-### `file_descriptor_cache_capacity`
-
-Default: 32768
-
-File handle cache capacity, 32768 file handles are cached by default.
-
-### `cache_clean_interval`
+#### `max_pushdown_conditions_per_column`
-Default: 1800(s)
-
-File handle cache cleaning interval, used to clean up file handles that have not been used for a long time.
-Also the clean interval of Segment Cache.
-
-### `flush_thread_num_per_store`
-
-Default: 2
-
-The number of threads used to refresh the memory table per store
-
-### `force_recovery`
-
-### `fragment_pool_queue_size`
-
-Default: 2048
-
-The upper limit of query requests that can be processed on a single node
-
-### `fragment_pool_thread_num_min`
-
-Default: 64
+* Type: int
+* Description: Used to limit the maximum number of conditions that can be pushed down to the storage engine for a single column in a query request. During the execution of the query plan, the filter conditions on some columns can be pushed down to the storage engine, so that the index information in the storage engine can be used for data filtering, reducing the amount of data that needs to be scanned by the query. Such as equivalent conditions, conditions in IN predicates, etc. In most [...]
+* Default value: 1024
-### `fragment_pool_thread_num_max`
+* Example
+ - The table structure is' id INT, col2 INT, col3 varchar (32),... '.
+ - The query request is'WHERE id IN (v1, v2, v3, ...)
+#### `max_send_batch_parallelism_per_job`
-Default: 256
+* Type: int
+* Description: Max send batch parallelism for OlapTableSink. The value set by the user for `send_batch_parallelism` is not allowed to exceed `max_send_batch_parallelism_per_job`, if exceed, the value of `send_batch_parallelism` would be `max_send_batch_parallelism_per_job`.
+* Default value: 5
-The above two parameters are to set the number of query threads. By default, a minimum of 64 threads will be started, subsequent query requests will dynamically create threads, and a maximum of 256 threads will be created.
+#### `serialize_batch`
-### `heartbeat_service_port`
-* Type: int32
-* Description: Heartbeat service port (thrift) on BE, used to receive heartbeat from FE
-* Default value: 9050
+* Type: bool
+* Description: Whether the rpc communication between BEs serializes RowBatch for data transmission between query layers
+* Default value: false
-### `heartbeat_service_thread_count`
+#### `doris_scan_range_max_mb`
* Type: int32
-* Description: The number of threads that execute the heartbeat service on BE. the default is 1, it is not recommended to modify
-* Default value: 1
-
-### `ignore_broken_disk`
-
-Default: false
-
-When BE start, If there is a broken disk, BE process will exit by default.Otherwise, we will ignore the broken disk
-
-### `ignore_load_tablet_failure`
-When BE starts, it will check all the paths under the storage_root_path in configuration.
-
-`ignore_broken_disk=true`
-
-If the path does not exist or the file under the path cannot be read or written (broken disk), it will be ignored. If there are any other available paths, the startup will not be interrupted.
-
-`ignore_broken_disk=false`
-
-If the path does not exist or the file under the path cannot be read or written (bad disk), the startup will fail and exit.
-
-The default value is false.
-```
-load tablets from header failed, failed tablets size: xxx, path=xxx
-```
-
-Indicates how many tablets in the data directory failed to load. At the same time, the log will also contain specific information about the tablet that failed to load. At this time, manual intervention is required to troubleshoot the cause of the error. After investigation, there are usually two ways to recover:
-
-1. The tablet information cannot be repaired. If the other copies are normal, you can delete the wrong tablet with the `meta_tool` tool.
-2. Set `ignore_load_tablet_failure` to true, BE will ignore these faulty tablets and start normally
-
-### ignore_rowset_stale_unconsistent_delete
-
-* Type: boolean
-* Description:It is used to decide whether to delete the outdated merged rowset if it cannot form a consistent version path.
-* Default: false
-
-The merged expired rowset version path will be deleted after half an hour. In abnormal situations, deleting these versions will result in the problem that the consistent path of the query cannot be constructed. When the configuration is false, the program check is strict and the program will directly report an error and exit.
-When configured as true, the program will run normally and ignore this error. In general, ignoring this error will not affect the query, only when the merged version is dispatched by fe, -230 error will appear.
-
-### inc_rowset_expired_sec
-
-Default: 1800 （s）
-
-Import activated data, storage engine retention time, used for incremental cloning
-
-### `index_stream_cache_capacity`
-
-Default: 10737418240
-
-BloomFilter/Min/Max and other statistical information cache capacity
-
-### `kafka_api_version_request`
-
-Default: true
-
-If the dependent Kafka version is lower than 0.10.0.0, this value should be set to false.
-
-### `kafka_broker_version_fallback`
-
-Default: 0.10.0
-
-If the dependent Kafka version is lower than 0.10.0.0, the value set by the fallback version kafka_broker_version_fallback will be used if the value of kafka_api_version_request is set to false, and the valid values are: 0.9.0.x, 0.8.x.y.
-
-### `load_data_reserve_hours`
-
-Default: 4（hour）
-
-Used for mini load. The mini load data file will be deleted after this time
-
-### `load_error_log_reserve_hours`
-
-Default: 48 （hour）
-
-The load error log will be deleted after this time
-
-### `load_process_max_memory_limit_bytes`
-
-Default: 107374182400
-
-The upper limit of memory occupied by all imported threads on a single node, default value: 100G
-
-Set these default values very large, because we don't want to affect load performance when users upgrade Doris. If necessary, the user should set these configurations correctly.
-
-### `load_process_max_memory_limit_percent`
-
-Default: 50 (%)
-
-The percentage of the upper memory limit occupied by all imported threads on a single node, the default is 50%
-
-Set these default values very large, because we don't want to affect load performance when users upgrade Doris. If necessary, the user should set these configurations correctly
-
-### `load_process_soft_mem_limit_percent`
-
-Default: 50 (%)
-
-The soft limit refers to the proportion of the load memory limit of a single node. For example, the load memory limit for all load tasks is 20GB, and the soft limit defaults to 50% of this value, that is, 10GB. When the load memory usage exceeds the soft limit, the job with the largest memory consuption will be selected to be flushed to release the memory space, the default is 50%
-
-### `log_buffer_level`
-
-Default: empty
-
-The log flushing strategy is kept in memory by default
-
-### `madvise_huge_pages`
-
-Default: false
-
-Whether to use linux memory huge pages, not enabled by default
-
-### `make_snapshot_worker_count`
+* Description: The maximum amount of data read by each OlapScanner.
+* Default value: 1024
-Default: 5
+### compaction
-Number of threads making snapshots
+#### `disable_auto_compaction`
-### `max_client_cache_size_per_host`
+* Type: bool
+* Description: Whether disable automatic compaction task
+ - Generally it needs to be turned off. When you want to manually operate the compaction task in the debugging or test environment, you can turn on the configuration.
+* Default value: false
-Default: 10
+#### `enable_vectorized_compaction`
-The maximum number of client caches per host. There are multiple client caches in BE, but currently we use the same cache size configuration. If necessary, use different configurations to set up different client-side caches
+* Type: bool
+* Description: Whether to enable vectorized compaction
+* Default value: true
-### `max_base_compaction_threads`
+#### `max_base_compaction_threads`
* Type: int32
* Description: The maximum of thread number in base compaction thread pool.
* Default value: 4
-### `max_cumu_compaction_threads`
-
-* Type: int32
-* Description: The maximum of thread number in cumulative compaction thread pool.
-* Default value: 10
-
-### `max_consumer_num_per_group`
-
-Default: 3
-
-The maximum number of consumers in a data consumer group, used for routine load
-
-### `cumulative_compaction_min_deltas`
-
-Default: 5
-
-Cumulative compaction strategy: the minimum number of incremental files
-
-### `cumulative_compaction_max_deltas`
-
-Default: 1000
-
-Cumulative compaction strategy: the maximum number of incremental files
-
-### `max_download_speed_kbps`
-
-Default: 50000 （KB/s）
+#### `generate_compaction_tasks_interval_ms`
-Maximum download speed limit
+* Description: Minimal interval (ms) to generate compaction tasks
+* Default value: 10 (ms)
-### `max_free_io_buffers`
+#### `base_compaction_min_rowset_num`
-Default: 128
-
-For each io buffer size, the maximum number of buffers that IoMgr will reserve ranges from 1024B to 8MB buffers, up to about 2GB buffers.
-
-### `max_garbage_sweep_interval`
-
-Default: 3600
+* Description: One of the triggering conditions of BaseCompaction: The limit of the number of Cumulative files to be reached. After reaching this limit, BaseCompaction will be triggered
+* Default value: 5
-The maximum interval for disk garbage cleaning, the default is one hour
+#### `base_compaction_min_data_ratio`
-### `max_memory_sink_batch_count`
+* Description: One of the trigger conditions of BaseCompaction: Cumulative file size reaches the proportion of Base file
+* Default value: 0.3 (30%)
-Default: 20
+#### `total_permits_for_compaction_score`
-The maximum external scan cache batch count, which means that the cache max_memory_cache_batch_count * batch_size row, the default is 20, and the default value of batch_size is 1024, which means that 20 * 1024 rows will be cached
+* Type: int64
+* Description: The upper limit of "permits" held by all compaction tasks. This config can be set to limit memory consumption for compaction.
+* Default value: 10000
+* Dynamically modifiable: Yes
-### `max_percentage_of_error_disk`
+#### `compaction_tablet_compaction_score_factor`
* Type: int32
-* Description: The storage engine allows the percentage of damaged hard disks to exist. After the damaged hard disk exceeds the changed ratio, BE will automatically exit.
-* Default value: 0
+* Description: Coefficient for compaction score when calculating tablet score to find a tablet for compaction.
+* Default value: 1
-### `max_pushdown_conditions_per_column`
+#### `compaction_promotion_size_mbytes`
-* Type: int
-* Description: Used to limit the maximum number of conditions that can be pushed down to the storage engine for a single column in a query request. During the execution of the query plan, the filter conditions on some columns can be pushed down to the storage engine, so that the index information in the storage engine can be used for data filtering, reducing the amount of data that needs to be scanned by the query. Such as equivalent conditions, conditions in IN predicates, etc. In most [...]
+* Type: int64
+* Description: The total disk size of the output rowset of cumulative compaction exceeds this configuration size, and the rowset will be used for base compaction. The unit is m bytes.
+ - Generally, if the configuration is less than 2G, in order to prevent the cumulative compression time from being too long, resulting in the version backlog.
* Default value: 1024
-* Example
-
- The table structure is `id INT, col2 INT, col3 varchar (32), ...`.
-
- The query is `... WHERE id IN (v1, v2, v3, ...)`
-
- If the number of conditions in the IN predicate exceeds the configuration, try to increase the configuration value and observe whether the query response has improved.
-
-### `max_runnings_transactions_per_txn_map`
+#### `compaction_promotion_ratio`
-Default: 100
-
-Max number of txns for every txn_partition_map in txn manager, this is a self protection to avoid too many txns saving in manager
+* Type: double
+* Description: When the total disk size of the cumulative compaction output rowset exceeds the configuration ratio of the base version rowset, the rowset will be used for base compaction.
+ - Generally, it is recommended that the configuration should not be higher than 0.1 and lower than 0.02.
+* Default value: 0.05
-### `max_send_batch_parallelism_per_job`
+#### `compaction_promotion_min_size_mbytes`
-* Type: int
-* Description: Max send batch parallelism for OlapTableSink. The value set by the user for `send_batch_parallelism` is not allowed to exceed `max_send_batch_parallelism_per_job`, if exceed, the value of `send_batch_parallelism` would be `max_send_batch_parallelism_per_job`.
-* Default value: 5
+* Type: int64
+* Description: If the total disk size of the output rowset of the cumulative compaction is lower than this configuration size, the rowset will not undergo base compaction and is still in the cumulative compaction process. The unit is m bytes.
+ - Generally, the configuration is within 512m. If the configuration is too large, the size of the early base version is too small, and base compaction has not been performed.
+* Default value: 64
-### `max_tablet_num_per_shard`
+#### `compaction_min_size_mbytes`
-Default: 1024
+* Type: int64
+* Description: When the cumulative compaction is merged, the selected rowsets to be merged have a larger disk size than this configuration, then they are divided and merged according to the level policy. When it is smaller than this configuration, merge directly. The unit is m bytes.
+ - Generally, the configuration is within 128m. Over configuration will cause more cumulative compaction write amplification.
+* Default value: 64
-The number of sliced tablets, plan the layout of the tablet, and avoid too many tablet subdirectories in the repeated directory
+#### `default_rowset_type`
-### `max_tablet_version_num`
+* Type: string
+* Description: Identifies the storage format selected by BE by default. The configurable parameters are: "**ALPHA**", "**BETA**". Mainly play the following two roles
+ - When the storage_format of the table is set to Default, select the storage format of BE through this configuration.
+ - Select the storage format of when BE performing Compaction
+* Default value: BETA
-* Type: int
-* Description: Limit the number of versions of a single tablet. It is used to prevent a large number of version accumulation problems caused by too frequent import or untimely compaction. When the limit is exceeded, the import task will be rejected.
-* Default value: 500
+#### `cumulative_compaction_min_deltas`
-### `mem_limit`
+* Description: Cumulative compaction strategy: the minimum number of incremental files
+* Default value: 5
-* Type: string
-* Description: Limit the percentage of the server's maximum memory used by the BE process. It is used to prevent BE memory from occupying to many the machine's memory. This parameter must be greater than 0. When the percentage is greater than 100%, the value will default to 100%.
-* Default value: 80%
+#### `cumulative_compaction_max_deltas`
-### `memory_mode`
+* Description: Cumulative compaction strategy: the maximum number of incremental files
+* Default value: 1000
-* Type: string
-* Description: Control gc of tcmalloc, in performance mode doirs releases memory of tcmalloc cache when usgae >= 90% * mem_limit, otherwise, doris releases memory of tcmalloc cache when usage >= 50% * mem_limit;
-* Default value: performance
+#### `base_compaction_trace_threshold`
-### `max_sys_mem_available_low_water_mark_bytes`
+* Type: int32
+* Description: Threshold to logging base compaction's trace information, in seconds
+* Default value: 10
-* Type: int64
-* Description: The maximum low water mark of the system `/proc/meminfo/MemAvailable`, Unit byte, default 1.6G, actual low water mark=min(1.6G, MemTotal * 10%), avoid wasting too much memory on machines with large memory larger than 16G. Turn up max. On machines with more than 16G memory, more memory buffers will be reserved for Full GC. Turn down max. will use as much memory as possible.
-* Default value: 1717986918
+Base compaction is a long time cost background task, this configuration is the threshold to logging trace information. Trace information in log file looks like:
-### `memory_limitation_per_thread_for_schema_change_bytes`
+```
+W0610 11:26:33.804431 56452 storage_engine.cpp:552] Trace:
+0610 11:23:03.727535 (+ 0us) storage_engine.cpp:554] start to perform base compaction
+0610 11:23:03.728961 (+ 1426us) storage_engine.cpp:560] found best tablet 546859
+0610 11:23:03.728963 (+ 2us) base_compaction.cpp:40] got base compaction lock
+0610 11:23:03.729029 (+ 66us) base_compaction.cpp:44] rowsets picked
+0610 11:24:51.784439 (+108055410us) compaction.cpp:46] got concurrency lock and start to do compaction
+0610 11:24:51.784818 (+ 379us) compaction.cpp:74] prepare finished
+0610 11:26:33.359265 (+101574447us) compaction.cpp:87] merge rowsets finished
+0610 11:26:33.484481 (+125216us) compaction.cpp:102] output rowset built
+0610 11:26:33.484482 (+ 1us) compaction.cpp:106] check correctness finished
+0610 11:26:33.513197 (+ 28715us) compaction.cpp:110] modify rowsets finished
+0610 11:26:33.513300 (+ 103us) base_compaction.cpp:49] compaction finished
+0610 11:26:33.513441 (+ 141us) base_compaction.cpp:56] unused rowsets have been moved to GC queue
+Metrics: {"filtered_rows":0,"input_row_num":3346807,"input_rowsets_count":42,"input_rowsets_data_size":1256413170,"input_segments_num":44,"merge_rowsets_latency_us":101574444,"merged_rows":0,"output_row_num":3346807,"output_rowset_data_size":1228439659,"output_segments_num":6}
+```
-Default: 2147483648
+#### `cumulative_compaction_trace_threshold`
-Maximum memory allowed for a single schema change task
+* Type: int32
+* Description: Threshold to logging cumulative compaction's trace information, in seconds
+ - Similar to `base_compaction_trace_threshold`.
+* Default value: 2
-### `memory_maintenance_sleep_time_s`
+#### `compaction_task_num_per_disk`
-Default: 10
+* Type: int32
+* Description: The number of compaction tasks which execute in parallel for a disk(HDD).
+* Default value: 2
-Sleep time (in seconds) between memory maintenance iterations
+#### `compaction_task_num_per_fast_disk`
-### `memory_max_alignment`
+* Type: int32
+* Description: The number of compaction tasks which execute in parallel for a fast disk(SSD).
+* Default value: 4
-Default: 16
+#### `cumulative_compaction_rounds_for_each_base_compaction_round`
-Maximum alignment memory
+* Type: int32
+* Description: How many rounds of cumulative compaction for each round of base compaction when compaction tasks generation.
+* Default value: 9
-### `read_size`
+#### `cumulative_compaction_policy`
-Default: 8388608
+* Type: string
+* Description: Configure the merge strategy in the cumulative compression phase. Currently, two merge strategies are implemented, num_based and size_based
+ - For details, "ordinary" is the initial version of the cumulative compression consolidation policy. After a cumulative compression, the base compression process is directly performed. size_The general policy is the optimized version of the ordinary policy. Version merging can only be performed when the disk volume of the rowset is the same order of magnitude. After merging, qualified rowsets are promoted to the base compaction stage. In the case of a large number of small batch import [...]
+* Default value: size_based
-The read size is the read size sent to the os. There is a trade-off between latency and the whole process, getting to keep the disk busy but not introducing seeks. For 8 MB reads, random io and sequential io have similar performance
+#### `max_cumu_compaction_threads`
-### `min_buffer_size`
+* Type: int32
+* Description: The maximum of thread number in cumulative compaction thread pool.
+* Default value: 10
-Default: 1024
+#### `enable_segcompaction`
-Minimum read buffer size (in bytes)
+* Type: bool
+* Description: Enable to use segment compaction during loading
+* Default value: false
-### `min_compaction_threads`
+#### `segcompaction_threshold_segment_num`
* Type: int32
-* Description: The minimum of thread number in compaction thread pool.
+* Description: Trigger segcompaction if the num of segments in a rowset exceeds this threshold
* Default value: 10
-### `min_file_descriptor_number`
+#### `segcompaction_small_threshold`
-Default: 60000
+* Type: int32
+* Description: The segment whose row number above the threshold will be compacted during segcompaction
+* Default value: 1048576
-The lower limit required by the file handle limit of the BE process
+#### `disable_compaction_trace_log`
-### `min_garbage_sweep_interval`
+* Type: bool
+* Description: disable the trace log of compaction
+ - If set to true, the `cumulative_compaction_trace_threshold` and `base_compaction_trace_threshold` won't work and log is disabled.
+* Default value: true
-Default: 180
-The minimum interval between disk garbage cleaning, time seconds
+### Load
-### `mmap_buffers`
+#### `enable_stream_load_record`
-Default: false
+* Type: bool
+* Description:Whether to enable stream load record function, the default is false.
+* Default value: false
-Whether to use mmap to allocate memory, not used by default
+#### `load_data_reserve_hours`
-### `num_cores`
+* Description: Used for mini load. The mini load data file will be deleted after this time
+* Default value: 4 (h)
-* Type: int32
-* Description: The number of CPU cores that BE can use. When the value is 0, BE will obtain the number of CPU cores of the machine from /proc/cpuinfo.
-* Default value: 0
+#### `push_worker_count_high_priority`
-### `num_disks`
+* Description: Import the number of threads for processing HIGH priority tasks
+* Default value: 3
-Defalut: 0
+#### `push_worker_count_normal_priority`
-Control the number of disks on the machine. If it is 0, it comes from the system settings
+* Description: Import the number of threads for processing NORMAL priority tasks
+* Default value: 3
-### `num_threads_per_core`
+#### `load_error_log_reserve_hours`
-Default: 3
+* Description: The load error log will be deleted after this time
+* Default value: 48 (h)
-Control the number of threads that each core runs. Usually choose 2 times or 3 times the number of cores. This keeps the core busy without causing excessive jitter
+#### `load_process_max_memory_limit_bytes`
-### `num_threads_per_disk`
+* Description: The upper limit of memory occupied by all imported threads on a single node.
+ - Set these default values very large, because we don't want to affect load performance when users upgrade Doris. If necessary, the user should set these configurations correctly.
+* Default value: 107374182400 (100G)
-Default: 0
+#### `load_process_max_memory_limit_percent`
-The maximum number of threads per disk is also the maximum queue depth of each disk
+* Description: The percentage of the upper memory limit occupied by all imported threads on a single node, the default is 50%
+ - Set these default values very large, because we don't want to affect load performance when users upgrade Doris. If necessary, the user should set these configurations correctly
+* Default value: 50 (%)
-### `number_slave_replica_download_threads`
+#### `load_process_soft_mem_limit_percent`
-Default: 64
+* Description: The soft limit refers to the proportion of the load memory limit of a single node. For example, the load memory limit for all load tasks is 20GB, and the soft limit defaults to 50% of this value, that is, 10GB. When the load memory usage exceeds the soft limit, the job with the largest memory consuption will be selected to be flushed to release the memory space, the default is 50%
+* Default value: 50 (%)
-Number of threads for slave replica synchronize data, used for single replica load.
+#### `routine_load_thread_pool_size`
-### `number_tablet_writer_threads`
+* Description: The thread pool size of the routine load task. This should be greater than the FE configuration'max_concurrent_task_num_per_be'
+* Default value: 10
-Default: 16
+#### `single_replica_load_brpc_num_threads`
-Number of tablet write threads
+* Type: int32
+* Description: This configuration is mainly used to modify the number of bthreads for single replica load brpc. When the load concurrency increases, you can adjust this parameter to ensure that the Slave replica synchronizes data files from the Master replica timely.
+* Default value: 64
-### `path_gc_check`
+#### `single_replica_load_download_num_workers`
-Default: true
+* Type: int32
+* Description:This configuration is mainly used to modify the number of http threads for segment download, used for single replica load. When the load concurrency increases, you can adjust this parameter to ensure that the Slave replica synchronizes data files from the Master replica timely.
+* Default value: 64
-Whether to enable the recycle scan data thread check, it is enabled by default
+#### `slave_replica_writer_rpc_timeout_sec`
-### `path_gc_check_interval_second`
+* Type: int32
+* Description: This configuration is mainly used to modify timeout of brpc between master replica and slave replica, used for single replica load.
+* Default value: 60
-Default: 86400
+#### `max_segment_num_per_rowset`
-Recycle scan data thread check interval, in seconds
+* Type: int32
+* Description: Used to limit the number of segments in the newly generated rowset when importing. If the threshold is exceeded, the import will fail with error -238. Too many segments will cause compaction to take up a lot of memory and cause OOM errors.
+* Default value: 200
-### `path_gc_check_step`
+#### `high_priority_flush_thread_num_per_store`
-Default: 1000
+* Type: int32
+* Description: The number of flush threads per store path allocated for the high priority import task.
+* Default value: 1
-### `path_gc_check_step_interval_ms`
+#### `routine_load_consumer_pool_size`
-Default: 10 (ms)
+* Type: int32
+* Description: The number of caches for the data consumer used by the routine load.
+* Default value: 10
-### `path_scan_interval_second`
+#### `load_task_high_priority_threshold_second`
-Default: 86400
+* Type: int32
+* Description: When the timeout of an import task is less than this threshold, Doris will consider it to be a high priority task. High priority tasks use a separate pool of flush threads.
+* Default: 120
-### `pending_data_expire_time_sec`
+#### `min_load_rpc_timeout_ms`
-Default: 1800
+* Type: int32
+* Description: The minimum timeout for each rpc in the load job.
+* Default: 20
-The maximum duration of unvalidated data retained by the storage engine, the default unit: seconds
+#### `kafka_api_version_request`
-### `periodic_counter_update_period_ms`
+* Type: bool
+* Description: If the dependent Kafka version is lower than 0.10.0.0, this value should be set to false.
+* Default: true
-Default: 500
+#### `kafka_broker_version_fallback`
-Update rate counter and sampling counter cycle, default unit: milliseconds
+* Description: If the dependent Kafka version is lower than 0.10.0.0, the value set by the fallback version kafka_broker_version_fallback will be used if the value of kafka_api_version_request is set to false, and the valid values are: 0.9.0.x, 0.8.x.y.
+* Default: 0.10.0
-### `plugin_path`
+#### `max_consumer_num_per_group`
-Default: ${DORIS_HOME}/plugin
+* Description: The maximum number of consumers in a data consumer group, used for routine load
+* Default: 3
-pliugin path
+#### `streaming_load_max_mb`
-### `port`
+* Type: int64
+* Description: Used to limit the maximum amount of csv data allowed in one Stream load.
+ - Stream Load is generally suitable for loading data less than a few GB, not suitable for loading` too large data.
+* Default value: 10240 (MB)
+* Dynamically modifiable: Yes
-* Type: int32
-* Description: The port used in UT. Meaningless in the actual environment and can be ignored.
-* Default value: 20001
+#### `streaming_load_json_max_mb`
-### `pprof_profile_dir`
+* Type: int64
+* Description: it is used to limit the maximum amount of json data allowed in one Stream load. The unit is MB.
+ - Some data formats, such as JSON, cannot be split. Doris must read all the data into the memory before parsing can begin. Therefore, this value is used to limit the maximum amount of data that can be loaded in a single Stream load.
+* Default value: 100
+* Dynamically modifiable: Yes
-Default : ${DORIS_HOME}/log
+### Thread
-pprof profile save directory
+#### `delete_worker_count`
-### `priority_networks`
+* Description: Number of threads performing data deletion tasks
+* Default value: 3
-Default: empty
+#### `clear_transaction_task_worker_count`
-Declare a selection strategy for those servers with many IPs. Note that at most one ip should match this list. This is a semicolon-separated list in CIDR notation, such as 10.10.10.0/24. If there is no IP matching this rule, one will be randomly selected
+* Description: Number of threads used to clean up transactions
+* Default value: 1
-### `priority_queue_remaining_tasks_increased_frequency`
+#### `clone_worker_count`
-Default: 512
+* Description: Number of threads used to perform cloning tasks
+* Default value: 3
- the increased frequency of priority for remaining tasks in BlockingPriorityQueue
+#### `be_service_threads`
-### `publish_version_worker_count`
+* Type: int32
+* Description: The number of execution threads of the thrift server service on BE which represents the number of threads that can be used to execute FE requests.
+* Default value: 64
-Default: 8
+#### `download_worker_count`
-the count of thread to publish version
+* Description: The number of download threads.
+* Default value: 1
-### `pull_load_task_dir`
+#### `drop_tablet_worker_count`
-Default: ${DORIS_HOME}/var/pull_load
+* Description: Number of threads to delete tablet
+* Default value: 3
-Pull the directory of the laod task
+#### `flush_thread_num_per_store`
-### `push_worker_count_high_priority`
+* Description: The number of threads used to refresh the memory table per store
+* Default value: 2
-Default: 3
+#### `num_threads_per_core`
-Import the number of threads for processing HIGH priority tasks
+* Description: Control the number of threads that each core runs. Usually choose 2 times or 3 times the number of cores. This keeps the core busy without causing excessive jitter
+* Default value: 3
-### `push_worker_count_normal_priority`
+#### `num_threads_per_disk`
-Default: 3
+* Description: The maximum number of threads per disk is also the maximum queue depth of each disk
+* Default value: 0
-Import the number of threads for processing NORMAL priority tasks
+#### `number_slave_replica_download_threads`
-### `query_scratch_dirs`
+* Description: Number of threads for slave replica synchronize data, used for single replica load.
+* Default value: 64
-+ Type: string
-+ Description: The directory selected by BE to store temporary data during spill to disk. which is similar to the storage path configuration, multiple directories are separated by ;.
-+ Default value: ${DORIS_HOME}
+#### `publish_version_worker_count`
-### `release_snapshot_worker_count`
+* Description: the count of thread to publish version
+* Default value: 8
-Default: 5
+#### `upload_worker_count`
-Number of threads releasing snapshots
+* Description: Maximum number of threads for uploading files
+* Default value: 1
-### `report_disk_state_interval_seconds`
+#### `webserver_num_workers`
-Default: 60
+* Description: Webserver default number of worker threads
+* Default value: 48
-The interval time for the agent to report the disk status to FE, unit (seconds)
+#### `send_batch_thread_pool_thread_num`
-### `report_tablet_interval_seconds`
+* Type: int32
+* Description: The number of threads in the SendBatch thread pool. In NodeChannels' sending data tasks, the SendBatch operation of each NodeChannel will be submitted as a thread task to the thread pool to be scheduled. This parameter determines the size of the SendBatch thread pool.
+* Default value: 64
-Default: 60
+#### `send_batch_thread_pool_queue_size`
-The interval time for the agent to report the olap table to the FE, in seconds
+* Type: int32
+* Description: The queue length of the SendBatch thread pool. In NodeChannels' sending data tasks, the SendBatch operation of each NodeChannel will be submitted as a thread task to the thread pool waiting to be scheduled, and after the number of submitted tasks exceeds the length of the thread pool queue, subsequent submitted tasks will be blocked until there is a empty slot in the queue.
+* Default value: 102400
-### `report_task_interval_seconds`
+#### `make_snapshot_worker_count`
-Default: 10
+* Description: Number of threads making snapshots
+* Default value: 5
-The interval time for the agent to report the task signature to FE, unit (seconds)
+#### `release_snapshot_worker_count`
-### `result_buffer_cancelled_interval_time`
+* Description: Number of threads releasing snapshots
+* Default value: 5
-Default: 300
+### Memory
-Result buffer cancellation time (unit: second)
+#### `disable_mem_pools`
-### `routine_load_thread_pool_size`
+* Type: bool
+* Description: Whether to disable the memory cache pool.
+* Default value: false
-Default: 10
+#### `buffer_pool_clean_pages_limit`
-The thread pool size of the routine load task. This should be greater than the FE configuration'max_concurrent_task_num_per_be' (default 5)
+* Description: Clean up pages that may be saved by the buffer pool
+* Default value: 50%
-### `row_nums_check`
+#### `buffer_pool_limit`
-Default: true
+* Type: string
+* Description: The largest allocatable memory of the buffer pool
+ - The maximum amount of memory available in the BE buffer pool. The buffer pool is a new memory management structure of BE, which manages the memory by the buffer page and enables spill data to disk. The memory for all concurrent queries will be allocated from the buffer pool. The current buffer pool only works on **AggregationNode** and **ExchangeNode**.
+* Default value: 20%
-Check row nums for BE/CE and schema change. true is open, false is closed
+#### `chunk_reserved_bytes_limit`
-### `scan_context_gc_interval_min`
+* Description: The reserved bytes limit of Chunk Allocator, usually set as a percentage of mem_limit. defaults to bytes if no unit is given, the number of bytes must be a multiple of 2. must larger than 0. and if larger than physical memory size, it will be set to physical memory size. increase this variable can improve performance, but will acquire more free memory which can not be used by other modules.
+* Default value: 20%
-Default: 5
+#### `madvise_huge_pages`
-This configuration is used for the context gc thread scheduling cycle. Note: The unit is minutes, and the default is 5 minutes
+* Type: bool
+* Description: Whether to use linux memory huge pages.
+* Default value: false
-### `send_batch_thread_pool_thread_num`
+#### `max_memory_sink_batch_count`
-* Type: int32
-* Description: The number of threads in the SendBatch thread pool. In NodeChannels' sending data tasks, the SendBatch operation of each NodeChannel will be submitted as a thread task to the thread pool to be scheduled. This parameter determines the size of the SendBatch thread pool.
-* Default value: 64
+* Description: The maximum external scan cache batch count, which means that the cache max_memory_cache_batch_count * batch_size row, the default is 20, and the default value of batch_size is 1024, which means that 20 * 1024 rows will be cached
+* Default value: 20
-### `send_batch_thread_pool_queue_size`
+#### `memory_limitation_per_thread_for_schema_change`
-* Type: int32
-* Description: The queue length of the SendBatch thread pool. In NodeChannels' sending data tasks, the SendBatch operation of each NodeChannel will be submitted as a thread task to the thread pool waiting to be scheduled, and after the number of submitted tasks exceeds the length of the thread pool queue, subsequent submitted tasks will be blocked until there is a empty slot in the queue.
-* Default value: 102400
+* Description: The maximum memory allowed for a single schema change task
+* Default value: 2 (GB)
-### `download_cache_thread_pool_thread_num`
+#### `memory_max_alignment`
-* Type: int32
-* Description: The number of threads in the DownloadCache thread pool. In the download cache task of FileCache, the download cache operation will be submitted to the thread pool as a thread task and wait to be scheduled. This parameter determines the size of the DownloadCache thread pool.
-* Default value: 48
+* Description: Maximum alignment memory
+* Default value: 16
-### `download_cache_thread_pool_queue_size`
+#### `mmap_buffers`
-* Type: int32
-* Description: The number of threads in the DownloadCache thread pool. In the download cache task of FileCache, the download cache operation will be submitted to the thread pool as a thread task and wait to be scheduled. After the number of submitted tasks exceeds the length of the thread pool queue, subsequent submitted tasks will be blocked until there is a empty slot in the queue.
-* Default value: 102400
+* Description: Whether to use mmap to allocate memory
+* Default value: false
-### `download_cache_buffer_size`
+#### `download_cache_buffer_size`
* Type: int64
* Description: The size of the buffer used to receive data when downloading the cache.
* Default value: 10485760
-### `single_replica_load_brpc_port`
+#### `zone_map_row_num_threshold`
* Type: int32
-* Description: The port of BRPC on BE, used for single replica load. There is a independent BRPC thread pool for the communication between the Master replica and Slave replica during single replica load, which prevents data synchronization between the replicas from preempt the thread resources for data distribution and query tasks when the load concurrency is large.
-* Default value: 9070
-
-### `single_replica_load_brpc_num_threads`
+* Description: If the number of rows in a page is less than this value, no zonemap will be created to reduce data expansion
+* Default value: 20
-* Type: int32
-* Description: This configuration is mainly used to modify the number of bthreads for single replica load brpc. When the load concurrency increases, you can adjust this parameter to ensure that the Slave replica synchronizes data files from the Master replica timely.
-* Default value: 64
+#### `enable_tcmalloc_hook`
-### `single_replica_load_download_port`
+* Type: bool
+* Description: Whether Hook TCmalloc new/delete, currently consume/release tls mem tracker in Hook.
+* Default value: true
-* Type: int32
-* Description: The port of http for segment download on BE, used for single replica load. There is a independent HTTP thread pool for the Slave replica to download segments during single replica load, which prevents data synchronization between the replicas from preempt the thread resources for other http tasks when the load concurrency is large.
-* Default value: 8050
+#### `memory_mode`
-### `single_replica_load_download_num_workers`
+* Type: string
+* Description: Control gc of tcmalloc, in performance mode doirs releases memory of tcmalloc cache when usgae >= 90% * mem_limit, otherwise, doris releases memory of tcmalloc cache when usage >= 50% * mem_limit;
+* Default value: performance
-* Type: int32
-* Description: This configuration is mainly used to modify the number of http threads for segment download, used for single replica load. When the load concurrency increases, you can adjust this parameter to ensure that the Slave replica synchronizes data files from the Master replica timely.
-* Default value: 64
+#### `max_sys_mem_available_low_water_mark_bytes`
-### `slave_replica_writer_rpc_timeout_sec`
+* Type: int64
+* Description: The maximum low water mark of the system `/proc/meminfo/MemAvailable`, Unit byte, default 1.6G, actual low water mark=min(1.6G, MemTotal * 10%), avoid wasting too much memory on machines with large memory larger than 16G. Turn up max. On machines with more than 16G memory, more memory buffers will be reserved for Full GC. Turn down max. will use as much memory as possible.
+* Default value: 1717986918
-* Type: int32
-* Description: This configuration is mainly used to modify timeout of brpc between master replica and slave replica, used for single replica load.
-* Default value: 60
+#### `memory_limitation_per_thread_for_schema_change_bytes`
-### `sleep_one_second`
+* Description: Maximum memory allowed for a single schema change task
+* Default value: 2147483648
-+ Type: int32
-+ Description: Global variables, used for BE thread sleep for 1 seconds, should not be modified
-+ Default value: 1
+#### `mem_tracker_consume_min_size_bytes`
-### `small_file_dir`
+* Type: int32
+* Description: The minimum length of TCMalloc Hook when consume/release MemTracker. Consume size smaller than this value will continue to accumulate to avoid frequent calls to consume/release of MemTracker. Decreasing this value will increase the frequency of consume/release. Increasing this value will cause MemTracker statistics to be inaccurate. Theoretically, the statistical value of a MemTracker differs from the true value = ( mem_tracker_consume_min_size_bytes * the number of BE thr [...]
+* Default value: 1,048,576
-Default: ${DORIS_HOME}/lib/small_file/
+#### `cache_clean_interval`
-Directory for saving files downloaded by SmallFileMgr
+* Description: File handle cache cleaning interval, used to clean up file handles that have not been used for a long time.Also the clean interval of Segment Cache.
+* Default value: 1800 (s)
-### `snapshot_expire_time_sec`
+#### `min_buffer_size`
-Default: 172800
+* Description: Minimum read buffer size
+* Default value: 1024 (byte)
-Snapshot file cleaning interval, default value: 48 hours
+#### `write_buffer_size`
-### `status_report_interval`
+* Description: The size of the buffer before flashing
+ - Imported data is first written to a memory block on the BE, and only written back to disk when this memory block reaches the threshold. The default size is 100MB. too small a threshold may result in a large number of small files on the BE. This threshold can be increased to reduce the number of files. However, too large a threshold may cause RPC timeouts
+* Default value: 104,857,600
-Default: 5
+#### `remote_storage_read_buffer_mb`
-Interval between profile reports; unit: seconds
+* Type: int32
+* Description: The cache size used when reading files on hdfs or object storage.
+ - Increasing this value can reduce the number of calls to read remote data, but it will increase memory overhead.
+* Default value: 16MB
-### `storage_flood_stage_left_capacity_bytes`
+#### `segment_cache_capacity`
-Default: 1073741824
+* Type: int32
+* Description: The maximum number of Segments cached by Segment Cache.
+ - The default value is currently only an empirical value, and may need to be modified according to actual scenarios. Increasing this value can cache more segments and avoid some IO. Decreasing this value will reduce memory usage.
+* Default value: 1000000
-The min bytes that should be left of a data dir，default value:1G
+#### `file_cache_type`
-### `storage_flood_stage_usage_percent`
+* Type: string
+* Description: Type of cache file. whole_ file_ Cache: download the entire segment file, sub_ file_ Cache: the segment file is divided into multiple files by size.
+* Default value: null
-Default: 90 （90%）
+### `file_cache_alive_time_sec`
-The storage_flood_stage_usage_percent and storage_flood_stage_left_capacity_bytes configurations limit the maximum usage of the capacity of the data directory.
+* Type: int64
+* Description: Save time of cache file
+* Default value: 604800 (1 week)
-### `storage_medium_migrate_count`
+#### `max_sub_cache_file_size`
-Default: 1
+* Type: int64
+* Description: Cache files using sub_ file_ The maximum size of the split file during cache
+* Default value: 104857600 (100MB)
-the count of thread to clone
+#### `download_cache_thread_pool_thread_num`
-### `storage_page_cache_limit`
+* Type: int32
+* Description: The number of threads in the DownloadCache thread pool. In the download cache task of FileCache, the download cache operation will be submitted to the thread pool as a thread task and wait to be scheduled. This parameter determines the size of the DownloadCache thread pool.
+* Default value: 48
-Default: 20%
-Cache for storage page size
+#### `download_cache_thread_pool_queue_size`
-### `storage_page_cache_shard_size`
+* Type: int32
+* Description: The number of threads in the DownloadCache thread pool. In the download cache task of FileCache, the download cache operation will be submitted to the thread pool as a thread task and wait to be scheduled. After the number of submitted tasks exceeds the length of the thread pool queue, subsequent submitted tasks will be blocked until there is a empty slot in the queue.
+* Default value: 102400
-Default: 16
+#### `generate_cache_cleaner_task_interval_sec`
-Shard size of StoragePageCache, the value must be power of two. It's recommended to set it to a value close to the number of BE cores in order to reduce lock contentions.
+* Type：int64
+* Description：Cleaning interval of cache files, in seconds
+* Default：43200（12 hours）
-### `index_page_cache_percentage`
-* Type: int32
-* Description: Index page cache as a percentage of total storage page cache, value range is [0, 100]
-* Default value: 10
+#### `path_gc_check`
-### `storage_root_path`
+* Type：bool
+* Description：Whether to enable the recycle scan data thread check
+* Default：true
-* Type: string
+#### `path_gc_check_interval_second`
-* Description: data root path, separate by ';'.you can specify the storage medium of each root path, HDD or SSD. you can add capacity limit at the end of each root path, separate by ','.
- If the user does not use a mix of SSD and HDD disks, they do not need to configure the configuration methods in Example 1 and Example 2 below, but only need to specify the storage directory; they also do not need to modify the default storage media configuration of FE.
+* Description：Recycle scan data thread check interval
+* Default：86400 (s)
- eg.1: `storage_root_path=/home/disk1/doris.HDD;/home/disk2/doris.SSD;/home/disk2/doris`
-
- * 1./home/disk1/doris.HDD, indicates that the storage medium is HDD;
- * 2./home/disk2/doris.SSD, indicates that the storage medium is SSD;
- * 3./home/disk2/doris, indicates that the storage medium is HDD by default
-
- eg.2: `storage_root_path=/home/disk1/doris,medium:hdd;/home/disk2/doris,medium:ssd`
-
- * 1./home/disk1/doris,medium:hdd，indicates that the storage medium is HDD;
- * 2./home/disk2/doris,medium:ssd，indicates that the storage medium is SSD;
+#### `path_gc_check_step`
-* Default: ${DORIS_HOME}
+* Default：1000
-### `storage_strict_check_incompatible_old_format`
-* Type: bool
-* Description: Used to check incompatible old format strictly
-* Default value: true
-* Dynamically modify: false
+#### `path_gc_check_step_interval_ms`
-This config is used to check incompatible old format hdr_ format whether doris uses strict way. When config is true,
-process will log fatal and exit. When config is false, process will only log warning.
+* Default：10 (ms)
-### `streaming_load_max_mb`
+#### `path_scan_interval_second`
-* Type: int64
-* Description: Used to limit the maximum amount of csv data allowed in one Stream load. The unit is MB.
-* Default value: 10240
-* Dynamically modify: yes
+* Default：86400
-Stream Load is generally suitable for loading data less than a few GB, not suitable for loading` too large data.
+#### `scan_context_gc_interval_min`
-### `streaming_load_json_max_mb`
+* Description：This configuration is used for the context gc thread scheduling cycle. Note: The unit is minutes, and the default is 5 minutes
+* Default：5
-* Type: int64
-* Description: it is used to limit the maximum amount of json data allowed in one Stream load. The unit is MB.
-* Default value: 100
-* Dynamically modify: yes
+### Storage
-Some data formats, such as JSON, cannot be split. Doris must read all the data into the memory before parsing can begin. Therefore, this value is used to limit the maximum amount of data that can be loaded in a single Stream load.
+#### `default_num_rows_per_column_file_block`
-### `streaming_load_rpc_max_alive_time_sec`
+* Type: int32
+* Description: Configure how many rows of data are contained in a single RowBlock.
+* Default value: 1024
-Default: 1200
+#### `disable_storage_page_cache`
-The lifetime of TabletsChannel. If the channel does not receive any data at this time, the channel will be deleted, unit: second
+* Type: bool
+* Description: Disable to use page cache for index caching, this configuration only takes effect in BETA storage format, usually it is recommended to false
+* Default value: false
-### `sync_tablet_meta`
+#### `disk_stat_monitor_interval`
-Default: false
+* Description: Disk status check interval
+* Default value: 5（s）
-Whether the storage engine opens sync and keeps it to the disk
+#### `max_free_io_buffers`
-### `sys_log_dir`
+* Description: For each io buffer size, the maximum number of buffers that IoMgr will reserve ranges from 1024B to 8MB buffers, up to about 2GB buffers.
+* Default value: 128
-* Type: string
-* Description: Storage directory of BE log data
-* Default: ${DORIS_HOME}/log
+#### `max_garbage_sweep_interval`
-### `sys_log_level`
+* Description: The maximum interval for disk garbage cleaning
+* Default value: 3600 (s)
-INFO
+#### `max_percentage_of_error_disk`
-Log Level: INFO < WARNING < ERROR < FATAL
+* Type: int32
+* Description: The storage engine allows the percentage of damaged hard disks to exist. After the damaged hard disk exceeds the changed ratio, BE will automatically exit.
+* Default value: 0
-### `sys_log_roll_mode`
+#### `read_size`
-Default: SIZE-MB-1024
+* Description: The read size is the read size sent to the os. There is a trade-off between latency and the whole process, getting to keep the disk busy but not introducing seeks. For 8 MB reads, random io and sequential io have similar performance.
+* Default value: 8388608
-The size of the log split, one log file is split every 1G
+#### `min_garbage_sweep_interval`
-### `sys_log_roll_num`
+* Description: The minimum interval between disk garbage cleaning
+* Default value: 180 (s)
-Default: 10
+#### `pprof_profile_dir`
-Number of log files kept
+* Description: pprof profile save directory
+* Default value: ${DORIS_HOME}/log
-### `sys_log_verbose_level`
+#### `small_file_dir`
-Defaultl: 10
+* Description: 用于保存 SmallFileMgr 下载的文件的目录
+* Default value: ${DORIS_HOME}/lib/small_file/
-Log display level, used to control the log output at the beginning of VLOG in the code
+#### `user_function_dir`
-### `sys_log_verbose_modules`
+* Description: udf function directory
+* Default value: ${DORIS_HOME}/lib/udf
-Default: empty
+#### `storage_flood_stage_left_capacity_bytes`
-Log printing module, writing olap will only print the log under the olap module
+* Description: The min bytes that should be left of a data dir.
+* Default value: 1073741824
-### `tablet_map_shard_size`
+#### `storage_flood_stage_usage_percent`
-Default: 1
+* Description: The storage_flood_stage_usage_percent and storage_flood_stage_left_capacity_bytes configurations limit the maximum usage of the capacity of the data directory.
+* Default value: 90 （90%）
-tablet_map_lock fragment size, the value is 2^n, n=0,1,2,3,4, this is for better tablet management
+#### `storage_medium_migrate_count`
-### `tablet_meta_checkpoint_min_interval_secs`
+* Description: the count of thread to clone
+* Default value: 1
-Default: 600（s）
+#### `storage_page_cache_limit`
-The polling interval of the TabletMeta Checkpoint thread
+* Description: Cache for storage page size
+* Default value: 20%
-### `tablet_meta_checkpoint_min_new_rowsets_num`
+#### `storage_page_cache_shard_size`
-### `tablet_scan_frequency_time_node_interval_second`
+* Description: Shard size of StoragePageCache, the value must be power of two. It's recommended to set it to a value close to the number of BE cores in order to reduce lock contentions.
+* Default value: 16
-* Type: int64
-* Description: Time interval to record the metric 'query_scan_count' and timestamp in second for the purpose of calculating tablet scan frequency during a latest period of time at the present.
-* Default: 300
+#### `index_page_cache_percentage`
-### `tablet_stat_cache_update_interval_second`
+* Type: int32
+* Description: Index page cache as a percentage of total storage page cache, value range is [0, 100]
+* Default value: 10
-default: 10
+#### `storage_strict_check_incompatible_old_format`
-The minimum number of Rowsets for TabletMeta Checkpoint
+* Type: bool
+* Description: Used to check incompatible old format strictly
+* Default value: true
+* Dynamically modify: false
-### `tablet_rowset_stale_sweep_time_sec`
+#### `sync_tablet_meta`
-* Type: int64
-* Description: It is used to control the expiration time of cleaning up the merged rowset version. When the current time now() minus the max created rowset‘s create time in a version path is greater than tablet_rowset_stale_sweep_time_sec, the current path is cleaned up and these merged rowsets are deleted, the unit is second.
-* Default: 1800
+* Description: Whether the storage engine opens sync and keeps it to the disk
+* Default value: false
-When writing is too frequent and the disk time is insufficient, you can configure less tablet_rowset_stale_sweep_time_sec. However, if this time is less than 5 minutes, it may cause fe to query the version that has been merged, causing a query -230 error.
+#### `pending_data_expire_time_sec`
-### `tablet_writer_open_rpc_timeout_sec`
+* Description: The maximum duration of unvalidated data retained by the storage engine
+* Default value: 1800 (s)
-Default: 300
+#### `ignore_rowset_stale_unconsistent_delete`
-Update interval of tablet state cache, unit: second
+* Type: boolean
+* Description:It is used to decide whether to delete the outdated merged rowset if it cannot form a consistent version path.
+ - The merged expired rowset version path will be deleted after half an hour. In abnormal situations, deleting these versions will result in the problem that the consistent path of the query cannot be constructed. When the configuration is false, the program check is strict and the program will directly report an error and exit.When configured as true, the program will run normally and ignore this error. In general, ignoring this error will not affect the query, only when the merged ver [...]
+* Default value: false
-The RPC timeout for sending a Batch (1024 lines) during import. The default is 60 seconds. Since this RPC may involve writing multiple batches of memory, the RPC timeout may be caused by writing batches, so this timeout can be adjusted to reduce timeout errors (such as send batch fail errors). Also, if you increase the write_buffer_size configuration, you need to increase this parameter as well.
+#### `create_tablet_worker_count`
-### `tablet_writer_ignore_eovercrowded`
+* Description: Number of worker threads for BE to create a tablet
+* Default value: 3
-* Type: bool
-* Description: Used to ignore brpc error '[E1011]The server is overcrowded' when writing data.
-* Default value: false
+#### `check_consistency_worker_count`
-When meet '[E1011]The server is overcrowded' error, you can tune the configuration `brpc_socket_max_unwritten_bytes`, but it can't be modified at runtime. Set it to `true` to avoid writing failed temporarily. Notice that, it only effects `write`, other rpc requests will still check if overcrowded.
+* Description: The number of worker threads to calculate the checksum of the tablet
+* Default value: 1
-### `thrift_client_retry_interval_ms`
+#### `max_tablet_version_num`
-* Type: int64
-* Description: Used to set retry interval for thrift client in be to avoid avalanche disaster in fe thrift server, the unit is ms.
-* Default: 1000
+* Type: int
+* Description: Limit the number of versions of a single tablet. It is used to prevent a large number of version accumulation problems caused by too frequent import or untimely compaction. When the limit is exceeded, the import task will be rejected.
+* Default value: 500
-### `thrift_connect_timeout_seconds`
+#### `number_tablet_writer_threads`
-Default: 3
+* Description: Number of tablet write threads
+* Default value: 16
-The default thrift client connection timeout time (unit: seconds)
+#### `tablet_map_shard_size`
-### `thrift_rpc_timeout_ms`
+* Description: tablet_map_lock fragment size, the value is 2^n, n=0,1,2,3,4, this is for better tablet management
+* Default value: 1
-Default: 5000
+#### `tablet_meta_checkpoint_min_interval_secs`
-thrift default timeout time, default: 5 seconds
+* Description: TabletMeta Checkpoint线程轮询的时间间隔
+* Default value: 600 (s)
-### `thrift_server_type_of_fe`
+#### `tablet_meta_checkpoint_min_new_rowsets_num`
-This configuration indicates the service model used by FE's Thrift service. The type is string and is case-insensitive. This parameter needs to be consistent with the setting of fe's thrift_server_type parameter. Currently there are two values for this parameter, `THREADED` and `THREAD_POOL`.
+* Description: The minimum number of Rowsets for storing TabletMeta Checkpoints
+* Default value: 10
-If the parameter is `THREADED`, the model is a non-blocking I/O model,
+#### `tablet_stat_cache_update_interval_second`
-If the parameter is `THREAD_POOL`, the model is a blocking I/O model.
+* Description: Update interval of tablet state cache
+* Default value:300 (s)
-### `total_permits_for_compaction_score`
+#### `tablet_rowset_stale_sweep_time_sec`
* Type: int64
-* Description: The upper limit of "permits" held by all compaction tasks. This config can be set to limit memory consumption for compaction.
-* Default: 10000
-* Dynamically modify: true
+* Description: It is used to control the expiration time of cleaning up the merged rowset version. When the current time now() minus the max created rowset‘s create time in a version path is greater than tablet_rowset_stale_sweep_time_sec, the current path is cleaned up and these merged rowsets are deleted, the unit is second.
+ - When writing is too frequent and the disk time is insufficient, you can configure less tablet_rowset_stale_sweep_time_sec. However, if this time is less than 5 minutes, it may cause fe to query the version that has been merged, causing a query -230 error.
+* Default value: 1800
+
+#### `tablet_writer_open_rpc_timeout_sec`
-### `trash_file_expire_time_sec`
+* Description: Update interval of tablet state cache
+ - The RPC timeout for sending a Batch (1024 lines) during import. The default is 60 seconds. Since this RPC may involve writing multiple batches of memory, the RPC timeout may be caused by writing batches, so this timeout can be adjusted to reduce timeout errors (such as send batch fail errors). Also, if you increase the write_buffer_size configuration, you need to increase this parameter as well.
+* Default value: 60
+
+#### `tablet_writer_ignore_eovercrowded`
-Default: 259200
+* Type: bool
+* Description: Used to ignore brpc error '[E1011]The server is overcrowded' when writing data.
+* Default value: false
-The interval for cleaning the recycle bin is 72 hours. When the disk space is insufficient, the file retention period under trash may not comply with this parameter
+#### `streaming_load_rpc_max_alive_time_sec`
-### `txn_commit_rpc_timeout_ms`
+* Description: The lifetime of TabletsChannel. If the channel does not receive any data at this time, the channel will be deleted.
+* Default value: 1200
-Default: 10000
+#### `alter_tablet_worker_count`
-txn submit rpc timeout, the default is 10 seconds
+* Description: The number of threads making schema changes
+* Default value: 3
-### `txn_map_shard_size`
+#### `ignore_load_tablet_failure`
-Default: 128
+* Type: bool
+* Description: It is used to decide whether to ignore errors and continue to start be in case of tablet loading failure
+* Default value: false
-txn_map_lock fragment size, the value is 2^n, n=0,1,2,3,4. This is an enhancement to improve the performance of managing txn
+When BE starts, a separate thread will be started for each data directory to load the meta information of the tablet header. In the default configuration, if a data directory fails to load a tablet, the startup process will terminate. At the same time, it will be displayed in the ` be The following error message is seen in the INFO ` log:
-### `txn_shard_size`
+```
+load tablets from header failed, failed tablets size: xxx, path=xxx
+```
-Default: 1024
+Indicates how many tablets failed to load in the data directory. At the same time, the log will also contain specific information about the tablets that failed to load. At this time, manual intervention is required to troubleshoot the cause of the error. After troubleshooting, there are usually two ways to restore:
+ - The tablet information cannot be repaired. If the other copies are normal, you can delete the wrong tablet with the `meta_tool` tool.
+ - Set `ignore_load_tablet_failure` to true, BE will ignore these faulty tablets and start normally
-txn_lock shard size, the value is 2^n, n=0,1,2,3,4, this is an enhancement function that can improve the performance of submitting and publishing txn
+#### `report_disk_state_interval_seconds`
-### `unused_rowset_monitor_interval`
+* Description: The interval time for the agent to report the disk status to FE
+* Default value: 60 (s)
-Default: 30
+#### `result_buffer_cancelled_interval_time`
-Time interval for clearing expired Rowset, unit: second
+* Description: Result buffer cancellation time
+* Default value: 300 (s)
-### `upload_worker_count`
+#### `snapshot_expire_time_sec`
-Default: 1
+* Description: Snapshot file cleaning interval.
+* Default value:172800 (48 hours)
-Maximum number of threads for uploading files
+#### `compress_rowbatches`
-### `user_function_dir`
+* Type: bool
+* Description: enable to use Snappy compression algorithm for data compression when serializing RowBatch
+* Default value: true
-${DORIS_HOME}/lib/udf
+### Log
-udf function directory
+#### `sys_log_dir`
-### `webserver_num_workers`
+* Type: string
+* Description: Storage directory of BE log data
+* Default value: ${DORIS_HOME}/log
-Default: 48
+#### `sys_log_level`
-Webserver default number of worker threads
+* Description: Log Level: INFO < WARNING < ERROR < FATAL
+* Default value: INFO
-### `webserver_port`
+#### `sys_log_roll_mode`
-* Type: int32
-* Description: Service port of http server on BE
-* Default: 8040
+* Description: The size of the log split, one log file is split every 1G
+* Default value: SIZE-MB-1024
-### `write_buffer_size`
+#### `sys_log_roll_num`
-Default: 104857600
+* Description: Number of log files kept
+* Default value: 10
-The size of the buffer before flashing
+#### `sys_log_verbose_level`
-Imported data is first written to a memory block on the BE, and only written back to disk when this memory block reaches the threshold. The default size is 100MB. too small a threshold may result in a large number of small files on the BE. This threshold can be increased to reduce the number of files. However, too large a threshold may cause RPC timeouts
+* Description: Log display level, used to control the log output at the beginning of VLOG in the code
+* Default value: 10
-### `zone_map_row_num_threshold`
+#### `sys_log_verbose_modules`
-* Type: int32
-* Description: If the number of rows in a page is less than this value, no zonemap will be created to reduce data expansion
-* Default: 20
+* Description: Log printing module, writing olap will only print the log under the olap module
+* Default value: 空
-### `aws_log_level`
+#### `aws_log_level`
* Type: int32
-
-* Description: log level of AWS SDK,
+* Description: log level of AWS SDK,
```
Off = 0,
Fatal = 1,
@@ -1451,152 +1283,77 @@ Imported data is first written to a memory block on the BE, and only written bac
Debug = 5,
Trace = 6
```
-
-* Default: 3
-
-### `enable_tcmalloc_hook`
-
-* Type: bool
-* Description: Whether Hook TCmalloc new/delete, currently consume/release tls mem tracker in Hook.
-* Default: true
-
-### `mem_tracker_consume_min_size_bytes`
-
-* Type: int32
-* Description: The minimum length of TCMalloc Hook when consume/release MemTracker. Consume size smaller than this value will continue to accumulate to avoid frequent calls to consume/release of MemTracker. Decreasing this value will increase the frequency of consume/release. Increasing this value will cause MemTracker statistics to be inaccurate. Theoretically, the statistical value of a MemTracker differs from the true value = ( mem_tracker_consume_min_size_bytes * the number of BE thr [...]
-* Default: 1048576
-
-### `max_segment_num_per_rowset`
-
-* Type: int32
-* Description: Used to limit the number of segments in the newly generated rowset when importing. If the threshold is exceeded, the import will fail with error -238. Too many segments will cause compaction to take up a lot of memory and cause OOM errors.
-* Default value: 200
-
-### `remote_storage_read_buffer_mb`
-
-* Type: int32
-* Description: The cache size used when reading files on hdfs or object storage.
-* Default value: 16MB
-
-Increasing this value can reduce the number of calls to read remote data, but it will increase memory overhead.
-
-### `external_table_connect_timeout_sec`
-
-* Type: int32
-* Description: The timeout when establishing connection with external table such as ODBC table.
-* Default value: 5 seconds
-
-### `segment_cache_capacity`
-
-* Type: int32
-* Description: The maximum number of Segments cached by Segment Cache.
-* Default value: 1000000
+* Default value: 3
-The default value is currently only an empirical value, and may need to be modified according to actual scenarios. Increasing this value can cache more segments and avoid some IO. Decreasing this value will reduce memory usage.
+#### `log_buffer_level`
-### `auto_refresh_brpc_channel`
-
-* Type: bool
-* Description: When obtaining a brpc connection, judge the availability of the connection through hand_shake rpc, and re-establish the connection if it is not available 。
-* Default value: false
+* Description: The log flushing strategy is kept in memory by default
+* Default value: 空
-### `high_priority_flush_thread_num_per_store`
+### Else
-* Type: int32
-* Description: The number of flush threads per store path allocated for the high priority import task.
-* Default value: 1
+#### `report_tablet_interval_seconds`
-### `routine_load_consumer_pool_size`
+* Description: The interval time for the agent to report the olap table to the FE
+* Default value: 60 (s)
-* Type: int32
-* Description: The number of caches for the data consumer used by the routine load.
-* Default: 10
+#### `report_task_interval_seconds`
-### `load_task_high_priority_threshold_second`
+* Description: The interval time for the agent to report the task signature to FE
+* Default value: 10 (s)
-* Type: int32
-* Description: When the timeout of an import task is less than this threshold, Doris will consider it to be a high priority task. High priority tasks use a separate pool of flush threads.
-* Default: 120
+#### `periodic_counter_update_period_ms`
-### `min_load_rpc_timeout_ms`
+* Description: Update rate counter and sampling counter cycle
+* Default value: 500 (ms)
-* Type: int32
-* Description: The minimum timeout for each rpc in the load job.
-* Default: 20
+#### `enable_metric_calculator`
-Translated with www.DeepL.com/Translator (free version)
+* Description: If set to true, the metric calculator will run to collect BE-related indicator information, if set to false, it will not run
+* Default value: true
-### `doris_scan_range_max_mb`
-* Type: int32
-* Description: The maximum amount of data read by each OlapScanner.
-* Default: 1024
+#### `enable_system_metrics`
-### `enable_quick_compaction`
-* Type: bool
-* Description: enable quick compaction,It is mainly used in the scenario of frequent import of small amount of data. The problem of -235 can be effectively avoided by merging the imported versions in time through the mechanism of rapid compaction. The definition of small amount of data is currently defined according to the number of rows
-* Default: true
+* Description: User control to turn on and off system indicators.
+* Default value: true
-### `quick_compaction_max_rows`
-* Type: int32
-* Description: When the number of imported rows is less than this value, it is considered that this import is an import of small amount of data, which will be selected during quick compaction
-* Default: 1000
+#### `enable_token_check`
-### `quick_compaction_batch_size`
-* Type: int32
-* Description: trigger time, when import times reach quick_compaction_batch_size will trigger immediately
-* Default: 10
+* Description: Used for forward compatibility, will be removed later.
+* Default value: true
-### `quick_compaction_min_rowsets`
-* Type: int32
-* Description: at least the number of versions to be compaction, and the number of rowsets with a small amount of data in the selection. If it is greater than this value, the real compaction will be carried out
-* Default: 10
+#### `max_runnings_transactions_per_txn_map`
-### `generate_cache_cleaner_task_interval_sec`
-* Type：int64
-* Description：Cleaning interval of cache files, in seconds
-* Default：43200（12 hours）
+* Description: Max number of txns for every txn_partition_map in txn manager, this is a self protection to avoid too many txns saving in manager
+* Default value: 100
-### `file_cache_type`
-* Type：string
-* Description：Type of cache file. whole_ file_ Cache: download the entire segment file, sub_ file_ Cache: the segment file is divided into multiple files by size.
-* Default：""
+#### `max_download_speed_kbps`
-### `max_sub_cache_file_size`
-* Type：int64
-* Description：Cache files using sub_ file_ The maximum size of the split file during cache, unit: B
-* Default：104857600（100MB）
+* Description: Maximum download speed limit
+* Default value: 50000 （kb/s）
-### `file_cache_alive_time_sec`
-* Type：int64
-* Description：Save time of cache file, in seconds
-* Default：604800（1 week）
+#### `download_low_speed_time`
-### `enable_segcompaction`
+* Description: Download time limit
+* Default value: 300 (s)
-* Type: bool
-* Description: Enable to use segment compaction during loading
-* Default value: false
+#### `download_low_speed_limit_kbps`
-### `segcompaction_threshold_segment_num`
+* Description: Minimum download speed
+* Default value: 50 (KB/s)
-* Type: int32
-* Description: Trigger segcompaction if the num of segments in a rowset exceeds this threshold
-* Default value: 10
+#### `doris_cgroups`
-### `segcompaction_small_threshold`
+* Description: Cgroups assigned to doris
+* Default value: empty
-* Type: int32
-* Description: The segment whose row number above the threshold will be compacted during segcompaction
-* Default value: 1048576
+#### `row_nums_check`
-### `big_column_size_buffer`
-* Type: int64
-* Description: When using the odbc external table, if a column type of the odbc source table is HLL, CHAR or VARCHAR, and the length of the column value exceeds this value, the query will report an error 'column value length longer than buffer length'. You can increase this value
-* Default value: 65535
+* Description: Check row nums for BE/CE and schema change
+* Default value: true
-### `small_column_size_buffer`
+#### `priority_queue_remaining_tasks_increased_frequency`
-* Type: int64
-* Description: When using the odbc external table, if a column type of the odbc source table is not HLL, CHAR or VARCHAR, and the length of the column value exceeds this value, the query will report an error 'column value length longer than buffer length'. You can increase this value
-* Default value: 100
+* Description: the increased frequency of priority for remaining tasks in BlockingPriorityQueue
+* Default value: 512
diff --git a/docs/zh-CN/docs/admin-manual/config/be-config.md b/docs/zh-CN/docs/admin-manual/config/be-config.md
index df0119b30d..a71618d449 100644
--- a/docs/zh-CN/docs/admin-manual/config/be-config.md
+++ b/docs/zh-CN/docs/admin-manual/config/be-config.md
@@ -1,7 +1,9 @@
---
{
"title": "BE 配置项",
- "language": "zh-CN"
+ "language": "zh-CN",
+ "toc_min_heading_level": 2,
+ "toc_max_heading_level": 4
}
---
@@ -46,17 +48,17 @@ BE 的配置项有两种方式进行配置：
1. 静态配置
- 在 `conf/be.conf` 文件中添加和设置配置项。`be.conf` 中的配置项会在 BE 进行启动时被读取。没有在 `be.conf` 中的配置项将使用默认值。
+在 `conf/be.conf` 文件中添加和设置配置项。`be.conf` 中的配置项会在 BE 进行启动时被读取。没有在 `be.conf` 中的配置项将使用默认值。
2. 动态配置
- BE 启动后，可以通过以下命令动态设置配置项。
+BE 启动后，可以通过以下命令动态设置配置项。
```
curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}'
```
- 在 0.13 版本及之前，通过该方式修改的配置项将在 BE 进程重启后失效。在 0.14 及之后版本中，可以通过以下命令持久化修改后的配置。修改后的配置项存储在 `be_custom.conf` 文件中。
+在 0.13 版本及之前，通过该方式修改的配置项将在 BE 进程重启后失效。在 0.14 及之后版本中，可以通过以下命令持久化修改后的配置。修改后的配置项存储在 `be_custom.conf` 文件中。
```
curl -X POST http://{be_ip}:{be_http_port}/api/update_config?{key}={value}&persist=true
@@ -66,19 +68,19 @@ BE 的配置项有两种方式进行配置：
1. 静态方式修改 `max_base_compaction_threads`
- 通过在 `be.conf` 文件中添加：
+通过在 `be.conf` 文件中添加：
- ```max_base_compaction_threads=5```
+```max_base_compaction_threads=5```
- 之后重启 BE 进程以生效该配置。
+之后重启 BE 进程以生效该配置。
2. 动态方式修改 `streaming_load_max_mb`
- BE 启动后，通过下面命令动态设置配置项 `streaming_load_max_mb`:
+BE 启动后，通过下面命令动态设置配置项 `streaming_load_max_mb`:
- ```curl -X POST http://{be_ip}:{be_http_port}/api/update_config?streaming_load_max_mb=1024```
+```curl -X POST http://{be_ip}:{be_http_port}/api/update_config?streaming_load_max_mb=1024```
- 返回值如下，则说明设置成功。
+返回值如下，则说明设置成功。
```
{
@@ -87,7 +89,7 @@ BE 的配置项有两种方式进行配置：
}
```
- BE 重启后该配置将失效。如果想持久化修改结果，使用如下命令：
+BE 重启后该配置将失效。如果想持久化修改结果，使用如下命令：
```
curl -X POST http://{be_ip}:{be_http_port}/api/update_config?streaming_load_max_mb=1024\&persist=true
@@ -95,1371 +97,1198 @@ BE 的配置项有两种方式进行配置：
## 配置项列表
-### `alter_tablet_worker_count`
+### 服务
-默认值：3
-
-进行schema change的线程数
-
-### `generate_compaction_tasks_interval_ms`
-
-默认值：10 （ms）
-
-生成compaction作业的最小间隔时间（ms）
-
-### `enable_vectorized_compaction`
-
-默认值：true
-
-是否开启向量化compaction
-
-### `base_compaction_min_rowset_num`
-
-默认值：5
-
-BaseCompaction触发条件之一：Cumulative文件数目要达到的限制，达到这个限制之后会触发BaseCompaction
-
-### `base_compaction_write_mbytes_per_sec`
-
-默认值：5（MB）
-
-BaseCompaction任务每秒写磁盘最大速度
-
-### `base_compaction_min_data_ratio`
-
-默认值：0.3 （30%）
-
-BaseCompaction触发条件之一：Cumulative文件大小达到Base文件的比例
-
-### `base_compaction_trace_threshold`
-
-* 类型：int32
-* 描述：打印base compaction的trace信息的阈值，单位秒
-* 默认值：10
-
-base compaction是一个耗时较长的后台操作，为了跟踪其运行信息，可以调整这个阈值参数来控制trace日志的打印。打印信息如下：
-
-```
-W0610 11:26:33.804431 56452 storage_engine.cpp:552] Trace:
-0610 11:23:03.727535 (+ 0us) storage_engine.cpp:554] start to perform base compaction
-0610 11:23:03.728961 (+ 1426us) storage_engine.cpp:560] found best tablet 546859
-0610 11:23:03.728963 (+ 2us) base_compaction.cpp:40] got base compaction lock
-0610 11:23:03.729029 (+ 66us) base_compaction.cpp:44] rowsets picked
-0610 11:24:51.784439 (+108055410us) compaction.cpp:46] got concurrency lock and start to do compaction
-0610 11:24:51.784818 (+ 379us) compaction.cpp:74] prepare finished
-0610 11:26:33.359265 (+101574447us) compaction.cpp:87] merge rowsets finished
-0610 11:26:33.484481 (+125216us) compaction.cpp:102] output rowset built
-0610 11:26:33.484482 (+ 1us) compaction.cpp:106] check correctness finished
-0610 11:26:33.513197 (+ 28715us) compaction.cpp:110] modify rowsets finished
-0610 11:26:33.513300 (+ 103us) base_compaction.cpp:49] compaction finished
-0610 11:26:33.513441 (+ 141us) base_compaction.cpp:56] unused rowsets have been moved to GC queue
-Metrics: {"filtered_rows":0,"input_row_num":3346807,"input_rowsets_count":42,"input_rowsets_data_size":1256413170,"input_segments_num":44,"merge_rowsets_latency_us":101574444,"merged_rows":0,"output_row_num":3346807,"output_rowset_data_size":1228439659,"output_segments_num":6}
-```
-
-### `be_port`
+#### `be_port`
* 类型：int32
* 描述：BE 上 thrift server 的端口号，用于接收来自 FE 的请求
* 默认值：9060
-### `be_service_threads`
-* 类型：int32
-* 描述：BE 上 thrift server service的执行线程数，代表可以用于执行FE请求的线程数。
-* 默认值：64
-
-### `brpc_max_body_size`
-
-这个配置主要用来修改 brpc 的参数 `max_body_size`。
-
-有时查询失败，在 BE 日志中会出现 `body_size is too large` 的错误信息。这可能发生在 SQL 模式为 multi distinct + 无 group by + 超过1T 数据量的情况下。这个错误表示 brpc 的包大小超过了配置值。此时可以通过调大该配置避免这个错误。
-
-### `brpc_socket_max_unwritten_bytes`
+#### `heartbeat_service_port`
-这个配置主要用来修改 brpc 的参数 `socket_max_unwritten_bytes`。
-
-有时查询失败，BE 日志中会出现 `The server is overcrowded` 的错误信息，表示连接上有过多的未发送数据。当查询需要发送较大的bitmap字段时，可能会遇到该问题，此时可能通过调大该配置避免该错误。
-
-### `transfer_large_data_by_brpc`
-
-* 类型: bool
-* 描述：该配置用来控制是否在 Tuple/Block data 长度大于1.8G时，将 protoBuf request 序列化后和 Tuple/Block data 一起嵌入到 controller attachment 后通过 http brpc 发送。为了避免 protoBuf request 的长度超过2G时的错误：Bad request, error_text=[E1003]Fail to compress request。在过去的版本中，曾将 Tuple/Block data 放入 attachment 后通过默认的 baidu_std brpc 发送，但 attachment 超过2G时将被截断，通过 http brpc 发送不存在2G的限制。
-* 默认值：true
-
-### `brpc_num_threads`
+* 类型：int32
+* 描述：BE 上心跳服务端口（thrift），用于接收来自 FE 的心跳
+* 默认值：9050
-该配置主要用来修改brpc中bthreads的数量. 该配置的默认值被设置为-1, 这意味着bthreads的数量将被设置为机器的cpu核数。
+#### `webserver_port`
-用户可以将该配置的值调大来获取更好的QPS性能。更多的信息可以参考`https://github.com/apache/incubator-brpc/blob/master/docs/cn/benchmark.md`。
+* 类型：int32
+* 描述：BE 上的 http server 的服务端口
+* 默认值：8040
-### `brpc_port`
+#### `brpc_port`
* 类型：int32
* 描述：BE 上的 brpc 的端口，用于 BE 之间通讯
* 默认值：8060
-### `buffer_pool_clean_pages_limit`
+#### `single_replica_load_brpc_port`
-默认值：50%
+* 类型: int32
+* 描述: 单副本数据导入功能中，Master副本和Slave副本之间通信的RPC端口。Master副本flush完成之后通过RPC通知Slave副本同步数据，以及Slave副本同步数据完成后通过RPC通知Master副本。系统为单副本数据导入过程中Master副本和Slave副本之间通信开辟了独立的BRPC线程池，以避免导入并发较大时副本之间的数据同步抢占导入数据分发和查询任务的线程资源。
+* 默认值: 9070
-清理可能被缓冲池保存的Page
+#### `single_replica_load_download_port`
-### `buffer_pool_limit`
+* 类型: int32
+* 描述: 单副本数据导入功能中，Slave副本通过HTTP从Master副本下载数据文件的端口。系统为单副本数据导入过程中Slave副本从Master副本下载数据文件开辟了独立的HTTP线程池，以避免导入并发较大时Slave副本下载数据文件抢占其他http任务的线程资源。
+* 默认值: 8050
-* 类型：string
-* 描述：buffer pool之中最大的可分配内存
-* 默认值：20%
+#### `priority_networks`
-BE缓存池最大的内存可用量，buffer pool是BE新的内存管理结构，通过buffer page来进行内存管理，并能够实现数据的落盘。并发的所有查询的内存申请都会通过buffer pool来申请。当前buffer pool仅作用在**AggregationNode**与**ExchangeNode**。
+* 描述: 为那些有很多 ip 的服务器声明一个选择策略。 请注意，最多应该有一个 ip 与此列表匹配。 这是一个以分号分隔格式的列表，用 CIDR 表示法，例如 10.10.10.0/24 ， 如果没有匹配这条规则的ip，会随机选择一个。
+* 默认值: 空
-### `check_consistency_worker_count`
+#### `storage_root_path`
-默认值：1
+* 类型：string
-计算tablet的校验和(checksum)的工作线程数
+* 描述：BE数据存储的目录,多目录之间用英文状态的分号`;`分隔。可以通过路径区别存储目录的介质，HDD或SSD。可以添加容量限制在每个路径的末尾，通过英文状态逗号`,`隔开。如果用户不是SSD和HDD磁盘混合使用的情况，不需要按照如下示例一和示例二的配置方法配置，只需指定存储目录即可；也不需要修改FE的默认存储介质配置
-### `chunk_reserved_bytes_limit`
+ 示例1如下：
-默认值：20%
+ **注意：如果是SSD磁盘要在目录后面加上`.SSD`,HDD磁盘在目录后面加`.HDD`**
-Chunk Allocator的reserved bytes限制，通常被设置为 mem_limit 的百分比。默认单位字节，值必须是2的倍数，且必须大于0，如果大于物理内存，将被设置为物理内存大小。增加这个变量可以提高性能，但是会获得更多其他模块无法使用的空闲内存。
+ `storage_root_path=/home/disk1/doris.HDD;/home/disk2/doris.SSD;/home/disk2/doris`
-### `clear_transaction_task_worker_count`
+ **说明**
-默认值：1
+ - /home/disk1/doris.HDD，表示存储介质是HDD;
+ - /home/disk2/doris.SSD，表示存储介质是SSD；
+ - /home/disk2/doris，存储介质默认为HDD
-用于清理事务的线程数
+ 示例2如下：
-### `clone_worker_count`
+ **注意：不论HHD磁盘目录还是SSD磁盘目录，都无需添加后缀，storage_root_path参数里指定medium即可**
-默认值：3
+ `storage_root_path=/home/disk1/doris,medium:hdd;/home/disk2/doris,medium:ssd`
-用于执行克隆任务的线程数
+ **说明**
-### `cluster_id`
+ - /home/disk1/doris,medium:hdd，表示存储介质是HHD;
+ - /home/disk2/doris,medium:ssd，表示存储介质是SSD;
-* 类型：int32
-* 描述：配置BE的所属于的集群id。
+* 默认值：${DORIS_HOME}/storage
-* 默认值：-1
+#### `heartbeat_service_thread_count`
-该值通常由FE通过心跳向BE下发，不需要额外进行配置。当确认某BE属于某一个确定的Drois集群时，可以进行配置，同时需要修改数据目录下的cluster_id文件，使二者相同。
+* 类型：int32
+* 描述：执行BE上心跳服务的线程数，默认为1，不建议修改
+* 默认值：1
-### `column_dictionary_key_ratio_threshold`
+#### `ignore_broken_disk`
-默认值：0
+* 类型：bool
+* 描述：当BE启动时，会检查``storage_root_path`` 配置下的所有路径。
-字符串类型的取值比例，小于这个比例采用字典压缩算法
+ - `ignore_broken_disk=true`
+
+ 如果路径不存在或路径下无法进行读写文件(坏盘)，将忽略此路径，如果有其他可用路径则不中断启动。
-### `column_dictionary_key_size_threshold`
+ - `ignore_broken_disk=false`
-默认值：0
+ 如果路径不存在或路径下无法进行读写文件(坏盘)，将中断启动失败退出。
-字典压缩列大小，小于这个值采用字典压缩算法
+* 默认值：false
-### `compaction_tablet_compaction_score_factor`
+#### `mem_limit`
-* 类型：int32
-* 描述：选择tablet进行compaction时，计算 tablet score 的公式中 compaction score的权重。
-* 默认值：1
+* 类型：string
+* 描述：限制BE进程使用服务器最大内存百分比。用于防止BE内存挤占太多的机器内存，该参数必须大于0，当百分大于100%之后，该值会默认为100%。
+* 默认值：80%
-### `compaction_tablet_scan_frequency_factor`
+#### `cluster_id`
* 类型：int32
-* 描述：选择tablet进行compaction时，计算 tablet score 的公式中 tablet scan frequency 的权重。
-* 默认值：0
-
-选择一个tablet执行compaction任务时，可以将tablet的scan频率作为一个选择依据，对当前最近一段时间频繁scan的tablet优先执行compaction。
-tablet score可以通过以下公式计算：
+* 描述：配置BE的所属于的集群id。
+ - 该值通常由FE通过心跳向BE下发，不需要额外进行配置。当确认某BE属于某一个确定的Drois集群时，可以进行配置，同时需要修改数据目录下的cluster_id文件，使二者相同。
+* 默认值：-1
-tablet_score = compaction_tablet_scan_frequency_factor * tablet_scan_frequency + compaction_tablet_compaction_score_factor * compaction_score
+#### `custom_config_dir`
-### `compaction_task_num_per_disk`
+* 描述：配置 `be_custom.conf` 文件的位置。默认为 `conf/` 目录下。
+ - 在某些部署环境下，`conf/`目录可能因为系统的版本升级被覆盖掉。这会导致用户在运行是持久化修改的配置项也被覆盖。这时，我们可以将 `be_custom.conf` 存储在另一个指定的目录中，以防止配置文件被覆盖。
+* 默认值: 空
-* 类型：int32
-* 描述：每个磁盘（HDD）可以并发执行的compaction任务数量。
-* 默认值：2
+#### `trash_file_expire_time_sec`
-### `compaction_task_num_per_fast_disk`
+* 描述：回收站清理的间隔，72个小时，当磁盘空间不足时，trash下的文件保存期可不遵守这个参数
+* 默认值: 259200
-* 类型：int32
-* 描述：每个高速磁盘（SSD）可以并发执行的compaction任务数量。
-* 默认值：4
+#### `es_http_timeout_ms`
-### `compress_rowbatches`
-* 类型：bool
+* 描述：通过http连接ES的超时时间，默认是5秒
+* 默认值: 5000 (ms)
-* 描述：序列化RowBatch时是否使用Snappy压缩算法进行数据压缩
+#### `es_scroll_keepalive`
-* 默认值：true
+* 描述：es scroll Keeplive保持时间，默认5分钟
+* 默认值: 5 (m)
-### `create_tablet_worker_count`
+#### `external_table_connect_timeout_sec`
-默认值：3
+* 类型: int32
+* 描述: 和外部表建立连接的超时时间。
+* 默认值: 5秒
-BE创建tablet的工作线程数
+#### `status_report_interval`
-### `cumulative_compaction_rounds_for_each_base_compaction_round`
+* 描述：配置文件报告之间的间隔；单位：秒
+* 默认值: 5
-* 类型：int32
-* 描述：Compaction任务的生产者每次连续生产多少轮cumulative compaction任务后生产一轮base compaction。
-* 默认值：9
+#### `brpc_max_body_size`
-### `disable_auto_compaction`
+* 描述：这个配置主要用来修改 brpc 的参数 `max_body_size`。
-* 类型：bool
-* 描述：关闭自动执行compaction任务
-* 默认值：false
+ - 有时查询失败，在 BE 日志中会出现 `body_size is too large` 的错误信息。这可能发生在 SQL 模式为 multi distinct + 无 group by + 超过1T 数据量的情况下。这个错误表示 brpc 的包大小超过了配置值。此时可以通过调大该配置避免这个错误。
-一般需要为关闭状态，当调试或测试环境中想要手动操作compaction任务时，可以对该配置进行开启
+#### `brpc_socket_max_unwritten_bytes`
-### `cumulative_compaction_budgeted_bytes`
+* 描述：这个配置主要用来修改 brpc 的参数 `socket_max_unwritten_bytes`。
-默认值：104857600
+ - 有时查询失败，BE 日志中会出现 `The server is overcrowded` 的错误信息，表示连接上有过多的未发送数据。当查询需要发送较大的bitmap字段时，可能会遇到该问题，此时可能通过调大该配置避免该错误。
-BaseCompaction触发条件之一：Singleton文件大小限制，100MB
+#### `transfer_large_data_by_brpc`
-### `cumulative_compaction_trace_threshold`
+* 类型: bool
+* 描述：该配置用来控制是否在 Tuple/Block data 长度大于1.8G时，将 protoBuf request 序列化后和 Tuple/Block data 一起嵌入到 controller attachment 后通过 http brpc 发送。为了避免 protoBuf request 的长度超过2G时的错误：Bad request, error_text=[E1003]Fail to compress request。在过去的版本中，曾将 Tuple/Block data 放入 attachment 后通过默认的 baidu_std brpc 发送，但 attachment 超过2G时将被截断，通过 http brpc 发送不存在2G的限制。
+* 默认值：true
-* 类型：int32
-* 描述：打印cumulative compaction的trace信息的阈值，单位秒
-* 默认值：2
+#### `brpc_num_threads`
-与base_compaction_trace_threshold类似。
+* 描述：该配置主要用来修改brpc中bthreads的数量. 该配置的默认值被设置为-1, 这意味着bthreads的数量将被设置为机器的cpu核数。
-### disable_compaction_trace_log
+ - 用户可以将该配置的值调大来获取更好的QPS性能。更多的信息可以参考`https://github.com/apache/incubator-brpc/blob/master/docs/cn/benchmark.md`。
+* 默认值：1
-* 类型: bool
-* 描述: 关闭compaction的trace日志
-* 默认值: true
+#### `thrift_rpc_timeout_ms`
-如果设置为true，`cumulative_compaction_trace_threshold` 和 `base_compaction_trace_threshold` 将不起作用。并且trace日志将关闭。
+* 描述：thrift默认超时时间
+* 默认值：5000
-### `compaction_promotion_size_mbytes`
+#### `thrift_client_retry_interval_ms`
* 类型：int64
-* 描述：cumulative compaction的输出rowset总磁盘大小超过了此配置大小，该rowset将用于base compaction。单位是m字节。
-* 默认值：1024
+* 描述：用来为be的thrift客户端设置重试间隔, 避免fe的thrift server发生雪崩问题，单位为ms。
+* 默认值：1000
-一般情况下，配置在2G以内，为了防止cumulative compaction时间过长，导致版本积压。
+#### `thrift_connect_timeout_seconds`
-### `compaction_promotion_ratio`
+* 描述：默认thrift客户端连接超时时间
+* 默认值：3 (m)
-* 类型：double
-* 描述：cumulative compaction的输出rowset总磁盘大小超过base版本rowset的配置比例时，该rowset将用于base compaction。
-* 默认值：0.05
+#### `thrift_server_type_of_fe`
-一般情况下，建议配置不要高于0.1，低于0.02。
+* 类型：string
+* 描述：该配置表示FE的Thrift服务使用的服务模型, 类型为string, 大小写不敏感,该参数需要和fe的thrift_server_type参数的设置保持一致。目前该参数的取值有两个,`THREADED`和`THREAD_POOL`。
-### `compaction_promotion_min_size_mbytes`
+ - 若该参数为`THREADED`, 该模型为非阻塞式I/O模型，
-* 类型：int64
-* 描述：Cumulative compaction的输出rowset总磁盘大小低于此配置大小，该rowset将不进行base compaction，仍然处于cumulative compaction流程中。单位是m字节。
-* 默认值：64
+ - 若该参数为`THREAD_POOL`, 该模型为阻塞式I/O模型。
-一般情况下，配置在512m以内，配置过大会导致base版本早期的大小过小，一直不进行base compaction。
+#### `txn_commit_rpc_timeout_ms`
-### `compaction_min_size_mbytes`
+* 描述：txn 提交 rpc 超时
+* 默认值：10000 (ms)
-* 类型：int64
-* 描述：cumulative compaction进行合并时，选出的要进行合并的rowset的总磁盘大小大于此配置时，才按级别策略划分合并。小于这个配置时，直接执行合并。单位是m字节。
-* 默认值：64
+#### `txn_map_shard_size`
-一般情况下，配置在128m以内，配置过大会导致cumulative compaction写放大较多。
+* 描述：txn_map_lock 分片大小，取值为2^n，n=0,1,2,3,4。这是一项增强功能，可提高管理 txn 的性能
+* 默认值：128
-### `custom_config_dir`
+#### `txn_shard_size`
-配置 `be_custom.conf` 文件的位置。默认为 `conf/` 目录下。
+* 描述：txn_lock 分片大小，取值为2^n，n=0,1,2,3,4， 这是一项增强功能，可提高提交和发布 txn 的性能
+* 默认值：1024
-在某些部署环境下，`conf/` 目录可能因为系统的版本升级被覆盖掉。这会导致用户在运行是持久化修改的配置项也被覆盖。这时，我们可以将 `be_custom.conf` 存储在另一个指定的目录中，以防止配置文件被覆盖。
+#### `unused_rowset_monitor_interval`
-### `default_num_rows_per_column_file_block`
-* 类型：int32
-* 描述：配置单个RowBlock之中包含多少行的数据。
-* 默认值：1024
+* 描述：清理过期Rowset的时间间隔
+* 默认值：30 (s)
-### `default_rowset_type`
-* 类型：string
-* 描述：标识BE默认选择的存储格式，可配置的参数为："**ALPHA**", "**BETA**"。主要起以下两个作用
-1. 当建表的storage_format设置为Default时，通过该配置来选取BE的存储格式。
-2. 进行Compaction时选择BE的存储格式
-* 默认值：BETA
+#### `max_client_cache_size_per_host`
-### `delete_worker_count`
+* 描述：每个主机的最大客户端缓存数，BE 中有多种客户端缓存，但目前我们使用相同的缓存大小配置。 如有必要，使用不同的配置来设置不同的客户端缓存。
+* 默认值：10
-默认值：3
+#### `string_type_length_soft_limit_bytes`
-执行数据删除任务的线程数
+* 类型: int32
+* 描述: String 类型最大长度的软限，单位是字节
+* 默认值: 1048576
-### `disable_mem_pools`
+#### `big_column_size_buffer`
-默认值：false
+* 类型：int64
+* 描述：当使用odbc外表时，如果odbc源表的某一列类型为HLL, CHAR或者VARCHAR，并且列值长度超过该值，则查询报错'column value length longer than buffer length'. 可增大该值
+* 默认值：65535
-是否禁用内存缓存池，默认不禁用
+#### `small_column_size_buffer`
-### `disable_storage_page_cache`
+* 类型：int64
+* 描述：当使用odbc外表时，如果odbc源表的某一列类型不是HLL, CHAR或者VARCHAR，并且列值长度超过该值，则查询报错'column value length longer than buffer length'. 可增大该值
+* 默认值：100
-* 类型：bool
-* 描述：是否进行使用page cache进行index的缓存，该配置仅在BETA存储格式时生效
-* 默认值：false
+### 查询
-### `disk_stat_monitor_interval`
+#### `fragment_pool_queue_size`
-默认值：5 （s）
+* 描述：单节点上能够处理的查询请求上限
+* 默认值：2048
-磁盘状态检查时间间隔
+#### `fragment_pool_thread_num_min`
-### `doris_cgroups`
+* 描述：查询线程数，默认最小启动64个线程。
+* 默认值：64
-默认值：空
+#### `fragment_pool_thread_num_max`
-分配给doris的cgroups
+* 描述：后续查询请求动态创建线程，最大创建256个线程。
+* 默认值：256
-### `doris_max_pushdown_conjuncts_return_rate`
+#### `doris_max_pushdown_conjuncts_return_rate`
* 类型：int32
* 描述：BE在进行HashJoin时，会采取动态分区裁剪的方式将join条件下推到OlapScanner上。当OlapScanner扫描的数据大于32768行时，BE会进行过滤条件检查，如果该过滤条件的过滤率低于该配置，则Doris会停止使用动态分区裁剪的条件进行数据过滤。
* 默认值：90
-
-### `doris_max_scan_key_num`
+#### `doris_max_scan_key_num`
* 类型：int
* 描述：用于限制一个查询请求中，scan node 节点能拆分的最大 scan key 的个数。当一个带有条件的查询请求到达 scan node 节点时，scan node 会尝试将查询条件中 key 列相关的条件拆分成多个 scan key range。之后这些 scan key range 会被分配给多个 scanner 线程进行数据扫描。较大的数值通常意味着可以使用更多的 scanner 线程来提升扫描操作的并行度。但在高并发场景下，过多的线程可能会带来更大的调度开销和系统负载，反而会降低查询响应速度。一个经验数值为 50。该配置可以单独进行会话级别的配置，具体可参阅 [变量](../../advanced/variables.md) 中 `max_scan_key_num` 的说明。
+ - 当在高并发场景下发下并发度无法提升时，可以尝试降低该数值并观察影响。
* 默认值：1024
-当在高并发场景下发下并发度无法提升时，可以尝试降低该数值并观察影响。
-
-### `doris_scan_range_row_count`
+#### `doris_scan_range_row_count`
* 类型：int32
* 描述：BE在进行数据扫描时，会将同一个扫描范围拆分为多个ScanRange。该参数代表了每个ScanRange代表扫描数据范围。通过该参数可以限制单个OlapScanner占用io线程的时间。
* 默认值：524288
-### `doris_scanner_queue_size`
+#### `doris_scanner_queue_size`
* 类型：int32
* 描述：TransferThread与OlapScanner之间RowBatch的缓存队列的长度。Doris进行数据扫描时是异步进行的，OlapScanner扫描上来的Rowbatch会放入缓存队列之中，等待上层TransferThread取走。
* 默认值：1024
-### `doris_scanner_row_num`
-
-默认值：16384
+#### `doris_scanner_row_num`
-每个扫描线程单次执行最多返回的数据行数
+* 描述：每个扫描线程单次执行最多返回的数据行数
+* 默认值：16384
-### `doris_scanner_thread_pool_queue_size`
+#### `doris_scanner_thread_pool_queue_size`
* 类型：int32
* 描述：Scanner线程池的队列长度。在Doris的扫描任务之中，每一个Scanner会作为一个线程task提交到线程池之中等待被调度，而提交的任务数目超过线程池队列的长度之后，后续提交的任务将阻塞直到队列之中有新的空缺。
* 默认值：102400
-### `doris_scanner_thread_pool_thread_num`
+#### `doris_scanner_thread_pool_thread_num`
* 类型：int32
* 描述：Scanner线程池线程数目。在Doris的扫描任务之中，每一个Scanner会作为一个线程task提交到线程池之中等待被调度，该参数决定了Scanner线程池的大小。
* 默认值：48
-### `download_low_speed_limit_kbps`
+#### `enable_prefetch`
-默认值：50 (KB/s)
+* 类型：bool
+* 描述：当使用PartitionedHashTable进行聚合和join计算时，是否进行HashBuket的预取，推荐设置为true。
+* 默认值：true
-下载最低限速
+#### `enable_quadratic_probing`
-### `download_low_speed_time`
+* 类型：bool
+* 描述：当使用PartitionedHashTable时发生Hash冲突时，是否采用平方探测法来解决Hash冲突。该值为false的话，则选用线性探测发来解决Hash冲突。关于平方探测法可参考：[quadratic_probing](https://en.wikipedia.org/wiki/Quadratic_probing)
+* 默认值：true
-默认值：300 (s)
+#### `exchg_node_buffer_size_bytes`
-下载时间限制，默认300秒
+* 类型：int32
+* 描述：ExchangeNode节点Buffer队列的大小，单位为byte。来自Sender端发送的数据量大于ExchangeNode的Buffer大小之后，后续发送的数据将阻塞直到Buffer腾出可写入的空间。
+* 默认值：10485760
-### `download_worker_count`
+#### `max_pushdown_conditions_per_column`
-默认值：1
+* 类型：int
+* 描述：用于限制一个查询请求中，针对单个列，能够下推到存储引擎的最大条件数量。在查询计划执行的过程中，一些列上的过滤条件可以下推到存储引擎，这样可以利用存储引擎中的索引信息进行数据过滤，减少查询需要扫描的数据量。比如等值条件、IN 谓词中的条件等。这个参数在绝大多数情况下仅影响包含 IN 谓词的查询。如 `WHERE colA IN (1,2,3,4,...)`。较大的数值意味值 IN 谓词中更多的条件可以推送给存储引擎，但过多的条件可能会导致随机读的增加，某些情况下可能会降低查询效率。该配置可以单独进行会话级别的配置，具体可参阅 [变量](../../advanced/variables.md) 中 `max_pushdown_conditions_per_column ` 的说明。
+* 默认值：1024
-下载线程数，默认1个
+* 示例
-### `drop_tablet_worker_count`
+ - 表结构为 `id INT, col2 INT, col3 varchar(32), ...`。
+ - 查询请求为 `... WHERE id IN (v1, v2, v3, ...)`
+ - 如果 IN 谓词中的条件数量超过了该配置，则可以尝试增加该配置值，观察查询响应是否有所改善。
-默认值：3
+#### `max_send_batch_parallelism_per_job`
-删除tablet的线程数
+* 类型：int
+* 描述：OlapTableSink 发送批处理数据的最大并行度，用户为 `send_batch_parallelism` 设置的值不允许超过 `max_send_batch_parallelism_per_job` ，如果超过， `send_batch_parallelism` 将被设置为 `max_send_batch_parallelism_per_job` 的值。
+* 默认值：5
-### `enable_metric_calculator`
+#### `serialize_batch`
-默认值：true
+* 类型：bool
+* 描述：BE之间rpc通信是否序列化RowBatch，用于查询层之间的数据传输
+* 默认值: false
-如果设置为 true，metric calculator 将运行，收集BE相关指标信息，如果设置成false将不运行
+#### `doris_scan_range_max_mb`
-### `enable_partitioned_aggregation`
+* 类型: int32
+* 描述: 每个OlapScanner 读取的最大数据量
+* 默认值: 1024
-* 类型：bool
-* 描述：BE节点是否通过PartitionAggregateNode来实现聚合操作，如果false的话将会执行AggregateNode完成聚合。非特殊需求场景不建议设置为false。
-* 默认值：true
+### compaction
-### `enable_prefetch`
+#### `disable_auto_compaction`
* 类型：bool
-* 描述：当使用PartitionedHashTable进行聚合和join计算时，是否进行HashBuket的预取，推荐设置为true。
-* 默认值：true
+* 描述：关闭自动执行compaction任务
+ - 一般需要为关闭状态，当调试或测试环境中想要手动操作compaction任务时，可以对该配置进行开启
+* 默认值：false
-### `enable_quadratic_probing`
+#### `enable_vectorized_compaction`
* 类型：bool
-* 描述：当使用PartitionedHashTable时发生Hash冲突时，是否采用平方探测法来解决Hash冲突。该值为false的话，则选用线性探测发来解决Hash冲突。关于平方探测法可参考：[quadratic_probing](https://en.wikipedia.org/wiki/Quadratic_probing)
+* 描述：是否开启向量化compaction
* 默认值：true
-### `enable_system_metrics`
+#### `max_base_compaction_threads`
-默认值：true
+* 类型：int32
+* 描述：Base Compaction线程池中线程数量的最大值。
+* 默认值：4
-用户控制打开和关闭系统指标
+#### `generate_compaction_tasks_interval_ms`
-### `enable_token_check`
+* 描述：生成compaction作业的最小间隔时间
+* 默认值：10 （ms）
-默认值：true
+#### `base_compaction_min_rowset_num`
-用于向前兼容，稍后将被删除
+* 描述：BaseCompaction触发条件之一：Cumulative文件数目要达到的限制，达到这个限制之后会触发BaseCompaction
+* 默认值：5
-### `enable_stream_load_record`
+#### `base_compaction_min_data_ratio`
-默认值：false
+* 描述：BaseCompaction触发条件之一：Cumulative文件大小达到Base文件的比例。
+* 默认值：0.3 （30%）
-是否开启 stream load 操作记录，默认是不启用
+#### `total_permits_for_compaction_score`
-### `es_http_timeout_ms`
+* 类型：int64
+* 描述：被所有的compaction任务所能持有的 "permits" 上限，用来限制compaction占用的内存。
+* 默认值：10000
+* 可动态修改：是
-默认值：5000 (ms)
+#### `compaction_tablet_compaction_score_factor`
-通过http连接ES的超时时间，默认是5秒
+* 类型：int32
+* 描述：选择tablet进行compaction时，计算 tablet score 的公式中 compaction score的权重。
+* 默认值：1
-### `es_scroll_keepalive`
+#### `compaction_promotion_size_mbytes`
-默认值：5m
+* 类型：int64
+* 描述：cumulative compaction的输出rowset总磁盘大小超过了此配置大小，该rowset将用于base compaction。单位是m字节。
+ - 一般情况下，配置在2G以内，为了防止cumulative compaction时间过长，导致版本积压。
+* 默认值：1024
-es scroll Keeplive保持时间，默认5分钟
+#### `compaction_promotion_ratio`
-### `etl_thread_pool_queue_size`
+* 类型：double
+* 描述：cumulative compaction的输出rowset总磁盘大小超过base版本rowset的配置比例时，该rowset将用于base compaction。
+ - 一般情况下，建议配置不要高于0.1，低于0.02。
+* 默认值：0.05
-默认值：256
+#### `compaction_promotion_min_size_mbytes`
-ETL线程池的大小
+* 类型：int64
+* 描述：Cumulative compaction的输出rowset总磁盘大小低于此配置大小，该rowset将不进行base compaction，仍然处于cumulative compaction流程中。单位是m字节。
+ - 一般情况下，配置在512m以内，配置过大会导致base版本早期的大小过小，一直不进行base compaction。
+* 默认值：64
-### `etl_thread_pool_size`
+#### `compaction_min_size_mbytes`
-### `exchg_node_buffer_size_bytes`
+* 类型：int64
+* 描述：cumulative compaction进行合并时，选出的要进行合并的rowset的总磁盘大小大于此配置时，才按级别策略划分合并。小于这个配置时，直接执行合并。单位是m字节。
+ - 一般情况下，配置在128m以内，配置过大会导致cumulative compaction写放大较多。
+* 默认值：64
-* 类型：int32
-* 描述：ExchangeNode节点Buffer队列的大小，单位为byte。来自Sender端发送的数据量大于ExchangeNode的Buffer大小之后，后续发送的数据将阻塞直到Buffer腾出可写入的空间。
-* 默认值：10485760
+#### `default_rowset_type`
-### `file_descriptor_cache_capacity`
+* 类型：string
+* 描述：标识BE默认选择的存储格式，可配置的参数为："**ALPHA**", "**BETA**"。主要起以下两个作用
+ - 当建表的storage_format设置为Default时，通过该配置来选取BE的存储格式。
+ - 进行Compaction时选择BE的存储格式
+* 默认值：BETA
-默认值：32768
+#### `cumulative_compaction_min_deltas`
-文件句柄缓存的容量，默认缓存32768个文件句柄
+* 描述：cumulative compaction策略：最小增量文件的数量
+* 默认值：5
-### `cache_clean_interval`
+#### `cumulative_compaction_max_deltas`
-默认值：1800 (s)
+* 描述：cumulative compaction策略：最大增量文件的数量
+* 默认值：1000
-文件句柄缓存清理的间隔，用于清理长期不用的文件句柄。
-同时也是Segment Cache的清理间隔时间。
+#### `base_compaction_trace_threshold`
-### `flush_thread_num_per_store`
+* 类型：int32
+* 描述：打印base compaction的trace信息的阈值，单位秒
+* 默认值：10
-默认值：2
+base compaction是一个耗时较长的后台操作，为了跟踪其运行信息，可以调整这个阈值参数来控制trace日志的打印。打印信息如下：
-每个store用于刷新内存表的线程数
+```
+W0610 11:26:33.804431 56452 storage_engine.cpp:552] Trace:
+0610 11:23:03.727535 (+ 0us) storage_engine.cpp:554] start to perform base compaction
+0610 11:23:03.728961 (+ 1426us) storage_engine.cpp:560] found best tablet 546859
+0610 11:23:03.728963 (+ 2us) base_compaction.cpp:40] got base compaction lock
+0610 11:23:03.729029 (+ 66us) base_compaction.cpp:44] rowsets picked
+0610 11:24:51.784439 (+108055410us) compaction.cpp:46] got concurrency lock and start to do compaction
+0610 11:24:51.784818 (+ 379us) compaction.cpp:74] prepare finished
+0610 11:26:33.359265 (+101574447us) compaction.cpp:87] merge rowsets finished
+0610 11:26:33.484481 (+125216us) compaction.cpp:102] output rowset built
+0610 11:26:33.484482 (+ 1us) compaction.cpp:106] check correctness finished
+0610 11:26:33.513197 (+ 28715us) compaction.cpp:110] modify rowsets finished
+0610 11:26:33.513300 (+ 103us) base_compaction.cpp:49] compaction finished
+0610 11:26:33.513441 (+ 141us) base_compaction.cpp:56] unused rowsets have been moved to GC queue
+Metrics: {"filtered_rows":0,"input_row_num":3346807,"input_rowsets_count":42,"input_rowsets_data_size":1256413170,"input_segments_num":44,"merge_rowsets_latency_us":101574444,"merged_rows":0,"output_row_num":3346807,"output_rowset_data_size":1228439659,"output_segments_num":6}
+```
-### `force_recovery`
+#### `cumulative_compaction_trace_threshold`
-### `fragment_pool_queue_size`
+* 类型：int32
+* 描述：打印cumulative compaction的trace信息的阈值，单位秒
+ - 与base_compaction_trace_threshold类似。
+* 默认值：2
-默认值：2048
+#### `compaction_task_num_per_disk`
-单节点上能够处理的查询请求上限
+* 类型：int32
+* 描述：每个磁盘（HDD）可以并发执行的compaction任务数量。
+* 默认值：2
-### `fragment_pool_thread_num_min`
+#### `compaction_task_num_per_fast_disk`
-默认值：64
+* 类型：int32
+* 描述：每个高速磁盘（SSD）可以并发执行的compaction任务数量。
+* 默认值：4
-### `fragment_pool_thread_num_max`
+#### `cumulative_compaction_rounds_for_each_base_compaction_round`
-默认值：256
+* 类型：int32
+* 描述：Compaction任务的生产者每次连续生产多少轮cumulative compaction任务后生产一轮base compaction。
+* 默认值：9
-查询线程数，默认最小启动64个线程，后续查询请求动态创建线程，最大创建256个线程
+#### `cumulative_compaction_policy`
-### `heartbeat_service_port`
+* 类型：string
+* 描述：配置 cumulative compaction 阶段的合并策略，目前实现了两种合并策略，num_based和size_based
+ - 详细说明，ordinary，是最初版本的cumulative compaction合并策略，做一次cumulative compaction之后直接base compaction流程。size_based，通用策略是ordinary策略的优化版本，仅当rowset的磁盘体积在相同数量级时才进行版本合并。合并之后满足条件的rowset进行晋升到base compaction阶段。能够做到在大量小批量导入的情况下：降低base compact的写入放大率，并在读取放大率和空间放大率之间进行权衡，同时减少了文件版本的数据。
+* 默认值：size_based
-* 类型：int32
-* 描述：BE 上心跳服务端口（thrift），用于接收来自 FE 的心跳
-* 默认值：9050
-
-### `heartbeat_service_thread_count`
+#### `max_cumu_compaction_threads`
* 类型：int32
-* 描述：执行BE上心跳服务的线程数，默认为1，不建议修改
-* 默认值：1
-
-### `ignore_broken_disk`
-
- 当BE启动时，会检查``storage_root_path`` 配置下的所有路径。
-
- - `ignore_broken_disk=true`
-
- 如果路径不存在或路径下无法进行读写文件(坏盘)，将忽略此路径，如果有其他可用路径则不中断启动。
-
- - `ignore_broken_disk=false`
-
- 如果路径不存在或路径下无法进行读写文件(坏盘)，将中断启动失败退出。
-
- 默认为false
-
-### `ignore_load_tablet_failure`
-
-* 类型：bool
-* 描述：用来决定在有tablet 加载失败的情况下是否忽略错误，继续启动be
-* 默认值：false
-
-BE启动时，会对每个数据目录单独启动一个线程进行 tablet header 元信息的加载。默认配置下，如果某个数据目录有 tablet 加载失败，则启动进程会终止。同时会在 `be.INFO` 日志中看到如下错误信息：
-
-```
-load tablets from header failed, failed tablets size: xxx, path=xxx
-```
-
-表示该数据目录共有多少 tablet 加载失败。同时，日志中也会有加载失败的 tablet 的具体信息。此时需要人工介入来对错误原因进行排查。排查后，通常有两种方式进行恢复：
-
-1. tablet 信息不可修复，在确保其他副本正常的情况下，可以通过 `meta_tool` 工具将错误的tablet删除。
-2. 将 `ignore_load_tablet_failure` 设置为 true，则 BE 会忽略这些错误的 tablet，正常启动。
+* 描述：Cumulative Compaction线程池中线程数量的最大值。
+* 默认值：10
-### `ignore_rowset_stale_unconsistent_delete`
+#### `enable_segcompaction`
* 类型：bool
-* 描述：用来决定当删除过期的合并过的rowset后无法构成一致的版本路径时，是否仍要删除。
+* 描述：在导入时进行 segment compaction 来减少 segment 数量
* 默认值：false
-合并的过期 rowset 版本路径会在半个小时后进行删除。在异常下，删除这些版本会出现构造不出查询一致路径的问题，当配置为false时，程序检查比较严格，程序会直接报错退出。
-当配置为true时，程序会正常运行，忽略这个错误。一般情况下，忽略这个错误不会对查询造成影响，仅会在fe下发了合并过的版本时出现-230错误。
-
-### `inc_rowset_expired_sec`
-
-默认值：1800 （s）
-
-导入激活的数据，存储引擎保留的时间，用于增量克隆
-
-### `index_stream_cache_capacity`
-
-默认值：10737418240
+#### `segcompaction_threshold_segment_num`
-BloomFilter/Min/Max等统计信息缓存的容量
-
-### `kafka_api_version_request`
-
-默认值：true
-
-如果依赖的 kafka 版本低于0.10.0.0, 该值应该被设置为 false。
+* 类型：int32
+* 描述：当 segment 数量超过此阈值时触发 segment compaction
+* 默认值：10
-### `kafka_broker_version_fallback`
+#### `segcompaction_small_threshold`
-默认值：0.10.0
+* 类型：int32
+* 描述：当 segment 文件超过此大小时则会在 segment compaction 时被 compact，否则跳过
+* 默认值：1048576
-如果依赖的 kafka 版本低于0.10.0.0, 当 kafka_api_version_request 值为 false 的时候，将使用回退版本 kafka_broker_version_fallback 设置的值，有效值为：0.9.0.x、0.8.x.y。
+#### `disable_compaction_trace_log`
-### `load_data_reserve_hours`
+* 类型: bool
+* 描述: 关闭compaction的trace日志
+ - 如果设置为true，`cumulative_compaction_trace_threshold` 和 `base_compaction_trace_threshold` 将不起作用。并且trace日志将关闭。
+* 默认值: true
-默认值：4 （小时）
-用于mini load。mini load数据文件将在此时间后被删除
+### 导入
-### `load_error_log_reserve_hours`
+#### `enable_stream_load_record`
-默认值：48（小时）
+* 类型: bool
+* 描述: 是否开启 stream load 操作记录，默认是不启用
+* 默认值: false
-load错误日志将在此时间后删除
+#### `load_data_reserve_hours`
-### `load_process_max_memory_limit_bytes`
+* 描述: 用于mini load。mini load数据文件将在此时间后被删除
+* 默认值: 4（h）
-默认值：107374182400
+#### `push_worker_count_high_priority`
-单节点上所有的导入线程占据的内存上限，默认值：100G
+* 描述: 导入线程数，用于处理HIGH优先级任务
+* 默认值: 3
-将这些默认值设置得很大，因为我们不想在用户升级 Doris 时影响负载性能。 如有必要，用户应正确设置这些配置。
+#### `push_worker_count_normal_priority`
-### `load_process_max_memory_limit_percent`
+* 描述: 导入线程数，用于处理NORMAL优先级任务
+* 默认值: 3
-默认值：50
+#### `load_error_log_reserve_hours`
-单节点上所有的导入线程占据的内存上限比例，默认50%
+* 描述: load错误日志将在此时间后删除
+* 默认值: 48（h）
-将这些默认值设置得很大，因为我们不想在用户升级 Doris 时影响负载性能。 如有必要，用户应正确设置这些配置。
+#### `load_process_max_memory_limit_bytes`
-### `load_process_soft_mem_limit_percent`
+* 描述: 单节点上所有的导入线程占据的内存上限。
+ - 将这些默认值设置得很大，因为我们不想在用户升级 Doris 时影响负载性能。 如有必要，用户应正确设置这些配置。
+* 默认值: 107374182400 （100G）
-默认值：50
+#### `load_process_max_memory_limit_percent`
-soft limit是指站单节点导入内存上限的比例。例如所有导入任务导入的内存上限是20GB，则soft limit默认为该值的50%，即10GB。导入内存占用超过soft limit时，会挑选占用内存最大的作业进行下刷以提前释放内存空间，默认50%
+* 描述: 单节点上所有的导入线程占据的内存上限比例
+ - 将这些默认值设置得很大，因为我们不想在用户升级 Doris 时影响负载性能。 如有必要，用户应正确设置这些配置。
+* 默认值: 50 （%）
-### `log_buffer_level`
+#### `load_process_soft_mem_limit_percent`
-默认值：空
+* 描述: soft limit是指站单节点导入内存上限的比例。例如所有导入任务导入的内存上限是20GB，则soft limit默认为该值的50%，即10GB。导入内存占用超过soft limit时，会挑选占用内存最大的作业进行下刷以提前释放内存空间。
+* 默认值: 50（%）
-日志刷盘的策略，默认保持在内存中
+#### `routine_load_thread_pool_size`
-### `madvise_huge_pages`
+* 描述: routine load任务的线程池大小。 这应该大于 FE 配置 'max_concurrent_task_num_per_be'
+* 默认值: 10
-默认值：false
+#### `single_replica_load_brpc_num_threads`
-是否使用linux内存大页，默认不启用
+* 类型: int32
+* 描述: 单副本数据导入功能中，Master副本和Slave副本之间通信的线程数量。导入并发增大时，可以适当调大该参数来保证Slave副本及时同步Master副本数据。
+* 默认值: 64
-### `make_snapshot_worker_count`
+#### `single_replica_load_download_num_workers`
-默认值：5
+* 类型: int32
+* 描述: 单副本数据导入功能中，Slave副本通过HTTP从Master副本下载数据文件的线程数。导入并发增大时，可以适当调大该参数来保证Slave副本及时同步Master副本数据。
+* 默认值: 64
-制作快照的线程数
+#### `slave_replica_writer_rpc_timeout_sec`
-### `max_client_cache_size_per_host`
+* 类型: int32
+* 描述: 单副本数据导入功能中，Master副本和Slave副本之间通信的RPC超时时间。
+* 默认值: 60
-默认值：10
+#### `max_segment_num_per_rowset`
-每个主机的最大客户端缓存数，BE 中有多种客户端缓存，但目前我们使用相同的缓存大小配置。 如有必要，使用不同的配置来设置不同的客户端缓存。
+* 类型: int32
+* 描述: 用于限制导入时，新产生的rowset中的segment数量。如果超过阈值，导入会失败并报错 -238。过多的 segment 会导致compaction占用大量内存引发 OOM 错误。
+* 默认值: 200
-### `max_base_compaction_threads`
+#### `high_priority_flush_thread_num_per_store`
* 类型：int32
-* 描述：Base Compaction线程池中线程数量的最大值。
-* 默认值：4
+* 描述：每个存储路径所分配的用于高优导入任务的 flush 线程数量。
+* 默认值：1
-### `max_cumu_compaction_threads`
+#### `routine_load_consumer_pool_size`
* 类型：int32
-* 描述：Cumulative Compaction线程池中线程数量的最大值。
+* 描述：routine load 所使用的 data consumer 的缓存数量。
* 默认值：10
-### `max_consumer_num_per_group`
-
-默认值：3
-
-一个数据消费者组中的最大消费者数量，用于routine load
-
-### `cumulative_compaction_min_deltas`
-
-默认值：5
-
-cumulative compaction策略：最小增量文件的数量
-
-### `cumulative_compaction_max_deltas`
-
-默认值：1000
-
-cumulative compaction策略：最大增量文件的数量
-
-### `max_download_speed_kbps`
-
-默认值：50000 （kb/s）
-
-最大下载速度限制
-
-### `max_free_io_buffers`
-
-默认值：128
-
-对于每个 io 缓冲区大小，IoMgr 将保留的最大缓冲区数从 1024B 到 8MB 的缓冲区，最多约为 2GB 的缓冲区。
-
-### `max_garbage_sweep_interval`
-
-默认值：3600
-
-磁盘进行垃圾清理的最大间隔，默认一个小时
-
-### `max_memory_sink_batch_count`
-
-默认值：20
-
-最大外部扫描缓存批次计数，表示缓存max_memory_cache_batch_count * batch_size row，默认为20，batch_size的默认值为1024，表示将缓存20 * 1024行
-
-### `max_percentage_of_error_disk`
+#### `load_task_high_priority_threshold_second`
* 类型：int32
-* 描述：存储引擎允许存在损坏硬盘的百分比，损坏硬盘超过改比例后，BE将会自动退出。
-* 默认值：0
-
-### `max_pushdown_conditions_per_column`
-
-* 类型：int
-* 描述：用于限制一个查询请求中，针对单个列，能够下推到存储引擎的最大条件数量。在查询计划执行的过程中，一些列上的过滤条件可以下推到存储引擎，这样可以利用存储引擎中的索引信息进行数据过滤，减少查询需要扫描的数据量。比如等值条件、IN 谓词中的条件等。这个参数在绝大多数情况下仅影响包含 IN 谓词的查询。如 `WHERE colA IN (1,2,3,4,...)`。较大的数值意味值 IN 谓词中更多的条件可以推送给存储引擎，但过多的条件可能会导致随机读的增加，某些情况下可能会降低查询效率。该配置可以单独进行会话级别的配置，具体可参阅 [变量](../../advanced/variables.md) 中 `max_pushdown_conditions_per_column ` 的说明。
-* 默认值：1024
-
-* 示例
-
- 表结构为 `id INT, col2 INT, col3 varchar(32), ...`。
-
- 查询请求为 `... WHERE id IN (v1, v2, v3, ...)`
-
- 如果 IN 谓词中的条件数量超过了该配置，则可以尝试增加该配置值，观察查询响应是否有所改善。
-
-### `max_runnings_transactions_per_txn_map`
-
-默认值：100
-
-txn 管理器中每个 txn_partition_map 的最大 txns 数，这是一种自我保护，以避免在管理器中保存过多的 txns
-
-### `max_send_batch_parallelism_per_job`
-
-* 类型：int
-* 描述：OlapTableSink 发送批处理数据的最大并行度，用户为 `send_batch_parallelism` 设置的值不允许超过 `max_send_batch_parallelism_per_job` ，如果超过， `send_batch_parallelism` 将被设置为 `max_send_batch_parallelism_per_job` 的值。
-* 默认值：5
-
-### `max_tablet_num_per_shard`
+* 描述：当一个导入任务的超时时间小于这个阈值是，Doris 将认为他是一个高优任务。高优任务会使用独立的 flush 线程池。
+* 默认：120
-默认：1024
+#### `min_load_rpc_timeout_ms`
-每个shard的tablet数目，用于划分tablet，防止单个目录下tablet子目录过多
+* 类型：int32
+* 描述：load 作业中各个rpc 的最小超时时间。
+* 默认：20
-### `max_tablet_version_num`
+#### `kafka_api_version_request`
-* 类型：int
-* 描述：限制单个 tablet 最大 version 的数量。用于防止导入过于频繁，或 compaction 不及时导致的大量 version 堆积问题。当超过限制后，导入任务将被拒绝。
-* 默认值：500
+* 类型：bool
+* 描述：如果依赖的 kafka 版本低于0.10.0.0, 该值应该被设置为 false。
+* 默认：true
-### `mem_limit`
+#### `kafka_broker_version_fallback`
-* 类型：string
-* 描述：限制BE进程使用服务器最大内存百分比。用于防止BE内存挤占太多的机器内存，该参数必须大于0，当百分大于100%之后，该值会默认为100%。
-* 默认值：80%
+* 描述：如果依赖的 kafka 版本低于0.10.0.0, 当 kafka_api_version_request 值为 false 的时候，将使用回退版本 kafka_broker_version_fallback 设置的值，有效值为：0.9.0.x、0.8.x.y。
+* 默认：0.10.0
-### `memory_mode`
+#### `max_consumer_num_per_group`
-* 类型：string
-* 描述：控制tcmalloc的回收。如果配置为performance，内存使用超过mem_limit的90%时，doris会释放tcmalloc cache中的内存，如果配置为compact，内存使用超过mem_limit的50%时，doris会释放tcmalloc cache中的内存。
-* 默认值：performance
+* 描述：一个数据消费者组中的最大消费者数量，用于routine load。
+* 默认：3
-### `max_sys_mem_available_low_water_mark_bytes`
+#### `streaming_load_max_mb`
* 类型：int64
-* 描述：系统`/proc/meminfo/MemAvailable` 的最大低水位线，单位字节，默认1.6G，实际低水位线=min(1.6G，MemTotal * 10%)，避免在大于16G的机器上浪费过多内存。调大max，在大于16G内存的机器上，将为Full GC预留更多的内存buffer；反之调小max，将尽可能充分使用内存。
-* 默认值：1717986918
-
-### `memory_limitation_per_thread_for_schema_change_bytes`
-
-默认值：2147483648
-
-单个schema change任务允许占用的最大内存
-
-### `memory_maintenance_sleep_time_s`
-
-默认值：10
-
-内存维护迭代之间的休眠时间（以秒为单位）
+* 描述：用于限制数据格式为 csv 的一次 Stream load 导入中，允许的最大数据量。
+ - Stream Load 一般适用于导入几个GB以内的数据，不适合导入过大的数据。
+* 默认值： 10240 （MB）
+* 可动态修改：是
-### `memory_max_alignment`
+#### `streaming_load_json_max_mb`
-默认值：16
+* 类型：int64
+* 描述：用于限制数据格式为 json 的一次 Stream load 导入中，允许的最大数据量。单位 MB。
+ - 一些数据格式，如 JSON，无法进行拆分处理，必须读取全部数据到内存后才能开始解析，因此，这个值用于限制此类格式数据单次导入最大数据量。
+* 默认值： 100
+* 可动态修改：是
-最大校对内存
+### 线程
-### `read_size`
+#### `delete_worker_count`
-默认值：8388608
+* 描述：执行数据删除任务的线程数
+* 默认值：3
-读取大小是发送到 os 的读取大小。 在延迟和整个过程之间进行权衡，试图让磁盘保持忙碌但不引入寻道。 对于 8 MB 读取，随机 io 和顺序 io 的性能相似
+#### `clear_transaction_task_worker_count`
-### `min_buffer_size`
+* 描述：用于清理事务的线程数
+* 默认值：1
-默认值：1024
+#### `clone_worker_count`
-最小读取缓冲区大小（以字节为单位）
+* 描述：用于执行克隆任务的线程数
+* 默认值：3
-### `min_compaction_threads`
+#### `be_service_threads`
* 类型：int32
-* 描述：Compaction线程池中线程数量的最小值。
-* 默认值：10
-
-### `min_file_descriptor_number`
+* 描述：BE 上 thrift server service的执行线程数，代表可以用于执行FE请求的线程数。
+* 默认值：64
-默认值：60000
+#### `download_worker_count`
-BE进程的文件句柄limit要求的下限
+* 描述：下载线程数
+* 默认值：1
-### `min_garbage_sweep_interval`
+#### `drop_tablet_worker_count`
-默认值：180
+* 描述：删除tablet的线程数
+* 默认值：3
-磁盘进行垃圾清理的最小间隔，时间秒
+#### `flush_thread_num_per_store`
-### `mmap_buffers`
+* 描述：每个store用于刷新内存表的线程数
+* 默认值：2
-默认值：false
+#### `num_threads_per_core`
-是否使用mmap分配内存，默认不使用
+* 描述：控制每个内核运行工作的线程数。 通常选择 2 倍或 3 倍的内核数量。 这使核心保持忙碌而不会导致过度抖动
+* 默认值：3
-### `num_cores`
+#### `num_threads_per_disk`
-* 类型：int32
-* 描述：BE可以使用CPU的核数。当该值为0时，BE将从/proc/cpuinfo之中获取本机的CPU核数。
+* 描述：每个磁盘的最大线程数也是每个磁盘的最大队列深度
* 默认值：0
-### `num_disks`
-
-默认值：0
-
-控制机器上的磁盘数量。 如果为 0，则来自系统设置。
-
-### `num_threads_per_core`
-
-默认值：3
-
-控制每个内核运行工作的线程数。 通常选择 2 倍或 3 倍的内核数量。 这使核心保持忙碌而不会导致过度抖动
-
-### `num_threads_per_disk`
-
-默认值：0
-
-每个磁盘的最大线程数也是每个磁盘的最大队列深度
-
-### `number_slave_replica_download_threads`
-
-默认值: 64
-
-每个BE节点上slave副本同步Master副本数据的线程数，用于单副本数据导入功能。
-
-### `number_tablet_writer_threads`
-
-默认值：16
-
-tablet写线程数
+#### `number_slave_replica_download_threads`
-### `path_gc_check`
-
-默认值：true
-
-是否启用回收扫描数据线程检查，默认启用
-
-### `path_gc_check_interval_second`
-
-默认值：86400
+* 描述：每个BE节点上slave副本同步Master副本数据的线程数，用于单副本数据导入功能。
+* 默认值：64
-回收扫描数据线程检查时间间隔，单位秒
+#### `publish_version_worker_count`
-### `path_gc_check_step`
+* 描述：生效版本的线程数
+* 默认值：8
-默认值：1000
+#### `upload_worker_count`
-### `path_gc_check_step_interval_ms`
+* 描述：上传文件最大线程数
+* 默认值：1
-默认值：10 (ms)
+#### `webserver_num_workers`
-### `path_scan_interval_second`
+* 描述：webserver默认工作线程数
+* 默认值：48
-默认值：86400
+#### `send_batch_thread_pool_thread_num`
-### `pending_data_expire_time_sec`
+* 类型：int32
+* 描述：SendBatch线程池线程数目。在NodeChannel的发送数据任务之中，每一个NodeChannel的SendBatch操作会作为一个线程task提交到线程池之中等待被调度，该参数决定了SendBatch线程池的大小。
+* 默认值：64
-默认值：1800
+#### `send_batch_thread_pool_queue_size`
-存储引擎保留的未生效数据的最大时长，默认单位：秒
+* 类型：int32
+* 描述：SendBatch线程池的队列长度。在NodeChannel的发送数据任务之中，每一个NodeChannel的SendBatch操作会作为一个线程task提交到线程池之中等待被调度，而提交的任务数目超过线程池队列的长度之后，后续提交的任务将阻塞直到队列之中有新的空缺。
+* 默认值：102400
-### `periodic_counter_update_period_ms`
+#### `make_snapshot_worker_count`
-默认值：500
+* 描述：制作快照的线程数
+* 默认值：5
-更新速率计数器和采样计数器的周期，默认单位：毫秒
+#### `release_snapshot_worker_count`
-### `plugin_path`
+* 描述：释放快照的线程数
+* 默认值：5
-默认值：${DORIS_HOME}/plugin
+### 内存
-插件路径
+#### `disable_mem_pools`
-### `port`
+* 类型：bool
+* 描述：是否禁用内存缓存池
+* 默认值：false
-* 类型：int32
-* 描述：BE单测时使用的端口，在实际环境之中无意义，可忽略。
-* 默认值：20001
+#### `buffer_pool_clean_pages_limit`
-### `pprof_profile_dir`
+* 描述：清理可能被缓冲池保存的Page
+* 默认值：50%
-默认值：${DORIS_HOME}/log
+#### `buffer_pool_limit`
-pprof profile保存目录
+* 类型：string
+* 描述：buffer pool之中最大的可分配内存
+ - BE缓存池最大的内存可用量，buffer pool是BE新的内存管理结构，通过buffer page来进行内存管理，并能够实现数据的落盘。并发的所有查询的内存申请都会通过buffer pool来申请。当前buffer pool仅作用在**AggregationNode**与**ExchangeNode**。
+* 默认值：20%
-### `priority_networks`
+#### `chunk_reserved_bytes_limit`
-默认值：空
+* 描述：Chunk Allocator的reserved bytes限制，通常被设置为 mem_limit 的百分比。默认单位字节，值必须是2的倍数，且必须大于0，如果大于物理内存，将被设置为物理内存大小。增加这个变量可以提高性能，但是会获得更多其他模块无法使用的空闲内存。
+* 默认值：20%
-为那些有很多 ip 的服务器声明一个选择策略。 请注意，最多应该有一个 ip 与此列表匹配。 这是一个以分号分隔格式的列表，用 CIDR 表示法，例如 10.10.10.0/24 ， 如果没有匹配这条规则的ip，会随机选择一个。
+#### `madvise_huge_pages`
-### `priority_queue_remaining_tasks_increased_frequency`
+* 类型：bool
+* 描述：是否使用linux内存大页
+* 默认值：false
-默认值：512
+#### `max_memory_sink_batch_count`
- the increased frequency of priority for remaining tasks in BlockingPriorityQueue
+* 描述：最大外部扫描缓存批次计数，表示缓存max_memory_cache_batch_count * batch_size row，默认为20，batch_size的默认值为1024，表示将缓存20 * 1024行
+* 默认值：20
-### `publish_version_worker_count`
+#### `memory_limitation_per_thread_for_schema_change`
-默认值：8
+* 描述：单个schema change任务允许占用的最大内存
+* 默认值：2 （GB）
-生效版本的线程数
+#### `memory_max_alignment`
-### `pull_load_task_dir`
+* 描述：最大校对内存
+* 默认值：16
-默认值：${DORIS_HOME}/var/pull_load
+#### `mmap_buffers`
-拉取laod任务的目录
+* 描述：是否使用mmap分配内存
+* 默认值：false
-### `push_worker_count_high_priority`
+#### `download_cache_buffer_size`
-默认值：3
+* 类型: int64
+* 描述: 下载缓存时用于接收数据的buffer的大小。
+* 默认值: 10485760
-导入线程数，用于处理HIGH优先级任务
+#### `zone_map_row_num_threshold`
-### `push_worker_count_normal_priority`
+* 类型： int32
+* 描述: 如果一个page中的行数小于这个值就不会创建zonemap，用来减少数据膨胀
+* 默认值: 20
-默认值：3
+#### `enable_tcmalloc_hook`
-导入线程数，用于处理NORMAL优先级任务
+* 类型：bool
+* 描述：是否Hook TCmalloc new/delete，目前在Hook中统计thread local MemTracker。
+* 默认值：true
-### `query_scratch_dirs`
+#### `memory_mode`
* 类型：string
-* 描述：BE进行数据落盘时选取的目录来存放临时数据，与存储路径配置类似，多目录之间用;分隔。
-* 默认值：${DORIS_HOME}
-
-### `release_snapshot_worker_count`
-
-默认值：5
-
-释放快照的线程数
-
-### `report_disk_state_interval_seconds`
-
-默认值：60
-
-代理向 FE 报告磁盘状态的间隔时间（秒）
+* 描述：控制tcmalloc的回收。如果配置为performance，内存使用超过mem_limit的90%时，doris会释放tcmalloc cache中的内存，如果配置为compact，内存使用超过mem_limit的50%时，doris会释放tcmalloc cache中的内存。
+* 默认值：performance
-### `report_tablet_interval_seconds`
+#### `max_sys_mem_available_low_water_mark_bytes`
-默认值：60
+* 类型：int64
+* 描述：系统`/proc/meminfo/MemAvailable` 的最大低水位线，单位字节，默认1.6G，实际低水位线=min(1.6G，MemTotal * 10%)，避免在大于16G的机器上浪费过多内存。调大max，在大于16G内存的机器上，将为Full GC预留更多的内存buffer；反之调小max，将尽可能充分使用内存。
+* 默认值：1717986918
-代理向 FE 报告 olap 表的间隔时间（秒）
+#### `memory_limitation_per_thread_for_schema_change_bytes`
-### `report_task_interval_seconds`
+* 描述：单个schema change任务允许占用的最大内存
+* 默认值：2147483648
-默认值：10
+#### `mem_tracker_consume_min_size_bytes`
-代理向 FE 报告任务签名的间隔时间（秒）
+* 类型: int32
+* 描述: TCMalloc Hook consume/release MemTracker时的最小长度，小于该值的consume size会持续累加，避免频繁调用MemTracker的consume/release，减小该值会增加consume/release的频率，增大该值会导致MemTracker统计不准，理论上一个MemTracker的统计值与真实值相差 = (mem_tracker_consume_min_size_bytes * 这个MemTracker所在的BE线程数)。
+* 默认值: 1048576
-### `result_buffer_cancelled_interval_time`
+#### `cache_clean_interval`
-默认值：300
+* 描述: 文件句柄缓存清理的间隔，用于清理长期不用的文件句柄。同时也是Segment Cache的清理间隔时间。
+* 默认值: 1800 (s)
-结果缓冲区取消时间（单位：秒）
+#### `min_buffer_size`
-### `routine_load_thread_pool_size`
+* 描述: 最小读取缓冲区大小
+* 默认值: 1024 (byte)
-默认值：10
+#### `write_buffer_size`
-routine load任务的线程池大小。 这应该大于 FE 配置 'max_concurrent_task_num_per_be'（默认 5）
+* 描述: 刷写前缓冲区的大小
+ - 导入数据在 BE 上会先写入到一个内存块，当这个内存块达到阈值后才会写回磁盘。默认大小是 100MB。过小的阈值可能导致 BE 上存在大量的小文件。可以适当提高这个阈值减少文件数量。但过大的阈值可能导致 RPC 超时
+* 默认值: 104857600
-### `row_nums_check`
+#### `remote_storage_read_buffer_mb`
-默认值：true
+* 类型: int32
+* 描述: 读取hdfs或者对象存储上的文件时，使用的缓存大小。
+ - 增大这个值，可以减少远端数据读取的调用次数，但会增加内存开销。
+* 默认值: 16MB
-检查 BE/CE 和schema更改的行号。 true 是打开的，false 是关闭的。
+#### `segment_cache_capacity`
-### `scan_context_gc_interval_min`
+* 类型: int32
+* 描述: Segment Cache 缓存的 Segment 最大数量
+ - 默认值目前只是一个经验值，可能需要根据实际场景修改。增大该值可以缓存更多的segment从而避免一些IO。减少该值则会降低内存使用。
+* 默认值: 1000000
-默认值：5
+#### `file_cache_type`
-此配置用于上下文gc线程调度周期 ， 注意：单位为分钟，默认为 5 分钟
+* 类型：string
+* 描述：缓存文件的类型。whole_file_cache：将segment文件整个下载，sub_file_cache：将segment文件按大小切分成多个文件。
+* 默认值：""
-### `send_batch_thread_pool_thread_num`
+### `file_cache_alive_time_sec`
-* 类型：int32
-* 描述：SendBatch线程池线程数目。在NodeChannel的发送数据任务之中，每一个NodeChannel的SendBatch操作会作为一个线程task提交到线程池之中等待被调度，该参数决定了SendBatch线程池的大小。
-* 默认值：64
+* 类型：int64
+* 描述：缓存文件的保存时间，单位：秒
+* 默认值：604800（1个星期）
-### `send_batch_thread_pool_queue_size`
+#### `max_sub_cache_file_size`
-* 类型：int32
-* 描述：SendBatch线程池的队列长度。在NodeChannel的发送数据任务之中，每一个NodeChannel的SendBatch操作会作为一个线程task提交到线程池之中等待被调度，而提交的任务数目超过线程池队列的长度之后，后续提交的任务将阻塞直到队列之中有新的空缺。
-* 默认值：102400
+* 类型：int64
+* 描述：缓存文件使用sub_file_cache时，切分文件的最大大小，单位B
+* 默认值：104857600（100MB）
-### `download_cache_thread_pool_thread_num`
+#### `download_cache_thread_pool_thread_num`
* 类型: int32
* 描述: DownloadCache线程池线程数目. 在FileCache的缓存下载任务之中, 缓存下载操作会作为一个线程task提交到线程池之中等待被调度，该参数决定了DownloadCache线程池的大小。
* 默认值：48
-### `download_cache_thread_pool_queue_size`
+#### `download_cache_thread_pool_queue_size`
* Type: int32
* 描述: DownloadCache线程池线程数目. 在FileCache的缓存下载任务之中, 缓存下载操作会作为一个线程task提交到线程池之中等待被调度，而提交的任务数目超过线程池队列的长度之后，后续提交的任务将阻塞直到队列之中有新的空缺。
* 默认值：102400
-### `download_cache_buffer_size`
+#### `generate_cache_cleaner_task_interval_sec`
-* 类型: int64
-* 描述: 下载缓存时用于接收数据的buffer的大小。
-* 默认值: 10485760
+* 类型：int64
+* 描述：缓存文件的清理间隔，单位：秒
+* 默认值：43200（12小时）
-### `serialize_batch`
+#### `path_gc_check`
-默认值：false
+* 类型：bool
+* 描述：是否启用回收扫描数据线程检查
+* 默认值：true
-BE之间rpc通信是否序列化RowBatch，用于查询层之间的数据传输
+#### `path_gc_check_interval_second`
-### `single_replica_load_brpc_port`
+* 描述：回收扫描数据线程检查时间间隔
+* 默认值：86400 (s)
-* 类型: int32
-* 描述: 单副本数据导入功能中，Master副本和Slave副本之间通信的RPC端口。Master副本flush完成之后通过RPC通知Slave副本同步数据，以及Slave副本同步数据完成后通过RPC通知Master副本。系统为单副本数据导入过程中Master副本和Slave副本之间通信开辟了独立的BRPC线程池，以避免导入并发较大时副本之间的数据同步抢占导入数据分发和查询任务的线程资源。
-* 默认值: 9070
+#### `path_gc_check_step`
-### `single_replica_load_brpc_num_threads`
+* 默认值：1000
-* 类型: int32
-* 描述: 单副本数据导入功能中，Master副本和Slave副本之间通信的线程数量。导入并发增大时，可以适当调大该参数来保证Slave副本及时同步Master副本数据。
-* 默认值: 64
+#### `path_gc_check_step_interval_ms`
-### `single_replica_load_download_port`
+* 默认值：10 (ms)
-* 类型: int32
-* 描述: 单副本数据导入功能中，Slave副本通过HTTP从Master副本下载数据文件的端口。系统为单副本数据导入过程中Slave副本从Master副本下载数据文件开辟了独立的HTTP线程池，以避免导入并发较大时Slave副本下载数据文件抢占其他http任务的线程资源。
-* 默认值: 8050
+#### `path_scan_interval_second`
-### `single_replica_load_download_num_workers`
+* 默认值：86400
-* 类型: int32
-* 描述: 单副本数据导入功能中，Slave副本通过HTTP从Master副本下载数据文件的线程数。导入并发增大时，可以适当调大该参数来保证Slave副本及时同步Master副本数据。
-* 默认值: 64
+#### `scan_context_gc_interval_min`
-### `slave_replica_writer_rpc_timeout_sec`
+* 描述：此配置用于上下文gc线程调度周期
+* 默认值：5 (分钟)
-* 类型: int32
-* 描述: 单副本数据导入功能中，Master副本和Slave副本之间通信的RPC超时时间。
-* 默认值: 60
+### 存储
-### `sleep_one_second`
-+ 类型：int32
-+ 描述：全局变量，用于BE线程休眠1秒，不应该被修改
-+ 默认值：1
+#### `default_num_rows_per_column_file_block`
-### `small_file_dir`
+* 类型：int32
+* 描述：配置单个RowBlock之中包含多少行的数据。
+* 默认值：1024
-默认值：${DORIS_HOME}/lib/small_file/
+#### `disable_storage_page_cache`
-用于保存 SmallFileMgr 下载的文件的目录
+* 类型：bool
+* 描述：是否进行使用page cache进行index的缓存，该配置仅在BETA存储格式时生效
+* 默认值：false
-### `snapshot_expire_time_sec`
+#### `disk_stat_monitor_interval`
-默认值：172800
+* 描述：磁盘状态检查时间间隔
+* 默认值：5 （s）
-快照文件清理的间隔，默认值：48小时
+#### `max_free_io_buffers`
-### `status_report_interval`
+* 描述：对于每个 io 缓冲区大小，IoMgr 将保留的最大缓冲区数从 1024B 到 8MB 的缓冲区，最多约为 2GB 的缓冲区。
+* 默认值：128
-默认值：5
+#### `max_garbage_sweep_interval`
-配置文件报告之间的间隔；单位：秒
+* 描述：磁盘进行垃圾清理的最大间隔
+* 默认值：3600 (s)
-### `storage_flood_stage_left_capacity_bytes`
+#### `max_percentage_of_error_disk`
-默认值：1073741824
+* 类型：int32
+* 描述：存储引擎允许存在损坏硬盘的百分比，损坏硬盘超过改比例后，BE将会自动退出。
+* 默认值：0
-数据目录应该剩下的最小存储空间，默认1G
+#### `read_size`
-### `storage_flood_stage_usage_percent`
+* 描述：读取大小是发送到 os 的读取大小。 在延迟和整个过程之间进行权衡，试图让磁盘保持忙碌但不引入寻道。 对于 8 MB 读取，随机 io 和顺序 io 的性能相似
+* 默认值：8388608
-默认值：90 （90%）
+#### `min_garbage_sweep_interval`
-storage_flood_stage_usage_percent和storage_flood_stage_left_capacity_bytes两个配置限制了数据目录的磁盘容量的最大使用。 如果这两个阈值都达到，则无法将更多数据写入该数据目录。 数据目录的最大已用容量百分比
+* 描述：磁盘进行垃圾清理的最小间隔
+* 默认值：180 (s)
-### `storage_medium_migrate_count`
+#### `pprof_profile_dir`
-默认值：1
+* 描述：pprof profile保存目录
+* 默认值：${DORIS_HOME}/log
-要克隆的线程数
+#### `small_file_dir`
-### `storage_page_cache_limit`
+* 描述：用于保存 SmallFileMgr 下载的文件的目录
+* 默认值：${DORIS_HOME}/lib/small_file/
-默认值：20%
+#### `user_function_dir`
-缓存存储页大小
+* 描述：udf函数目录
+* 默认值：${DORIS_HOME}/lib/udf
-### `storage_page_cache_shard_size`
+#### `storage_flood_stage_left_capacity_bytes`
-默认值: 16
+* 描述：数据目录应该剩下的最小存储空间，默认1G
+* 默认值：1073741824
-StoragePageCache的分片大小，值为 2^n (n=0,1,2,...)。建议设置为接近BE CPU核数的值，可减少StoragePageCache的锁竞争。
-### `index_page_cache_percentage`
-* 类型：int32
-* 描述：索引页缓存占总页面缓存的百分比，取值为[0, 100]。
-* 默认值：10
+#### `storage_flood_stage_usage_percent`
-### `storage_root_path`
+* 描述：storage_flood_stage_usage_percent和storage_flood_stage_left_capacity_bytes两个配置限制了数据目录的磁盘容量的最大使用。 如果这两个阈值都达到，则无法将更多数据写入该数据目录。 数据目录的最大已用容量百分比
+* 默认值：90 （90%）
-* 类型：string
+#### `storage_medium_migrate_count`
-* 描述：BE数据存储的目录,多目录之间用英文状态的分号`;`分隔。可以通过路径区别存储目录的介质，HDD或SSD。可以添加容量限制在每个路径的末尾，通过英文状态逗号`,`隔开。
- 如果用户不是SSD和HDD磁盘混合使用的情况，不需要按照如下示例一和示例二的配置方法配置，只需指定存储目录即可；也不需要修改FE的默认存储介质配置
+* 描述：要克隆的线程数
+* 默认值：1
- 示例1如下：
-
- **注意：如果是SSD磁盘要在目录后面加上`.SSD`,HDD磁盘在目录后面加`.HDD`**
+#### `storage_page_cache_limit`
- `storage_root_path=/home/disk1/doris.HDD;/home/disk2/doris.SSD;/home/disk2/doris`
+* 描述：缓存存储页大小
+* 默认值：20%
- **说明**
+#### `storage_page_cache_shard_size`
- * /home/disk1/doris.HDD，表示存储介质是HDD;
- * /home/disk2/doris.SSD，表示存储介质是SSD；
- * /home/disk2/doris，存储介质默认为HDD
-
- 示例2如下：
-
- **注意：不论HHD磁盘目录还是SSD磁盘目录，都无需添加后缀，storage_root_path参数里指定medium即可**
-
- `storage_root_path=/home/disk1/doris,medium:hdd;/home/disk2/doris,medium:ssd`
-
- **说明**
-
- - /home/disk1/doris,medium:hdd，表示存储介质是HHD;
- - /home/disk2/doris,medium:ssd，表示存储介质是SSD;
+* 描述：StoragePageCache的分片大小，值为 2^n (n=0,1,2,...)。建议设置为接近BE CPU核数的值，可减少StoragePageCache的锁竞争。
+* 默认值：16
+
+#### `index_page_cache_percentage`
+* 类型：int32
+* 描述：索引页缓存占总页面缓存的百分比，取值为[0, 100]。
+* 默认值：10
-* 默认值：${DORIS_HOME}
+#### `storage_strict_check_incompatible_old_format`
-### `storage_strict_check_incompatible_old_format`
* 类型：bool
* 描述：用来检查不兼容的旧版本格式时是否使用严格的验证方式
+ - 配置用来检查不兼容的旧版本格式时是否使用严格的验证方式，当含有旧版本的 hdr 格式时，使用严谨的方式时，程序会打出 fatal log 并且退出运行；否则，程序仅打印 warn log.
* 默认值： true
* 可动态修改：否
-配置用来检查不兼容的旧版本格式时是否使用严格的验证方式，当含有旧版本的 hdr 格式时，使用严谨的方式时，程序会
-打出 fatal log 并且退出运行；否则，程序仅打印 warn log.
-
-### `streaming_load_max_mb`
-
-* 类型：int64
-* 描述：用于限制数据格式为 csv 的一次 Stream load 导入中，允许的最大数据量。单位 MB。
-* 默认值： 10240
-* 可动态修改：是
+#### `sync_tablet_meta`
-Stream Load 一般适用于导入几个GB以内的数据，不适合导入过大的数据。
-
-### `streaming_load_json_max_mb`
-
-* 类型：int64
-* 描述：用于限制数据格式为 json 的一次 Stream load 导入中，允许的最大数据量。单位 MB。
-* 默认值： 100
-* 可动态修改：是
-
-一些数据格式，如 JSON，无法进行拆分处理，必须读取全部数据到内存后才能开始解析，因此，这个值用于限制此类格式数据单次导入最大数据量。
-
-### `streaming_load_rpc_max_alive_time_sec`
-
-默认值：1200
-
-TabletsChannel 的存活时间。如果此时通道没有收到任何数据， 通道将被删除。
-
-### `sync_tablet_meta`
-
-默认值：false
-
-存储引擎是否开sync保留到磁盘上
-
-### `sys_log_dir`
-
-* 类型：string
-* 描述：BE日志数据的存储目录
-* 默认值：${DORIS_HOME}/log
-
-### `sys_log_level`
-
-默认值：INFO
-
-日志级别，INFO < WARNING < ERROR < FATAL
-
-### `sys_log_roll_mode`
-
-默认值：SIZE-MB-1024
-
-日志拆分的大小，每1G拆分一个日志文件
-
-### `sys_log_roll_num`
-
-默认值：10
-
-日志文件保留的数目
+* 描述：存储引擎是否开sync保留到磁盘上
+* 默认值：false
-### `sys_log_verbose_level`
+#### `pending_data_expire_time_sec`
-默认值：10
+* 描述：存储引擎保留的未生效数据的最大时长
+* 默认值：1800 (s)
-日志显示的级别，用于控制代码中VLOG开头的日志输出
+#### `ignore_rowset_stale_unconsistent_delete`
-### `sys_log_verbose_modules`
+* 类型：bool
+* 描述：用来决定当删除过期的合并过的rowset后无法构成一致的版本路径时，是否仍要删除。
+ - 合并的过期 rowset 版本路径会在半个小时后进行删除。在异常下，删除这些版本会出现构造不出查询一致路径的问题，当配置为false时，程序检查比较严格，程序会直接报错退出。
+ 当配置为true时，程序会正常运行，忽略这个错误。一般情况下，忽略这个错误不会对查询造成影响，仅会在fe下发了合并过的版本时出现-230错误。
+* 默认值：false
-默认值：空
+#### `create_tablet_worker_count`
-日志打印的模块，写olap就只打印olap模块下的日志
+* 描述：BE创建tablet的工作线程数
+* 默认值：3
-### `tablet_map_shard_size`
+#### `check_consistency_worker_count`
-默认值：1
+* 描述：计算tablet的校验和(checksum)的工作线程数
+* 默认值：1
-tablet_map_lock 分片大小，值为 2^n, n=0,1,2,3,4 ，这是为了更好地管理tablet
+#### `max_tablet_version_num`
-### `tablet_meta_checkpoint_min_interval_secs`
+* 类型：int
+* 描述：限制单个 tablet 最大 version 的数量。用于防止导入过于频繁，或 compaction 不及时导致的大量 version 堆积问题。当超过限制后，导入任务将被拒绝。
+* 默认值：500
-默认值：600 （秒）
+#### `number_tablet_writer_threads`
-TabletMeta Checkpoint线程轮询的时间间隔
+* 描述：tablet写线程数
+* 默认值：16
-### `tablet_meta_checkpoint_min_new_rowsets_num`
+#### `tablet_map_shard_size`
-默认值：10
+* 描述：tablet_map_lock 分片大小，值为 2^n, n=0,1,2,3,4 ，这是为了更好地管理tablet
+* 默认值：1
-TabletMeta Checkpoint的最小Rowset数目
+#### `tablet_meta_checkpoint_min_interval_secs`
-### `tablet_scan_frequency_time_node_interval_second`
+* 描述：TabletMeta Checkpoint线程轮询的时间间隔
+* 默认值：600 (s)
-* 类型：int64
-* 描述：用来表示记录 metric 'query_scan_count' 的时间间隔。为了计算当前一段时间的tablet的scan频率，需要每隔一段时间记录一次 metric 'query_scan_count'。
-* 默认值：300
+#### `tablet_meta_checkpoint_min_new_rowsets_num`
-### `tablet_stat_cache_update_interval_second`
+* 描述：TabletMeta Checkpoint的最小Rowset数目
+* 默认值：10
-默认值：300
+#### `tablet_stat_cache_update_interval_second`
-tablet状态缓存的更新间隔，单位：秒
+* 描述：tablet状态缓存的更新间隔
+* 默认值：300 (s)
-### `tablet_rowset_stale_sweep_time_sec`
+#### `tablet_rowset_stale_sweep_time_sec`
* 类型：int64
* 描述：用来表示清理合并版本的过期时间，当当前时间 now() 减去一个合并的版本路径中rowset最近创建创建时间大于tablet_rowset_stale_sweep_time_sec时，对当前路径进行清理，删除这些合并过的rowset, 单位为s。
+ - 当写入过于频繁，磁盘空间不足时，可以配置较少这个时间。不过这个时间过短小于5分钟时，可能会引发fe查询不到已经合并过的版本，引发查询-230错误。
* 默认值：1800
-当写入过于频繁，磁盘空间不足时，可以配置较少这个时间。不过这个时间过短小于5分钟时，可能会引发fe查询不到已经合并过的版本，引发查询-230错误。
-
-### `tablet_writer_open_rpc_timeout_sec`
-
-默认值：60
-
-在远程BE 中打开tablet writer的 rpc 超时。 操作时间短，可设置短超时时间
+#### `tablet_writer_open_rpc_timeout_sec`
-导入过程中，发送一个 Batch（1024行）的 RPC 超时时间。默认 60 秒。因为该 RPC 可能涉及多个 分片内存块的写盘操作，所以可能会因为写盘导致 RPC 超时，可以适当调整这个超时时间来减少超时错误（如 send batch fail 错误）。同时，如果调大 write_buffer_size 配置，也需要适当调大这个参数
+* 描述：在远程BE 中打开tablet writer的 rpc 超时。 操作时间短，可设置短超时时间
+ - 导入过程中，发送一个 Batch（1024行）的 RPC 超时时间。默认 60 秒。因为该 RPC 可能涉及多个 分片内存块的写盘操作，所以可能会因为写盘导致 RPC 超时，可以适当调整这个超时时间来减少超时错误（如 send batch fail 错误）。同时，如果调大 write_buffer_size 配置，也需要适当调大这个参数
+* 默认值：60
-### `tablet_writer_ignore_eovercrowded`
+#### `tablet_writer_ignore_eovercrowded`
* 类型：bool
* 描述：写入时可忽略brpc的'[E1011]The server is overcrowded'错误。
+ - 当遇到'[E1011]The server is overcrowded'的错误时，可以调整配置项`brpc_socket_max_unwritten_bytes`，但这个配置项不能动态调整。所以可通过设置此项为`true`来临时避免写失败。注意，此配置项只影响写流程，其他的rpc请求依旧会检查是否overcrowded。
* 默认值：false
-当遇到'[E1011]The server is overcrowded'的错误时，可以调整配置项`brpc_socket_max_unwritten_bytes`，但这个配置项不能动态调整。所以可通过设置此项为`true`来临时避免写失败。注意，此配置项只影响写流程，其他的rpc请求依旧会检查是否overcrowded。
-
-### `thrift_client_retry_interval_ms`
-
-* 类型：int64
-* 描述：用来为be的thrift客户端设置重试间隔, 避免fe的thrift server发生雪崩问题，单位为ms。
-* 默认值：1000
-
-### `thrift_connect_timeout_seconds`
-
-默认值：3
-
-默认thrift客户端连接超时时间（单位：秒）
-
-### `thrift_rpc_timeout_ms`
-
-默认值：5000
-
-thrift默认超时时间，默认：5秒
-
-### `thrift_server_type_of_fe`
-
-该配置表示FE的Thrift服务使用的服务模型, 类型为string, 大小写不敏感,该参数需要和fe的thrift_server_type参数的设置保持一致。目前该参数的取值有两个,`THREADED`和`THREAD_POOL`。
-
-若该参数为`THREADED`, 该模型为非阻塞式I/O模型，
-
-若该参数为`THREAD_POOL`, 该模型为阻塞式I/O模型。
-
-### `total_permits_for_compaction_score`
-
-* 类型：int64
-* 描述：被所有的compaction任务所能持有的 "permits" 上限，用来限制compaction占用的内存。
-* 默认值：10000
-* 可动态修改：是
+#### `streaming_load_rpc_max_alive_time_sec`
-### `trash_file_expire_time_sec`
+* 描述：TabletsChannel 的存活时间。如果此时通道没有收到任何数据， 通道将被删除。
+* 默认值：1200
-默认值：259200
+#### `alter_tablet_worker_count`
-回收站清理的间隔，72个小时，当磁盘空间不足时，trash下的文件保存期可不遵守这个参数
+* 描述：进行schema change的线程数
+* 默认值：3
-### `txn_commit_rpc_timeout_ms`
+#### `ignore_load_tablet_failure`
-默认值：10000
-
-txn 提交 rpc 超时，默认10秒
+* 类型：bool
+* 描述：用来决定在有tablet 加载失败的情况下是否忽略错误，继续启动be
+* 默认值：false
-### `txn_map_shard_size`
+BE启动时，会对每个数据目录单独启动一个线程进行 tablet header 元信息的加载。默认配置下，如果某个数据目录有 tablet 加载失败，则启动进程会终止。同时会在 `be.INFO` 日志中看到如下错误信息：
-默认值：128
+```
+load tablets from header failed, failed tablets size: xxx, path=xxx
+```
-txn_map_lock 分片大小，取值为2^n，n=0,1,2,3,4。这是一项增强功能，可提高管理 txn 的性能
+表示该数据目录共有多少 tablet 加载失败。同时，日志中也会有加载失败的 tablet 的具体信息。此时需要人工介入来对错误原因进行排查。排查后，通常有两种方式进行恢复：
-### `txn_shard_size`
+1. tablet 信息不可修复，在确保其他副本正常的情况下，可以通过 `meta_tool` 工具将错误的tablet删除。
+2. 将 `ignore_load_tablet_failure` 设置为 true，则 BE 会忽略这些错误的 tablet，正常启动。
-默认值：1024
+#### `report_disk_state_interval_seconds`
-txn_lock 分片大小，取值为2^n，n=0,1,2,3,4， 这是一项增强功能，可提高提交和发布 txn 的性能
+* 描述：代理向 FE 报告磁盘状态的间隔时间
+* 默认值：60 (s)
-### `unused_rowset_monitor_interval`
+#### `result_buffer_cancelled_interval_time`
-默认值：30
+* 描述：结果缓冲区取消时间
+* 默认值：300 (s)
-清理过期Rowset的时间间隔，单位：秒
+#### `snapshot_expire_time_sec`
-### `upload_worker_count`
+* 描述：快照文件清理的间隔
+* 默认值：172800 (48小时)
-默认值：1
+#### `compress_rowbatches`
-上传文件最大线程数
+* 类型：bool
+* 描述：序列化RowBatch时是否使用Snappy压缩算法进行数据压缩
+* 默认值：true
-### `user_function_dir`
+### 日志
-默认值：${DORIS_HOME}/lib/udf
+#### `sys_log_dir`
-udf函数目录
+* 类型：string
+* 描述：BE日志数据的存储目录
+* 默认值：${DORIS_HOME}/log
-### `webserver_num_workers`
+#### `sys_log_level`
-默认值：48
+* 描述：日志级别，INFO < WARNING < ERROR < FATAL
+* 默认值：INFO
-webserver默认工作线程数
+#### `sys_log_roll_mode`
-### `webserver_port`
-* 类型：int32
-* 描述：BE 上的 http server 的服务端口
-* 默认值：8040
+* 描述：日志拆分的大小，每1G拆分一个日志文件
+* 默认值：SIZE-MB-1024
-### `write_buffer_size`
+#### `sys_log_roll_num`
-默认值：104857600
+* 描述：日志文件保留的数目
+* 默认值：10
-刷写前缓冲区的大小
+#### `sys_log_verbose_level`
-导入数据在 BE 上会先写入到一个内存块，当这个内存块达到阈值后才会写回磁盘。默认大小是 100MB。过小的阈值可能导致 BE 上存在大量的小文件。可以适当提高这个阈值减少文件数量。但过大的阈值可能导致 RPC 超时
+* 描述：日志显示的级别，用于控制代码中VLOG开头的日志输出
+* 默认值：10
-### `zone_map_row_num_threshold`
+#### `sys_log_verbose_modules`
-* 类型： int32
-* 描述: 如果一个page中的行数小于这个值就不会创建zonemap，用来减少数据膨胀
-* 默认值: 20
+* 描述：日志打印的模块，写olap就只打印olap模块下的日志
+* 默认值：空
-### `aws_log_level`
+#### `aws_log_level`
* 类型: int32
* 描述: AWS SDK 的日志级别
@@ -1474,150 +1303,74 @@ webserver默认工作线程数
```
* 默认值: 3
-### `enable_tcmalloc_hook`
+#### `log_buffer_level`
-* 类型：bool
-* 描述：是否Hook TCmalloc new/delete，目前在Hook中统计thread local MemTracker。
-* 默认值：true
+* 描述: 日志刷盘的策略，默认保持在内存中
+* 默认值: 空
-### `mem_tracker_consume_min_size_bytes`
+### 其他
-* 类型: int32
-* 描述: TCMalloc Hook consume/release MemTracker时的最小长度，小于该值的consume size会持续累加，避免频繁调用MemTracker的consume/release，减小该值会增加consume/release的频率，增大该值会导致MemTracker统计不准，理论上一个MemTracker的统计值与真实值相差 = (mem_tracker_consume_min_size_bytes * 这个MemTracker所在的BE线程数)。
-* 默认值: 1048576
+#### `report_tablet_interval_seconds`
-### `max_segment_num_per_rowset`
+* 描述: 代理向 FE 报告 olap 表的间隔时间
+* 默认值: 60 (s)
-* 类型: int32
-* 描述: 用于限制导入时，新产生的rowset中的segment数量。如果超过阈值，导入会失败并报错 -238。过多的 segment 会导致compaction占用大量内存引发 OOM 错误。
-* 默认值: 200
+#### `report_task_interval_seconds`
-### `remote_storage_read_buffer_mb`
+* 描述: 代理向 FE 报告任务签名的间隔时间
+* 默认值: 10 (s)
-* 类型: int32
-* 描述: 读取hdfs或者对象存储上的文件时，使用的缓存大小。
-* 默认值: 16MB
-
-增大这个值，可以减少远端数据读取的调用次数，但会增加内存开销。
+#### `periodic_counter_update_period_ms`
-### `external_table_connect_timeout_sec`
+* 描述: 更新速率计数器和采样计数器的周期
+* 默认值: 500 (ms)
-* 类型: int32
-* 描述: 和外部表建立连接的超时时间。
-* 默认值: 5秒
+#### `enable_metric_calculator`
-### `segment_cache_capacity`
-
-* 类型: int32
-* 描述: Segment Cache 缓存的 Segment 最大数量
-* 默认值: 1000000
-
-默认值目前只是一个经验值，可能需要根据实际场景修改。增大该值可以缓存更多的segment从而避免一些IO。减少该值则会降低内存使用。
-
-### `auto_refresh_brpc_channel`
-
-* 类型: bool
-* 描述: 获取brpc连接时，通过hand_shake rpc 判断连接的可用性，如果不可用则重新建立连接
-* 默认值: false
-
-### `high_priority_flush_thread_num_per_store`
-
-* 类型：int32
-* 描述：每个存储路径所分配的用于高优导入任务的 flush 线程数量。
-* 默认值：1
-
-### `routine_load_consumer_pool_size`
-
-* 类型：int32
-* 描述：routine load 所使用的 data consumer 的缓存数量。
-* 默认值：10
-
-### `load_task_high_priority_threshold_second`
-
-* 类型：int32
-* 描述：当一个导入任务的超时时间小于这个阈值是，Doris 将认为他是一个高优任务。高优任务会使用独立的 flush 线程池。
-* 默认：120
-
-### `min_load_rpc_timeout_ms`
+* 描述: 如果设置为 true，metric calculator 将运行，收集BE相关指标信息，如果设置成false将不运行
+* 默认值: true
-* 类型：int32
-* 描述：load 作业中各个rpc 的最小超时时间。
-* 默认：20
+#### `enable_system_metrics`
-### `doris_scan_range_max_mb`
-* 类型: int32
-* 描述: 每个OlapScanner 读取的最大数据量
-* 默认值: 1024
+* 描述: 用户控制打开和关闭系统指标
+* 默认值: true
-### `string_type_length_soft_limit_bytes`
-* 类型: int32
-* 描述: String 类型最大长度的软限，单位是字节
-* 默认值: 1048576
+#### `enable_token_check`
-### `enable_quick_compaction`
-* 类型: bool
-* 描述: 是否开启quick_compaction,主要用在小数据量频繁导入的场景,通过快速compaction的机制及时合并导入版本可以有效避免-235的问题，小数据量的定义目前是根据行数来定义
+* 描述: 用于向前兼容，稍后将被删除
* 默认值: true
-### `quick_compaction_max_rows`
-* 类型: int32
-* 描述: 当导入的行数小于这个值认为这次导入是小数据量的导入，在快速合并时会被选中
-* 默认值: 1000
+#### `max_runnings_transactions_per_txn_map`
-### `quick_compaction_batch_size`
-* 类型: int32
-* 描述: 快速合并的触发时机，导入次数达到quick_compaction_batch_size时触发一次
-* 默认值: 10
-
-### `quick_compaction_min_rowsets`
-* 类型: int32
-* 描述: 最少进行合并的版本数，当选中的小数据量的rowset个数，大于这个值是才会进行真正的合并
-* 默认值: 10
+* 描述: txn 管理器中每个 txn_partition_map 的最大 txns 数，这是一种自我保护，以避免在管理器中保存过多的 txns
+* 默认值: 100
-### `generate_cache_cleaner_task_interval_sec`
-* 类型：int64
-* 描述：缓存文件的清理间隔，单位：秒
-* 默认值：43200（12小时）
+#### `max_download_speed_kbps`
-### `file_cache_type`
-* 类型：string
-* 描述：缓存文件的类型。whole_file_cache：将segment文件整个下载，sub_file_cache：将segment文件按大小切分成多个文件。
-* 默认值：""
+* 描述: 最大下载速度限制
+* 默认值: 50000 （kb/s）
-### `max_sub_cache_file_size`
-* 类型：int64
-* 描述：缓存文件使用sub_file_cache时，切分文件的最大大小，单位B
-* 默认值：104857600（100MB）
+#### `download_low_speed_time`
-### `file_cache_alive_time_sec`
-* 类型：int64
-* 描述：缓存文件的保存时间，单位：秒
-* 默认值：604800（1个星期）
+* 描述: 下载时间限制
+* 默认值: 300 (s)
-### `enable_segcompaction`
+#### `download_low_speed_limit_kbps`
-* 类型：bool
-* 描述：在导入时进行 segment compaction 来减少 segment 数量
-* 默认值：false
+* 描述: 下载最低限速
+* 默认值: 50 (KB/s)
-### `segcompaction_threshold_segment_num`
+#### `doris_cgroups`
-* 类型：int32
-* 描述：当 segment 数量超过此阈值时触发 segment compaction
-* 默认值：10
+* 描述: 分配给doris的cgroups
+* 默认值: 空
-### `segcompaction_small_threshold`
+#### `row_nums_check`
-* 类型：int32
-* 描述：当 segment 文件超过此大小时则会在 segment compaction 时被 compact，否则跳过
-* 默认值：1048576
+* 描述: 检查 BE/CE 和schema更改的行号。
+* 默认值: true
-### `big_column_size_buffer`
-* 类型：int64
-* 描述：当使用odbc外表时，如果odbc源表的某一列类型为HLL, CHAR或者VARCHAR，并且列值长度超过该值，则查询报错'column value length longer than buffer length'. 可增大该值
-* 默认值：65535
+#### `priority_queue_remaining_tasks_increased_frequency`
-### `small_column_size_buffer`
-* 类型：int64
-* 描述：当使用odbc外表时，如果odbc源表的某一列类型不是HLL, CHAR或者VARCHAR，并且列值长度超过该值，则查询报错'column value length longer than buffer length'. 可增大该值
-* 默认值：100
+* 描述: BlockingPriorityQueue中剩余任务的优先级频率增加
+* 默认值:512
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org