You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "15767714253 (via GitHub)" <gi...@apache.org> on 2026/10/08 06:01:35 UTC
[I] [Bug] BE ls_normal thread_pool_queue_size grows monotonically and saturates at capacity — phantom queue count in the time-sharing scan pool (4.1.2) [doris]
15767714253 opened a new issue, #68771:
URL: https://github.com/apache/doris/issues/68771
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
4.1.2
### What's Wrong?
The Prometheus metric `doris_be_thread_pool_queue_size{thread_pool_name="ls_normal", workload_group="normal"}` grows **monotonically** on **2 of 5 BEs** (`172.31.20.48`, `172.31.17.165`), starting around **2026-09-17**, until it saturates near **102400** (which equals `doris_scanner_thread_pool_queue_size`) around **2026-10-05**, then drops vertically.
The vertical drop is **not** the queue draining — it coincides exactly with a **BE restart**:
| BE | `LastStartTime` (SHOW BACKENDS) | Growing? |
| ----------------- | ------------------------------- | --------- |
| 172.31.17.66 | 2026-06-24 14:52:14 | No (flat) |
| 172.31.25.179 | 2026-06-24 14:50:20 | No (flat) |
| 172.31.19.197 | 2026-06-24 14:50:48 | No (flat) |
| **172.31.20.48** | **2026-10-05 14:17:36** | **Yes** |
| **172.31.17.165** | **2026-10-05 14:16:37** | **Yes** |
The two affected BEs also have ~1/33 of the cumulative `task_execution_count_total` of the other three, consistent with a recent restart.
**Key paradox — the queue is a phantom.** At the time of sampling, the affected pools report:
```
doris_be_thread_pool_active_threads{thread_pool_name="ls_normal"} = 0 (max_threads = 48)
doris_be_thread_pool_submit_failed{thread_pool_name="ls_normal"} = 0
doris_be_thread_pool_queue_size{thread_pool_name="ls_normal"} = 14905 (!!)
```
**Idle workers (active = 0) together with a non-empty queue of ~15,000 entries is impossible for a correct implementation** — the dispatcher would drain any real pending split immediately. This indicates the gauge is **over-counted (phantom queue depth)**, while the real queue is empty.
**It is still growing right now** (two samples, 4 minutes apart):
| BE | `queue_size` | `task_execution_count_total` | `active_threads` |
| ---------------------- | ----------------------- | ----------------------------------- | ---------------- |
| **172.31.20.48** | 14893 → **14905** (+12) | 102,235,465 → 102,275,040 (+39,575) | 1 → 0 |
| **172.31.17.165** | 14750 → **14764** (+14) | 106,108,764 → 106,152,122 (+43,358) | 0 → 0 |
| 172.31.17.66 (healthy) | 19030 → **19030** (+0) | 3,361,977,363 → 3,362,017,498 | 0 → 0 |
Rate ≈ **3–3.5 / minute ≈ 4,700 / day**. From the 2026-10-05 restart to the 2026-10-08 sample (2.9 days) this accumulates to ~14,900 — matching the observed value exactly. Extrapolating, the pool will hit the **102400** capacity again around **2026-10-25**, at which point `submit_failed` should become non-zero.
Note that a **healthy** BE processing the same volume of tasks (≈40,000 in 4 minutes) keeps `queue_size` **perfectly flat** — so this is a **state-dependent accounting leak**, not a throughput/backlog problem.
**config：**
# 基础参数
# 系统日志文件保留的滚动数量，设置为 1 表示只保留最新的日志文件。
sys_log_roll_num = 7
# Stream Load 导入 JSON 数据的最大文件大小限制（2048 MB = 2GB）。
streaming_load_json_max_mb = 2048
# 允许 STRING 类型字段存储的最大字节数的软限制（1 GB），超出可能影响性能。
string_type_length_soft_limit_bytes = 1073741824
# 合并参数
# 累计合并（Cumulative Compaction）任务的最大并发线程数。
max_cumu_compaction_threads = 24
max_base_compaction_threads = 8
# 每个磁盘同时运行的 Compaction 任务数量上限。
compaction_task_num_per_disk = 32
# 每个SSD磁盘同时运行的 Compaction 任务数量上限。
compaction_task_num_per_fast_disk = 32
# 触发 Cumulative Compaction 的最小数据版本增量（Delta）数量阈值。
cumulative_compaction_min_deltas = 5
# 触发 Base Compaction 的条件
base_compaction_interval_seconds_since_last_operation = 43200
base_compaction_min_data_ratio = 0.2
# 用 Segment 层级压缩（节省 CPU 资源，推荐关闭）。
enable_segcompaction = true
# 列compatction
enable_vertical_compaction = true
# 禁用 Compaction 任务的优先级调度机制（减少调度开销）。
enable_compaction_priority_scheduling = false
# 写入优化
# 写入缓冲区的内存大小（1 GB），提升高频写场景的吞吐量。
write_buffer_size = 1073741824
# 线程池
# Pipeline 执行线程数（用于并行查询处理）。
pipeline_executor_size = 12
# 阻塞式 Pipeline 执行线程数（处理阻塞 I/O 操作）。
blocking_pipeline_executor_size = 8
# 数据版本发布任务的并发工作线程数。
publish_version_worker_count = 16
# 文件上传任务的线程数（影响存算分离上传性能）。
upload_worker_count = 16
# Tablet 数据写入线程数（提升并发写入能力）。
number_tablet_writer_threads = 32
# 高优先级数据推送任务的线程数（如副本同步）。
push_worker_count_high_priority = 6
# 普通优先级数据推送任务的线程数。
push_worker_count_normal_priority = 6
# 批量数据发送线程池的线程数（影响节点间数据传输）。
send_batch_thread_pool_thread_num = 96
#默认只有 2 个线程，在高写入量下是瓶颈。可增加该值让更多 CPU 参与刷盘
flush_thread_num_per_store = 8
# S3 存算分离
# S3 文件上传线程池的最大线程数。
num_s3_file_upload_thread_pool_max_thread = 64
# S3 文件上传线程池的最小线程数（核心数）。
num_s3_file_upload_thread_pool_min_thread = 32
# 操作 S3 文件系统的最大线程数。
max_s3_file_system_thread_num = 128
# 操作 S3 文件系统的最小线程数（核心数）。
min_s3_file_system_thread_num = 32
remote_storage_read_buffer_mb = 128
doris_remote_scanner_thread_pool_thread_num = 128
#S3 写入缓冲区大小（字节），需 ≥ 5MB
s3_write_buffer_size = 10485760
#S3 文件系统本地上传缓冲区大小
s3_file_system_local_upload_buffer_size = 10485760
# 缓存配置
#开启行缓存
disable_storage_row_cache = false
#行缓存内存比例
row_cache_mem_limit = 10%
#缓存
storage_page_cache_limit = 15%
max_tablet_version_num = 4000
enable_packed_file = false
# delete bitmap 计算调整
calc_delete_bitmap_worker_count = 16
calc_tablet_delete_bitmap_task_max_thread = 64
delete_bitmap_agg_cache_capacity = 1073741824
calc_delete_bitmap_max_thread = 64
calc_delete_bitmap_for_load_max_thread = 16
tablet_publish_txn_max_thread = 64
### What You Expected?
1. `doris_be_thread_pool_queue_size` for `ls_normal` should fall back to ~0 when the pool is idle (`active_threads = 0`), and must never grow without bound.
2. The counter should stay consistent across token lifecycle events (token shutdown / task removal) — every entry removed from the queue must decrement `_total_queued_tasks`.
3. `submit_failed` should not begin incrementing merely because of a leaked/phantom queue count.
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE ls_normal thread_pool_queue_size grows monotonically and saturates at capacity — phantom queue count in the time-sharing scan pool (4.1.2) [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on issue #68771:
URL: https://github.com/apache/doris/issues/68771#issuecomment-6096970041
You mean the queue size is large, but active thread number is very low?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE ls_normal thread_pool_queue_size grows monotonically and saturates at capacity — phantom queue count in the time-sharing scan pool (4.1.2) [doris]
Posted by "Doris-Breakwater (via GitHub)" <gi...@apache.org>.
Doris-Breakwater commented on issue #68771:
URL: https://github.com/apache/doris/issues/68771#issuecomment-6053708760
Breakwater-GitHub-Analysis-Slot: slot_841d0d77defc
**Initial assessment:** This strongly matches a known `TimeSharingTaskExecutor` queue-accounting defect. The defect is present in the upstream **4.1.2** source; the matching fix was merged as [PR #63568](https://github.com/apache/doris/pull/63568), with the 4.1 backport in [PR #64855](https://github.com/apache/doris/pull/64855). I verified that backport commit `1f197f6c931217f546463371865c4d756b1a6058` is absent from the 4.1.2 ancestry and present in 4.1.3. Confidence is high for the source-level defect and its match to these symptoms; confirming the exact running BE builds is still necessary.
**Verified code mechanism:** I inspected upstream tag `4.1.2` (commit `aec169d20256a788de448f46737b5fb53ee3b4e3`) without changing the checkout.
- With `enable_task_executor_in_internal_table=true` (the default), `WorkloadGroup` creates `TaskExecutorSimplifiedScanScheduler` for `ls_<workload_group>`, using `doris_scanner_thread_pool_queue_size` as its queue limit. See [workload_group.cpp](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/runtime/workload_group/workload_group.cpp#L561-L573).
- The metric hook reads `get_queue_size()`, which returns `_total_queued_tasks`, rather than the actual split-queue size. `_do_submit()` increments that count when offering a split; a worker decrements it when taking the split. See [executor metric/submission code](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp#L242-L254) and [queue-size getter](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.h#L208-L211).
- `ScannerContext::stop_scanners()` and its destructor call `remove_task()`. That function closes the task handle and physically removes its queued splits with `remove_all()`, **without decrementing `_total_queued_tasks`**. Workers decide whether to sleep from the real split-queue size, so they can be idle while the reported count retains removed entries. See [scanner cleanup](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/scanner_context.cpp#L366-L385), [remove_task](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp#L746-L775), and [worker dispatch](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp#L503-L547).
A minimal sequence is therefore: enqueue a split (+1), remove its task before a worker takes it (real queue -1, accounting unchanged), repeat. This explains accumulated phantom entries and why continuing to execute unrelated scan tasks does not drain the leaked offset. It does not require token shutdown itself to omit a decrement.
This can affect query execution, not just the dashboard. The [capacity check](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp#L383-L391) uses:
`capacity_remaining = max_threads - active_threads + max_queue_size - _total_queued_tasks`
It rejects submissions and increments `submit_failed` when this is below 1. Thus the exact boundary depends on active workers; with 48 maximum workers and zero active workers, it is 102448, rather than exactly 102400. The projected exhaustion date remains an estimate.
**What remains unverified:** The supplied samples support this mechanism but do not identify which queries removed queued splits. Equal execution rates do not imply equal cancellation/cleanup histories. The supposedly healthy BE's flat, nonzero count of 19030 could also be an older leaked offset. Also, active-thread and queue gauges are read under separate lock acquisitions, so one `active=0` sample alone is not conclusive; sustained observations and source evidence are stronger.
Please provide:
1. Full BE version/build Git hashes for all five nodes, any local patches, and effective `enable_task_executor_in_internal_table`, scanner queue limit, and workload-group scan-thread settings.
2. Several timestamped raw `/metrics` captures retaining all labels, including pool `id` and BE identity, plus the dashboard PromQL. This will exclude aggregation across different pool instances.
3. Representative query IDs, sanitized SQL/reproduction steps, and BE/FE logs showing cancellation, timeout, early scan termination, or cleanup around count increases. Please also include any `TooManyTasks` / `Thread pool ... is at capacity` errors and workload/configuration changes near the reported onset. These events are candidates, not established triggers here.
**Recommended next steps:** Use a tested maintenance build containing the 4.1 backport; upstream 4.1.3 contains this specific fix, but verify the packaged BE hash. The patch subtracts only entries actually removed from the executor queue, centralizes offer accounting, and handles the idle transition/notification. Subtracting the entire task's split list would be incorrect because it can include running or otherwise nonqueued splits.
For validation, use the existing [`TimeSharingTaskExecutorTest.test_remove_task_clears_queued_task_count`](https://github.com/apache/doris/blob/1f197f6c931217f546463371865c4d756b1a6058/be/test/exec/executor/time_sharing/time_sharing_task_executor_test.cpp#L373-L423) from the backport, then exercise scan cancellation/cleanup in staging and check that an idle pool returns to zero and accepts subsequent submissions. I inspected that test; I did not build or run the BE suite or reproduce this against your cluster. A BE restart recreates the counter and can temporarily restore headroom, but does not remove the defect from an unpatched binary. Increasing the queue limit only postpones rejection.
If the deployed build already includes the fix and still exhibits growth, keep this issue open for further investigation: obtain a maintainer-assisted snapshot of the actual split-queue size and `_total_queued_tasks` under the same executor lock, plus worker thread stacks. That comparison distinguishes an accounting mismatch from a real queue that is failing to dispatch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE ls_normal thread_pool_queue_size grows monotonically and saturates at capacity — phantom queue count in the time-sharing scan pool (4.1.2) [doris]
Posted by "15767714253 (via GitHub)" <gi...@apache.org>.
15767714253 commented on issue #68771:
URL: https://github.com/apache/doris/issues/68771#issuecomment-6186382547
> 5\. You mean the queue size is large, but active thread number is very low?
YES
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE ls_normal thread_pool_queue_size grows monotonically and saturates at capacity — phantom queue count in the time-sharing scan pool (4.1.2) [doris]
Posted by "15767714253 (via GitHub)" <gi...@apache.org>.
15767714253 closed issue #68771: [Bug] BE ls_normal thread_pool_queue_size grows monotonically and saturates at capacity — phantom queue count in the time-sharing scan pool (4.1.2)
URL: https://github.com/apache/doris/issues/68771
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE ls_normal thread_pool_queue_size grows monotonically and saturates at capacity — phantom queue count in the time-sharing scan pool (4.1.2) [doris]
Posted by "15767714253 (via GitHub)" <gi...@apache.org>.
15767714253 commented on issue #68771:
URL: https://github.com/apache/doris/issues/68771#issuecomment-6094630691
> Breakwater-GitHub-Analysis-Slot: slot_841d0d77defc
>
> **Initial assessment:** This strongly matches a known `TimeSharingTaskExecutor` queue-accounting defect. The defect is present in the upstream **4.1.2** source; the matching fix was merged as [PR #63568](https://github.com/apache/doris/pull/63568), with the 4.1 backport in [PR #64855](https://github.com/apache/doris/pull/64855). I verified that backport commit `1f197f6c931217f546463371865c4d756b1a6058` is absent from the 4.1.2 ancestry and present in 4.1.3. Confidence is high for the source-level defect and its match to these symptoms; confirming the exact running BE builds is still necessary.
>
> **Verified code mechanism:** I inspected upstream tag `4.1.2` (commit `aec169d20256a788de448f46737b5fb53ee3b4e3`) without changing the checkout.
>
> * With `enable_task_executor_in_internal_table=true` (the default), `WorkloadGroup` creates `TaskExecutorSimplifiedScanScheduler` for `ls_<workload_group>`, using `doris_scanner_thread_pool_queue_size` as its queue limit. See [workload_group.cpp](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/runtime/workload_group/workload_group.cpp#L561-L573).
> * The metric hook reads `get_queue_size()`, which returns `_total_queued_tasks`, rather than the actual split-queue size. `_do_submit()` increments that count when offering a split; a worker decrements it when taking the split. See [executor metric/submission code](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp#L242-L254) and [queue-size getter](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.h#L208-L211).
> * `ScannerContext::stop_scanners()` and its destructor call `remove_task()`. That function closes the task handle and physically removes its queued splits with `remove_all()`, **without decrementing `_total_queued_tasks`**. Workers decide whether to sleep from the real split-queue size, so they can be idle while the reported count retains removed entries. See [scanner cleanup](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/scanner_context.cpp#L366-L385), [remove_task](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp#L746-L775), and [worker dispatch](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp#L503-L547).
>
> A minimal sequence is therefore: enqueue a split (+1), remove its task before a worker takes it (real queue -1, accounting unchanged), repeat. This explains accumulated phantom entries and why continuing to execute unrelated scan tasks does not drain the leaked offset. It does not require token shutdown itself to omit a decrement.
>
> This can affect query execution, not just the dashboard. The [capacity check](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp#L383-L391) uses:
>
> `capacity_remaining = max_threads - active_threads + max_queue_size - _total_queued_tasks`
>
> It rejects submissions and increments `submit_failed` when this is below 1. Thus the exact boundary depends on active workers; with 48 maximum workers and zero active workers, it is 102448, rather than exactly 102400. The projected exhaustion date remains an estimate.
>
> **What remains unverified:** The supplied samples support this mechanism but do not identify which queries removed queued splits. Equal execution rates do not imply equal cancellation/cleanup histories. The supposedly healthy BE's flat, nonzero count of 19030 could also be an older leaked offset. Also, active-thread and queue gauges are read under separate lock acquisitions, so one `active=0` sample alone is not conclusive; sustained observations and source evidence are stronger.
>
> Please provide:
>
> 1. Full BE version/build Git hashes for all five nodes, any local patches, and effective `enable_task_executor_in_internal_table`, scanner queue limit, and workload-group scan-thread settings.
> 2. Several timestamped raw `/metrics` captures retaining all labels, including pool `id` and BE identity, plus the dashboard PromQL. This will exclude aggregation across different pool instances.
> 3. Representative query IDs, sanitized SQL/reproduction steps, and BE/FE logs showing cancellation, timeout, early scan termination, or cleanup around count increases. Please also include any `TooManyTasks` / `Thread pool ... is at capacity` errors and workload/configuration changes near the reported onset. These events are candidates, not established triggers here.
>
> **Recommended next steps:** Use a tested maintenance build containing the 4.1 backport; upstream 4.1.3 contains this specific fix, but verify the packaged BE hash. The patch subtracts only entries actually removed from the executor queue, centralizes offer accounting, and handles the idle transition/notification. Subtracting the entire task's split list would be incorrect because it can include running or otherwise nonqueued splits.
>
> For validation, use the existing [`TimeSharingTaskExecutorTest.test_remove_task_clears_queued_task_count`](https://github.com/apache/doris/blob/1f197f6c931217f546463371865c4d756b1a6058/be/test/exec/executor/time_sharing/time_sharing_task_executor_test.cpp#L373-L423) from the backport, then exercise scan cancellation/cleanup in staging and check that an idle pool returns to zero and accepts subsequent submissions. I inspected that test; I did not build or run the BE suite or reproduce this against your cluster. A BE restart recreates the counter and can temporarily restore headroom, but does not remove the defect from an unpatched binary. Increasing the queue limit only postpones rejection.
>
> If the deployed build already includes the fix and still exhibits growth, keep this issue open for further investigation: obtain a maintainer-assisted snapshot of the actual split-queue size and `_total_queued_tasks` under the same executor lock, plus worker thread stacks. That comparison distinguishes an accounting mismatch from a real queue that is failing to dispatch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE ls_normal thread_pool_queue_size grows monotonically and saturates at capacity — phantom queue count in the time-sharing scan pool (4.1.2) [doris]
Posted by "15767714253 (via GitHub)" <gi...@apache.org>.
15767714253 commented on issue #68771:
URL: https://github.com/apache/doris/issues/68771#issuecomment-6094680572
**Key paradox — the queue is a phantom *and* it grows while the pool is idle.**
#### Sampling / 采样（primary evidence）
Consecutive scrapes of `/metrics` on `172.31.17.165` (`ls_normal`, `workload_group=normal`) — **10 samples, 10 s apart, 2026-10-10 14:16:29 → 14:17:59 (GMT+8)**:
```bash
for i in $(seq 1 10); do
curl -s http://172.31.17.165:8040/metrics \
| grep -E 'thread_pool_(queue_size|active_threads|submit_failed|task_execution_count_total)\{[^}]*thread_pool_name="ls_normal"'
sleep 10
done
```
| time (GMT+8) | `queue_size` | `active_threads` | `task_execution_count_total` | `submit_failed` |
| ------------ | ------------ | ---------------- | ---------------------------- | --------------- |
| 14:16:29 | 25,610 | 0 | 181,602,147 | 0 |
| 14:16:39 | 25,610 | 0 | 181,606,709 | 0 |
| 14:16:49 | 25,611 | 1 | 181,611,036 | 0 |
| 14:16:59 | 25,611 | 1 | 181,615,106 | 0 |
| 14:17:09 | 25,612 | 0 | 181,618,385 | 0 |
| 14:17:19 | 25,613 | 0 | 181,622,243 | 0 |
| 14:17:29 | 25,613 | 0 | 181,625,538 | 0 |
| 14:17:39 | 25,614 | 0 | 181,630,293 | 0 |
| 14:17:49 | 25,615 | 0 | 181,634,608 | 0 |
| 14:17:59 | 25,615 | 0 | 181,639,119 | 0 |
**Derived:**
- `queue_size`: **+5 / 90 s ⇒ ≈ 3.3 / min ⇒ ≈ 4,800 / day**.
- `task_execution_count_total`: **+36,972 / 90 s ⇒ ≈ 411 splits/s** actually consumed by the pool.
- `max_threads = 48`; `active_threads ∈ {0, 1}` for the whole window; `submit_failed = 0` throughout.
**Why this proves the counter is wrong (independent of the "two gauges are read under separate locks" caveat):**
The pool reports it is consuming **≈411 splits/s**. If a **25,610-entry real queue** were feeding those ≤48 workers, even a single active worker would drain it in far under a minute; with **≥47 workers idle at every sample**, a real queue of 25,610 would be **emptied in well under 65 s**. Yet across the entire 90 s window the value **does not fall by a single unit — it only rises**. A *real* backlog cannot simultaneously be huge, almost-idle-drained (411/s), and monotonically growing. **⇒ The reported depth cannot correspond to real queued splits; the gauge is over-counted (phantom queue depth).**
This argument does **not** rely on the atomicity of reading `active_threads` and `queue_size` separately — it only relies on the fact that a *real* queue of that size cannot survive an almost-idle worker set that is draining the pool at ~411 splits/s.
#### Earlier corroborating sample / 早期佐证采样（2026-10-08，间隔 4 分钟两次读数）
| BE | `queue_size` | `task_execution_count_total` | `active_threads` |
| ---------------------- | ------------------------- | ----------------------------------- | ---------------- |
| **172.31.20.48** | 14,893 → **14,905** (+12) | 102,235,465 → 102,275,040 (+39,575) | 1 → 0 |
| **172.31.17.165** | 14,750 → **14,764** (+14) | 106,108,764 → 106,152,122 (+43,358) | 0 → 0 |
| 172.31.17.66 (healthy) | 19,030 → **19,030** (+0) | 3,361,977,363 → 3,362,017,498 | 0 → 0 |
Rate ≈ **3–3.5 / min ≈ 4,700–5,100 / day**. From the **2026-10-05 14:16 restart** to the **2026-10-10 14:16 sample (5.00 days)** this predicts ~25,600 — matching the observed **25,610** almost exactly. Extrapolating, the pool will hit the **102400** capacity again around **2026-10-25**, at which point `submit_failed` should become non-zero.
Note that a **healthy** BE processes the same volume of tasks (≈40,000 in 4 minutes) while its `queue_size` stays **perfectly flat at the same ~19,000 floor** — so this is a **state-dependent accounting leak**, not a throughput/backlog problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org