You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jaehui Lee (Jira)" <ji...@apache.org> on 2025/12/15 23:42:00 UTC
[jira] [Updated] (HBASE-29781) Dynamic configurations for call queue length doesn't work correctly when increasing the limit
[ https://issues.apache.org/jira/browse/HBASE-29781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jaehui Lee updated HBASE-29781:
-------------------------------
Description:
h2. Problem
The dynamic configurations for call queue length (such as {{{}ipc.server.max.callqueue.length{}}}, {{{}ipc.server.priority.max.callqueue.length{}}}, {{{}ipc.server.replication.max.callqueue.length{}}}, {{{}ipc.server.bulkload.max.callqueue.length{}}}) only works when *decreasing* the limit. When *increasing* the limit, the configuration change has no effect - tasks are still rejected at the hard limit.
*Example:*
* Initial configuration: {{ipc.server.max.callqueue.length = 100}}
* Change configuration to: {{ipc.server.max.callqueue.length = 200}}
* Expected: Queue accepts up to 200 tasks
* Actual: Queue still rejects tasks at 100
h2. Root Cause
{{RpcExecutor}} uses two limit mechanisms: (This was introduced by HBASE-15306)
# *Soft limit* ({{{}currentQueueLimit{}}} variable): Updated by {{resizeQueues()}} when configuration changes
# *Hard limit* ({{{}BlockingQueue{}}} capacity): Set once during queue initialization and *cannot be changed*
During initialization, queues are created with a fixed capacity based on the initial configuration value. When configuration changes, {{resizeQueues()}} only updates the {{currentQueueLimit}} variable but cannot modify the underlying {{BlockingQueue}} capacity, which is immutable.
h2. Proposed Solutions
*Option 1: Set hard limit to Integer.MAX_VALUE (or sufficiently large value)*
Modify {{initializeQueues()}} to set the queue capacity to {{Integer.MAX_VALUE}} instead of the configured value, and rely solely on {{currentQueueLimit}} for enforcement. This is simple and enables dynamic resizing in both directions. Note that this may allow slight overshooting of the soft limit due to race conditions in concurrent dispatch.
*Option 2: Recreate queues when increasing capacity*
When {{resizeQueues()}} detects an increase in limit, drain existing queues and create new ones with the larger capacity. This is more complex but preserves hard limit safety.
*Option 3: Use Semaphore for limit enforcement*
Maintain a separate {{Semaphore}} per queue for atomic limit control. This eliminates race conditions but adds overhead.
I'm uncertain whether this behavior is intentional or needs fixing. Is this something that should be addressed? If so, which approach would be most appropriate? Any feedback would be greatly appreciated.
was:
h2. Problem
The dynamic configurations for call queue length (such as {{{}ipc.server.max.callqueue.length{}}}, {{{}ipc.server.priority.max.callqueue.length{}}}, {{{}ipc.server.replication.max.callqueue.length{}}}, {{{}ipc.server.bulkload.max.callqueue.length{}}}) only works when *decreasing* the limit. When *increasing* the limit, the configuration change has no effect - tasks are still rejected at the hard limit.
*Example:*
* Initial configuration: {{ipc.server.max.callqueue.length = 100}}
* Change configuration to: {{ipc.server.max.callqueue.length = 200}}
* Expected: Queue accepts up to 200 tasks
* Actual: Queue still rejects tasks at 100
h2. Root Cause
{{RpcExecutor}} uses two limit mechanisms: (This was introduced by HBASE-15306)
# *Soft limit* ({{{}currentQueueLimit{}}} variable): Updated by {{resizeQueues()}} when configuration changes
# *Hard limit* ({{{}BlockingQueue{}}} capacity): Set once during queue initialization and *cannot be changed*
During initialization, queues are created with a fixed capacity based on the initial configuration value. When configuration changes, {{resizeQueues()}} only updates the {{currentQueueLimit}} variable but cannot modify the underlying {{BlockingQueue}} capacity, which is immutable.
h2. Proposed Solutions
*Option 1: Set hard limit to Integer.MAX_VALUE (or sufficiently large value)*
Modify {{initializeQueues()}} to set the queue capacity to {{Integer.MAX_VALUE}} instead of the configured value, and rely solely on {{currentQueueLimit}} for enforcement. This is simple and enables dynamic resizing in both directions. Note that this may allow slight overshooting of the soft limit due to race conditions in concurrent dispatch.
*Option 2: Recreate queues when increasing capacity*
When {{resizeQueues()}} detects an increase in limit, drain existing queues and create new ones with the larger capacity. This is more complex but preserves hard limit safety.
*Option 3: Use Semaphore for limit enforcement*
Maintain a separate {{Semaphore}} per queue for atomic limit control. This eliminates race conditions but adds overhead.
I'm uncertain whether this behavior is intentional or needs fixing. Is this something that should be addressed? If so, which approach would be most appropriate for HBase's architecture? Any feedback would be greatly appreciated.
> Dynamic configurations for call queue length doesn't work correctly when increasing the limit
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-29781
> URL: https://issues.apache.org/jira/browse/HBASE-29781
> Project: HBase
> Issue Type: Bug
> Components: rpc
> Reporter: Jaehui Lee
> Assignee: Jaehui Lee
> Priority: Minor
>
> h2. Problem
> The dynamic configurations for call queue length (such as {{{}ipc.server.max.callqueue.length{}}}, {{{}ipc.server.priority.max.callqueue.length{}}}, {{{}ipc.server.replication.max.callqueue.length{}}}, {{{}ipc.server.bulkload.max.callqueue.length{}}}) only works when *decreasing* the limit. When *increasing* the limit, the configuration change has no effect - tasks are still rejected at the hard limit.
> *Example:*
> * Initial configuration: {{ipc.server.max.callqueue.length = 100}}
> * Change configuration to: {{ipc.server.max.callqueue.length = 200}}
> * Expected: Queue accepts up to 200 tasks
> * Actual: Queue still rejects tasks at 100
> h2. Root Cause
> {{RpcExecutor}} uses two limit mechanisms: (This was introduced by HBASE-15306)
> # *Soft limit* ({{{}currentQueueLimit{}}} variable): Updated by {{resizeQueues()}} when configuration changes
> # *Hard limit* ({{{}BlockingQueue{}}} capacity): Set once during queue initialization and *cannot be changed*
>
> During initialization, queues are created with a fixed capacity based on the initial configuration value. When configuration changes, {{resizeQueues()}} only updates the {{currentQueueLimit}} variable but cannot modify the underlying {{BlockingQueue}} capacity, which is immutable.
> h2. Proposed Solutions
> *Option 1: Set hard limit to Integer.MAX_VALUE (or sufficiently large value)*
> Modify {{initializeQueues()}} to set the queue capacity to {{Integer.MAX_VALUE}} instead of the configured value, and rely solely on {{currentQueueLimit}} for enforcement. This is simple and enables dynamic resizing in both directions. Note that this may allow slight overshooting of the soft limit due to race conditions in concurrent dispatch.
> *Option 2: Recreate queues when increasing capacity*
> When {{resizeQueues()}} detects an increase in limit, drain existing queues and create new ones with the larger capacity. This is more complex but preserves hard limit safety.
> *Option 3: Use Semaphore for limit enforcement*
> Maintain a separate {{Semaphore}} per queue for atomic limit control. This eliminates race conditions but adds overhead.
>
> I'm uncertain whether this behavior is intentional or needs fixing. Is this something that should be addressed? If so, which approach would be most appropriate? Any feedback would be greatly appreciated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)