[jira] [Comment Edited] (HBASE-26703) Allow configuration of IPC queue balancer

["Andrew Kyle Purtell (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-26703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17493546#comment-17493546 ] 

Andrew Kyle Purtell edited comment on HBASE-26703 at 2/16/22, 11:36 PM:
------------------------------------------------------------------------

Thanks for sharing [~bbeaudreault]. 

bq. One might also ask why not just use quotas or other rate limiting, which is actually what we've had bespoke built into our hbase fork for years and are hoping to drop in our upgrade to hbase2.

Rate limiting can help but it's not effective if you cannot cleanly segregate users by priority band based on table or namespace or user. For example, we have a highly multienant application managing a billion+ unique identities but all access at the HBase level is via a single Kerberos principal, and many tables are accessed by a significant subset of application identities. I think this is very common. Also, conditions can dynamically change after calls have been accepted into the queues. So for these reasons and more the techniques you describe and this extension surface that supports them is of value. (IMHO)

bq. We're probably going to do a blog post with more details at some point in the future, and maybe upstream parts of it if there's interest.

The Apache Phoenix project might be interested. They have design goals on avoiding starvation of base table updates due to index table updates, and vice versa; and there was earlier work like PHOENIX-938 and HBASE-11048. Also a noisy neighbor mitigation would be of interest to the Phoenix focused dev teams adjacent to me at my employer. 


was (Author: apurtell):
Thanks for sharing [~bbeaudreault]. 

bq. One might also ask why not just use quotas or other rate limiting, which is actually what we've had bespoke built into our hbase fork for years and are hoping to drop in our upgrade to hbase2.

Rate limiting can help but it's not effective if you cannot cleanly segregate users by priority band based on table or namespace or user. For example, we have a highly multienant application managing a billion+ unique identities but all access at the HBase level is via a single Kerberos principal. I think this is very common. Also, conditions can dynamically change after calls have been accepted into the queues. So for these reasons and more the techniques you describe and this extension surface that supports them is of value. (IMHO)

bq. We're probably going to do a blog post with more details at some point in the future, and maybe upstream parts of it if there's interest.

The Apache Phoenix project might be interested. They have design goals on avoiding starvation of base table updates due to index table updates, and vice versa; and there was earlier work like PHOENIX-938 and HBASE-11048. Also a noisy neighbor mitigation would be of interest to the Phoenix focused dev teams adjacent to me at my employer. 

> Allow configuration of IPC queue balancer
> -----------------------------------------
>
>                 Key: HBASE-26703
>                 URL: https://issues.apache.org/jira/browse/HBASE-26703
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Bryan Beaudreault
>            Assignee: Bryan Beaudreault
>            Priority: Minor
>             Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> Currently we randomly assign IPC calls to queues using a RandomQueueBalancer, which relies on ThreadLocalRandom. I would like to make that configurable so that one can plug in their own queue balancer. This usefully combines with the existing ability to specify a pluggable queue type.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)