[jira] [Comment Edited] (HBASE-30348) Throttles should work for system tables (excluding quotas, meta)

["Ray Mattingly (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-30348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18110767#comment-18110767 ] 

Ray Mattingly edited comment on HBASE-30348 at 9/2/26 7:26 PM:
---------------------------------------------------------------

I'm working on [a draft|https://github.com/apache/hbase/compare/master...HubSpot:hbase:HBASE-30348-v2] that solves this by:
 * Allowing quotas against system tables, except for {{{}hbase:meta{}}}, {{{}hbase:quota{}}}, and {{{}hbase:namespace{}}}. Adding the namespace table is what fixes the immediate 2.x deadlock – {{TableNamespaceManager}} reads it during {{{}initClusterSchemaService{}}}, which runs before {{initQuotaManager}} creates {{{}hbase:quota{}}}.
 * Adding defense in depth for this genre of deadlock: {{QuotaCache#ensureInitialized}} will run the initial load on an executor, waiting on it for a configurable bounded duration rather than loading inline on the RPC handler. A request that waits out the timeout proceeds with default, unthrottled quota state because that's better than a deadlock and there's no way for this code to really be certain that the master is appropriately initialized.

Does that strategy sound reasonable?


was (Author: JIRAUSER286879):
I'm working on a draft that solves this by:

* Allowing quotas against system tables, except for {{hbase:meta}}, {{hbase:quota}}, and {{hbase:namespace}}. Adding the namespace table is what fixes the immediate 2.x deadlock -- {{TableNamespaceManager}} reads it during {{initClusterSchemaService}}, which runs before {{initQuotaManager}} creates {{hbase:quota}}.
* Adding defense in depth for this genre of deadlock: {{QuotaCache#ensureInitialized}} will run the initial load on an executor, waiting on it for a configurable bounded duration rather than loading inline on the RPC handler. A request that waits out the timeout proceeds with default, unthrottled quota state because that's better than a deadlock and there's no way for this code to really be certain that the master is appropriately initialized.

Does that strategy sound reasonable?

> Throttles should work for system tables (excluding quotas, meta)
> ----------------------------------------------------------------
>
>                 Key: HBASE-30348
>                 URL: https://issues.apache.org/jira/browse/HBASE-30348
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 2.6.7
>            Reporter: Ray Mattingly
>            Assignee: Ray Mattingly
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0-alpha-1, 2.7.0, 3.1.0, 3.0.1, 2.6.8
>
>
> Right now throttles silently no-op for all system tables. Outside of the quotas table itself, of course, and perhaps the meta table I don't think this actually makes sense. We saw a pathologic build up of huge cells in incremental backups tables, for example, and it resulted in significant IO hotspots throughout our HBase clusters during backups and during bulk loads. We will fix that backups bug separately, but we would also love to have throttles in place on these backup system tables.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)