[jira] [Updated] (HBASE-27784) Custom quota groupings

["Bryan Beaudreault (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-27784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bryan Beaudreault updated HBASE-27784:
--------------------------------------
    Description: 
Currently we provide the ability to define quotas for namespaces, tables, or users. On multi-tenant clusters, users may be broken down into groups based on their use-case. For us this comes down to 2 main cases:
 # Hadoop jobs – it would be good to be able to limit all hadoop jobs in aggregate
 # Proxy APIs - this is common where upstream callers don't hit hbase directly, instead they go through one of many proxy api's.  For us we have a custom auth plugin which sets the username to the upstream caller name. But it would still be useful to be able to limit all usage from some particular proxy API in aggregate.

I think this could build upon the idea for Connection attributes in HBASE-27657. Basically when a Connection is established we can set an attribute (i.e. quotaGrouping=hadoop or quotaGrouping=MyProxyAPI).  In QuotaCache, we can add a {{getQuotaGroupLimiter(String groupName)}} and also allow someone to define quotas using {{set_quota TYPE => THROTTLE, GROUP => 'hadoop', LIMIT => '100M/sec'}}

I need to do more investigation into whether we'd want to return a simple group limiter (more similar to table/namespace handling) or treat it more like the USER limiters which returns a QuotaState (so you can limit by-group-by-table).

We need to consider how GROUP quotas interact with USER quotas. If a user has a quota defined, and that user is also part of a group with a quota defined, does the request need to honor both quotas? Maybe we provide a GROUP_BYPASS setting, similar to GLOBAL_BYPASS?

  was:
Currently we provide the ability to define quotas for namespaces, tables, or users. On multi-tenant clusters, users may be broken down into groups based on their use-case. For us this comes down to 2 main cases:
 # Hadoop jobs – it would be good to be able to limit all hadoop jobs in aggregate
 # Proxy APIs - this is common where upstream callers don't hit hbase directly, instead they go through one of many proxy api's.  For us we have a custom auth plugin which sets the username to the upstream caller name. But it would still be useful to be able to limit all usage from some particular proxy API in aggregate.

I think this could build upon the idea for Connection attributes in HBASE-27657. Basically when a Connection is established we can set an attribute (i.e. quotaGrouping=hadoop or quotaGrouping=MyProxyAPI).  In QuotaCache, we can add a {{getQuotaGroupLimiter(String groupName)}} and also allow someone to define quotas using {{set_quota TYPE => THROTTLE, GROUP => 'hadoop', LIMIT => '100M/sec'}}

I need to do more investigation into whether we'd want to return a simple group limiter (more similar to table/namespace handling) or treat it more like the USER limiters which returns a QuotaState (so you can limit by-group-by-table).


> Custom quota groupings
> ----------------------
>
>                 Key: HBASE-27784
>                 URL: https://issues.apache.org/jira/browse/HBASE-27784
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: Bryan Beaudreault
>            Priority: Major
>
> Currently we provide the ability to define quotas for namespaces, tables, or users. On multi-tenant clusters, users may be broken down into groups based on their use-case. For us this comes down to 2 main cases:
>  # Hadoop jobs – it would be good to be able to limit all hadoop jobs in aggregate
>  # Proxy APIs - this is common where upstream callers don't hit hbase directly, instead they go through one of many proxy api's.  For us we have a custom auth plugin which sets the username to the upstream caller name. But it would still be useful to be able to limit all usage from some particular proxy API in aggregate.
> I think this could build upon the idea for Connection attributes in HBASE-27657. Basically when a Connection is established we can set an attribute (i.e. quotaGrouping=hadoop or quotaGrouping=MyProxyAPI).  In QuotaCache, we can add a {{getQuotaGroupLimiter(String groupName)}} and also allow someone to define quotas using {{set_quota TYPE => THROTTLE, GROUP => 'hadoop', LIMIT => '100M/sec'}}
> I need to do more investigation into whether we'd want to return a simple group limiter (more similar to table/namespace handling) or treat it more like the USER limiters which returns a QuotaState (so you can limit by-group-by-table).
> We need to consider how GROUP quotas interact with USER quotas. If a user has a quota defined, and that user is also part of a group with a quota defined, does the request need to honor both quotas? Maybe we provide a GROUP_BYPASS setting, similar to GLOBAL_BYPASS?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)