[jira] [Updated] (RANGER-4761) Reduce memory footprint of hbase plugin

["Fateh Singh (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/RANGER-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fateh Singh updated RANGER-4761:
--------------------------------
    Summary: Reduce memory footprint of hbase plugin  (was: Revisit ColumnFamilyCache to reduce memory footprint of hbase plugin)

> Reduce memory footprint of hbase plugin
> ---------------------------------------
>
>                 Key: RANGER-4761
>                 URL: https://issues.apache.org/jira/browse/RANGER-4761
>             Project: Ranger
>          Issue Type: Improvement
>          Components: Ranger
>            Reporter: Fateh Singh
>            Assignee: Fateh Singh
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> * Map<String, Set<String>> getColumnFamilies(Map<byte[], ? extends Collection<?>> families) was the original bottleneck for resulting in Ranger Authz CP taking 60% of Put RPC time — it is a computationally heavy function converting bytes to string and type-casting Collection to set of strings.
>  * ColumnFamilyCache was introduced to fix this issue. But this caching does not work because the columns and column families accessed for a table are inconsistent and results in many many *cache misses* *and thus results in getColumnFamilies()* getting called – so additional memory requirement for cache (grows exponentially due to large number of columns) and also double computation (serialization of new family map to check in cache and also getColumnFamilies is computed). E.g. First call comes for cf1:c1,c3,c4 Second call comes for cf1:c1,c2,c5 then the second call is a cache miss because the set of columns accessed is different from the first call. Both these entries get added to the cache. 
>  * Validation: Added debug statements for cache hit and miss counts.  word count in log file for Cache Miss and Cache Hit, Cache size reaches max default size of 1024
> {code:java}
> [root@ccycloud-2 hbase]# grep 'Cache Miss' hbase-cmf-HBASE-1-REGIONSERVER-ccycloud-2.fs2-7192.root.comops.site.log.out | wc -l
> 10584
> [root@ccycloud-2 hbase]# grep 'Cache Hit' hbase-cmf-HBASE-1-REGIONSERVER-ccycloud-2.fs2-7192.root.comops.site.log.out | wc -l
> 0
> {code}
> {code:java}
> 2024-02-09 17:12:31,093 DEBUG org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor: evaluateAccess: Cache Size:1024
> 2024-02-09 17:12:31,096 DEBUG org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor: evaluateAccess: Cache Size:1024
> {code}
> Another issue:
> Key computation for cache has a bug wherein address of byte array is being used as key in hashmap instead of value which results in cache miss always irrespective of what the request is
> {code:java}
> 2024-02-09 18:12:12,310 DEBUG org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor: evaluateAccess: Cache Miss for user[atlas] for key: atlas_janus:[[B@1b59403b=null]{code}
>  
> The implementation needs to be revisited to reduce memory footprint



--
This message was sent by Atlassian Jira
(v8.20.10#820010)