[jira] [Commented] (RANGER-2341) Support for Incremental policy updates to improve performance of ranger-admin and plugins by optimal building of policy-engine

["Don Bosco Durai (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/RANGER-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781211#comment-16781211 ] 

Don Bosco Durai commented on RANGER-2341:
-----------------------------------------

[~abhayk] this would be a good feature. Few questions...

> Cache management in ranger-admin is enhanced to use this table to figure out changes using a previously known version number (provided by module requesting updated policies).
Seems more like more like redo logs in database, which I feel is a good approach.

> Backward compatibility is maintained with older plugins by adding another parameter to REST API for downloading policies.
Should we do the other way? New plugins should pass the addition param, so that older plugins will work without change?
 
> Policy deltas are disabled by default. 
I feel, we should enable this by default. This is a good feature and let the plugins decide whether to use or not.

> Policy delta table is cleared of records older than a week on restart of ranger-admin.
I not sure whether restart should be the trigger, but might be okay for now till have an inbuilt scheduler. I assume, we will make the the retention period configurable.



> Support for Incremental policy updates to improve performance of ranger-admin and plugins by optimal building of policy-engine
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: RANGER-2341
>                 URL: https://issues.apache.org/jira/browse/RANGER-2341
>             Project: Ranger
>          Issue Type: Improvement
>          Components: Ranger
>    Affects Versions: master
>            Reporter: Abhay Kulkarni
>            Assignee: Abhay Kulkarni
>            Priority: Major
>             Fix For: master
>
>
> Requirements:
> Currently, every change to any policy causes rebuilding of policy-engine from scratch. There are several disadvantages:
> 1. Compute time for rebuilding
> 2. Large traffic from ranger-admin to each of the plugins
> 3. Large demand on JVM memory system resulting in frequent garbage collection and pauses of JVM.
> It will be more optimal to communicate only the changes and apply them to existing policy-engine.
> Design notes:
> Policy changes are logged into a new database table.
> Cache management in ranger-admin is enhanced to use this table to figure out changes using a previously known version number (provided by module requesting updated policies).
> Policy engine supports update operation that accepts policy-deltas and returns a new policy engine with deltas applied.
> Resource Trie structures are copied from older policy-engine selectively, and not rebuilt from scratch.
> Backward compatibility is maintained with older plugins by adding another parameter to REST API for downloading policies.
> Ranger admin as well as component plugins may be configured to optionally use policy deltas for its internal policy-engines. Policy deltas are disabled by default. In ranger-admin, policy-deltas are enabled in the ranger-admin by setting configuration variable 'ranger.admin.supports.policy.deltas' to true. In individual plugins, policy-deltas are enabled by setting configuration variable 'ranger.plugin.<service-type>.policy.rest.supports.policy.deltas' to "true".
> Policy delta table is cleared of records older than a week on restart of ranger-admin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)