[jira] [Commented] (RANGER-2341) Support for Incremental policy updates to improve performance of ranger-admin and plugins by optimal building of policy-engine

["Pradeep Agrawal (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/RANGER-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17416842#comment-17416842 ] 

Pradeep Agrawal commented on RANGER-2341:
-----------------------------------------

Review request for Addendum patch : https://reviews.apache.org/r/73587/

> Support for Incremental policy updates to improve performance of ranger-admin and plugins by optimal building of policy-engine
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: RANGER-2341
>                 URL: https://issues.apache.org/jira/browse/RANGER-2341
>             Project: Ranger
>          Issue Type: Improvement
>          Components: Ranger
>    Affects Versions: master
>            Reporter: Abhay Kulkarni
>            Assignee: Abhay Kulkarni
>            Priority: Major
>             Fix For: 2.0.0
>
>
> Requirements:
> Currently, every change to any policy causes rebuilding of policy-engine from scratch. There are several disadvantages:
> 1. Compute time for rebuilding
> 2. Large traffic from ranger-admin to each of the plugins
> 3. Large demand on JVM memory system resulting in frequent garbage collection and pauses of JVM.
> It will be more optimal to communicate only the changes and apply them to existing policy-engine.
> Design notes:
> Policy changes are logged into a new database table.
> Cache management in ranger-admin is enhanced to use this table to figure out changes using a previously known version number (provided by module requesting updated policies).
> Policy engine supports update operation that accepts policy-deltas and returns a new policy engine with deltas applied.
> Resource Trie structures are copied from older policy-engine selectively, and not rebuilt from scratch.
> Backward compatibility is maintained with older plugins by adding another parameter to REST API for downloading policies.
> Ranger admin as well as component plugins may be configured to optionally use policy deltas for its internal policy-engines. Policy deltas are disabled by default. In ranger-admin, policy-deltas are enabled in the ranger-admin by setting configuration variable 'ranger.admin.supports.policy.deltas' to true. In individual plugins, policy-deltas are enabled by setting configuration variable 'ranger.plugin.<service-type>.policy.rest.supports.policy.deltas' to "true".
> Policy delta table is cleared of records older than a week on restart of ranger-admin.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)