You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Pradeep Agrawal <pr...@freestoneinfotech.com> on 2016/04/07 12:49:34 UTC
Review Request 45091: RANGER-900 : Remove support for DB based
auditing
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45091/
-----------------------------------------------------------
Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
Bugs: RANGER-900
https://issues.apache.org/jira/browse/RANGER-900
Repository: ranger
Description
-------
**Problem Statement :**
Remove option to store audit to DB as storing audit logs in db requires lots of data management activity and frequent backup-restore process might hamper Ranger application performance. Production team might face frequent down-time issues due to db disk space reclaim activities.
**Proposed Solution :**
Proposed solution is having below mentioned approch :
1. Remove audit to DB related properties from install.properties of all components.
2. Disable shell script code to read audit to DB related properties from install.properties of all components.
3. Disable code from dba_script.py which is invoked to create audit DB schema, audit User and executes grants privileges.
4. Disable code from db_setup.py to skip 'xa_access_audit' table creation and stop executing audit to Db related sql patches.
5. Make solr as mandatory audit data store/source.
Diffs
-----
agents-audit/src/main/java/org/apache/ranger/audit/test/TestEvents.java 3e89cc4
agents-common/scripts/enable-agent.sh b9511d2
agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java 8ee6bea
hbase-agent/conf/ranger-hbase-audit-changes.cfg e29ccd5
hbase-agent/scripts/install.properties 795ea3e
hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 9c88450
hdfs-agent/scripts/install.properties b4dda13
hive-agent/conf/ranger-hive-audit-changes.cfg 4e61c7d
hive-agent/scripts/install.properties 6b71a85
kms/scripts/install.properties d30b28c
knox-agent/conf/ranger-knox-audit-changes.cfg f722e53
knox-agent/scripts/install.properties 1febd49
plugin-kafka/conf/ranger-kafka-audit-changes.cfg 46ee29a
plugin-kafka/scripts/install.properties 79ea6db
plugin-kms/conf/ranger-kms-audit-changes.cfg 5a51455
plugin-kms/scripts/enable-kms-plugin.sh 7bf6c62
plugin-solr/conf/ranger-solr-audit-changes.cfg 2742bc1
plugin-solr/scripts/install.properties a3d9887
plugin-yarn/conf/ranger-yarn-audit-changes.cfg b650be1
plugin-yarn/scripts/install.properties 3780068
security-admin/scripts/db_setup.py f2cc9b9
security-admin/scripts/dba_script.py 0ebd90b
security-admin/scripts/install.properties 1d9d207
security-admin/scripts/setup.sh bf29ed6
storm-agent/conf/ranger-storm-audit-changes.cfg b650be1
storm-agent/scripts/install.properties f2aa5c4
Diff: https://reviews.apache.org/r/45091/diff/
Testing
-------
**Steps Performaed (With patch) :**
**Use-case 1:**
Fresh Ranger Admin Installation :
Steps:
1. After Ranger installation did not find any audit to DB related properties in install.properties file so provided 'solr' as audit data store and configured solr URL in solr_url property.
2. Executed setup.sh to install Ranger
Expected Behaviour :
1. Installation script should complete successfully and after starting Ranger, Ranger UI should work; user should able to create services, policies, users and groups.
Actual Behaviour :
1. In installation log it was observed that installation process skipped creation of audit DB, audit user and execution of audit db related sql patches.
2. Ranger installation was finished successfully.
3. After starting Ranger; was able to login to Ranger and Ranger UI was working fine. Was able to create services, policies, users and groups.
**Use-case 2:**
Enabling Ranger plugin and writing audit logs To solr :
Steps:
1. Enabled HDFS plugin with solr and provided solr url so that hdfs component should write audit logs in solr.
2. Excecuted HDFS command to READ/CREATE resources of hdfs for which Ranger policies were created.
Expected Behaviour :
HDFS plugin should write audit logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
Actual Behaviour :
Expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
**Use-case 3:**
Ranger admin and Ranger plugins upgrade:
Steps:
1. Installed Ranger admin without patch and started Ranger admin with audit source as DB; enabled HDFS plugin with Audit logs to all three audit destination DB, HDFS and solr.
2. Created HDFS service and policies; assigned policies to users with different combination of access permissions.
3. From console window excecuted HDFS command to READ/CREATE HDFS resources on which Ranger policies was created.
4. It was observed that HDFS plugin was writing audit logs to all three audit stores.
5. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
6. Stopped Ranger Admin, disabled plugins to stop communicating to Ranger Admin, Stopped HDFS component.
7. Copied Ranger admin install.properties and installed Ranger with patch and used same properties of previous installation, since new install.properties did not have audit to DB related properties so skipped that and provided solr url which was used in solr related config of HDFS plugin.
7. Executed Ranger setup script and restarted Ranger admin.
8. Now Ranger UI was reading audit logs from solr source and expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
9. Enabled HDFS plugin(patched version) with audit destination as HDFS and solr.
10. Excecuted HDFS command to READ/CREATE resources on which Ranger policies exist.
Expected Behaviour :
1. HDFS plugin should write new logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
2. Expected logs should appear in Solr UI also.
3. HDFS plugin must not write any new audit logs to DB.
Actual Behaviour :
1. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
2. Expected logs were available in solr UI.
3. There was no new logs in 'xa_access_audit' table of Ranger audit DB.
Thanks,
Pradeep Agrawal
Re: Review Request 45091: RANGER-900 : Remove support for DB based
auditing
Posted by Velmurugan Periasamy <vp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45091/#review129010
-----------------------------------------------------------
Ship it!
Ship It!
- Velmurugan Periasamy
On April 14, 2016, 6:39 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45091/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 6:39 p.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-900
> https://issues.apache.org/jira/browse/RANGER-900
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement :**
> Remove option to store audit to DB as storing audit logs in db requires lots of data management activity and frequent backup-restore process might hamper Ranger application performance. Production team might face frequent down-time issues due to db disk space reclaim activities.
>
>
> **Proposed Solution :**
> Proposed solution is having below mentioned approch :
> 1. Remove audit to DB related properties from install.properties of all components.
> 2. Disable shell script code to read audit to DB related properties from install.properties of all components.
> 3. Disable code from dba_script.py which is invoked to create audit DB schema, audit User and executes grants privileges.
> 4. Disable code from db_setup.py to skip 'xa_access_audit' table creation and stop executing audit to Db related sql patches.
> 5. Make solr as mandatory audit data store/source.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/test/TestEvents.java 3e89cc4
> agents-common/scripts/enable-agent.sh b9511d2
> agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java 8ee6bea
> hbase-agent/conf/ranger-hbase-audit-changes.cfg e29ccd5
> hbase-agent/scripts/install.properties 795ea3e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 9c88450
> hdfs-agent/scripts/install.properties b4dda13
> hive-agent/conf/ranger-hive-audit-changes.cfg 4e61c7d
> hive-agent/scripts/install.properties 6b71a85
> kms/scripts/install.properties 7762948
> knox-agent/conf/ranger-knox-audit-changes.cfg f722e53
> knox-agent/scripts/install.properties 1febd49
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg 46ee29a
> plugin-kafka/scripts/install.properties 79ea6db
> plugin-kms/conf/ranger-kms-audit-changes.cfg 5a51455
> plugin-kms/scripts/enable-kms-plugin.sh 7bf6c62
> plugin-solr/conf/ranger-solr-audit-changes.cfg 2742bc1
> plugin-solr/scripts/install.properties a3d9887
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg b650be1
> plugin-yarn/scripts/install.properties 3780068
> security-admin/scripts/db_setup.py 3d20fcd
> security-admin/scripts/dba_script.py 0ebd90b
> security-admin/scripts/install.properties 1d9d207
> security-admin/scripts/setup.sh bf29ed6
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 3333827
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml 6ee48f4
> storm-agent/conf/ranger-storm-audit-changes.cfg b650be1
> storm-agent/scripts/install.properties f2aa5c4
>
> Diff: https://reviews.apache.org/r/45091/diff/
>
>
> Testing
> -------
>
> **Steps Performaed (With patch) :**
>
> **Use-case 1:**
> Fresh Ranger Admin Installation :
>
> Steps:
> 1. After Ranger installation did not find any audit to DB related properties in install.properties file so provided 'solr' as audit data store and configured solr URL in solr_url property.
> 2. Executed setup.sh to install Ranger
>
> Expected Behaviour :
> 1. Installation script should complete successfully and after starting Ranger, Ranger UI should work; user should able to create services, policies, users and groups.
>
> Actual Behaviour :
> 1. In installation log it was observed that installation process skipped creation of audit DB, audit user and execution of audit db related sql patches.
> 2. Ranger installation was finished successfully.
> 3. After starting Ranger; was able to login to Ranger and Ranger UI was working fine. Was able to create services, policies, users and groups.
>
> **Use-case 2:**
> Enabling Ranger plugin and writing audit logs To solr :
>
> Steps:
> 1. Enabled HDFS plugin with solr and provided solr url so that hdfs component should write audit logs in solr.
> 2. Excecuted HDFS command to READ/CREATE resources of hdfs for which Ranger policies were created.
>
> Expected Behaviour :
> HDFS plugin should write audit logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
>
> Actual Behaviour :
> Expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
>
>
> **Use-case 3:**
> Ranger admin and Ranger plugins upgrade:
>
> Steps:
> 1. Installed Ranger admin without patch and started Ranger admin with audit source as DB; enabled HDFS plugin with Audit logs to all three audit destination DB, HDFS and solr.
> 2. Created HDFS service and policies; assigned policies to users with different combination of access permissions.
> 3. From console window excecuted HDFS command to READ/CREATE HDFS resources on which Ranger policies was created.
> 4. It was observed that HDFS plugin was writing audit logs to all three audit stores.
> 5. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
> 6. Stopped Ranger Admin, disabled plugins to stop communicating to Ranger Admin, Stopped HDFS component.
> 7. Copied Ranger admin install.properties and installed Ranger with patch and used same properties of previous installation, since new install.properties did not have audit to DB related properties so skipped that and provided solr url which was used in solr related config of HDFS plugin.
> 7. Executed Ranger setup script and restarted Ranger admin.
> 8. Now Ranger UI was reading audit logs from solr source and expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
> 9. Enabled HDFS plugin(patched version) with audit destination as HDFS and solr.
> 10. Excecuted HDFS command to READ/CREATE resources on which Ranger policies exist.
>
> Expected Behaviour :
> 1. HDFS plugin should write new logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
> 2. Expected logs should appear in Solr UI also.
> 3. HDFS plugin must not write any new audit logs to DB.
>
>
> Actual Behaviour :
> 1. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
> 2. Expected logs were available in solr UI.
> 3. There was no new logs in 'xa_access_audit' table of Ranger audit DB.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 45091: RANGER-900 : Remove support for DB based
auditing
Posted by Pradeep Agrawal <pr...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45091/
-----------------------------------------------------------
(Updated April 14, 2016, 6:39 p.m.)
Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
Changes
-------
Addressed review comments and tested setup script for all db flavor in local environment
Bugs: RANGER-900
https://issues.apache.org/jira/browse/RANGER-900
Repository: ranger
Description
-------
**Problem Statement :**
Remove option to store audit to DB as storing audit logs in db requires lots of data management activity and frequent backup-restore process might hamper Ranger application performance. Production team might face frequent down-time issues due to db disk space reclaim activities.
**Proposed Solution :**
Proposed solution is having below mentioned approch :
1. Remove audit to DB related properties from install.properties of all components.
2. Disable shell script code to read audit to DB related properties from install.properties of all components.
3. Disable code from dba_script.py which is invoked to create audit DB schema, audit User and executes grants privileges.
4. Disable code from db_setup.py to skip 'xa_access_audit' table creation and stop executing audit to Db related sql patches.
5. Make solr as mandatory audit data store/source.
Diffs (updated)
-----
agents-audit/src/main/java/org/apache/ranger/audit/test/TestEvents.java 3e89cc4
agents-common/scripts/enable-agent.sh b9511d2
agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java 8ee6bea
hbase-agent/conf/ranger-hbase-audit-changes.cfg e29ccd5
hbase-agent/scripts/install.properties 795ea3e
hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 9c88450
hdfs-agent/scripts/install.properties b4dda13
hive-agent/conf/ranger-hive-audit-changes.cfg 4e61c7d
hive-agent/scripts/install.properties 6b71a85
kms/scripts/install.properties 7762948
knox-agent/conf/ranger-knox-audit-changes.cfg f722e53
knox-agent/scripts/install.properties 1febd49
plugin-kafka/conf/ranger-kafka-audit-changes.cfg 46ee29a
plugin-kafka/scripts/install.properties 79ea6db
plugin-kms/conf/ranger-kms-audit-changes.cfg 5a51455
plugin-kms/scripts/enable-kms-plugin.sh 7bf6c62
plugin-solr/conf/ranger-solr-audit-changes.cfg 2742bc1
plugin-solr/scripts/install.properties a3d9887
plugin-yarn/conf/ranger-yarn-audit-changes.cfg b650be1
plugin-yarn/scripts/install.properties 3780068
security-admin/scripts/db_setup.py 3d20fcd
security-admin/scripts/dba_script.py 0ebd90b
security-admin/scripts/install.properties 1d9d207
security-admin/scripts/setup.sh bf29ed6
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 3333827
security-admin/src/main/resources/conf.dist/ranger-admin-site.xml 6ee48f4
storm-agent/conf/ranger-storm-audit-changes.cfg b650be1
storm-agent/scripts/install.properties f2aa5c4
Diff: https://reviews.apache.org/r/45091/diff/
Testing
-------
**Steps Performaed (With patch) :**
**Use-case 1:**
Fresh Ranger Admin Installation :
Steps:
1. After Ranger installation did not find any audit to DB related properties in install.properties file so provided 'solr' as audit data store and configured solr URL in solr_url property.
2. Executed setup.sh to install Ranger
Expected Behaviour :
1. Installation script should complete successfully and after starting Ranger, Ranger UI should work; user should able to create services, policies, users and groups.
Actual Behaviour :
1. In installation log it was observed that installation process skipped creation of audit DB, audit user and execution of audit db related sql patches.
2. Ranger installation was finished successfully.
3. After starting Ranger; was able to login to Ranger and Ranger UI was working fine. Was able to create services, policies, users and groups.
**Use-case 2:**
Enabling Ranger plugin and writing audit logs To solr :
Steps:
1. Enabled HDFS plugin with solr and provided solr url so that hdfs component should write audit logs in solr.
2. Excecuted HDFS command to READ/CREATE resources of hdfs for which Ranger policies were created.
Expected Behaviour :
HDFS plugin should write audit logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
Actual Behaviour :
Expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
**Use-case 3:**
Ranger admin and Ranger plugins upgrade:
Steps:
1. Installed Ranger admin without patch and started Ranger admin with audit source as DB; enabled HDFS plugin with Audit logs to all three audit destination DB, HDFS and solr.
2. Created HDFS service and policies; assigned policies to users with different combination of access permissions.
3. From console window excecuted HDFS command to READ/CREATE HDFS resources on which Ranger policies was created.
4. It was observed that HDFS plugin was writing audit logs to all three audit stores.
5. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
6. Stopped Ranger Admin, disabled plugins to stop communicating to Ranger Admin, Stopped HDFS component.
7. Copied Ranger admin install.properties and installed Ranger with patch and used same properties of previous installation, since new install.properties did not have audit to DB related properties so skipped that and provided solr url which was used in solr related config of HDFS plugin.
7. Executed Ranger setup script and restarted Ranger admin.
8. Now Ranger UI was reading audit logs from solr source and expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
9. Enabled HDFS plugin(patched version) with audit destination as HDFS and solr.
10. Excecuted HDFS command to READ/CREATE resources on which Ranger policies exist.
Expected Behaviour :
1. HDFS plugin should write new logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
2. Expected logs should appear in Solr UI also.
3. HDFS plugin must not write any new audit logs to DB.
Actual Behaviour :
1. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
2. Expected logs were available in solr UI.
3. There was no new logs in 'xa_access_audit' table of Ranger audit DB.
Thanks,
Pradeep Agrawal
Re: Review Request 45091: RANGER-900 : Remove support for DB based
auditing
Posted by Pradeep Agrawal <pr...@freestoneinfotech.com>.
> On April 14, 2016, 4:39 p.m., Velmurugan Periasamy wrote:
> > security-admin/scripts/db_setup.py, line 2109
> > <https://reviews.apache.org/r/45091/diff/2/?file=1341082#file1341082line2109>
> >
> > Is it required to use db_name as audit_db_name here? Same for other properties?
Have commented these assignments; please review.
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45091/#review128903
-----------------------------------------------------------
On April 14, 2016, 6:39 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45091/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 6:39 p.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-900
> https://issues.apache.org/jira/browse/RANGER-900
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement :**
> Remove option to store audit to DB as storing audit logs in db requires lots of data management activity and frequent backup-restore process might hamper Ranger application performance. Production team might face frequent down-time issues due to db disk space reclaim activities.
>
>
> **Proposed Solution :**
> Proposed solution is having below mentioned approch :
> 1. Remove audit to DB related properties from install.properties of all components.
> 2. Disable shell script code to read audit to DB related properties from install.properties of all components.
> 3. Disable code from dba_script.py which is invoked to create audit DB schema, audit User and executes grants privileges.
> 4. Disable code from db_setup.py to skip 'xa_access_audit' table creation and stop executing audit to Db related sql patches.
> 5. Make solr as mandatory audit data store/source.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/test/TestEvents.java 3e89cc4
> agents-common/scripts/enable-agent.sh b9511d2
> agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java 8ee6bea
> hbase-agent/conf/ranger-hbase-audit-changes.cfg e29ccd5
> hbase-agent/scripts/install.properties 795ea3e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 9c88450
> hdfs-agent/scripts/install.properties b4dda13
> hive-agent/conf/ranger-hive-audit-changes.cfg 4e61c7d
> hive-agent/scripts/install.properties 6b71a85
> kms/scripts/install.properties 7762948
> knox-agent/conf/ranger-knox-audit-changes.cfg f722e53
> knox-agent/scripts/install.properties 1febd49
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg 46ee29a
> plugin-kafka/scripts/install.properties 79ea6db
> plugin-kms/conf/ranger-kms-audit-changes.cfg 5a51455
> plugin-kms/scripts/enable-kms-plugin.sh 7bf6c62
> plugin-solr/conf/ranger-solr-audit-changes.cfg 2742bc1
> plugin-solr/scripts/install.properties a3d9887
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg b650be1
> plugin-yarn/scripts/install.properties 3780068
> security-admin/scripts/db_setup.py 3d20fcd
> security-admin/scripts/dba_script.py 0ebd90b
> security-admin/scripts/install.properties 1d9d207
> security-admin/scripts/setup.sh bf29ed6
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 3333827
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml 6ee48f4
> storm-agent/conf/ranger-storm-audit-changes.cfg b650be1
> storm-agent/scripts/install.properties f2aa5c4
>
> Diff: https://reviews.apache.org/r/45091/diff/
>
>
> Testing
> -------
>
> **Steps Performaed (With patch) :**
>
> **Use-case 1:**
> Fresh Ranger Admin Installation :
>
> Steps:
> 1. After Ranger installation did not find any audit to DB related properties in install.properties file so provided 'solr' as audit data store and configured solr URL in solr_url property.
> 2. Executed setup.sh to install Ranger
>
> Expected Behaviour :
> 1. Installation script should complete successfully and after starting Ranger, Ranger UI should work; user should able to create services, policies, users and groups.
>
> Actual Behaviour :
> 1. In installation log it was observed that installation process skipped creation of audit DB, audit user and execution of audit db related sql patches.
> 2. Ranger installation was finished successfully.
> 3. After starting Ranger; was able to login to Ranger and Ranger UI was working fine. Was able to create services, policies, users and groups.
>
> **Use-case 2:**
> Enabling Ranger plugin and writing audit logs To solr :
>
> Steps:
> 1. Enabled HDFS plugin with solr and provided solr url so that hdfs component should write audit logs in solr.
> 2. Excecuted HDFS command to READ/CREATE resources of hdfs for which Ranger policies were created.
>
> Expected Behaviour :
> HDFS plugin should write audit logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
>
> Actual Behaviour :
> Expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
>
>
> **Use-case 3:**
> Ranger admin and Ranger plugins upgrade:
>
> Steps:
> 1. Installed Ranger admin without patch and started Ranger admin with audit source as DB; enabled HDFS plugin with Audit logs to all three audit destination DB, HDFS and solr.
> 2. Created HDFS service and policies; assigned policies to users with different combination of access permissions.
> 3. From console window excecuted HDFS command to READ/CREATE HDFS resources on which Ranger policies was created.
> 4. It was observed that HDFS plugin was writing audit logs to all three audit stores.
> 5. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
> 6. Stopped Ranger Admin, disabled plugins to stop communicating to Ranger Admin, Stopped HDFS component.
> 7. Copied Ranger admin install.properties and installed Ranger with patch and used same properties of previous installation, since new install.properties did not have audit to DB related properties so skipped that and provided solr url which was used in solr related config of HDFS plugin.
> 7. Executed Ranger setup script and restarted Ranger admin.
> 8. Now Ranger UI was reading audit logs from solr source and expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
> 9. Enabled HDFS plugin(patched version) with audit destination as HDFS and solr.
> 10. Excecuted HDFS command to READ/CREATE resources on which Ranger policies exist.
>
> Expected Behaviour :
> 1. HDFS plugin should write new logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
> 2. Expected logs should appear in Solr UI also.
> 3. HDFS plugin must not write any new audit logs to DB.
>
>
> Actual Behaviour :
> 1. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
> 2. Expected logs were available in solr UI.
> 3. There was no new logs in 'xa_access_audit' table of Ranger audit DB.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 45091: RANGER-900 : Remove support for DB based
auditing
Posted by Velmurugan Periasamy <vp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45091/#review128903
-----------------------------------------------------------
Fix it, then Ship it!
Fix then ship it!
security-admin/scripts/db_setup.py (line 2109)
<https://reviews.apache.org/r/45091/#comment192347>
Is it required to use db_name as audit_db_name here? Same for other properties?
security-admin/scripts/dba_script.py (line 1677)
<https://reviews.apache.org/r/45091/#comment192348>
Same comment as above
- Velmurugan Periasamy
On April 12, 2016, 10:21 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45091/
> -----------------------------------------------------------
>
> (Updated April 12, 2016, 10:21 a.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-900
> https://issues.apache.org/jira/browse/RANGER-900
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement :**
> Remove option to store audit to DB as storing audit logs in db requires lots of data management activity and frequent backup-restore process might hamper Ranger application performance. Production team might face frequent down-time issues due to db disk space reclaim activities.
>
>
> **Proposed Solution :**
> Proposed solution is having below mentioned approch :
> 1. Remove audit to DB related properties from install.properties of all components.
> 2. Disable shell script code to read audit to DB related properties from install.properties of all components.
> 3. Disable code from dba_script.py which is invoked to create audit DB schema, audit User and executes grants privileges.
> 4. Disable code from db_setup.py to skip 'xa_access_audit' table creation and stop executing audit to Db related sql patches.
> 5. Make solr as mandatory audit data store/source.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/test/TestEvents.java 3e89cc4
> agents-common/scripts/enable-agent.sh b9511d2
> agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java 8ee6bea
> hbase-agent/conf/ranger-hbase-audit-changes.cfg e29ccd5
> hbase-agent/scripts/install.properties 795ea3e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 9c88450
> hdfs-agent/scripts/install.properties b4dda13
> hive-agent/conf/ranger-hive-audit-changes.cfg 4e61c7d
> hive-agent/scripts/install.properties 6b71a85
> kms/scripts/install.properties d30b28c
> knox-agent/conf/ranger-knox-audit-changes.cfg f722e53
> knox-agent/scripts/install.properties 1febd49
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg 46ee29a
> plugin-kafka/scripts/install.properties 79ea6db
> plugin-kms/conf/ranger-kms-audit-changes.cfg 5a51455
> plugin-kms/scripts/enable-kms-plugin.sh 7bf6c62
> plugin-solr/conf/ranger-solr-audit-changes.cfg 2742bc1
> plugin-solr/scripts/install.properties a3d9887
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg b650be1
> plugin-yarn/scripts/install.properties 3780068
> security-admin/scripts/db_setup.py 3d20fcd
> security-admin/scripts/dba_script.py 0ebd90b
> security-admin/scripts/install.properties 1d9d207
> security-admin/scripts/setup.sh bf29ed6
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 3333827
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml 6ee48f4
> storm-agent/conf/ranger-storm-audit-changes.cfg b650be1
> storm-agent/scripts/install.properties f2aa5c4
>
> Diff: https://reviews.apache.org/r/45091/diff/
>
>
> Testing
> -------
>
> **Steps Performaed (With patch) :**
>
> **Use-case 1:**
> Fresh Ranger Admin Installation :
>
> Steps:
> 1. After Ranger installation did not find any audit to DB related properties in install.properties file so provided 'solr' as audit data store and configured solr URL in solr_url property.
> 2. Executed setup.sh to install Ranger
>
> Expected Behaviour :
> 1. Installation script should complete successfully and after starting Ranger, Ranger UI should work; user should able to create services, policies, users and groups.
>
> Actual Behaviour :
> 1. In installation log it was observed that installation process skipped creation of audit DB, audit user and execution of audit db related sql patches.
> 2. Ranger installation was finished successfully.
> 3. After starting Ranger; was able to login to Ranger and Ranger UI was working fine. Was able to create services, policies, users and groups.
>
> **Use-case 2:**
> Enabling Ranger plugin and writing audit logs To solr :
>
> Steps:
> 1. Enabled HDFS plugin with solr and provided solr url so that hdfs component should write audit logs in solr.
> 2. Excecuted HDFS command to READ/CREATE resources of hdfs for which Ranger policies were created.
>
> Expected Behaviour :
> HDFS plugin should write audit logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
>
> Actual Behaviour :
> Expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
>
>
> **Use-case 3:**
> Ranger admin and Ranger plugins upgrade:
>
> Steps:
> 1. Installed Ranger admin without patch and started Ranger admin with audit source as DB; enabled HDFS plugin with Audit logs to all three audit destination DB, HDFS and solr.
> 2. Created HDFS service and policies; assigned policies to users with different combination of access permissions.
> 3. From console window excecuted HDFS command to READ/CREATE HDFS resources on which Ranger policies was created.
> 4. It was observed that HDFS plugin was writing audit logs to all three audit stores.
> 5. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
> 6. Stopped Ranger Admin, disabled plugins to stop communicating to Ranger Admin, Stopped HDFS component.
> 7. Copied Ranger admin install.properties and installed Ranger with patch and used same properties of previous installation, since new install.properties did not have audit to DB related properties so skipped that and provided solr url which was used in solr related config of HDFS plugin.
> 7. Executed Ranger setup script and restarted Ranger admin.
> 8. Now Ranger UI was reading audit logs from solr source and expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
> 9. Enabled HDFS plugin(patched version) with audit destination as HDFS and solr.
> 10. Excecuted HDFS command to READ/CREATE resources on which Ranger policies exist.
>
> Expected Behaviour :
> 1. HDFS plugin should write new logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
> 2. Expected logs should appear in Solr UI also.
> 3. HDFS plugin must not write any new audit logs to DB.
>
>
> Actual Behaviour :
> 1. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
> 2. Expected logs were available in solr UI.
> 3. There was no new logs in 'xa_access_audit' table of Ranger audit DB.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 45091: RANGER-900 : Remove support for DB based
auditing
Posted by Pradeep Agrawal <pr...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45091/
-----------------------------------------------------------
(Updated April 12, 2016, 10:21 a.m.)
Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
Changes
-------
Moved audit db properties from ranger-admin-site.xml to ranger-admin-default-site.xml
Bugs: RANGER-900
https://issues.apache.org/jira/browse/RANGER-900
Repository: ranger
Description
-------
**Problem Statement :**
Remove option to store audit to DB as storing audit logs in db requires lots of data management activity and frequent backup-restore process might hamper Ranger application performance. Production team might face frequent down-time issues due to db disk space reclaim activities.
**Proposed Solution :**
Proposed solution is having below mentioned approch :
1. Remove audit to DB related properties from install.properties of all components.
2. Disable shell script code to read audit to DB related properties from install.properties of all components.
3. Disable code from dba_script.py which is invoked to create audit DB schema, audit User and executes grants privileges.
4. Disable code from db_setup.py to skip 'xa_access_audit' table creation and stop executing audit to Db related sql patches.
5. Make solr as mandatory audit data store/source.
Diffs (updated)
-----
agents-audit/src/main/java/org/apache/ranger/audit/test/TestEvents.java 3e89cc4
agents-common/scripts/enable-agent.sh b9511d2
agents-common/src/test/java/org/apache/ranger/plugin/policyengine/TestPolicyEngine.java 8ee6bea
hbase-agent/conf/ranger-hbase-audit-changes.cfg e29ccd5
hbase-agent/scripts/install.properties 795ea3e
hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 9c88450
hdfs-agent/scripts/install.properties b4dda13
hive-agent/conf/ranger-hive-audit-changes.cfg 4e61c7d
hive-agent/scripts/install.properties 6b71a85
kms/scripts/install.properties d30b28c
knox-agent/conf/ranger-knox-audit-changes.cfg f722e53
knox-agent/scripts/install.properties 1febd49
plugin-kafka/conf/ranger-kafka-audit-changes.cfg 46ee29a
plugin-kafka/scripts/install.properties 79ea6db
plugin-kms/conf/ranger-kms-audit-changes.cfg 5a51455
plugin-kms/scripts/enable-kms-plugin.sh 7bf6c62
plugin-solr/conf/ranger-solr-audit-changes.cfg 2742bc1
plugin-solr/scripts/install.properties a3d9887
plugin-yarn/conf/ranger-yarn-audit-changes.cfg b650be1
plugin-yarn/scripts/install.properties 3780068
security-admin/scripts/db_setup.py 3d20fcd
security-admin/scripts/dba_script.py 0ebd90b
security-admin/scripts/install.properties 1d9d207
security-admin/scripts/setup.sh bf29ed6
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 3333827
security-admin/src/main/resources/conf.dist/ranger-admin-site.xml 6ee48f4
storm-agent/conf/ranger-storm-audit-changes.cfg b650be1
storm-agent/scripts/install.properties f2aa5c4
Diff: https://reviews.apache.org/r/45091/diff/
Testing
-------
**Steps Performaed (With patch) :**
**Use-case 1:**
Fresh Ranger Admin Installation :
Steps:
1. After Ranger installation did not find any audit to DB related properties in install.properties file so provided 'solr' as audit data store and configured solr URL in solr_url property.
2. Executed setup.sh to install Ranger
Expected Behaviour :
1. Installation script should complete successfully and after starting Ranger, Ranger UI should work; user should able to create services, policies, users and groups.
Actual Behaviour :
1. In installation log it was observed that installation process skipped creation of audit DB, audit user and execution of audit db related sql patches.
2. Ranger installation was finished successfully.
3. After starting Ranger; was able to login to Ranger and Ranger UI was working fine. Was able to create services, policies, users and groups.
**Use-case 2:**
Enabling Ranger plugin and writing audit logs To solr :
Steps:
1. Enabled HDFS plugin with solr and provided solr url so that hdfs component should write audit logs in solr.
2. Excecuted HDFS command to READ/CREATE resources of hdfs for which Ranger policies were created.
Expected Behaviour :
HDFS plugin should write audit logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
Actual Behaviour :
Expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
**Use-case 3:**
Ranger admin and Ranger plugins upgrade:
Steps:
1. Installed Ranger admin without patch and started Ranger admin with audit source as DB; enabled HDFS plugin with Audit logs to all three audit destination DB, HDFS and solr.
2. Created HDFS service and policies; assigned policies to users with different combination of access permissions.
3. From console window excecuted HDFS command to READ/CREATE HDFS resources on which Ranger policies was created.
4. It was observed that HDFS plugin was writing audit logs to all three audit stores.
5. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
6. Stopped Ranger Admin, disabled plugins to stop communicating to Ranger Admin, Stopped HDFS component.
7. Copied Ranger admin install.properties and installed Ranger with patch and used same properties of previous installation, since new install.properties did not have audit to DB related properties so skipped that and provided solr url which was used in solr related config of HDFS plugin.
7. Executed Ranger setup script and restarted Ranger admin.
8. Now Ranger UI was reading audit logs from solr source and expected logs were available in Ranger Admin UI -> Audit menu -> Access tab.
9. Enabled HDFS plugin(patched version) with audit destination as HDFS and solr.
10. Excecuted HDFS command to READ/CREATE resources on which Ranger policies exist.
Expected Behaviour :
1. HDFS plugin should write new logs to provided solr url and same logs should appear in Ranger admin UI -> Audit menu -> Access tab.
2. Expected logs should appear in Solr UI also.
3. HDFS plugin must not write any new audit logs to DB.
Actual Behaviour :
1. Expected logs were available in Ranger admin UI -> Audit menu -> Access tab.
2. Expected logs were available in solr UI.
3. There was no new logs in 'xa_access_audit' table of Ranger audit DB.
Thanks,
Pradeep Agrawal