You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Pradeep Agrawal <pr...@freestoneinfotech.com> on 2016/05/04 05:51:43 UTC
Review Request 46932: RANGER-965 : Validate Audit DB to Solr Migration
script from any previous version to 0.6 upgrade
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46932/
-----------------------------------------------------------
Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
Bugs: RANGER-965
https://issues.apache.org/jira/browse/RANGER-965
Repository: ranger
Description
-------
**Problem Statement :**
Currently migration script can migrate Audit DB log to Solr only if 'xa_access_audit' table schema is updated according to 0.6 version. Ranger should support migration of Access audit logs from previous version also.
** Proposed Solution :**
Before starting migration process java code will fetch columns of 'xa_access_audit' table using native query; Received columns name and its count can be used to decide on which version of Ranger 'xa_access_audit' table was created; Separate sql statement for underligned version of 'xa_access_audit' can be executed to fetch audit logs from 'xa_access_audit' table.
Diffs
-----
security-admin/db/mysql/patches/022-split-service-table.sql 962e715
security-admin/db/oracle/patches/022-split-service-table.sql 7059be2
security-admin/src/main/java/org/apache/ranger/common/AppConstants.java e9c7ac0
security-admin/src/main/java/org/apache/ranger/db/XXAccessAuditDao.java 432dfa9
security-admin/src/main/java/org/apache/ranger/patch/cliutil/DbToSolrMigrationUtil.java 9ce1a12
security-admin/src/main/resources/META-INF/jpa_named_queries.xml f209bc4
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 0d6679c
security-admin/src/main/webapp/META-INF/applicationContext.xml 17f35a2
tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java 6e2902e
Diff: https://reviews.apache.org/r/46932/diff/
Testing
-------
**Testing Done (with patch) :**
Tested following use cases :
1. Ranger 0.4 to 0.6 (MySQL DB Flavor)
2. Ranger 0.4 to 0.6 (Oracle DB Flavor)
3. Ranger 0.5 to 0.6 (MySQL DB Flavor)
4. Ranger 0.5 to 0.6 (Oracle DB Flavor)
5. Ranger 0.5 to 0.6 (Postgres DB Flavor)
6. Ranger 0.5 to 0.6 (SQLAnywhere DB Flavor)
7. Ranger 0.5 to 0.6 (MSSQL Server DB Flavor)
Thanks,
Pradeep Agrawal
Re: Review Request 46932: RANGER-965 : Validate Audit DB to Solr
Migration script from any previous version to 0.6 upgrade
Posted by Pradeep Agrawal <pr...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46932/#review131631
-----------------------------------------------------------
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml
<https://reviews.apache.org/r/46932/#comment195646>
I am removing this entry from here as its a duplicate entry, line 301 of the same file has same property.
- Pradeep Agrawal
On May 4, 2016, 3:51 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46932/
> -----------------------------------------------------------
>
> (Updated May 4, 2016, 3:51 a.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-965
> https://issues.apache.org/jira/browse/RANGER-965
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement :**
> Currently migration script can migrate Audit DB log to Solr only if 'xa_access_audit' table schema is updated according to 0.6 version. Ranger should support migration of Access audit logs from previous version also.
>
> ** Proposed Solution :**
> Before starting migration process java code will fetch columns of 'xa_access_audit' table using native query; Received columns name and its count can be used to decide on which version of Ranger 'xa_access_audit' table was created; Separate sql statement for underligned version of 'xa_access_audit' can be executed to fetch audit logs from 'xa_access_audit' table.
>
>
> Diffs
> -----
>
> security-admin/db/mysql/patches/022-split-service-table.sql 962e715
> security-admin/db/oracle/patches/022-split-service-table.sql 7059be2
> security-admin/src/main/java/org/apache/ranger/common/AppConstants.java e9c7ac0
> security-admin/src/main/java/org/apache/ranger/db/XXAccessAuditDao.java 432dfa9
> security-admin/src/main/java/org/apache/ranger/patch/cliutil/DbToSolrMigrationUtil.java 9ce1a12
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml f209bc4
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 0d6679c
> security-admin/src/main/webapp/META-INF/applicationContext.xml 17f35a2
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java 6e2902e
>
> Diff: https://reviews.apache.org/r/46932/diff/
>
>
> Testing
> -------
>
> **Testing Done (with patch) :**
> Tested following use cases :
> 1. Ranger 0.4 to 0.6 (MySQL DB Flavor)
> 2. Ranger 0.4 to 0.6 (Oracle DB Flavor)
> 3. Ranger 0.5 to 0.6 (MySQL DB Flavor)
> 4. Ranger 0.5 to 0.6 (Oracle DB Flavor)
> 5. Ranger 0.5 to 0.6 (Postgres DB Flavor)
> 6. Ranger 0.5 to 0.6 (SQLAnywhere DB Flavor)
> 7. Ranger 0.5 to 0.6 (MSSQL Server DB Flavor)
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 46932: RANGER-965 : Validate Audit DB to Solr
Migration script from any previous version to 0.6 upgrade
Posted by Madhan Neethiraj <ma...@apache.org>.
> On May 9, 2016, 6:39 a.m., Madhan Neethiraj wrote:
> > security-admin/src/main/resources/META-INF/jpa_named_queries.xml, line 1015
> > <https://reviews.apache.org/r/46932/diff/1/?file=1369771#file1369771line1015>
> >
> > Instead of queries listing version specific columns to retrieve, did you consider using version specific XXAccessAudit object - XXAccessAuditV4, XXAccessAuditV5 and XXAccessAuditV6 (all of them mapping to the same table)? This approach would help not having to deal with Object[] for each row.
>
> Pradeep Agrawal wrote:
> Yes, But in that case we have to create at least two more entity classes(for V4 and V5) ? I was not sure whether that approach will be right or not so i implemented this way.
> If additional entity classes approach is okay then i can try that implementation.
I think use of version specific entities classes would make it easier to maintain/understand. Can you please try/evaluate?
- Madhan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46932/#review132215
-----------------------------------------------------------
On May 4, 2016, 3:51 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46932/
> -----------------------------------------------------------
>
> (Updated May 4, 2016, 3:51 a.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-965
> https://issues.apache.org/jira/browse/RANGER-965
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement :**
> Currently migration script can migrate Audit DB log to Solr only if 'xa_access_audit' table schema is updated according to 0.6 version. Ranger should support migration of Access audit logs from previous version also.
>
> ** Proposed Solution :**
> Before starting migration process java code will fetch columns of 'xa_access_audit' table using native query; Received columns name and its count can be used to decide on which version of Ranger 'xa_access_audit' table was created; Separate sql statement for underligned version of 'xa_access_audit' can be executed to fetch audit logs from 'xa_access_audit' table.
>
>
> Diffs
> -----
>
> security-admin/db/mysql/patches/022-split-service-table.sql 962e715
> security-admin/db/oracle/patches/022-split-service-table.sql 7059be2
> security-admin/src/main/java/org/apache/ranger/common/AppConstants.java e9c7ac0
> security-admin/src/main/java/org/apache/ranger/db/XXAccessAuditDao.java 432dfa9
> security-admin/src/main/java/org/apache/ranger/patch/cliutil/DbToSolrMigrationUtil.java 9ce1a12
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml f209bc4
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 0d6679c
> security-admin/src/main/webapp/META-INF/applicationContext.xml 17f35a2
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java 6e2902e
>
> Diff: https://reviews.apache.org/r/46932/diff/
>
>
> Testing
> -------
>
> **Testing Done (with patch) :**
> Tested following use cases :
> 1. Ranger 0.4 to 0.6 (MySQL DB Flavor)
> 2. Ranger 0.4 to 0.6 (Oracle DB Flavor)
> 3. Ranger 0.5 to 0.6 (MySQL DB Flavor)
> 4. Ranger 0.5 to 0.6 (Oracle DB Flavor)
> 5. Ranger 0.5 to 0.6 (Postgres DB Flavor)
> 6. Ranger 0.5 to 0.6 (SQLAnywhere DB Flavor)
> 7. Ranger 0.5 to 0.6 (MSSQL Server DB Flavor)
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 46932: RANGER-965 : Validate Audit DB to Solr
Migration script from any previous version to 0.6 upgrade
Posted by Pradeep Agrawal <pr...@freestoneinfotech.com>.
> On May 9, 2016, 6:39 a.m., Madhan Neethiraj wrote:
> > security-admin/src/main/resources/META-INF/jpa_named_queries.xml, line 1015
> > <https://reviews.apache.org/r/46932/diff/1/?file=1369771#file1369771line1015>
> >
> > Instead of queries listing version specific columns to retrieve, did you consider using version specific XXAccessAudit object - XXAccessAuditV4, XXAccessAuditV5 and XXAccessAuditV6 (all of them mapping to the same table)? This approach would help not having to deal with Object[] for each row.
Yes, But in that case we have to create at least two more entity classes(for V4 and V5) ? I was not sure whether that approach will be right or not so i implemented this way.
If additional entity classes approach is okay then i can try that implementation.
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46932/#review132215
-----------------------------------------------------------
On May 4, 2016, 3:51 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46932/
> -----------------------------------------------------------
>
> (Updated May 4, 2016, 3:51 a.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-965
> https://issues.apache.org/jira/browse/RANGER-965
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement :**
> Currently migration script can migrate Audit DB log to Solr only if 'xa_access_audit' table schema is updated according to 0.6 version. Ranger should support migration of Access audit logs from previous version also.
>
> ** Proposed Solution :**
> Before starting migration process java code will fetch columns of 'xa_access_audit' table using native query; Received columns name and its count can be used to decide on which version of Ranger 'xa_access_audit' table was created; Separate sql statement for underligned version of 'xa_access_audit' can be executed to fetch audit logs from 'xa_access_audit' table.
>
>
> Diffs
> -----
>
> security-admin/db/mysql/patches/022-split-service-table.sql 962e715
> security-admin/db/oracle/patches/022-split-service-table.sql 7059be2
> security-admin/src/main/java/org/apache/ranger/common/AppConstants.java e9c7ac0
> security-admin/src/main/java/org/apache/ranger/db/XXAccessAuditDao.java 432dfa9
> security-admin/src/main/java/org/apache/ranger/patch/cliutil/DbToSolrMigrationUtil.java 9ce1a12
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml f209bc4
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 0d6679c
> security-admin/src/main/webapp/META-INF/applicationContext.xml 17f35a2
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java 6e2902e
>
> Diff: https://reviews.apache.org/r/46932/diff/
>
>
> Testing
> -------
>
> **Testing Done (with patch) :**
> Tested following use cases :
> 1. Ranger 0.4 to 0.6 (MySQL DB Flavor)
> 2. Ranger 0.4 to 0.6 (Oracle DB Flavor)
> 3. Ranger 0.5 to 0.6 (MySQL DB Flavor)
> 4. Ranger 0.5 to 0.6 (Oracle DB Flavor)
> 5. Ranger 0.5 to 0.6 (Postgres DB Flavor)
> 6. Ranger 0.5 to 0.6 (SQLAnywhere DB Flavor)
> 7. Ranger 0.5 to 0.6 (MSSQL Server DB Flavor)
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 46932: RANGER-965 : Validate Audit DB to Solr
Migration script from any previous version to 0.6 upgrade
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46932/#review132215
-----------------------------------------------------------
The changes look good. Please review the comment given in jpa_named_queries.xml.
security-admin/src/main/resources/META-INF/jpa_named_queries.xml (line 1015)
<https://reviews.apache.org/r/46932/#comment196392>
Instead of queries listing version specific columns to retrieve, did you consider using version specific XXAccessAudit object - XXAccessAuditV4, XXAccessAuditV5 and XXAccessAuditV6 (all of them mapping to the same table)? This approach would help not having to deal with Object[] for each row.
- Madhan Neethiraj
On May 4, 2016, 3:51 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46932/
> -----------------------------------------------------------
>
> (Updated May 4, 2016, 3:51 a.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-965
> https://issues.apache.org/jira/browse/RANGER-965
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement :**
> Currently migration script can migrate Audit DB log to Solr only if 'xa_access_audit' table schema is updated according to 0.6 version. Ranger should support migration of Access audit logs from previous version also.
>
> ** Proposed Solution :**
> Before starting migration process java code will fetch columns of 'xa_access_audit' table using native query; Received columns name and its count can be used to decide on which version of Ranger 'xa_access_audit' table was created; Separate sql statement for underligned version of 'xa_access_audit' can be executed to fetch audit logs from 'xa_access_audit' table.
>
>
> Diffs
> -----
>
> security-admin/db/mysql/patches/022-split-service-table.sql 962e715
> security-admin/db/oracle/patches/022-split-service-table.sql 7059be2
> security-admin/src/main/java/org/apache/ranger/common/AppConstants.java e9c7ac0
> security-admin/src/main/java/org/apache/ranger/db/XXAccessAuditDao.java 432dfa9
> security-admin/src/main/java/org/apache/ranger/patch/cliutil/DbToSolrMigrationUtil.java 9ce1a12
> security-admin/src/main/resources/META-INF/jpa_named_queries.xml f209bc4
> security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 0d6679c
> security-admin/src/main/webapp/META-INF/applicationContext.xml 17f35a2
> tagsync/src/main/java/org/apache/ranger/tagsync/process/TagSyncConfig.java 6e2902e
>
> Diff: https://reviews.apache.org/r/46932/diff/
>
>
> Testing
> -------
>
> **Testing Done (with patch) :**
> Tested following use cases :
> 1. Ranger 0.4 to 0.6 (MySQL DB Flavor)
> 2. Ranger 0.4 to 0.6 (Oracle DB Flavor)
> 3. Ranger 0.5 to 0.6 (MySQL DB Flavor)
> 4. Ranger 0.5 to 0.6 (Oracle DB Flavor)
> 5. Ranger 0.5 to 0.6 (Postgres DB Flavor)
> 6. Ranger 0.5 to 0.6 (SQLAnywhere DB Flavor)
> 7. Ranger 0.5 to 0.6 (MSSQL Server DB Flavor)
>
>
> Thanks,
>
> Pradeep Agrawal
>
>