You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Pradeep Agrawal <pr...@gmail.com> on 2021/12/10 20:08:45 UTC
Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/
-----------------------------------------------------------
Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Bugs: RANGER-3540
https://issues.apache.org/jira/browse/RANGER-3540
Repository: ranger
Description
-------
**Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
**Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
**Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
Diffs
-----
agents-audit/pom.xml 33fa256bb
agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
hbase-agent/scripts/install.properties 87a24819e
hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
hdfs-agent/scripts/install.properties 323b878cf
hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
hive-agent/scripts/install.properties 3720b66c8
kms/scripts/install.properties 6b6b66270
knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
knox-agent/scripts/install.properties 470400499
plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
plugin-atlas/scripts/install.properties 3b777bd6a
plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
plugin-elasticsearch/scripts/install.properties 4111afe3f
plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
plugin-kafka/scripts/install.properties 1e325e0ec
plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
plugin-kylin/scripts/install.properties 013433837
plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
plugin-ozone/scripts/install.properties 1891d565f
plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
plugin-presto/scripts/install.properties ce162a2bd
plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
plugin-solr/scripts/install.properties d1852e695
plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
plugin-sqoop/scripts/install.properties 81b4526a6
plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
plugin-yarn/scripts/install.properties e73ab8b14
pom.xml 6d3cafcf4
security-admin/pom.xml 3e7a64f2a
security-admin/scripts/install.properties 5a8b00c13
security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
security-admin/scripts/setup.sh c3f51a03a
security-admin/scripts/upgrade_admin.py 10fa485bd
security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java e902e65d0
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
security-admin/src/test/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestinationTest.java dde8bb568
storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
storm-agent/scripts/install.properties d219abf59
Diff: https://reviews.apache.org/r/73756/diff/1/
Testing
-------
Tested by creating IAM user in AWS and provided required configuration in the install.properties.
**Note:** AWS region name, access key and secret key should be provided in the environment.
Thanks,
Pradeep Agrawal
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Pradeep Agrawal <pr...@gmail.com>.
> On Dec. 21, 2021, 12:12 a.m., Yao Zhou wrote:
> > agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java
> > Lines 67 (patched)
> > <https://reviews.apache.org/r/73756/diff/2/?file=2256850#file2256850line67>
> >
> > do you want to keep "amazon" in the prefix? I am seeing inconsistent patterns in this CR. In the config, many places do have "amazon" prefix but here you are removing it.
amazon_cloudwatch
> On Dec. 21, 2021, 12:12 a.m., Yao Zhou wrote:
> > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> > Lines 137-166 (patched)
> > <https://reviews.apache.org/r/73756/diff/2/?file=2256889#file2256889line137>
> >
> > Please see my response in rev1. When log stream is not provided, simply skip the input so that the API searches all streams within the given log group.
Done, Please review
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223888
-----------------------------------------------------------
On Dec. 10, 2021, 8:08 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2021, 8:08 p.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java e902e65d0
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/3/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Yao Zhou <zh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223888
-----------------------------------------------------------
agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java
Lines 67 (patched)
<https://reviews.apache.org/r/73756/#comment312924>
do you want to keep "amazon" in the prefix? I am seeing inconsistent patterns in this CR. In the config, many places do have "amazon" prefix but here you are removing it.
security-admin/scripts/install.properties
Lines 116-119 (patched)
<https://reviews.apache.org/r/73756/#comment312925>
Here we are using "cloudwatch" instead of "amazon_cloudwatch". Either way is fine but let's be consistent across java code and config files.
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
Lines 137-166 (patched)
<https://reviews.apache.org/r/73756/#comment312927>
Please see my response in rev1. When log stream is not provided, simply skip the input so that the API searches all streams within the given log group.
- Yao Zhou
On Dec. 10, 2021, 8:08 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2021, 8:08 p.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java e902e65d0
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/2/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Yao Zhou <zh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223905
-----------------------------------------------------------
Ship it!
Ship It!
- Yao Zhou
On Dec. 23, 2021, 10:48 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 23, 2021, 10:48 a.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java 4d97f28fd
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/6/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Yao Zhou <zh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223906
-----------------------------------------------------------
Ship it!
Ship It!
- Yao Zhou
On Dec. 23, 2021, 10:48 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 23, 2021, 10:48 a.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java 4d97f28fd
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/6/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/
-----------------------------------------------------------
(Updated Dec. 23, 2021, 10:48 a.m.)
Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Changes
-------
updated patch
Bugs: RANGER-3540
https://issues.apache.org/jira/browse/RANGER-3540
Repository: ranger
Description
-------
**Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
**Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
**Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
Diffs (updated)
-----
agents-audit/pom.xml 5d031cca1
agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
hbase-agent/scripts/install.properties 87a24819e
hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
hdfs-agent/scripts/install.properties 323b878cf
hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
hive-agent/scripts/install.properties 3720b66c8
kms/scripts/install.properties 6b6b66270
knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
knox-agent/scripts/install.properties 470400499
plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
plugin-atlas/scripts/install.properties 3b777bd6a
plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
plugin-elasticsearch/scripts/install.properties 4111afe3f
plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
plugin-kafka/scripts/install.properties 1e325e0ec
plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
plugin-kylin/scripts/install.properties 013433837
plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
plugin-ozone/scripts/install.properties 1891d565f
plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
plugin-presto/scripts/install.properties ce162a2bd
plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
plugin-solr/scripts/install.properties d1852e695
plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
plugin-sqoop/scripts/install.properties 81b4526a6
plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
plugin-yarn/scripts/install.properties e73ab8b14
pom.xml f9c46f669
security-admin/pom.xml e9e9a537b
security-admin/scripts/install.properties 5a8b00c13
security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
security-admin/scripts/setup.sh c3f51a03a
security-admin/scripts/upgrade_admin.py 10fa485bd
security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java 4d97f28fd
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
storm-agent/scripts/install.properties d219abf59
Diff: https://reviews.apache.org/r/73756/diff/6/
Changes: https://reviews.apache.org/r/73756/diff/5-6/
Testing
-------
Tested by creating IAM user in AWS and provided required configuration in the install.properties.
**Note:** AWS region name, access key and secret key should be provided in the environment.
Thanks,
Pradeep Agrawal
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Pradeep Agrawal <pr...@gmail.com>.
> On Dec. 23, 2021, 9:15 a.m., Yao Zhou wrote:
> > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> > Lines 239-245 (original), 203-216 (patched)
> > <https://reviews.apache.org/r/73756/diff/4-5/?file=2257093#file2257093line240>
> >
> > Minor comment:
> >
> > Instead of this, just do
> >
> > ```
> > filterLogEventsRequest = new FilterLogEventsRequest().with()...
> >
> > if (StringUtils.isNotBlank(logStreamPrefix)) {
> > filterLogEventsRequest.setLogStreamNamePrefix(logStreamPrefix)
> > }
> > ```
Yes, last time i tried this. let me retry.
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223901
-----------------------------------------------------------
On Dec. 23, 2021, 6:56 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 23, 2021, 6:56 a.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java 4d97f28fd
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/5/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Yao Zhou <zh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223901
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
Lines 239-245 (original), 203-216 (patched)
<https://reviews.apache.org/r/73756/#comment312939>
Minor comment:
Instead of this, just do
```
filterLogEventsRequest = new FilterLogEventsRequest().with()...
if (StringUtils.isNotBlank(logStreamPrefix)) {
filterLogEventsRequest.setLogStreamNamePrefix(logStreamPrefix)
}
```
- Yao Zhou
On Dec. 23, 2021, 6:56 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 23, 2021, 6:56 a.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java 4d97f28fd
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/5/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Pradeep Agrawal <pr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/
-----------------------------------------------------------
(Updated Dec. 23, 2021, 6:56 a.m.)
Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
Changes
-------
Addressed review comments and tested the updated patch.
Bugs: RANGER-3540
https://issues.apache.org/jira/browse/RANGER-3540
Repository: ranger
Description
-------
**Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
**Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
**Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
Diffs (updated)
-----
agents-audit/pom.xml 5d031cca1
agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
hbase-agent/scripts/install.properties 87a24819e
hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
hdfs-agent/scripts/install.properties 323b878cf
hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
hive-agent/scripts/install.properties 3720b66c8
kms/scripts/install.properties 6b6b66270
knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
knox-agent/scripts/install.properties 470400499
plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
plugin-atlas/scripts/install.properties 3b777bd6a
plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
plugin-elasticsearch/scripts/install.properties 4111afe3f
plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
plugin-kafka/scripts/install.properties 1e325e0ec
plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
plugin-kylin/scripts/install.properties 013433837
plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
plugin-ozone/scripts/install.properties 1891d565f
plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
plugin-presto/scripts/install.properties ce162a2bd
plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
plugin-solr/scripts/install.properties d1852e695
plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
plugin-sqoop/scripts/install.properties 81b4526a6
plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
plugin-yarn/scripts/install.properties e73ab8b14
pom.xml f9c46f669
security-admin/pom.xml e9e9a537b
security-admin/scripts/install.properties 5a8b00c13
security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
security-admin/scripts/setup.sh c3f51a03a
security-admin/scripts/upgrade_admin.py 10fa485bd
security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java 4d97f28fd
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
storm-agent/scripts/install.properties d219abf59
Diff: https://reviews.apache.org/r/73756/diff/5/
Changes: https://reviews.apache.org/r/73756/diff/4-5/
Testing
-------
Tested by creating IAM user in AWS and provided required configuration in the install.properties.
**Note:** AWS region name, access key and secret key should be provided in the environment.
Thanks,
Pradeep Agrawal
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Pradeep Agrawal <pr...@gmail.com>.
> On Dec. 21, 2021, 7:09 p.m., Yao Zhou wrote:
> > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> > Lines 136-166 (patched)
> > <https://reviews.apache.org/r/73756/diff/4/?file=2257093#file2257093line136>
> >
> > Fetching all streams from a given log group could be quite slow and can easily trigger throttling given every AWS account has API limits.
> >
> > 1/ Have you verified that filterLogEvents() does not work if we don't provide a log stream? I was able to do it through AWS CLI but haven't tried SDK yet.
> > ```
> > aws logs filter-log-events --log-group-name "xxx"
> >
> > ....(all events across streams)
> > ```
> > 2/ I would suggest we change the log stream to log stream prefix, which matches the client-side config and also gives you the benefit of searching events across streams (instead of withLogStreamNames, you do withLogStreamPrefix see https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/logs/model/FilterLogEventsRequest.html#setLogStreamNamePrefix-java.lang.String-)
> >
> > ```
> > logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + "log_stream");
> > ```
>
> Pradeep Agrawal wrote:
> I tried with below given diff and its not returning any records. I added debug logs also and its not returning anything.
>
> ========
>
> diff --git a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> index 896ed05e1..cc300eacb 100644
> --- a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> +++ b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> @@ -21,6 +21,7 @@ package org.apache.ranger.amazon.cloudwatch;
>
> import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.CONFIG_PREFIX;
> import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_GROUP_NAME;
> +import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_STREAM_PREFIX;
>
> import java.text.SimpleDateFormat;
> import java.util.ArrayList;
> @@ -45,12 +46,9 @@ import org.springframework.beans.factory.annotation.Autowired;
> import org.springframework.stereotype.Component;
>
> import com.amazonaws.services.logs.AWSLogs;
> -import com.amazonaws.services.logs.model.DescribeLogStreamsRequest;
> -import com.amazonaws.services.logs.model.DescribeLogStreamsResult;
> import com.amazonaws.services.logs.model.FilterLogEventsRequest;
> import com.amazonaws.services.logs.model.FilterLogEventsResult;
> import com.amazonaws.services.logs.model.FilteredLogEvent;
> -import com.amazonaws.services.logs.model.LogStream;
>
> @Component
> public class CloudWatchUtil {
> @@ -66,7 +64,7 @@ public class CloudWatchUtil {
>
> public CloudWatchUtil() {
> logGroupName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + PROP_LOG_GROUP_NAME, "ranger_audits");
> - logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + "log_stream");
> + logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + PROP_LOG_STREAM_PREFIX, "ranger");
> String timeZone = PropertiesUtil.getProperty("ranger.cloudwatch.timezone");
> if (timeZone != null) {
> LOGGER.info("Setting timezone to " + timeZone);
> @@ -131,7 +129,7 @@ public class CloudWatchUtil {
> Date fromDate = null;
> Date toDate = null;
>
> - String nextToken = null;
> + /*String nextToken = null;
> boolean done = false;
> // load log stream names from cloudwatch if logStreamName is not provided
> List<String> logStreamNames = new ArrayList<String>();
> @@ -163,7 +161,7 @@ public class CloudWatchUtil {
> } while (!done);
> } else {
> logStreamNames.add(logStreamName);
> - }
> + }*/
>
> if (searchCriteria.getParamList() != null) {
> List<String> filterExpr = new ArrayList<String>();
> @@ -238,7 +236,7 @@ public class CloudWatchUtil {
> // Add FilterPattern which will only fetch logs required
> filterLogEventsRequest = new FilterLogEventsRequest()
> .withLogGroupName(logGroupName)
> - .withLogStreamNames(logStreamNames)
> + .withLogStreamNamePrefix(logStreamName)
> .withStartTime(fromDate.getTime())
> .withEndTime(toDate.getTime())
> .withFilterPattern(filterPattern.toString());
Can you confirm the same at your end.
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223893
-----------------------------------------------------------
On Dec. 10, 2021, 8:08 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2021, 8:08 p.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java e902e65d0
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/4/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Pradeep Agrawal <pr...@gmail.com>.
> On Dec. 21, 2021, 7:09 p.m., Yao Zhou wrote:
> > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> > Lines 136-166 (patched)
> > <https://reviews.apache.org/r/73756/diff/4/?file=2257093#file2257093line136>
> >
> > Fetching all streams from a given log group could be quite slow and can easily trigger throttling given every AWS account has API limits.
> >
> > 1/ Have you verified that filterLogEvents() does not work if we don't provide a log stream? I was able to do it through AWS CLI but haven't tried SDK yet.
> > ```
> > aws logs filter-log-events --log-group-name "xxx"
> >
> > ....(all events across streams)
> > ```
> > 2/ I would suggest we change the log stream to log stream prefix, which matches the client-side config and also gives you the benefit of searching events across streams (instead of withLogStreamNames, you do withLogStreamPrefix see https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/logs/model/FilterLogEventsRequest.html#setLogStreamNamePrefix-java.lang.String-)
> >
> > ```
> > logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + "log_stream");
> > ```
>
> Pradeep Agrawal wrote:
> I tried with below given diff and its not returning any records. I added debug logs also and its not returning anything.
>
> ========
>
> diff --git a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> index 896ed05e1..cc300eacb 100644
> --- a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> +++ b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> @@ -21,6 +21,7 @@ package org.apache.ranger.amazon.cloudwatch;
>
> import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.CONFIG_PREFIX;
> import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_GROUP_NAME;
> +import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_STREAM_PREFIX;
>
> import java.text.SimpleDateFormat;
> import java.util.ArrayList;
> @@ -45,12 +46,9 @@ import org.springframework.beans.factory.annotation.Autowired;
> import org.springframework.stereotype.Component;
>
> import com.amazonaws.services.logs.AWSLogs;
> -import com.amazonaws.services.logs.model.DescribeLogStreamsRequest;
> -import com.amazonaws.services.logs.model.DescribeLogStreamsResult;
> import com.amazonaws.services.logs.model.FilterLogEventsRequest;
> import com.amazonaws.services.logs.model.FilterLogEventsResult;
> import com.amazonaws.services.logs.model.FilteredLogEvent;
> -import com.amazonaws.services.logs.model.LogStream;
>
> @Component
> public class CloudWatchUtil {
> @@ -66,7 +64,7 @@ public class CloudWatchUtil {
>
> public CloudWatchUtil() {
> logGroupName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + PROP_LOG_GROUP_NAME, "ranger_audits");
> - logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + "log_stream");
> + logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + PROP_LOG_STREAM_PREFIX, "ranger");
> String timeZone = PropertiesUtil.getProperty("ranger.cloudwatch.timezone");
> if (timeZone != null) {
> LOGGER.info("Setting timezone to " + timeZone);
> @@ -131,7 +129,7 @@ public class CloudWatchUtil {
> Date fromDate = null;
> Date toDate = null;
>
> - String nextToken = null;
> + /*String nextToken = null;
> boolean done = false;
> // load log stream names from cloudwatch if logStreamName is not provided
> List<String> logStreamNames = new ArrayList<String>();
> @@ -163,7 +161,7 @@ public class CloudWatchUtil {
> } while (!done);
> } else {
> logStreamNames.add(logStreamName);
> - }
> + }*/
>
> if (searchCriteria.getParamList() != null) {
> List<String> filterExpr = new ArrayList<String>();
> @@ -238,7 +236,7 @@ public class CloudWatchUtil {
> // Add FilterPattern which will only fetch logs required
> filterLogEventsRequest = new FilterLogEventsRequest()
> .withLogGroupName(logGroupName)
> - .withLogStreamNames(logStreamNames)
> + .withLogStreamNamePrefix(logStreamName)
> .withStartTime(fromDate.getTime())
> .withEndTime(toDate.getTime())
> .withFilterPattern(filterPattern.toString());
>
> Pradeep Agrawal wrote:
> Can you confirm the same at your end.
>
> Yao Zhou wrote:
> I tested with below code and I was able to see events from different log streams. The code is written in Scala but it's using the latest AWS JAVA SDK (1.12.122). I suspect that you are seeing empty result because your filters (e.g. startTime/endTime/filterPattern) are excluding all the events.
>
> ```
> val logs: AWSLogs = AWSLogsClientBuilder
> .standard()
> .withCredentials(credProvider)
> .withRegion("us-east-1")
> .build()
>
> val req = new FilterLogEventsRequest()
> req.setLogGroupName("xxx")
> val events: List[FilteredLogEvent] = logs.filterLogEvents(req).getEvents.asScala.toList
> events.foreach(println)
> ```
Done. It seems there was jar conflict in my last attempt. Probably old jar was being referred somehow(though i deleted them) and was causing the issue.
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223893
-----------------------------------------------------------
On Dec. 23, 2021, 6:56 a.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 23, 2021, 6:56 a.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java 4d97f28fd
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/5/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Yao Zhou <zh...@gmail.com>.
> On Dec. 21, 2021, 7:09 p.m., Yao Zhou wrote:
> > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> > Lines 136-166 (patched)
> > <https://reviews.apache.org/r/73756/diff/4/?file=2257093#file2257093line136>
> >
> > Fetching all streams from a given log group could be quite slow and can easily trigger throttling given every AWS account has API limits.
> >
> > 1/ Have you verified that filterLogEvents() does not work if we don't provide a log stream? I was able to do it through AWS CLI but haven't tried SDK yet.
> > ```
> > aws logs filter-log-events --log-group-name "xxx"
> >
> > ....(all events across streams)
> > ```
> > 2/ I would suggest we change the log stream to log stream prefix, which matches the client-side config and also gives you the benefit of searching events across streams (instead of withLogStreamNames, you do withLogStreamPrefix see https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/logs/model/FilterLogEventsRequest.html#setLogStreamNamePrefix-java.lang.String-)
> >
> > ```
> > logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + "log_stream");
> > ```
>
> Pradeep Agrawal wrote:
> I tried with below given diff and its not returning any records. I added debug logs also and its not returning anything.
>
> ========
>
> diff --git a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> index 896ed05e1..cc300eacb 100644
> --- a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> +++ b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> @@ -21,6 +21,7 @@ package org.apache.ranger.amazon.cloudwatch;
>
> import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.CONFIG_PREFIX;
> import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_GROUP_NAME;
> +import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_STREAM_PREFIX;
>
> import java.text.SimpleDateFormat;
> import java.util.ArrayList;
> @@ -45,12 +46,9 @@ import org.springframework.beans.factory.annotation.Autowired;
> import org.springframework.stereotype.Component;
>
> import com.amazonaws.services.logs.AWSLogs;
> -import com.amazonaws.services.logs.model.DescribeLogStreamsRequest;
> -import com.amazonaws.services.logs.model.DescribeLogStreamsResult;
> import com.amazonaws.services.logs.model.FilterLogEventsRequest;
> import com.amazonaws.services.logs.model.FilterLogEventsResult;
> import com.amazonaws.services.logs.model.FilteredLogEvent;
> -import com.amazonaws.services.logs.model.LogStream;
>
> @Component
> public class CloudWatchUtil {
> @@ -66,7 +64,7 @@ public class CloudWatchUtil {
>
> public CloudWatchUtil() {
> logGroupName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + PROP_LOG_GROUP_NAME, "ranger_audits");
> - logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + "log_stream");
> + logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + PROP_LOG_STREAM_PREFIX, "ranger");
> String timeZone = PropertiesUtil.getProperty("ranger.cloudwatch.timezone");
> if (timeZone != null) {
> LOGGER.info("Setting timezone to " + timeZone);
> @@ -131,7 +129,7 @@ public class CloudWatchUtil {
> Date fromDate = null;
> Date toDate = null;
>
> - String nextToken = null;
> + /*String nextToken = null;
> boolean done = false;
> // load log stream names from cloudwatch if logStreamName is not provided
> List<String> logStreamNames = new ArrayList<String>();
> @@ -163,7 +161,7 @@ public class CloudWatchUtil {
> } while (!done);
> } else {
> logStreamNames.add(logStreamName);
> - }
> + }*/
>
> if (searchCriteria.getParamList() != null) {
> List<String> filterExpr = new ArrayList<String>();
> @@ -238,7 +236,7 @@ public class CloudWatchUtil {
> // Add FilterPattern which will only fetch logs required
> filterLogEventsRequest = new FilterLogEventsRequest()
> .withLogGroupName(logGroupName)
> - .withLogStreamNames(logStreamNames)
> + .withLogStreamNamePrefix(logStreamName)
> .withStartTime(fromDate.getTime())
> .withEndTime(toDate.getTime())
> .withFilterPattern(filterPattern.toString());
>
> Pradeep Agrawal wrote:
> Can you confirm the same at your end.
I tested with below code and I was able to see events from different log streams. The code is written in Scala but it's using the latest AWS JAVA SDK (1.12.122). I suspect that you are seeing empty result because your filters (e.g. startTime/endTime/filterPattern) are excluding all the events.
```
val logs: AWSLogs = AWSLogsClientBuilder
.standard()
.withCredentials(credProvider)
.withRegion("us-east-1")
.build()
val req = new FilterLogEventsRequest()
req.setLogGroupName("xxx")
val events: List[FilteredLogEvent] = logs.filterLogEvents(req).getEvents.asScala.toList
events.foreach(println)
```
- Yao
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223893
-----------------------------------------------------------
On Dec. 10, 2021, 8:08 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2021, 8:08 p.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java e902e65d0
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/4/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Pradeep Agrawal <pr...@gmail.com>.
> On Dec. 21, 2021, 7:09 p.m., Yao Zhou wrote:
> > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> > Lines 136-166 (patched)
> > <https://reviews.apache.org/r/73756/diff/4/?file=2257093#file2257093line136>
> >
> > Fetching all streams from a given log group could be quite slow and can easily trigger throttling given every AWS account has API limits.
> >
> > 1/ Have you verified that filterLogEvents() does not work if we don't provide a log stream? I was able to do it through AWS CLI but haven't tried SDK yet.
> > ```
> > aws logs filter-log-events --log-group-name "xxx"
> >
> > ....(all events across streams)
> > ```
> > 2/ I would suggest we change the log stream to log stream prefix, which matches the client-side config and also gives you the benefit of searching events across streams (instead of withLogStreamNames, you do withLogStreamPrefix see https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/logs/model/FilterLogEventsRequest.html#setLogStreamNamePrefix-java.lang.String-)
> >
> > ```
> > logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + "log_stream");
> > ```
I tried with below given diff and its not returning any records. I added debug logs also and its not returning anything.
========
diff --git a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
index 896ed05e1..cc300eacb 100644
--- a/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
+++ b/security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
@@ -21,6 +21,7 @@ package org.apache.ranger.amazon.cloudwatch;
import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.CONFIG_PREFIX;
import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_GROUP_NAME;
+import static org.apache.ranger.audit.destination.AmazonCloudWatchAuditDestination.PROP_LOG_STREAM_PREFIX;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
@@ -45,12 +46,9 @@ import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import com.amazonaws.services.logs.AWSLogs;
-import com.amazonaws.services.logs.model.DescribeLogStreamsRequest;
-import com.amazonaws.services.logs.model.DescribeLogStreamsResult;
import com.amazonaws.services.logs.model.FilterLogEventsRequest;
import com.amazonaws.services.logs.model.FilterLogEventsResult;
import com.amazonaws.services.logs.model.FilteredLogEvent;
-import com.amazonaws.services.logs.model.LogStream;
@Component
public class CloudWatchUtil {
@@ -66,7 +64,7 @@ public class CloudWatchUtil {
public CloudWatchUtil() {
logGroupName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + PROP_LOG_GROUP_NAME, "ranger_audits");
- logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + "log_stream");
+ logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + PROP_LOG_STREAM_PREFIX, "ranger");
String timeZone = PropertiesUtil.getProperty("ranger.cloudwatch.timezone");
if (timeZone != null) {
LOGGER.info("Setting timezone to " + timeZone);
@@ -131,7 +129,7 @@ public class CloudWatchUtil {
Date fromDate = null;
Date toDate = null;
- String nextToken = null;
+ /*String nextToken = null;
boolean done = false;
// load log stream names from cloudwatch if logStreamName is not provided
List<String> logStreamNames = new ArrayList<String>();
@@ -163,7 +161,7 @@ public class CloudWatchUtil {
} while (!done);
} else {
logStreamNames.add(logStreamName);
- }
+ }*/
if (searchCriteria.getParamList() != null) {
List<String> filterExpr = new ArrayList<String>();
@@ -238,7 +236,7 @@ public class CloudWatchUtil {
// Add FilterPattern which will only fetch logs required
filterLogEventsRequest = new FilterLogEventsRequest()
.withLogGroupName(logGroupName)
- .withLogStreamNames(logStreamNames)
+ .withLogStreamNamePrefix(logStreamName)
.withStartTime(fromDate.getTime())
.withEndTime(toDate.getTime())
.withFilterPattern(filterPattern.toString());
- Pradeep
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223893
-----------------------------------------------------------
On Dec. 10, 2021, 8:08 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2021, 8:08 p.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java e902e65d0
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/4/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Yao Zhou <zh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223893
-----------------------------------------------------------
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
Lines 136-166 (patched)
<https://reviews.apache.org/r/73756/#comment312933>
Fetching all streams from a given log group could be quite slow and can easily trigger throttling given every AWS account has API limits.
1/ Have you verified that filterLogEvents() does not work if we don't provide a log stream? I was able to do it through AWS CLI but haven't tried SDK yet.
```
aws logs filter-log-events --log-group-name "xxx"
....(all events across streams)
```
2/ I would suggest we change the log stream to log stream prefix, which matches the client-side config and also gives you the benefit of searching events across streams (instead of withLogStreamNames, you do withLogStreamPrefix see https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/logs/model/FilterLogEventsRequest.html#setLogStreamNamePrefix-java.lang.String-)
```
logStreamName = PropertiesUtil.getProperty(CONFIG_PREFIX + "." + "log_stream");
```
- Yao Zhou
On Dec. 10, 2021, 8:08 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2021, 8:08 p.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java e902e65d0
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/4/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Yao Zhou <zh...@gmail.com>.
> On Dec. 15, 2021, 8:46 p.m., Yao Zhou wrote:
> > agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java
> > Line 159 (original), 176 (patched)
> > <https://reviews.apache.org/r/73756/diff/1/?file=2256539#file2256539line177>
> >
> > DescribeLogStreams API supports pagination hence you may not get the stream by just calling it once.
> >
> > I saw you changed the log stream name to get rid of the UUID. There were two reasons behind it.
> >
> > 1/ We have a use case that one engine/service would generate audit events not only on master nodes but also on core nodes hence we prefer an unique stream for each node.
> >
> > 2/ By appending the UUID, we no longer need to pre-check if the log stream exists when ranger plugin gets restarted on the same machine.
>
> Pradeep Agrawal wrote:
> I am reverting my changes here.
>
>
> Also is it true that at every restart a new stream will be created ?
Yes. The init() method gets invoked whenever the plugin gets re-loaded (i.e. service restart) hence always creates a new log stream.
> On Dec. 15, 2021, 8:46 p.m., Yao Zhou wrote:
> > security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
> > Lines 137-141 (patched)
> > <https://reviews.apache.org/r/73756/diff/1/?file=2256578#file2256578line137>
> >
> > same comment: The DescribeLogStreams API supports pagination which needs to be handled.
>
> Pradeep Agrawal wrote:
> Thanks for review.
> How can we display audit logs in the ranger ui by consolidating them from so many streams, also it seems the cloudwatch logs APIs does not provide sorting of logs in descending order of timestamp.
Q: How can we display audit logs in the ranger ui by consolidating them from so many streams?
A: The API you are using should support searching across all the streams under a log group by not putting the log stream input (only log group).
Q: it seems the cloudwatch logs APIs does not provide sorting of logs in descending order of timestamp
A: The returned log events are sorted by event timestamp (needs to verify descending vs ascending).
Ref: https://docs.aws.amazon.com/cli/latest/reference/logs/filter-log-events.html
- Yao
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223859
-----------------------------------------------------------
On Dec. 10, 2021, 8:08 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2021, 8:08 p.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 5d031cca1
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml f9c46f669
> security-admin/pom.xml e9e9a537b
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java e902e65d0
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/4/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>
Re: Review Request 73756: RANGER-3540: Add support to read audit logs from Amazon CloudWatch
Posted by Yao Zhou <zh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73756/#review223859
-----------------------------------------------------------
agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java
Line 159 (original), 176 (patched)
<https://reviews.apache.org/r/73756/#comment312896>
DescribeLogStreams API supports pagination hence you may not get the stream by just calling it once.
I saw you changed the log stream name to get rid of the UUID. There were two reasons behind it.
1/ We have a use case that one engine/service would generate audit events not only on master nodes but also on core nodes hence we prefer an unique stream for each node.
2/ By appending the UUID, we no longer need to pre-check if the log stream exists when ranger plugin gets restarted on the same machine.
security-admin/scripts/install.properties
Line 86 (original), 86 (patched)
<https://reviews.apache.org/r/73756/#comment312897>
nit: cloudwatch -> cloudwatch logs
security-admin/scripts/install.properties
Lines 117-119 (patched)
<https://reviews.apache.org/r/73756/#comment312898>
Prefer to not have any default settings given there's no default setup at plugin side.
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java
Lines 58 (patched)
<https://reviews.apache.org/r/73756/#comment312901>
Why do we want to create log stream here? Isn't that Ranger Admin Server only needs to query audit events?
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
Lines 99 (patched)
<https://reviews.apache.org/r/73756/#comment312899>
nit: add a comment on this.
security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java
Lines 137-141 (patched)
<https://reviews.apache.org/r/73756/#comment312900>
same comment: The DescribeLogStreams API supports pagination which needs to be handled.
- Yao Zhou
On Dec. 10, 2021, 8:08 p.m., Pradeep Agrawal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73756/
> -----------------------------------------------------------
>
> (Updated Dec. 10, 2021, 8:08 p.m.)
>
>
> Review request for ranger, Dineshkumar Yadav, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-3540
> https://issues.apache.org/jira/browse/RANGER-3540
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement: ** This is related to RANGER-2967 which includes changes only at the plugin end. Access audit logs should be accessible and appear at Ranger admin UI end as well.
>
> **Proposed Solution: ** Proposed patch make use of AWS API's to read access audit logs from cloudwatch loggroup.
>
> **Known issue:** Cloudwatch APIs does not provide sorting of recording in descending order of timestamp, hence read operation will be slow. Hence its recommended to use the filter to minimise the resultset which shall reduce the response time and access audit page will load faster.
> Due to this issue as of now maximum 10k records will be loaded at a time to handle out of memory issue.
>
>
> Diffs
> -----
>
> agents-audit/pom.xml 33fa256bb
> agents-audit/src/main/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestination.java b236a2653
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java f58b813f8
> hbase-agent/conf/ranger-hbase-audit-changes.cfg a6c7ffd41
> hbase-agent/scripts/install.properties 87a24819e
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 92d2a4b08
> hdfs-agent/scripts/install.properties 323b878cf
> hive-agent/conf/ranger-hive-audit-changes.cfg 52c715ef5
> hive-agent/scripts/install.properties 3720b66c8
> kms/scripts/install.properties 6b6b66270
> knox-agent/conf/ranger-knox-audit-changes.cfg 52c715ef5
> knox-agent/scripts/install.properties 470400499
> plugin-atlas/conf/ranger-atlas-audit-changes.cfg 2d8251b5f
> plugin-atlas/scripts/install.properties 3b777bd6a
> plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg 52c715ef5
> plugin-elasticsearch/scripts/install.properties 4111afe3f
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg bc5a0890d
> plugin-kafka/scripts/install.properties 1e325e0ec
> plugin-kms/conf/ranger-kms-audit-changes.cfg e5e9ae489
> plugin-kylin/conf/ranger-kylin-audit-changes.cfg 52c715ef5
> plugin-kylin/scripts/install.properties 013433837
> plugin-ozone/conf/ranger-ozone-audit-changes.cfg 0eace6d29
> plugin-ozone/scripts/install.properties 1891d565f
> plugin-presto/conf/ranger-presto-audit-changes.cfg bc5a0890d
> plugin-presto/scripts/install.properties ce162a2bd
> plugin-solr/conf/ranger-solr-audit-changes.cfg ffa0a7696
> plugin-solr/scripts/install.properties d1852e695
> plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg 52c715ef5
> plugin-sqoop/scripts/install.properties 81b4526a6
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg 52c715ef5
> plugin-yarn/scripts/install.properties e73ab8b14
> pom.xml 6d3cafcf4
> security-admin/pom.xml 3e7a64f2a
> security-admin/scripts/install.properties 5a8b00c13
> security-admin/scripts/ranger-admin-site-template.xml 72ff66eaf
> security-admin/scripts/setup.sh c3f51a03a
> security-admin/scripts/upgrade_admin.py 10fa485bd
> security-admin/src/main/java/org/apache/ranger/AccessAuditsService.java e902e65d0
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchAccessAuditsService.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchMgr.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/amazon/cloudwatch/CloudWatchUtil.java PRE-CREATION
> security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java d3ce25158
> security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java 75ebae6f5
> security-admin/src/main/java/org/apache/ranger/biz/XAuditMgr.java 4e5410e8b
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java 0b2e7df7f
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java 9bee640a5
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java 0aea46d1b
> security-admin/src/main/java/org/apache/ranger/solr/SolrUtil.java 239698f3f
> security-admin/src/main/resources/conf.dist/ranger-admin-site.xml d32a324ec
> security-admin/src/test/java/org/apache/ranger/audit/destination/AmazonCloudWatchAuditDestinationTest.java dde8bb568
> storm-agent/conf/ranger-storm-audit-changes.cfg 52c715ef5
> storm-agent/scripts/install.properties d219abf59
>
>
> Diff: https://reviews.apache.org/r/73756/diff/1/
>
>
> Testing
> -------
>
> Tested by creating IAM user in AWS and provided required configuration in the install.properties.
>
> **Note:** AWS region name, access key and secret key should be provided in the environment.
>
>
> Thanks,
>
> Pradeep Agrawal
>
>