You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Ramesh Mani <rm...@hortonworks.com> on 2017/02/06 18:36:15 UTC
Review Request 56335: Ranger Audit framework enhancement to provide an
option to allow audit records to be spooled to local disk first before
sending it to destinations
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56335/
-----------------------------------------------------------
Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan Periasamy.
Repository: ranger
Description
-------
Ranger Audit framework enhancement to provide an option to allow audit records to be spooled to local disk first before sending it to destinations
Diffs
-----
agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 7c37cfa
agents-audit/src/main/java/org/apache/ranger/audit/provider/AuditFileCacheProvider.java PRE-CREATION
agents-audit/src/main/java/org/apache/ranger/audit/provider/AuditProviderFactory.java e3c3508
agents-audit/src/main/java/org/apache/ranger/audit/queue/AuditFileCacheProviderSpool.java PRE-CREATION
Diff: https://reviews.apache.org/r/56335/diff/
Testing
-------
Test all the plugins in Local VM
To enable the file cache provider for each of the components please do the following
For HDFS Plugin
===============
mkdir -p /var/log/hadoop/hdfs/audit/spool
cd /var/log/hadoop/hdfs/audit/
chown hdfs:hadoop spool
Add the following properties to the "custom ranger-hive-audit\u201d in the Ambari for hdfs.
xasecure.audit.provider.filecache.is.enabled=true
xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
xasecure.audit.provider.filecache.filespool.dir=/var/log/hadoop/hdfs/audit/spool
NOTE:
xasecure.audit.provider.filecache.is.enabled = true
This property will enable file cache provider which will store the audit locally first before sending it to destinations to avoid lose of data
xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
This property will close each of local file every 300 sec ( 5 min ) and send it destinations. For testing we maded to 30 sec.
xasecure.audit.provider.filecache.filespool.dir=/var/log/hadoop/hdfs/audit/spool
This property is the directory where the local audit cache is present.
For Hive Plugin
=============
mkdir -p /var/log/hive/audit/spool
cd /var/log/hive/audit/
chown hdfs:hadoop spool
Add the following properties to the "custom ranger-hive-audit\u201d in the Ambari for hdfs.
xasecure.audit.provider.filecache.is.enabled=true
xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
xasecure.audit.provider.filecache.filespool.dir=/var/log/hive/audit/spool
Please do the same steps mentioned for all the components which need this audit file cache provider.
---------------
Issues:
- Audit to HDFS destination gets 0 bytes file or missing records in the file from HDFS plugin when HDFS get restarted and
audit from hdfs plugin is logged into destination.
- Audit to HDFS destination gets partial records from HIVE/HBASE/KNOX/STORM plugin when HDFS is restarted and there are active
spooling into hdfs is happening.
Scenarios to test
1) Audit to HDFS / Solr destination with FileCache enabled- HDFS/HIVESERVER2/HBASE/KNOX/STORM/KAFKA.
- Mentioned issue should not happen.
- Audit will be getting pushed every 5 minutes ( we are setting it to 300 sec in the parameter)
2) Audit to HDFS / Solr destination with FileCache enabled with one of the destination is down and brought back up later.
- Audit from the local cache should be present in destination when the destination is up
- In case of HDFS as destination audit might show up during next rollover of hdfs file or if the corresponding component
is restarted ( say if it is hiveserver2 plugin, when Hiveserver2 is restarted audit into HDFS appears as this will close the
existing opened hdfsfile)
- Mentioned issue should not be present
-
-
3) Same has to be done for each for the plugins ( HBASE, STORM, KAFKA, KMS)
Thanks,
Ramesh Mani
Re: Review Request 56335: RANGER-1310: Ranger Audit framework
enhancement to
provide an option to allow audit records to be spooled to local disk first
before sending it to destinations
Posted by Endre Zoltan Kovacs via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56335/#review185142
-----------------------------------------------------------
agents-audit/src/main/java/org/apache/ranger/audit/provider/AuditFileCacheProvider.java
Lines 74 (patched)
<https://reviews.apache.org/r/56335/#comment261432>
Shouldn't this be:
ret &= log(event)
so that its not only the last log's result we guard but the whole collections. true would be returned only in case all the events's logging is succesful.
- Endre Zoltan Kovacs
On Feb. 10, 2017, 10:50 p.m., Ramesh Mani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56335/
> -----------------------------------------------------------
>
> (Updated Feb. 10, 2017, 10:50 p.m.)
>
>
> Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-1310
> https://issues.apache.org/jira/browse/RANGER-1310
>
>
> Repository: ranger
>
>
> Description
> -------
>
> RANGER-1310: Ranger Audit framework enhancement to provide an option to allow audit records to be spooled to local disk first before sending it to destinations
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 7c37cfa
> agents-audit/src/main/java/org/apache/ranger/audit/provider/AuditFileCacheProvider.java PRE-CREATION
> agents-audit/src/main/java/org/apache/ranger/audit/provider/AuditProviderFactory.java e3c3508
> agents-audit/src/main/java/org/apache/ranger/audit/queue/AuditFileCacheProviderSpool.java PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/56335/diff/1/
>
>
> Testing
> -------
>
> Test all the plugins in Local VM
> To enable the file cache provider for each of the components please do the following
>
> For HDFS Plugin
> ===============
> mkdir -p /var/log/hadoop/hdfs/audit/spool
> cd /var/log/hadoop/hdfs/audit/
> chown hdfs:hadoop spool
> Add the following properties to the "custom ranger-hive-audit” in the Ambari for hdfs.
> xasecure.audit.provider.filecache.is.enabled=true
> xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
> xasecure.audit.provider.filecache.filespool.dir=/var/log/hadoop/hdfs/audit/spool
>
> NOTE:
> xasecure.audit.provider.filecache.is.enabled = true
> This property will enable file cache provider which will store the audit locally first before sending it to destinations to avoid lose of data
> xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
> This property will close each of local file every 300 sec ( 5 min ) and send it destinations. For testing we maded to 30 sec.
> xasecure.audit.provider.filecache.filespool.dir=/var/log/hadoop/hdfs/audit/spool
> This property is the directory where the local audit cache is present.
>
> For Hive Plugin
> =============
>
> mkdir -p /var/log/hive/audit/spool
> cd /var/log/hive/audit/
> chown hdfs:hadoop spool
> Add the following properties to the "custom ranger-hive-audit” in the Ambari for hdfs.
> xasecure.audit.provider.filecache.is.enabled=true
> xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
> xasecure.audit.provider.filecache.filespool.dir=/var/log/hive/audit/spool
>
> Please do the same steps mentioned for all the components which need this audit file cache provider.
>
>
> ---------------
> Issues:
> - Audit to HDFS destination gets 0 bytes file or missing records in the file from HDFS plugin when HDFS get restarted and
> audit from hdfs plugin is logged into destination.
>
> - Audit to HDFS destination gets partial records from HIVE/HBASE/KNOX/STORM plugin when HDFS is restarted and there are active
> spooling into hdfs is happening.
>
> Scenarios to test
>
> 1) Audit to HDFS / Solr destination with FileCache enabled- HDFS/HIVESERVER2/HBASE/KNOX/STORM/KAFKA.
> - Mentioned issue should not happen.
> - Audit will be getting pushed every 5 minutes ( we are setting it to 300 sec in the parameter)
>
> 2) Audit to HDFS / Solr destination with FileCache enabled with one of the destination is down and brought back up later.
> - Audit from the local cache should be present in destination when the destination is up
> - In case of HDFS as destination audit might show up during next rollover of hdfs file or if the corresponding component
> is restarted ( say if it is hiveserver2 plugin, when Hiveserver2 is restarted audit into HDFS appears as this will close the
> existing opened hdfsfile)
> - Mentioned issue should not be present
> -
> -
> 3) Same has to be done for each for the plugins ( HBASE, STORM, KAFKA, KMS)
>
>
> Thanks,
>
> Ramesh Mani
>
>
Re: Review Request 56335: RANGER-1310: Ranger Audit framework
enhancement to
provide an option to allow audit records to be spooled to local disk first
before sending it to destinations
Posted by Ramesh Mani <rm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56335/
-----------------------------------------------------------
(Updated Feb. 10, 2017, 10:50 p.m.)
Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan Periasamy.
Changes
-------
Bug number and Branch added
Summary (updated)
-----------------
RANGER-1310: Ranger Audit framework enhancement to provide an option to allow audit records to be spooled to local disk first before sending it to destinations
Bugs: RANGER-1310
https://issues.apache.org/jira/browse/RANGER-1310
Repository: ranger
Description (updated)
-------
RANGER-1310: Ranger Audit framework enhancement to provide an option to allow audit records to be spooled to local disk first before sending it to destinations
Diffs
-----
agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 7c37cfa
agents-audit/src/main/java/org/apache/ranger/audit/provider/AuditFileCacheProvider.java PRE-CREATION
agents-audit/src/main/java/org/apache/ranger/audit/provider/AuditProviderFactory.java e3c3508
agents-audit/src/main/java/org/apache/ranger/audit/queue/AuditFileCacheProviderSpool.java PRE-CREATION
Diff: https://reviews.apache.org/r/56335/diff/
Testing
-------
Test all the plugins in Local VM
To enable the file cache provider for each of the components please do the following
For HDFS Plugin
===============
mkdir -p /var/log/hadoop/hdfs/audit/spool
cd /var/log/hadoop/hdfs/audit/
chown hdfs:hadoop spool
Add the following properties to the "custom ranger-hive-audit\u201d in the Ambari for hdfs.
xasecure.audit.provider.filecache.is.enabled=true
xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
xasecure.audit.provider.filecache.filespool.dir=/var/log/hadoop/hdfs/audit/spool
NOTE:
xasecure.audit.provider.filecache.is.enabled = true
This property will enable file cache provider which will store the audit locally first before sending it to destinations to avoid lose of data
xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
This property will close each of local file every 300 sec ( 5 min ) and send it destinations. For testing we maded to 30 sec.
xasecure.audit.provider.filecache.filespool.dir=/var/log/hadoop/hdfs/audit/spool
This property is the directory where the local audit cache is present.
For Hive Plugin
=============
mkdir -p /var/log/hive/audit/spool
cd /var/log/hive/audit/
chown hdfs:hadoop spool
Add the following properties to the "custom ranger-hive-audit\u201d in the Ambari for hdfs.
xasecure.audit.provider.filecache.is.enabled=true
xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
xasecure.audit.provider.filecache.filespool.dir=/var/log/hive/audit/spool
Please do the same steps mentioned for all the components which need this audit file cache provider.
---------------
Issues:
- Audit to HDFS destination gets 0 bytes file or missing records in the file from HDFS plugin when HDFS get restarted and
audit from hdfs plugin is logged into destination.
- Audit to HDFS destination gets partial records from HIVE/HBASE/KNOX/STORM plugin when HDFS is restarted and there are active
spooling into hdfs is happening.
Scenarios to test
1) Audit to HDFS / Solr destination with FileCache enabled- HDFS/HIVESERVER2/HBASE/KNOX/STORM/KAFKA.
- Mentioned issue should not happen.
- Audit will be getting pushed every 5 minutes ( we are setting it to 300 sec in the parameter)
2) Audit to HDFS / Solr destination with FileCache enabled with one of the destination is down and brought back up later.
- Audit from the local cache should be present in destination when the destination is up
- In case of HDFS as destination audit might show up during next rollover of hdfs file or if the corresponding component
is restarted ( say if it is hiveserver2 plugin, when Hiveserver2 is restarted audit into HDFS appears as this will close the
existing opened hdfsfile)
- Mentioned issue should not be present
-
-
3) Same has to be done for each for the plugins ( HBASE, STORM, KAFKA, KMS)
Thanks,
Ramesh Mani
Re: Review Request 56335: Ranger Audit framework enhancement to
provide an
option to allow audit records to be spooled to local disk first before
sending it to destinations
Posted by Velmurugan Periasamy <vp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56335/#review165103
-----------------------------------------------------------
Ship it!
Please add JIRA number and branch numbers - I believe this patch is required in master and ranger-0.7
- Velmurugan Periasamy
On Feb. 6, 2017, 6:36 p.m., Ramesh Mani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56335/
> -----------------------------------------------------------
>
> (Updated Feb. 6, 2017, 6:36 p.m.)
>
>
> Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan Periasamy.
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Ranger Audit framework enhancement to provide an option to allow audit records to be spooled to local disk first before sending it to destinations
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 7c37cfa
> agents-audit/src/main/java/org/apache/ranger/audit/provider/AuditFileCacheProvider.java PRE-CREATION
> agents-audit/src/main/java/org/apache/ranger/audit/provider/AuditProviderFactory.java e3c3508
> agents-audit/src/main/java/org/apache/ranger/audit/queue/AuditFileCacheProviderSpool.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/56335/diff/
>
>
> Testing
> -------
>
> Test all the plugins in Local VM
> To enable the file cache provider for each of the components please do the following
>
> For HDFS Plugin
> ===============
> mkdir -p /var/log/hadoop/hdfs/audit/spool
> cd /var/log/hadoop/hdfs/audit/
> chown hdfs:hadoop spool
> Add the following properties to the "custom ranger-hive-audit\u201d in the Ambari for hdfs.
> xasecure.audit.provider.filecache.is.enabled=true
> xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
> xasecure.audit.provider.filecache.filespool.dir=/var/log/hadoop/hdfs/audit/spool
>
> NOTE:
> xasecure.audit.provider.filecache.is.enabled = true
> This property will enable file cache provider which will store the audit locally first before sending it to destinations to avoid lose of data
> xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
> This property will close each of local file every 300 sec ( 5 min ) and send it destinations. For testing we maded to 30 sec.
> xasecure.audit.provider.filecache.filespool.dir=/var/log/hadoop/hdfs/audit/spool
> This property is the directory where the local audit cache is present.
>
> For Hive Plugin
> =============
>
> mkdir -p /var/log/hive/audit/spool
> cd /var/log/hive/audit/
> chown hdfs:hadoop spool
> Add the following properties to the "custom ranger-hive-audit\u201d in the Ambari for hdfs.
> xasecure.audit.provider.filecache.is.enabled=true
> xasecure.audit.provider.filecache.filespool.file.rollover.sec=300
> xasecure.audit.provider.filecache.filespool.dir=/var/log/hive/audit/spool
>
> Please do the same steps mentioned for all the components which need this audit file cache provider.
>
>
> ---------------
> Issues:
> - Audit to HDFS destination gets 0 bytes file or missing records in the file from HDFS plugin when HDFS get restarted and
> audit from hdfs plugin is logged into destination.
>
> - Audit to HDFS destination gets partial records from HIVE/HBASE/KNOX/STORM plugin when HDFS is restarted and there are active
> spooling into hdfs is happening.
>
> Scenarios to test
>
> 1) Audit to HDFS / Solr destination with FileCache enabled- HDFS/HIVESERVER2/HBASE/KNOX/STORM/KAFKA.
> - Mentioned issue should not happen.
> - Audit will be getting pushed every 5 minutes ( we are setting it to 300 sec in the parameter)
>
> 2) Audit to HDFS / Solr destination with FileCache enabled with one of the destination is down and brought back up later.
> - Audit from the local cache should be present in destination when the destination is up
> - In case of HDFS as destination audit might show up during next rollover of hdfs file or if the corresponding component
> is restarted ( say if it is hiveserver2 plugin, when Hiveserver2 is restarted audit into HDFS appears as this will close the
> existing opened hdfsfile)
> - Mentioned issue should not be present
> -
> -
> 3) Same has to be done for each for the plugins ( HBASE, STORM, KAFKA, KMS)
>
>
> Thanks,
>
> Ramesh Mani
>
>