You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Alok Lal <al...@hortonworks.com> on 2015/10/07 21:24:35 UTC
Review Request 39101: Ranger to support Azure Blob Datastore as an
audit destination via HDFS audit handler
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39101/
-----------------------------------------------------------
Review request for ranger, Don Bosco Durai, Gautam Borad, Madhan Neethiraj, and Velmurugan Periasamy.
Bugs: RANGER-682
https://issues.apache.org/jira/browse/RANGER-682
Repository: ranger
Description
-------
Hadoop has added native support for Azure Blob Storage. Ranger allows to write audit messages to it via Hadoops support for writing to Azure Blob Storage by existing HDFS Audit Provider.
- Also fixed a issue to reconnect to Filesystem in case of error
- Configuration support added only for v3 style of configuration.
- This patch applies to master. However, it will be ported to Ranger-0.5 branch, too.
Diffs
-----
agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 49e5fbb
hbase-agent/conf/ranger-hbase-audit.xml c5f416e
hdfs-agent/conf/ranger-hdfs-audit.xml 4c60c02
hive-agent/conf/ranger-hive-audit.xml 867080c
knox-agent/conf/ranger-knox-audit.xml ccc2691
plugin-kafka/conf/ranger-kafka-audit.xml bff6cbf
plugin-kms/conf/ranger-kms-audit.xml f1c4187
plugin-solr/conf/ranger-solr-audit.xml a42716f
plugin-yarn/conf/ranger-yarn-audit.xml c30f963
storm-agent/conf/ranger-storm-audit.xml 6a5ff69
Diff: https://reviews.apache.org/r/39101/diff/
Testing
-------
Write to Azure blob storage via TestEvents utlity
- Microsoft HDInisight currently clusters don't have support for kerberos. However, testing was done with and without wire encryption (WASB and WASBS schemas).
Thanks,
Alok Lal
Re: Review Request 39101: Ranger to support Azure Blob Datastore as
an audit destination via HDFS audit handler
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39101/#review102123
-----------------------------------------------------------
Ship it!
agents-audit/src/main/java/org/apache/ranger/audit/provider/BaseAuditHandler.java (line 118)
<https://reviews.apache.org/r/39101/#comment159675>
It might be safer to include the trailing "." as well in the prefix, like:
propPrefix + "." + PROP_CONFIG + "."
- Madhan Neethiraj
On Oct. 8, 2015, 11:27 p.m., Alok Lal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39101/
> -----------------------------------------------------------
>
> (Updated Oct. 8, 2015, 11:27 p.m.)
>
>
> Review request for ranger, Don Bosco Durai, Gautam Borad, Madhan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-682
> https://issues.apache.org/jira/browse/RANGER-682
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Hadoop has added native support for Azure Blob Storage. Ranger allows to write audit messages to it via Hadoops support for writing to Azure Blob Storage by existing HDFS Audit Provider.
> - Also fixed a issue to reconnect to Filesystem in case of error
> - Configuration support added only for v3 style of configuration.
> - This patch applies to master. However, it will be ported to Ranger-0.5 branch, too.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 49e5fbb
> agents-audit/src/main/java/org/apache/ranger/audit/provider/BaseAuditHandler.java 30db18b
> agents-audit/src/main/java/org/apache/ranger/audit/provider/hdfs/HdfsAuditProvider.java a18e3e9
> agents-audit/src/main/java/org/apache/ranger/audit/provider/hdfs/HdfsLogDestination.java 49f4e65
> agents-installer/src/main/java/org/apache/ranger/utils/install/XmlConfigChanger.java 05fbb23
> hbase-agent/conf/ranger-hbase-audit-changes.cfg b540e18
> hbase-agent/scripts/install.properties 9170f41
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 93e7b86
> hdfs-agent/scripts/install.properties 289117b
> hive-agent/conf/ranger-hive-audit-changes.cfg daa8b74
> hive-agent/scripts/install.properties 9370600
> kms/scripts/install.properties 563a0e0
> knox-agent/conf/ranger-knox-audit-changes.cfg 57858f7
> knox-agent/scripts/install.properties 2fee7ee
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg 0445576
> plugin-kafka/scripts/install.properties 1e2854e
> plugin-kms/conf/ranger-kms-audit-changes.cfg f7d3543
> plugin-solr/conf/ranger-solr-audit-changes.cfg 7e77a30
> plugin-solr/scripts/install.properties 1f3852e
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg bfc2cd8
> plugin-yarn/scripts/install.properties 9358033
> storm-agent/conf/ranger-storm-audit-changes.cfg bfc2cd8
> storm-agent/scripts/install.properties 0cee655
>
> Diff: https://reviews.apache.org/r/39101/diff/
>
>
> Testing
> -------
>
> Write to Azure blob storage via TestEvents utlity
> - Microsoft HDInisight currently clusters don't have support for kerberos. However, testing was done with and without wire encryption (WASB and WASBS schemas).
> - Tested with both V3 and V2 libraries
> - Did manual install of hbase plugin to confirm that ranger-hbase-audit.xml file gets created with right property names and values.
>
>
> Thanks,
>
> Alok Lal
>
>
Re: Review Request 39101: Ranger to support Azure Blob Datastore as
an audit destination via HDFS audit handler
Posted by Alok Lal <al...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39101/
-----------------------------------------------------------
(Updated Oct. 8, 2015, 4:27 p.m.)
Review request for ranger, Don Bosco Durai, Gautam Borad, Madhan Neethiraj, and Velmurugan Periasamy.
Changes
-------
Addressed rework comments:
- Move config-map to base class
- Added support to both V2 and V3 audit libraries
- Empty properties would not be added to config so that install is simplified
Also made change to allow for manual install. Changes in install.properties and audit-change.cfg files.
- Change to install framework is needed because azure support needs properties whose name changes based on account name used.
Bugs: RANGER-682
https://issues.apache.org/jira/browse/RANGER-682
Repository: ranger
Description
-------
Hadoop has added native support for Azure Blob Storage. Ranger allows to write audit messages to it via Hadoops support for writing to Azure Blob Storage by existing HDFS Audit Provider.
- Also fixed a issue to reconnect to Filesystem in case of error
- Configuration support added only for v3 style of configuration.
- This patch applies to master. However, it will be ported to Ranger-0.5 branch, too.
Diffs (updated)
-----
agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 49e5fbb
agents-audit/src/main/java/org/apache/ranger/audit/provider/BaseAuditHandler.java 30db18b
agents-audit/src/main/java/org/apache/ranger/audit/provider/hdfs/HdfsAuditProvider.java a18e3e9
agents-audit/src/main/java/org/apache/ranger/audit/provider/hdfs/HdfsLogDestination.java 49f4e65
agents-installer/src/main/java/org/apache/ranger/utils/install/XmlConfigChanger.java 05fbb23
hbase-agent/conf/ranger-hbase-audit-changes.cfg b540e18
hbase-agent/scripts/install.properties 9170f41
hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 93e7b86
hdfs-agent/scripts/install.properties 289117b
hive-agent/conf/ranger-hive-audit-changes.cfg daa8b74
hive-agent/scripts/install.properties 9370600
kms/scripts/install.properties 563a0e0
knox-agent/conf/ranger-knox-audit-changes.cfg 57858f7
knox-agent/scripts/install.properties 2fee7ee
plugin-kafka/conf/ranger-kafka-audit-changes.cfg 0445576
plugin-kafka/scripts/install.properties 1e2854e
plugin-kms/conf/ranger-kms-audit-changes.cfg f7d3543
plugin-solr/conf/ranger-solr-audit-changes.cfg 7e77a30
plugin-solr/scripts/install.properties 1f3852e
plugin-yarn/conf/ranger-yarn-audit-changes.cfg bfc2cd8
plugin-yarn/scripts/install.properties 9358033
storm-agent/conf/ranger-storm-audit-changes.cfg bfc2cd8
storm-agent/scripts/install.properties 0cee655
Diff: https://reviews.apache.org/r/39101/diff/
Testing (updated)
-------
Write to Azure blob storage via TestEvents utlity
- Microsoft HDInisight currently clusters don't have support for kerberos. However, testing was done with and without wire encryption (WASB and WASBS schemas).
- Tested with both V3 and V2 libraries
- Did manual install of hbase plugin to confirm that ranger-hbase-audit.xml file gets created with right property names and values.
Thanks,
Alok Lal
Re: Review Request 39101: Ranger to support Azure Blob Datastore as
an audit destination via HDFS audit handler
Posted by Alok Lal <al...@hortonworks.com>.
> On Oct. 7, 2015, 3:53 p.m., Madhan Neethiraj wrote:
> > agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java, line 149
> > <https://reviews.apache.org/r/39101/diff/1/?file=1092478#file1092478line149>
> >
> > Can out.close() throw exception? If yes, move the call to addDeferredCount() after out.close() call.
close() does not throw exception.
> On Oct. 7, 2015, 3:53 p.m., Madhan Neethiraj wrote:
> > hbase-agent/conf/ranger-hbase-audit.xml, line 165
> > <https://reviews.apache.org/r/39101/diff/1/?file=1092479#file1092479line165>
> >
> > It is not clear what settings to comment and what to uncomment. Perhaps you can list the property names to be commented and uncommented for Azure.
> >
> > Other option is to include Azure specific properties, with empty value (and handle in code appropriately). This will keep the conf file clean and easier to read. Also, note that all this should be documented so that users can configure audit properties via Ambari.
Simplified the properties and changed code so that properties with empty-value won't get added to the Configuration.
- Alok
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39101/#review101829
-----------------------------------------------------------
On Oct. 8, 2015, 4:27 p.m., Alok Lal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39101/
> -----------------------------------------------------------
>
> (Updated Oct. 8, 2015, 4:27 p.m.)
>
>
> Review request for ranger, Don Bosco Durai, Gautam Borad, Madhan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-682
> https://issues.apache.org/jira/browse/RANGER-682
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Hadoop has added native support for Azure Blob Storage. Ranger allows to write audit messages to it via Hadoops support for writing to Azure Blob Storage by existing HDFS Audit Provider.
> - Also fixed a issue to reconnect to Filesystem in case of error
> - Configuration support added only for v3 style of configuration.
> - This patch applies to master. However, it will be ported to Ranger-0.5 branch, too.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 49e5fbb
> agents-audit/src/main/java/org/apache/ranger/audit/provider/BaseAuditHandler.java 30db18b
> agents-audit/src/main/java/org/apache/ranger/audit/provider/hdfs/HdfsAuditProvider.java a18e3e9
> agents-audit/src/main/java/org/apache/ranger/audit/provider/hdfs/HdfsLogDestination.java 49f4e65
> agents-installer/src/main/java/org/apache/ranger/utils/install/XmlConfigChanger.java 05fbb23
> hbase-agent/conf/ranger-hbase-audit-changes.cfg b540e18
> hbase-agent/scripts/install.properties 9170f41
> hdfs-agent/conf/ranger-hdfs-audit-changes.cfg 93e7b86
> hdfs-agent/scripts/install.properties 289117b
> hive-agent/conf/ranger-hive-audit-changes.cfg daa8b74
> hive-agent/scripts/install.properties 9370600
> kms/scripts/install.properties 563a0e0
> knox-agent/conf/ranger-knox-audit-changes.cfg 57858f7
> knox-agent/scripts/install.properties 2fee7ee
> plugin-kafka/conf/ranger-kafka-audit-changes.cfg 0445576
> plugin-kafka/scripts/install.properties 1e2854e
> plugin-kms/conf/ranger-kms-audit-changes.cfg f7d3543
> plugin-solr/conf/ranger-solr-audit-changes.cfg 7e77a30
> plugin-solr/scripts/install.properties 1f3852e
> plugin-yarn/conf/ranger-yarn-audit-changes.cfg bfc2cd8
> plugin-yarn/scripts/install.properties 9358033
> storm-agent/conf/ranger-storm-audit-changes.cfg bfc2cd8
> storm-agent/scripts/install.properties 0cee655
>
> Diff: https://reviews.apache.org/r/39101/diff/
>
>
> Testing
> -------
>
> Write to Azure blob storage via TestEvents utlity
> - Microsoft HDInisight currently clusters don't have support for kerberos. However, testing was done with and without wire encryption (WASB and WASBS schemas).
> - Tested with both V3 and V2 libraries
> - Did manual install of hbase plugin to confirm that ranger-hbase-audit.xml file gets created with right property names and values.
>
>
> Thanks,
>
> Alok Lal
>
>
Re: Review Request 39101: Ranger to support Azure Blob Datastore as
an audit destination via HDFS audit handler
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39101/#review101829
-----------------------------------------------------------
agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java (line 101)
<https://reviews.apache.org/r/39101/#comment159327>
Notion of 'configProps' might be useful for other destinations as well. Consider moving this to base class AuditDestination.
agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java (line 145)
<https://reviews.apache.org/r/39101/#comment159328>
Can out.close() throw exception? If yes, move the call to addDeferredCount() after out.close() call.
hbase-agent/conf/ranger-hbase-audit.xml (line 165)
<https://reviews.apache.org/r/39101/#comment159325>
It is not clear what settings to comment and what to uncomment. Perhaps you can list the property names to be commented and uncommented for Azure.
Other option is to include Azure specific properties, with empty value (and handle in code appropriately). This will keep the conf file clean and easier to read. Also, note that all this should be documented so that users can configure audit properties via Ambari.
- Madhan Neethiraj
On Oct. 7, 2015, 7:24 p.m., Alok Lal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39101/
> -----------------------------------------------------------
>
> (Updated Oct. 7, 2015, 7:24 p.m.)
>
>
> Review request for ranger, Don Bosco Durai, Gautam Borad, Madhan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-682
> https://issues.apache.org/jira/browse/RANGER-682
>
>
> Repository: ranger
>
>
> Description
> -------
>
> Hadoop has added native support for Azure Blob Storage. Ranger allows to write audit messages to it via Hadoops support for writing to Azure Blob Storage by existing HDFS Audit Provider.
> - Also fixed a issue to reconnect to Filesystem in case of error
> - Configuration support added only for v3 style of configuration.
> - This patch applies to master. However, it will be ported to Ranger-0.5 branch, too.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 49e5fbb
> hbase-agent/conf/ranger-hbase-audit.xml c5f416e
> hdfs-agent/conf/ranger-hdfs-audit.xml 4c60c02
> hive-agent/conf/ranger-hive-audit.xml 867080c
> knox-agent/conf/ranger-knox-audit.xml ccc2691
> plugin-kafka/conf/ranger-kafka-audit.xml bff6cbf
> plugin-kms/conf/ranger-kms-audit.xml f1c4187
> plugin-solr/conf/ranger-solr-audit.xml a42716f
> plugin-yarn/conf/ranger-yarn-audit.xml c30f963
> storm-agent/conf/ranger-storm-audit.xml 6a5ff69
>
> Diff: https://reviews.apache.org/r/39101/diff/
>
>
> Testing
> -------
>
> Write to Azure blob storage via TestEvents utlity
> - Microsoft HDInisight currently clusters don't have support for kerberos. However, testing was done with and without wire encryption (WASB and WASBS schemas).
>
>
> Thanks,
>
> Alok Lal
>
>