You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Gautam Borad <gb...@gmail.com> on 2016/01/27 05:21:14 UTC
Review Request 42601: RANGER-798 - Approach 1 : Handle different
timezone issue while saving audit logs to Solr
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42601/
-----------------------------------------------------------
Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
Bugs: RANGER-798
https://issues.apache.org/jira/browse/RANGER-798
Repository: ranger
Description
-------
**Problem Statement**:
The current implementation of “auditEvent.getEventTime()” contains time in UTC and since it's a date object it will contain component machine's local timezone. When Solr receives this date object and timezone, it tries to convert it from given timezone to UTC timestamp, which leads to double conversion of actual time before it get stored in Solr.
**Proposed Solution**:
If we can provide server local time and timezone to Solr then Solr will convert the received time from given timezone to UTC.
As an alternate solution, replaced getUTCDate() with new Date() object at various places for audit event time, all audit destination will receive local Date object, for Solr there will be no conversion on received Date object but for all other audit destination we need to convert the received Date value to UTC timestamp as audit logs are being stored in UTC timestamp for all service/component. If all destination thread are enabled then changing the received event object may create issue in other audit destination as same event object is refferred everywhere. Hence received event object attributes value are being copied in another local event object and the updated event time can be stored there, after this local event object will be used to convert that in JSON to write in HDFS, or can be persisted in DB.
Diffs
-----
agents-audit/src/main/java/org/apache/ranger/audit/destination/DBAuditDestination.java 376e724
agents-audit/src/main/java/org/apache/ranger/audit/destination/FileAuditDestination.java c6cd8b2
agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 96755be
agents-audit/src/main/java/org/apache/ranger/audit/destination/Log4JAuditDestination.java 9521a4a
agents-audit/src/main/java/org/apache/ranger/audit/model/AuditEventBase.java 2a07e94
agents-audit/src/main/java/org/apache/ranger/audit/provider/BaseAuditHandler.java 6717c92
agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java 9586f73
agents-common/src/main/java/org/apache/ranger/authorization/utils/StringUtil.java f6f3d2d
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java fe50ca6
hbase-agent/src/main/java/org/apache/ranger/authorization/hbase/RangerAuthorizationCoprocessor.java 8762bf5
hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java 5125af7
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java 2ae4149
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java 0f13577
plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java bb6a337
plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java 04b8b91
plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java a8ecf15
plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java ab9b7a9
Diff: https://reviews.apache.org/r/42601/diff/
Testing
-------
Steps performed(after patch) :
1) Changed plugin system time zone to IST and restarted all components.
2) Initiated an HDFS audit event.
3) Checked event time of newly created audit log in Solr, Audit log event time was matching with UTC.
4) Checked event time in Ranger UI, newly generated Audit event is matching with current time.
5) Checked event time of newly created audit log in xa_access_audit table, Audit log event time was matching with UTC.
6) Checked event time of newly created audit log in HDFS logs, Audit log event time was matching with UTC.
Note: Will test other services audit logs after this approach is reviewed.
Thanks,
Gautam Borad
Re: Review Request 42601: RANGER-798 - Approach 1 : Handle different
timezone issue while saving audit logs to Solr
Posted by Don Bosco Durai <bo...@apache.org>.
> On Jan. 28, 2016, 8:32 a.m., Madhan Neethiraj wrote:
> > agents-audit/src/main/java/org/apache/ranger/audit/destination/DBAuditDestination.java, line 97
> > <https://reviews.apache.org/r/42601/diff/1/?file=1205404#file1205404line97>
> >
> > Instead of cloning and updating the eventDate in every destination (db/hdfs/solr/log4j), consider sending the "local" time (as set in AuthzAuditEvent.eventTime) to all audit destinations - exception DB, which require the time in UTC. For DB, consider updating the time in AuthzAuditEventDbObj(AuthzAuditEvent event) constructor.
>
> Gautam Borad wrote:
> Do we need to add timezone info also with event time, if yes then what should be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS".
Sorry, late for this discussion. Make whatever JSON we write in HDFS is loadable by Hive. On the previous release, I explicitly has to set the format that is compatible with Hive loader with date/timestamp. For applying this patch, that should be one of the pre-check/testing. Thanks
- Don Bosco
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42601/#review116741
-----------------------------------------------------------
On Feb. 1, 2016, 12:37 p.m., Gautam Borad wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42601/
> -----------------------------------------------------------
>
> (Updated Feb. 1, 2016, 12:37 p.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-798
> https://issues.apache.org/jira/browse/RANGER-798
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement**:
> The current implementation of “auditEvent.getEventTime()” contains time in UTC and since it's a date object it will contain component machine's local timezone. When Solr receives this date object and timezone, it tries to convert it from given timezone to UTC timestamp, which leads to double conversion of actual time before it get stored in Solr.
>
> **Proposed Solution**:
> If we can provide server local time and timezone to Solr then Solr will convert the received time from given timezone to UTC.
> As an alternate solution, replaced getUTCDate() with new Date() object at various places for audit event time, all audit destination will receive local Date object, for Solr there will be no conversion on received Date object but for all other audit destination we need to convert the received Date value to UTC timestamp as audit logs are being stored in UTC timestamp for all service/component. If all destination thread are enabled then changing the received event object may create issue in other audit destination as same event object is refferred everywhere. Hence received event object attributes value are being copied in another local event object and the updated event time can be stored there, after this local event object will be used to convert that in JSON to write in HDFS, or can be persisted in DB.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java d52a60a
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java 9586f73
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java fe50ca6
> hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java 5125af7
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java 2ae4149
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java 0f13577
> plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java bb6a337
> plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java 04b8b91
> plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java a8ecf15
> plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java ab9b7a9
>
> Diff: https://reviews.apache.org/r/42601/diff/
>
>
> Testing
> -------
>
> Steps performed(after patch) :
> 1) Changed plugin system time zone to IST and restarted all components.
> 2) Initiated an HDFS audit event.
> 3) Checked event time of newly created audit log in Solr, Audit log event time was matching with UTC.
> 4) Checked event time in Ranger UI, newly generated Audit event is matching with current time.
> 5) Checked event time of newly created audit log in xa_access_audit table, Audit log event time was matching with UTC.
> 6) Checked event time of newly created audit log in HDFS logs, Audit log event time was matching with UTC.
>
> Note: Will test other services audit logs after this approach is reviewed.
>
>
> Thanks,
>
> Gautam Borad
>
>
Re: Review Request 42601: RANGER-798 - Approach 1 : Handle
different timezone issue while saving audit logs to Solr
Posted by Madhan Neethiraj <mn...@hortonworks.com>.
>> Do you mean just for HDFS/File/Log4j or for DB and Solr also?
It depends on how Solr expects the date field to be formatted in.
For HDFS/Log4j - we should publish the date format used to write audit log, so that the readers can be configured appropriately. Going forward, I would suggest using a format that includes the timezone. For backward compatibility sake, it should be possible to configure each destination to use the legacy format "yyyy-MM-dd HH:mm:ss.SSS” - a configuration like “xasecure.audit.destination.<type>.date.format.legacy=false".
Thanks,
Madhan
On 2/1/16, 12:52 AM, "Gautam Borad" <gb...@gmail.com> wrote:
>>>For JSON, how about including the timezone?
>
>Do you mean just for HDFS/File/Log4j or for DB and Solr also? Thanks.
>
>
>On Mon, Feb 1, 2016 at 12:59 PM, Madhan Neethiraj <
>mneethiraj@hortonworks.com> wrote:
>
>> Gautam,
>>
>> >> Do we need to add timezone info also with event time, if yes then what
>> should be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS".
>> For JSON, how about including the timezone? Please find how/if the new
>> format (that includes timezone) can co-exist with the earlier format? If
>> not, then we would have to support writing in older format – depending upon
>> a configuration; by default the audit log should be written with the
>> timezone.
>>
>> Thanks,
>> Madhan
>>
>>
>> From: Gautam Borad <no...@reviews.apache.org> on behalf of Gautam Borad
>> <gb...@gmail.com>
>> Reply-To: Gautam Borad <gb...@gmail.com>
>> Date: Thursday, January 28, 2016 at 5:13 AM
>> To: Abhay Kulkarni <ak...@hortonworks.com>, Ramesh Mani <
>> rmani@hortonworks.com>, Velmurugan Periasamy <vp...@hortonworks.com>,
>> Alok Lal <al...@hortonworks.com>, Selvamohan Neethiraj <
>> sneethiraj@hortonworks.com>, Madhan Neethiraj <ma...@apache.org>, Don
>> Bosco Durai <bo...@apache.org>
>> Cc: ranger <de...@ranger.incubator.apache.org>, Gautam Borad <
>> gborad@gmail.com>
>> Subject: Re: Review Request 42601: RANGER-798 - Approach 1 : Handle
>> different timezone issue while saving audit logs to Solr
>>
>> This is an automatically generated e-mail. To reply, visit:
>> https://reviews.apache.org/r/42601/
>>
>> On January 28th, 2016, 8:32 a.m. UTC, *Madhan Neethiraj* wrote:
>>
>>
>> agents-audit/src/main/java/org/apache/ranger/audit/destination/DBAuditDestination.java
>> <https://reviews.apache.org/r/42601/diff/1/?file=1205404#file1205404line97>(Diff
>> revision 1)
>>
>> public boolean log(Collection<AuditEventBase> events) {
>>
>> 97
>>
>> eventCopy=cloneAuthzAuditEvent((AuthzAuditEvent)event);
>>
>> Instead of cloning and updating the eventDate in every destination (db/hdfs/solr/log4j), consider sending the "local" time (as set in AuthzAuditEvent.eventTime) to all audit destinations - exception DB, which require the time in UTC. For DB, consider updating the time in AuthzAuditEventDbObj(AuthzAuditEvent event) constructor.
>>
>> Do we need to add timezone info also with event time, if yes then what should be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS".
>>
>>
>> - Gautam
>>
>> On January 28th, 2016, 1:12 p.m. UTC, Gautam Borad wrote:
>> Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni,
>> Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan
>> Periasamy.
>> By Gautam Borad.
>>
>> *Updated Jan. 28, 2016, 1:12 p.m.*
>> *Bugs: *RANGER-798 <https://issues.apache.org/jira/browse/RANGER-798>
>> *Repository: *ranger
>> Description
>>
>> *Problem Statement*:
>> The current implementation of “auditEvent.getEventTime()” contains time in UTC and since it's a date object it will contain component machine's local timezone. When Solr receives this date object and timezone, it tries to convert it from given timezone to UTC timestamp, which leads to double conversion of actual time before it get stored in Solr.
>>
>> *Proposed Solution*:
>> If we can provide server local time and timezone to Solr then Solr will convert the received time from given timezone to UTC.
>> As an alternate solution, replaced getUTCDate() with new Date() object at various places for audit event time, all audit destination will receive local Date object, for Solr there will be no conversion on received Date object but for all other audit destination we need to convert the received Date value to UTC timestamp as audit logs are being stored in UTC timestamp for all service/component. If all destination thread are enabled then changing the received event object may create issue in other audit destination as same event object is refferred everywhere. Hence received event object attributes value are being copied in another local event object and the updated event time can be stored there, after this local event object will be used to convert that in JSON to write in HDFS, or can be persisted in DB.
>>
>> Testing
>>
>> Steps performed(after patch) :
>> 1) Changed plugin system time zone to IST and restarted all components.
>> 2) Initiated an HDFS audit event.
>> 3) Checked event time of newly created audit log in Solr, Audit log event time was matching with UTC.
>> 4) Checked event time in Ranger UI, newly generated Audit event is matching with current time.
>> 5) Checked event time of newly created audit log in xa_access_audit table, Audit log event time was matching with UTC.
>> 6) Checked event time of newly created audit log in HDFS logs, Audit log event time was matching with UTC.
>>
>> Note: Will test other services audit logs after this approach is reviewed.
>>
>> Diffs
>>
>> - agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java
>> (d52a60a)
>> - agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java
>> (9586f73)
>> - agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java
>> (fe50ca6)
>> - hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java
>> (5125af7)
>> - hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java
>> (2ae4149)
>> - hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java
>> (0f13577)
>> - plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java
>> (bb6a337)
>> - plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java
>> (04b8b91)
>> - plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java
>> (a8ecf15)
>> - plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java
>> (ab9b7a9)
>>
>> View Diff <https://reviews.apache.org/r/42601/diff/>
>>
>
>
>
>--
>Regards,
>Gautam.
Re: Review Request 42601: RANGER-798 - Approach 1 : Handle different
timezone issue while saving audit logs to Solr
Posted by Gautam Borad <gb...@gmail.com>.
>>For JSON, how about including the timezone?
Do you mean just for HDFS/File/Log4j or for DB and Solr also? Thanks.
On Mon, Feb 1, 2016 at 12:59 PM, Madhan Neethiraj <
mneethiraj@hortonworks.com> wrote:
> Gautam,
>
> >> Do we need to add timezone info also with event time, if yes then what
> should be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS".
> For JSON, how about including the timezone? Please find how/if the new
> format (that includes timezone) can co-exist with the earlier format? If
> not, then we would have to support writing in older format – depending upon
> a configuration; by default the audit log should be written with the
> timezone.
>
> Thanks,
> Madhan
>
>
> From: Gautam Borad <no...@reviews.apache.org> on behalf of Gautam Borad
> <gb...@gmail.com>
> Reply-To: Gautam Borad <gb...@gmail.com>
> Date: Thursday, January 28, 2016 at 5:13 AM
> To: Abhay Kulkarni <ak...@hortonworks.com>, Ramesh Mani <
> rmani@hortonworks.com>, Velmurugan Periasamy <vp...@hortonworks.com>,
> Alok Lal <al...@hortonworks.com>, Selvamohan Neethiraj <
> sneethiraj@hortonworks.com>, Madhan Neethiraj <ma...@apache.org>, Don
> Bosco Durai <bo...@apache.org>
> Cc: ranger <de...@ranger.incubator.apache.org>, Gautam Borad <
> gborad@gmail.com>
> Subject: Re: Review Request 42601: RANGER-798 - Approach 1 : Handle
> different timezone issue while saving audit logs to Solr
>
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42601/
>
> On January 28th, 2016, 8:32 a.m. UTC, *Madhan Neethiraj* wrote:
>
>
> agents-audit/src/main/java/org/apache/ranger/audit/destination/DBAuditDestination.java
> <https://reviews.apache.org/r/42601/diff/1/?file=1205404#file1205404line97>(Diff
> revision 1)
>
> public boolean log(Collection<AuditEventBase> events) {
>
> 97
>
> eventCopy=cloneAuthzAuditEvent((AuthzAuditEvent)event);
>
> Instead of cloning and updating the eventDate in every destination (db/hdfs/solr/log4j), consider sending the "local" time (as set in AuthzAuditEvent.eventTime) to all audit destinations - exception DB, which require the time in UTC. For DB, consider updating the time in AuthzAuditEventDbObj(AuthzAuditEvent event) constructor.
>
> Do we need to add timezone info also with event time, if yes then what should be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS".
>
>
> - Gautam
>
> On January 28th, 2016, 1:12 p.m. UTC, Gautam Borad wrote:
> Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni,
> Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan
> Periasamy.
> By Gautam Borad.
>
> *Updated Jan. 28, 2016, 1:12 p.m.*
> *Bugs: *RANGER-798 <https://issues.apache.org/jira/browse/RANGER-798>
> *Repository: *ranger
> Description
>
> *Problem Statement*:
> The current implementation of “auditEvent.getEventTime()” contains time in UTC and since it's a date object it will contain component machine's local timezone. When Solr receives this date object and timezone, it tries to convert it from given timezone to UTC timestamp, which leads to double conversion of actual time before it get stored in Solr.
>
> *Proposed Solution*:
> If we can provide server local time and timezone to Solr then Solr will convert the received time from given timezone to UTC.
> As an alternate solution, replaced getUTCDate() with new Date() object at various places for audit event time, all audit destination will receive local Date object, for Solr there will be no conversion on received Date object but for all other audit destination we need to convert the received Date value to UTC timestamp as audit logs are being stored in UTC timestamp for all service/component. If all destination thread are enabled then changing the received event object may create issue in other audit destination as same event object is refferred everywhere. Hence received event object attributes value are being copied in another local event object and the updated event time can be stored there, after this local event object will be used to convert that in JSON to write in HDFS, or can be persisted in DB.
>
> Testing
>
> Steps performed(after patch) :
> 1) Changed plugin system time zone to IST and restarted all components.
> 2) Initiated an HDFS audit event.
> 3) Checked event time of newly created audit log in Solr, Audit log event time was matching with UTC.
> 4) Checked event time in Ranger UI, newly generated Audit event is matching with current time.
> 5) Checked event time of newly created audit log in xa_access_audit table, Audit log event time was matching with UTC.
> 6) Checked event time of newly created audit log in HDFS logs, Audit log event time was matching with UTC.
>
> Note: Will test other services audit logs after this approach is reviewed.
>
> Diffs
>
> - agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java
> (d52a60a)
> - agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java
> (9586f73)
> - agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java
> (fe50ca6)
> - hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java
> (5125af7)
> - hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java
> (2ae4149)
> - hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java
> (0f13577)
> - plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java
> (bb6a337)
> - plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java
> (04b8b91)
> - plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java
> (a8ecf15)
> - plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java
> (ab9b7a9)
>
> View Diff <https://reviews.apache.org/r/42601/diff/>
>
--
Regards,
Gautam.
Re: Review Request 42601: RANGER-798 - Approach 1 : Handle
different timezone issue while saving audit logs to Solr
Posted by Madhan Neethiraj <mn...@hortonworks.com>.
Gautam,
>> Do we need to add timezone info also with event time, if yes then what should be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS".
For JSON, how about including the timezone? Please find how/if the new format (that includes timezone) can co-exist with the earlier format? If not, then we would have to support writing in older format – depending upon a configuration; by default the audit log should be written with the timezone.
Thanks,
Madhan
From: Gautam Borad <no...@reviews.apache.org>> on behalf of Gautam Borad <gb...@gmail.com>>
Reply-To: Gautam Borad <gb...@gmail.com>>
Date: Thursday, January 28, 2016 at 5:13 AM
To: Abhay Kulkarni <ak...@hortonworks.com>>, Ramesh Mani <rm...@hortonworks.com>>, Velmurugan Periasamy <vp...@hortonworks.com>>, Alok Lal <al...@hortonworks.com>>, Selvamohan Neethiraj <sn...@hortonworks.com>>, Madhan Neethiraj <ma...@apache.org>>, Don Bosco Durai <bo...@apache.org>>
Cc: ranger <de...@ranger.incubator.apache.org>>, Gautam Borad <gb...@gmail.com>>
Subject: Re: Review Request 42601: RANGER-798 - Approach 1 : Handle different timezone issue while saving audit logs to Solr
This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42601/
On January 28th, 2016, 8:32 a.m. UTC, Madhan Neethiraj wrote:
agents-audit/src/main/java/org/apache/ranger/audit/destination/DBAuditDestination.java<https://reviews.apache.org/r/42601/diff/1/?file=1205404#file1205404line97>(Diff revision 1)
public boolean log(Collection<AuditEventBase> events) {
97
eventCopy=cloneAuthzAuditEvent((AuthzAuditEvent)event);
Instead of cloning and updating the eventDate in every destination (db/hdfs/solr/log4j), consider sending the "local" time (as set in AuthzAuditEvent.eventTime) to all audit destinations - exception DB, which require the time in UTC. For DB, consider updating the time in AuthzAuditEventDbObj(AuthzAuditEvent event) constructor.
Do we need to add timezone info also with event time, if yes then what should be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS".
- Gautam
On January 28th, 2016, 1:12 p.m. UTC, Gautam Borad wrote:
Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
By Gautam Borad.
Updated Jan. 28, 2016, 1:12 p.m.
Bugs: RANGER-798<https://issues.apache.org/jira/browse/RANGER-798>
Repository: ranger
Description
Problem Statement:
The current implementation of “auditEvent.getEventTime()” contains time in UTC and since it's a date object it will contain component machine's local timezone. When Solr receives this date object and timezone, it tries to convert it from given timezone to UTC timestamp, which leads to double conversion of actual time before it get stored in Solr.
Proposed Solution:
If we can provide server local time and timezone to Solr then Solr will convert the received time from given timezone to UTC.
As an alternate solution, replaced getUTCDate() with new Date() object at various places for audit event time, all audit destination will receive local Date object, for Solr there will be no conversion on received Date object but for all other audit destination we need to convert the received Date value to UTC timestamp as audit logs are being stored in UTC timestamp for all service/component. If all destination thread are enabled then changing the received event object may create issue in other audit destination as same event object is refferred everywhere. Hence received event object attributes value are being copied in another local event object and the updated event time can be stored there, after this local event object will be used to convert that in JSON to write in HDFS, or can be persisted in DB.
Testing
Steps performed(after patch) :
1) Changed plugin system time zone to IST and restarted all components.
2) Initiated an HDFS audit event.
3) Checked event time of newly created audit log in Solr, Audit log event time was matching with UTC.
4) Checked event time in Ranger UI, newly generated Audit event is matching with current time.
5) Checked event time of newly created audit log in xa_access_audit table, Audit log event time was matching with UTC.
6) Checked event time of newly created audit log in HDFS logs, Audit log event time was matching with UTC.
Note: Will test other services audit logs after this approach is reviewed.
Diffs
* agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java (d52a60a)
* agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java (9586f73)
* agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java (fe50ca6)
* hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java (5125af7)
* hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java (2ae4149)
* hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java (0f13577)
* plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java (bb6a337)
* plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java (04b8b91)
* plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java (a8ecf15)
* plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java (ab9b7a9)
View Diff<https://reviews.apache.org/r/42601/diff/>
Re: Review Request 42601: RANGER-798 - Approach 1 : Handle different
timezone issue while saving audit logs to Solr
Posted by Gautam Borad <gb...@gmail.com>.
> On Jan. 28, 2016, 8:32 a.m., Madhan Neethiraj wrote:
> > agents-audit/src/main/java/org/apache/ranger/audit/destination/DBAuditDestination.java, line 97
> > <https://reviews.apache.org/r/42601/diff/1/?file=1205404#file1205404line97>
> >
> > Instead of cloning and updating the eventDate in every destination (db/hdfs/solr/log4j), consider sending the "local" time (as set in AuthzAuditEvent.eventTime) to all audit destinations - exception DB, which require the time in UTC. For DB, consider updating the time in AuthzAuditEventDbObj(AuthzAuditEvent event) constructor.
Do we need to add timezone info also with event time, if yes then what should be the new format ? current format is "yyyy-MM-dd HH:mm:ss.SSS".
- Gautam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42601/#review116741
-----------------------------------------------------------
On Jan. 28, 2016, 1:12 p.m., Gautam Borad wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42601/
> -----------------------------------------------------------
>
> (Updated Jan. 28, 2016, 1:12 p.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-798
> https://issues.apache.org/jira/browse/RANGER-798
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement**:
> The current implementation of “auditEvent.getEventTime()” contains time in UTC and since it's a date object it will contain component machine's local timezone. When Solr receives this date object and timezone, it tries to convert it from given timezone to UTC timestamp, which leads to double conversion of actual time before it get stored in Solr.
>
> **Proposed Solution**:
> If we can provide server local time and timezone to Solr then Solr will convert the received time from given timezone to UTC.
> As an alternate solution, replaced getUTCDate() with new Date() object at various places for audit event time, all audit destination will receive local Date object, for Solr there will be no conversion on received Date object but for all other audit destination we need to convert the received Date value to UTC timestamp as audit logs are being stored in UTC timestamp for all service/component. If all destination thread are enabled then changing the received event object may create issue in other audit destination as same event object is refferred everywhere. Hence received event object attributes value are being copied in another local event object and the updated event time can be stored there, after this local event object will be used to convert that in JSON to write in HDFS, or can be persisted in DB.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java d52a60a
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java 9586f73
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java fe50ca6
> hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java 5125af7
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java 2ae4149
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java 0f13577
> plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java bb6a337
> plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java 04b8b91
> plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java a8ecf15
> plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java ab9b7a9
>
> Diff: https://reviews.apache.org/r/42601/diff/
>
>
> Testing
> -------
>
> Steps performed(after patch) :
> 1) Changed plugin system time zone to IST and restarted all components.
> 2) Initiated an HDFS audit event.
> 3) Checked event time of newly created audit log in Solr, Audit log event time was matching with UTC.
> 4) Checked event time in Ranger UI, newly generated Audit event is matching with current time.
> 5) Checked event time of newly created audit log in xa_access_audit table, Audit log event time was matching with UTC.
> 6) Checked event time of newly created audit log in HDFS logs, Audit log event time was matching with UTC.
>
> Note: Will test other services audit logs after this approach is reviewed.
>
>
> Thanks,
>
> Gautam Borad
>
>
Re: Review Request 42601: RANGER-798 - Approach 1 : Handle different
timezone issue while saving audit logs to Solr
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42601/#review116741
-----------------------------------------------------------
agents-audit/src/main/java/org/apache/ranger/audit/destination/DBAuditDestination.java (line 97)
<https://reviews.apache.org/r/42601/#comment177806>
Instead of cloning and updating the eventDate in every destination (db/hdfs/solr/log4j), consider sending the "local" time (as set in AuthzAuditEvent.eventTime) to all audit destinations - exception DB, which require the time in UTC. For DB, consider updating the time in AuthzAuditEventDbObj(AuthzAuditEvent event) constructor.
- Madhan Neethiraj
On Jan. 27, 2016, 4:21 a.m., Gautam Borad wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42601/
> -----------------------------------------------------------
>
> (Updated Jan. 27, 2016, 4:21 a.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-798
> https://issues.apache.org/jira/browse/RANGER-798
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement**:
> The current implementation of “auditEvent.getEventTime()” contains time in UTC and since it's a date object it will contain component machine's local timezone. When Solr receives this date object and timezone, it tries to convert it from given timezone to UTC timestamp, which leads to double conversion of actual time before it get stored in Solr.
>
> **Proposed Solution**:
> If we can provide server local time and timezone to Solr then Solr will convert the received time from given timezone to UTC.
> As an alternate solution, replaced getUTCDate() with new Date() object at various places for audit event time, all audit destination will receive local Date object, for Solr there will be no conversion on received Date object but for all other audit destination we need to convert the received Date value to UTC timestamp as audit logs are being stored in UTC timestamp for all service/component. If all destination thread are enabled then changing the received event object may create issue in other audit destination as same event object is refferred everywhere. Hence received event object attributes value are being copied in another local event object and the updated event time can be stored there, after this local event object will be used to convert that in JSON to write in HDFS, or can be persisted in DB.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/destination/DBAuditDestination.java 376e724
> agents-audit/src/main/java/org/apache/ranger/audit/destination/FileAuditDestination.java c6cd8b2
> agents-audit/src/main/java/org/apache/ranger/audit/destination/HDFSAuditDestination.java 96755be
> agents-audit/src/main/java/org/apache/ranger/audit/destination/Log4JAuditDestination.java 9521a4a
> agents-audit/src/main/java/org/apache/ranger/audit/model/AuditEventBase.java 2a07e94
> agents-audit/src/main/java/org/apache/ranger/audit/provider/BaseAuditHandler.java 6717c92
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java 9586f73
> agents-common/src/main/java/org/apache/ranger/authorization/utils/StringUtil.java f6f3d2d
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java fe50ca6
> hbase-agent/src/main/java/org/apache/ranger/authorization/hbase/RangerAuthorizationCoprocessor.java 8762bf5
> hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java 5125af7
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java 2ae4149
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java 0f13577
> plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java bb6a337
> plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java 04b8b91
> plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java a8ecf15
> plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java ab9b7a9
>
> Diff: https://reviews.apache.org/r/42601/diff/
>
>
> Testing
> -------
>
> Steps performed(after patch) :
> 1) Changed plugin system time zone to IST and restarted all components.
> 2) Initiated an HDFS audit event.
> 3) Checked event time of newly created audit log in Solr, Audit log event time was matching with UTC.
> 4) Checked event time in Ranger UI, newly generated Audit event is matching with current time.
> 5) Checked event time of newly created audit log in xa_access_audit table, Audit log event time was matching with UTC.
> 6) Checked event time of newly created audit log in HDFS logs, Audit log event time was matching with UTC.
>
> Note: Will test other services audit logs after this approach is reviewed.
>
>
> Thanks,
>
> Gautam Borad
>
>
Re: Review Request 42601: RANGER-798 - Approach 1 : Handle different
timezone issue while saving audit logs to Solr
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42601/#review117191
-----------------------------------------------------------
Fix it, then Ship it!
Ok to commit after updating for the comment below.
agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java (line 149)
<https://reviews.apache.org/r/42601/#comment178294>
"event != null" - not necessary. Please remove.
- Madhan Neethiraj
On Jan. 28, 2016, 1:12 p.m., Gautam Borad wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42601/
> -----------------------------------------------------------
>
> (Updated Jan. 28, 2016, 1:12 p.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-798
> https://issues.apache.org/jira/browse/RANGER-798
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement**:
> The current implementation of “auditEvent.getEventTime()” contains time in UTC and since it's a date object it will contain component machine's local timezone. When Solr receives this date object and timezone, it tries to convert it from given timezone to UTC timestamp, which leads to double conversion of actual time before it get stored in Solr.
>
> **Proposed Solution**:
> If we can provide server local time and timezone to Solr then Solr will convert the received time from given timezone to UTC.
> As an alternate solution, replaced getUTCDate() with new Date() object at various places for audit event time, all audit destination will receive local Date object, for Solr there will be no conversion on received Date object but for all other audit destination we need to convert the received Date value to UTC timestamp as audit logs are being stored in UTC timestamp for all service/component. If all destination thread are enabled then changing the received event object may create issue in other audit destination as same event object is refferred everywhere. Hence received event object attributes value are being copied in another local event object and the updated event time can be stored there, after this local event object will be used to convert that in JSON to write in HDFS, or can be persisted in DB.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java d52a60a
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java 9586f73
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java fe50ca6
> hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java 5125af7
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java 2ae4149
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java 0f13577
> plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java bb6a337
> plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java 04b8b91
> plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java a8ecf15
> plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java ab9b7a9
>
> Diff: https://reviews.apache.org/r/42601/diff/
>
>
> Testing
> -------
>
> Steps performed(after patch) :
> 1) Changed plugin system time zone to IST and restarted all components.
> 2) Initiated an HDFS audit event.
> 3) Checked event time of newly created audit log in Solr, Audit log event time was matching with UTC.
> 4) Checked event time in Ranger UI, newly generated Audit event is matching with current time.
> 5) Checked event time of newly created audit log in xa_access_audit table, Audit log event time was matching with UTC.
> 6) Checked event time of newly created audit log in HDFS logs, Audit log event time was matching with UTC.
>
> Note: Will test other services audit logs after this approach is reviewed.
>
>
> Thanks,
>
> Gautam Borad
>
>
Re: Review Request 42601: RANGER-798 - Approach 1 : Handle different
timezone issue while saving audit logs to Solr
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42601/#review117272
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On Feb. 1, 2016, 12:37 p.m., Gautam Borad wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42601/
> -----------------------------------------------------------
>
> (Updated Feb. 1, 2016, 12:37 p.m.)
>
>
> Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-798
> https://issues.apache.org/jira/browse/RANGER-798
>
>
> Repository: ranger
>
>
> Description
> -------
>
> **Problem Statement**:
> The current implementation of “auditEvent.getEventTime()” contains time in UTC and since it's a date object it will contain component machine's local timezone. When Solr receives this date object and timezone, it tries to convert it from given timezone to UTC timestamp, which leads to double conversion of actual time before it get stored in Solr.
>
> **Proposed Solution**:
> If we can provide server local time and timezone to Solr then Solr will convert the received time from given timezone to UTC.
> As an alternate solution, replaced getUTCDate() with new Date() object at various places for audit event time, all audit destination will receive local Date object, for Solr there will be no conversion on received Date object but for all other audit destination we need to convert the received Date value to UTC timestamp as audit logs are being stored in UTC timestamp for all service/component. If all destination thread are enabled then changing the received event object may create issue in other audit destination as same event object is refferred everywhere. Hence received event object attributes value are being copied in another local event object and the updated event time can be stored there, after this local event object will be used to convert that in JSON to write in HDFS, or can be persisted in DB.
>
>
> Diffs
> -----
>
> agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java d52a60a
> agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java 9586f73
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java fe50ca6
> hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java 5125af7
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java 2ae4149
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java 0f13577
> plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java bb6a337
> plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java 04b8b91
> plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java a8ecf15
> plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java ab9b7a9
>
> Diff: https://reviews.apache.org/r/42601/diff/
>
>
> Testing
> -------
>
> Steps performed(after patch) :
> 1) Changed plugin system time zone to IST and restarted all components.
> 2) Initiated an HDFS audit event.
> 3) Checked event time of newly created audit log in Solr, Audit log event time was matching with UTC.
> 4) Checked event time in Ranger UI, newly generated Audit event is matching with current time.
> 5) Checked event time of newly created audit log in xa_access_audit table, Audit log event time was matching with UTC.
> 6) Checked event time of newly created audit log in HDFS logs, Audit log event time was matching with UTC.
>
> Note: Will test other services audit logs after this approach is reviewed.
>
>
> Thanks,
>
> Gautam Borad
>
>
Re: Review Request 42601: RANGER-798 - Approach 1 : Handle different
timezone issue while saving audit logs to Solr
Posted by Gautam Borad <gb...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42601/
-----------------------------------------------------------
(Updated Feb. 1, 2016, 12:37 p.m.)
Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
Changes
-------
Addressed review comments
Bugs: RANGER-798
https://issues.apache.org/jira/browse/RANGER-798
Repository: ranger
Description
-------
**Problem Statement**:
The current implementation of “auditEvent.getEventTime()” contains time in UTC and since it's a date object it will contain component machine's local timezone. When Solr receives this date object and timezone, it tries to convert it from given timezone to UTC timestamp, which leads to double conversion of actual time before it get stored in Solr.
**Proposed Solution**:
If we can provide server local time and timezone to Solr then Solr will convert the received time from given timezone to UTC.
As an alternate solution, replaced getUTCDate() with new Date() object at various places for audit event time, all audit destination will receive local Date object, for Solr there will be no conversion on received Date object but for all other audit destination we need to convert the received Date value to UTC timestamp as audit logs are being stored in UTC timestamp for all service/component. If all destination thread are enabled then changing the received event object may create issue in other audit destination as same event object is refferred everywhere. Hence received event object attributes value are being copied in another local event object and the updated event time can be stored there, after this local event object will be used to convert that in JSON to write in HDFS, or can be persisted in DB.
Diffs (updated)
-----
agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java d52a60a
agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java 9586f73
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java fe50ca6
hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java 5125af7
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java 2ae4149
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java 0f13577
plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java bb6a337
plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java 04b8b91
plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java a8ecf15
plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java ab9b7a9
Diff: https://reviews.apache.org/r/42601/diff/
Testing
-------
Steps performed(after patch) :
1) Changed plugin system time zone to IST and restarted all components.
2) Initiated an HDFS audit event.
3) Checked event time of newly created audit log in Solr, Audit log event time was matching with UTC.
4) Checked event time in Ranger UI, newly generated Audit event is matching with current time.
5) Checked event time of newly created audit log in xa_access_audit table, Audit log event time was matching with UTC.
6) Checked event time of newly created audit log in HDFS logs, Audit log event time was matching with UTC.
Note: Will test other services audit logs after this approach is reviewed.
Thanks,
Gautam Borad
Re: Review Request 42601: RANGER-798 - Approach 1 : Handle different
timezone issue while saving audit logs to Solr
Posted by Gautam Borad <gb...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42601/
-----------------------------------------------------------
(Updated Jan. 28, 2016, 1:12 p.m.)
Review request for ranger, Alok Lal, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, Ramesh Mani, Selvamohan Neethiraj, and Velmurugan Periasamy.
Changes
-------
Addressed review comments
Bugs: RANGER-798
https://issues.apache.org/jira/browse/RANGER-798
Repository: ranger
Description
-------
**Problem Statement**:
The current implementation of “auditEvent.getEventTime()” contains time in UTC and since it's a date object it will contain component machine's local timezone. When Solr receives this date object and timezone, it tries to convert it from given timezone to UTC timestamp, which leads to double conversion of actual time before it get stored in Solr.
**Proposed Solution**:
If we can provide server local time and timezone to Solr then Solr will convert the received time from given timezone to UTC.
As an alternate solution, replaced getUTCDate() with new Date() object at various places for audit event time, all audit destination will receive local Date object, for Solr there will be no conversion on received Date object but for all other audit destination we need to convert the received Date value to UTC timestamp as audit logs are being stored in UTC timestamp for all service/component. If all destination thread are enabled then changing the received event object may create issue in other audit destination as same event object is refferred everywhere. Hence received event object attributes value are being copied in another local event object and the updated event time can be stored there, after this local event object will be used to convert that in JSON to write in HDFS, or can be persisted in DB.
Diffs (updated)
-----
agents-audit/src/main/java/org/apache/ranger/audit/entity/AuthzAuditEventDbObj.java d52a60a
agents-audit/src/main/java/org/apache/ranger/audit/provider/MiscUtil.java 9586f73
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerAccessRequestImpl.java fe50ca6
hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java 5125af7
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAccessRequest.java 2ae4149
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuditHandler.java 0f13577
plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java bb6a337
plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java 04b8b91
plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java a8ecf15
plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java ab9b7a9
Diff: https://reviews.apache.org/r/42601/diff/
Testing
-------
Steps performed(after patch) :
1) Changed plugin system time zone to IST and restarted all components.
2) Initiated an HDFS audit event.
3) Checked event time of newly created audit log in Solr, Audit log event time was matching with UTC.
4) Checked event time in Ranger UI, newly generated Audit event is matching with current time.
5) Checked event time of newly created audit log in xa_access_audit table, Audit log event time was matching with UTC.
6) Checked event time of newly created audit log in HDFS logs, Audit log event time was matching with UTC.
Note: Will test other services audit logs after this approach is reviewed.
Thanks,
Gautam Borad