[jira] [Comment Edited] (RANGER-1501) Audit Flush to HDFS does not actually cause the audit logs to be flushed to HDFS

["Ramesh Mani (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/RANGER-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961462#comment-15961462 ] 

Ramesh Mani edited comment on RANGER-1501 at 4/7/17 9:27 PM:
-------------------------------------------------------------

 [~yzhou2001]This patch might need a good amount of testing especially KAFKA and HDFS ranger plugin where  there would be a continues logging happening and this doesn't get blocked in synchronized call to hflush() and when number of data nodes are more. 
Can we  come up with testing scenarios which can be done for this?
To address the loss of audit logs we already did an enhancement via https://issues.apache.org/jira/browse/RANGER-1310


was (Author: rmani):
[~ yzhou2001] This patch might need a good amount of testing especially KAFKA and HDFS ranger plugin where  there would be a continues logging happening and this doesn't get blocked in synchronized call to hflush() and when number of data nodes are more. 
Can we  come up with testing scenarios which can be done for this?
To address the loss of audit logs we already did an enhancement via https://issues.apache.org/jira/browse/RANGER-1310

> Audit Flush to HDFS does not actually cause the audit logs to be flushed to HDFS 
> ---------------------------------------------------------------------------------
>
>                 Key: RANGER-1501
>                 URL: https://issues.apache.org/jira/browse/RANGER-1501
>             Project: Ranger
>          Issue Type: Bug
>          Components: audit
>    Affects Versions: 0.7.0
>            Reporter: Yan
>            Assignee: Yan
>             Fix For: master
>
>         Attachments: 0001-RANGER-1501-Audit-Flush-to-HDFS-does-not-actually-ca.patch
>
>
> The reason is that HDFS file stream's flush() call does not really flush the data all the way to disk, nor even makes the data visible to HDFS users. See the HDFS semantics of the flush/sync at https://issues.apache.org/jira/browse/HADOOP-6313.
> Consequently the audit logs on HDFS won't be visible/durable from HDFS client until the log file is closed. This will, among other issues, boost chances of losing audit logs in case of system failure.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)