[jira] [Resolved] (ARGUS-5) Ability to write audit log in HDFS

["Ramesh Mani (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARGUS-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramesh Mani resolved ARGUS-5.
-----------------------------
       Resolution: Fixed
    Fix Version/s: 0.1.0

Patch available for review

> Ability to write audit log in HDFS
> ----------------------------------
>
>                 Key: ARGUS-5
>                 URL: https://issues.apache.org/jira/browse/ARGUS-5
>             Project: Argus
>          Issue Type: New Feature
>            Reporter: Selvamohan Neethiraj
>            Assignee: Ramesh Mani
>             Fix For: 0.1.0
>
>
> {panel:title=Ability to write Logs into HDFS}
> HdfsFileAppender is log4J appender  used to log into hdfs the logs.
> Following are configuration parameters.
> # HDFS appender
> #
> hdfs.xaaudit.logger=INFO,console,HDFSLOG
> log4j.logger.xaaudit=${hdfs.xaaudit.logger}
> log4j.additivity.xaaudit=false
> log4j.appender.HDFSLOG=com.xasecure.authorization.hadoop.log.HdfsFileAppender
> log4j.appender.HDFSLOG.File=/grid/0/var/log/hadoop/hdfs/argus_audit.log
> log4j.appender.HDFSLOG.HdfsDestination=hdfs://ec2-54-88-128.112.compute.1.amazonaws.com:8020:/audit/hdfs/%hostname%/argus_audit.log
> log4j.appender.HDFSLOG.layout=org.apache.log4j.PatternLayout
> log4j.appender.HDFSLOG.layout.ConversionPattern=%d\X{ISO8601} %p %c{2}: %m%n %X{LogPath}
> HdfsFileRollingInterval -> Hdfs file Rollover time e.g 1min, 5min,... 1hr, 2hrs,.. 1day, 2days... 1week, 2weeks.. 1month, 2months...
> log4j.appender.HDFSLOG.HdfsFileRollingInterval=3min
> LocalFileRollingInterval -> Hdfs file Rollover time e.g 1min, 5min,... 1hr, 2hrs,.. 1day, 2days... 1week, 2weeks.. 1month, 2months..
> log4j.appender.HDFSLOG.FileRollingInterval=1min
> log4j.appender.HDFSLOG.HdfsLiveUpdate=true
> log4j.appender.HDFSLOG.HdfsCheckInterval=2min
> 1)	HdfsFileAppender will log into given HDFSDestination Path.
> 2)	Incase of unavailability of configured hdfs, a Local file in the given log4j parameter FILE will be created with extension. cache.
> 3)	This local .cache file will be rolled over based on the FileRollingInterval parameter.
> 4)	 Once when the hdfs is available and ready, logging will be done in the HDFSDestination provided.
> 5)	Local .cache file will be moved into HDFSDestination.
> 6)	Log File created in the hdfs destination will be rolled over based on the HdfsFileRollingInterval parameter
> 7)	Parameter HdfsLiveUpdate = True mean when ever the hdfs is available appender will send the logs to hdfsfile. If False Local .cache file will be created and these files will be moved periodically into HDFSDestination
> 8)	Parameter HdfsCheckInterval is the interval to check for the availability of HDFS after the first failure. It that time local .cache file will hold the logs.
>   Argus Audit Logging into HDFS:
> 	.  For Audit logs “Policy Manager” should exclude the hdfs file Path from auditing to avoid recursive call that is there when logging the audit.
> 	.  Configure log4j parameter in the xasecure-audit.xml. Make it Async.
> 	  (Note that each agent will have its own xasecure-aduit.xml ) properties.
> 	. For Auditing Hdfs Agent, have the appender part of NameNode and SecondaryNamenode.
>          . For Auditing Hbase Agent , have the appender part of  Master and RegionServer.
> 	. For Auditing Hive Agent have it part of the HiverServer2
>           Regular Logging Usage:
> 		For regular functionality of enabling the logging ,  do the same way other Log4J appenders are configured.
> {panel}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)