Syslog-ng and hadoop logs

[Joe Hammerman <jh...@videoegg.com>]

Hi all,

	We are working on integrating our hadoop logs which live on the datanodes into a central syslog-ng logging server. There seems to be a dearth of information available on the internet as how to effect this; I was hoping someone here could help out, either with a link or by providing examples from their configuration.

	Thanks in advance!

Regards,
	Joseph Hammerman

Re: Syslog-ng and hadoop logs

[Brian Bockelman <bb...@cse.unl.edu>]

Hey Joe,

We added the following to log4j.properties:

log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.facility=local0
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.ConversionPattern=%p %c{2}: %m%n
log4j.appender.SYSLOG.SyslogHost=red
log4j.appender.SYSLOG.threshold=ERROR
log4j.appender.SYSLOG.Header=true
log4j.appender.SYSLOG.FacilityPrinting=true

IIRC, if you don't have have .Header=true, the generated packets are  
invalid and syslog chokes on them.  Fun!

Then, we add whatever we'd like to go to syslog; i.e.

log4j.logger.org.apache.hadoop.hdfs=INFO,SYSLOGM

or

log4j.rootLogger=DEBUG,SYSLOG,console,DRFA,EventCounter

Brian

On Oct 22, 2009, at 7:01 PM, Joe Hammerman wrote:

> Hi all,
>
> 	We are working on integrating our hadoop logs which live on the  
> datanodes into a central syslog-ng logging server. There seems to be  
> a dearth of information available on the internet as how to effect  
> this; I was hoping someone here could help out, either with a link  
> or by providing examples from their configuration.
>
> 	Thanks in advance!
>
> Regards,
> 	Joseph Hammerman