Syslog-ng, Flume (old gen) and metadata

[Bertrand Dechoux <de...@gmail.com>]

Hi,

I would like to deepen my understanding of syslog-ng and flume integration.

I had initial three questions :
1) Does flume timestamp is the same as the received syslog-ng event
timestamp?
2) Are the syslog-ng metadata kept by flume as extra metadata (> syslog-ng
OSE 3.0)?
3) Are the extra metadata (and potientially syslog-ng ones) usable with the
collector sink template?

>From what I understand :
1) Yes, acoording the cookbook. "use the timestamp found in the original
data,"
http://archive.cloudera.com/cdh/3/flume/Cookbook/
2) No but I haven't found a clear answer about it.
3) No but I haven't found a clear answer about it.

Could someone confirm me that?
I would believe 2) and 3) be generally asked features for those integrating
existent syslog-ng system with Flume but maybe it isn't. Or maybe there are
third party libraries for that? What is the status of Flume-ng with regards
to the same questions?

Regards

Bertrand

Re: Syslog-ng, Flume (old gen) and metadata

[Brock Noland <br...@cloudera.com>]

Hi,

I don't actually know the answer to your question, hopefully someone
else will help with that, but I have some hopefully useful comments
below.

On Mon, Oct 29, 2012 at 10:51 AM, Bertrand Dechoux <de...@gmail.com> wrote:
> Hi,
>
> I would like to deepen my understanding of syslog-ng and flume integration.
>
> I had initial three questions :
> 1) Does flume timestamp is the same as the received syslog-ng event
> timestamp?
> 2) Are the syslog-ng metadata kept by flume as extra metadata (> syslog-ng
> OSE 3.0)?
> 3) Are the extra metadata (and potientially syslog-ng ones) usable with the
> collector sink template?
>
> From what I understand :
> 1) Yes, acoording the cookbook. "use the timestamp found in the original
> data,"
> http://archive.cloudera.com/cdh/3/flume/Cookbook/

That is Flume OG which is still available but as far as I am aware no
one is actively taking that forward. Flume NG is actively being worked
on.  The Cloudera link for NG is here:
http://archive.cloudera.com/cdh/3/flume-ng-1.2.0-cdh3u5/ and the
apache webpage for flume is all NG content as far as I am aware:

http://flume.apache.org/

> 2) No but I haven't found a clear answer about it.
> 3) No but I haven't found a clear answer about it.
>
> Could someone confirm me that?
> I would believe 2) and 3) be generally asked features for those integrating
> existent syslog-ng system with Flume but maybe it isn't. Or maybe there are
> third party libraries for that? What is the status of Flume-ng with regards
> to the same questions?
>
> Regards
>
> Bertrand



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/