Kafka syslog producer

[Flinkster <fl...@gmail.com>]

Hi there,

Couple of questions:

1) Is there a standard way to push syslog messages into a Kafka
broker? I'd need a producer on a central syslog server that converts
syslog into kafka messages.

2) How about tailing a file to a central logging like you can do with
scribe/flume agents?

Thanks,

-raaka

Re: Kafka syslog producer

[Jay Kreps <ja...@gmail.com>]

At LinkedIn we do this from within the application. Our reason is that
we send all messsages as structured Avro, which allows us to
automatically carry this schema through to Hadoop/Hive/Pig and live
consumers. In other words we don't use syslog.

However, since much of the world uses syslog not our custom avro stuff
it would be great to have some kind of syslog support. If anyone is
interested in contributing such a thing we would be very interested to
take it. Not sure if folks have ideas on the best way to support it...

-Jay

On Fri, May 4, 2012 at 2:29 AM, navneet sharma
<na...@gmail.com> wrote:
> I am working on a similar use-case.
> I think there are 2 approaches which you can think of:
> 1) Application generating syslog should have a parallel producer to send
> syslog to kafka broker.
> 2) try the kafka log4j appender, but again it will be in parallel with
> syslogs
>
> Alternatively, i am trying with file APIs - infinite while loop on the
> syslog file, but that is very inefficient.
>
> -navneet sharma
>
> On Tue, Apr 24, 2012 at 1:31 AM, Flinkster <fl...@gmail.com> wrote:
>
>> How does LinkedIn handle syslog? Is Kafka at LinkedIN dedicated to
>> application logs & metrics and syslog messages (e.g. network, OS logs)
>> is a separate flow and datastore?
>>
>> On Mon, Apr 23, 2012 at 11:26 AM, Joel Koshy <jj...@gmail.com> wrote:
>> > This is an interesting use-case and similar to using the log4j appender
>> for
>> > application logs. What you describe sounds reasonable: i.e., have a
>> > producer process on the syslog server to send syslog messages to your
>> Kafka
>> > brokers.
>> >
>> >
>> >> 2) How about tailing a file to a central logging like you can do with
>> >> scribe/flume agents?
>> >>
>> >
>> > You could use tail and pipe (or a named pipe) to a console-based
>> producer.
>> > I don't know enough about syslog.d to tell if this would be too much of a
>> > hack or not, but I think it would be more reliable than using tail+pipes:
>> > you could configure your local machines to direct syslog messages to the
>> > remote server and you could write a simple component that listens on the
>> > syslog socket and redirects incoming messages to a Kafka producer.
>> >
>> > Joel
>>

Re: Kafka syslog producer

[navneet sharma <na...@gmail.com>]

I am working on a similar use-case.
I think there are 2 approaches which you can think of:
1) Application generating syslog should have a parallel producer to send
syslog to kafka broker.
2) try the kafka log4j appender, but again it will be in parallel with
syslogs

Alternatively, i am trying with file APIs - infinite while loop on the
syslog file, but that is very inefficient.

-navneet sharma

On Tue, Apr 24, 2012 at 1:31 AM, Flinkster <fl...@gmail.com> wrote:

> How does LinkedIn handle syslog? Is Kafka at LinkedIN dedicated to
> application logs & metrics and syslog messages (e.g. network, OS logs)
> is a separate flow and datastore?
>
> On Mon, Apr 23, 2012 at 11:26 AM, Joel Koshy <jj...@gmail.com> wrote:
> > This is an interesting use-case and similar to using the log4j appender
> for
> > application logs. What you describe sounds reasonable: i.e., have a
> > producer process on the syslog server to send syslog messages to your
> Kafka
> > brokers.
> >
> >
> >> 2) How about tailing a file to a central logging like you can do with
> >> scribe/flume agents?
> >>
> >
> > You could use tail and pipe (or a named pipe) to a console-based
> producer.
> > I don't know enough about syslog.d to tell if this would be too much of a
> > hack or not, but I think it would be more reliable than using tail+pipes:
> > you could configure your local machines to direct syslog messages to the
> > remote server and you could write a simple component that listens on the
> > syslog socket and redirects incoming messages to a Kafka producer.
> >
> > Joel
>

Re: Kafka syslog producer

[Flinkster <fl...@gmail.com>]

How does LinkedIn handle syslog? Is Kafka at LinkedIN dedicated to
application logs & metrics and syslog messages (e.g. network, OS logs)
is a separate flow and datastore?

On Mon, Apr 23, 2012 at 11:26 AM, Joel Koshy <jj...@gmail.com> wrote:
> This is an interesting use-case and similar to using the log4j appender for
> application logs. What you describe sounds reasonable: i.e., have a
> producer process on the syslog server to send syslog messages to your Kafka
> brokers.
>
>
>> 2) How about tailing a file to a central logging like you can do with
>> scribe/flume agents?
>>
>
> You could use tail and pipe (or a named pipe) to a console-based producer.
> I don't know enough about syslog.d to tell if this would be too much of a
> hack or not, but I think it would be more reliable than using tail+pipes:
> you could configure your local machines to direct syslog messages to the
> remote server and you could write a simple component that listens on the
> syslog socket and redirects incoming messages to a Kafka producer.
>
> Joel

Re: Kafka syslog producer

[Joel Koshy <jj...@gmail.com>]

This is an interesting use-case and similar to using the log4j appender for
application logs. What you describe sounds reasonable: i.e., have a
producer process on the syslog server to send syslog messages to your Kafka
brokers.


> 2) How about tailing a file to a central logging like you can do with
> scribe/flume agents?
>

You could use tail and pipe (or a named pipe) to a console-based producer.
I don't know enough about syslog.d to tell if this would be too much of a
hack or not, but I think it would be more reliable than using tail+pipes:
you could configure your local machines to direct syslog messages to the
remote server and you could write a simple component that listens on the
syslog socket and redirects incoming messages to a Kafka producer.

Joel