You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Scott Chapman <sc...@woofplanet.com> on 2015/02/22 22:36:29 UTC
Anyone using log4j Appender for Kafka?
I am just starting to use it and could use a little guidance. I was able to
get it working with 0.8.2 but am not clear on best practices for using it.
Anyway willing to help me out a bit? Got a few questions, like how to
protect applications from when kafka is down or unreachable.
It seems like a great idea for being able to get logs from existing
applications to be collected by kafka.
Thanks in advance!
Re: Anyone using log4j Appender for Kafka?
Posted by Joe Stein <jo...@stealth.ly>.
Here is sample log4j.properties
https://github.com/stealthly/scala-kafka/blob/master/src/test/resources/log4j.properties#L54-L67
I _almost_ have always pulled the class
https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/producer/KafkaLog4jAppender.scala
internal
to private repo and changed it as things came up... e.g. setSource(),
setTags() blah blah...
Paul Otto has an open source version
https://github.com/potto007/kafka-appender-layout that you could try out
too that he built to tackle some of the layout things.
~ Joe Stein
- - - - - - - - - - - - - - - - -
http://www.stealth.ly
- - - - - - - - - - - - - - - - -
On Mon, Feb 23, 2015 at 4:42 PM, Alex Melville <am...@g.hmc.edu> wrote:
> ^^ I would really appreciate this as well. It's unclear how to get log4j
> working with Kafka when you have no prior experience with log4j.
>
> On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman <sc...@woofplanet.com>
> wrote:
>
> > Thanks. But we're using log4j. I tried setting the type to async but it
> > generated a warning of no such field. Is there any real documentation on
> > the log4j appender?
> >
> > On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker <
> > sschlansker@opentable.com> wrote:
> >
> > > We just configure our logback.xml to have two Appenders, an
> AsyncAppender
> > > -> KafkaAppender, and FileAppender (or ConsoleAppender as appropriate).
> > >
> > > AsyncAppender removes more failure cases too, e.g. a health check
> hanging
> > > rather than returning rapidly could block you application.
> > >
> > > On Feb 22, 2015, at 11:26 PM, anthony musyoki <
> anthony.musyoki@gmail.com
> > >
> > > wrote:
> > >
> > > > Theres also another one here.
> > > >
> > > > https://github.com/danielwegener/logback-kafka-appender.
> > > >
> > > > It has a fallback appender which might address the issue of Kafka
> being
> > > > un-available.
> > > >
> > > >
> > > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker <
> > > > sschlansker@opentable.com> wrote:
> > > >
> > > >> Here’s my attempt at a Logback version, should be fairly easily
> > ported:
> > > >>
> > > >> https://github.com/opentable/otj-logging/blob/master/kafka/
> > > src/main/java/com/opentable/logging/KafkaAppender.java
> > > >>
> > > >> On Feb 22, 2015, at 1:36 PM, Scott Chapman <sc...@woofplanet.com>
> > > wrote:
> > > >>
> > > >>> I am just starting to use it and could use a little guidance. I was
> > > able
> > > >> to
> > > >>> get it working with 0.8.2 but am not clear on best practices for
> > using
> > > >> it.
> > > >>>
> > > >>> Anyway willing to help me out a bit? Got a few questions, like how
> to
> > > >>> protect applications from when kafka is down or unreachable.
> > > >>>
> > > >>> It seems like a great idea for being able to get logs from existing
> > > >>> applications to be collected by kafka.
> > > >>>
> > > >>> Thanks in advance!
> > > >>
> > > >>
> > >
> > >
> >
>
Re: Anyone using log4j Appender for Kafka?
Posted by Alex Melville <am...@g.hmc.edu>.
^^ I would really appreciate this as well. It's unclear how to get log4j
working with Kafka when you have no prior experience with log4j.
On Mon, Feb 23, 2015 at 4:39 AM, Scott Chapman <sc...@woofplanet.com> wrote:
> Thanks. But we're using log4j. I tried setting the type to async but it
> generated a warning of no such field. Is there any real documentation on
> the log4j appender?
>
> On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker <
> sschlansker@opentable.com> wrote:
>
> > We just configure our logback.xml to have two Appenders, an AsyncAppender
> > -> KafkaAppender, and FileAppender (or ConsoleAppender as appropriate).
> >
> > AsyncAppender removes more failure cases too, e.g. a health check hanging
> > rather than returning rapidly could block you application.
> >
> > On Feb 22, 2015, at 11:26 PM, anthony musyoki <anthony.musyoki@gmail.com
> >
> > wrote:
> >
> > > Theres also another one here.
> > >
> > > https://github.com/danielwegener/logback-kafka-appender.
> > >
> > > It has a fallback appender which might address the issue of Kafka being
> > > un-available.
> > >
> > >
> > > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker <
> > > sschlansker@opentable.com> wrote:
> > >
> > >> Here’s my attempt at a Logback version, should be fairly easily
> ported:
> > >>
> > >> https://github.com/opentable/otj-logging/blob/master/kafka/
> > src/main/java/com/opentable/logging/KafkaAppender.java
> > >>
> > >> On Feb 22, 2015, at 1:36 PM, Scott Chapman <sc...@woofplanet.com>
> > wrote:
> > >>
> > >>> I am just starting to use it and could use a little guidance. I was
> > able
> > >> to
> > >>> get it working with 0.8.2 but am not clear on best practices for
> using
> > >> it.
> > >>>
> > >>> Anyway willing to help me out a bit? Got a few questions, like how to
> > >>> protect applications from when kafka is down or unreachable.
> > >>>
> > >>> It seems like a great idea for being able to get logs from existing
> > >>> applications to be collected by kafka.
> > >>>
> > >>> Thanks in advance!
> > >>
> > >>
> >
> >
>
Re: Anyone using log4j Appender for Kafka?
Posted by Scott Chapman <sc...@woofplanet.com>.
Thanks. But we're using log4j. I tried setting the type to async but it
generated a warning of no such field. Is there any real documentation on
the log4j appender?
On Mon Feb 23 2015 at 2:58:54 AM Steven Schlansker <
sschlansker@opentable.com> wrote:
> We just configure our logback.xml to have two Appenders, an AsyncAppender
> -> KafkaAppender, and FileAppender (or ConsoleAppender as appropriate).
>
> AsyncAppender removes more failure cases too, e.g. a health check hanging
> rather than returning rapidly could block you application.
>
> On Feb 22, 2015, at 11:26 PM, anthony musyoki <an...@gmail.com>
> wrote:
>
> > Theres also another one here.
> >
> > https://github.com/danielwegener/logback-kafka-appender.
> >
> > It has a fallback appender which might address the issue of Kafka being
> > un-available.
> >
> >
> > On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker <
> > sschlansker@opentable.com> wrote:
> >
> >> Here’s my attempt at a Logback version, should be fairly easily ported:
> >>
> >> https://github.com/opentable/otj-logging/blob/master/kafka/
> src/main/java/com/opentable/logging/KafkaAppender.java
> >>
> >> On Feb 22, 2015, at 1:36 PM, Scott Chapman <sc...@woofplanet.com>
> wrote:
> >>
> >>> I am just starting to use it and could use a little guidance. I was
> able
> >> to
> >>> get it working with 0.8.2 but am not clear on best practices for using
> >> it.
> >>>
> >>> Anyway willing to help me out a bit? Got a few questions, like how to
> >>> protect applications from when kafka is down or unreachable.
> >>>
> >>> It seems like a great idea for being able to get logs from existing
> >>> applications to be collected by kafka.
> >>>
> >>> Thanks in advance!
> >>
> >>
>
>
Re: Anyone using log4j Appender for Kafka?
Posted by Steven Schlansker <ss...@opentable.com>.
We just configure our logback.xml to have two Appenders, an AsyncAppender -> KafkaAppender, and FileAppender (or ConsoleAppender as appropriate).
AsyncAppender removes more failure cases too, e.g. a health check hanging rather than returning rapidly could block you application.
On Feb 22, 2015, at 11:26 PM, anthony musyoki <an...@gmail.com> wrote:
> Theres also another one here.
>
> https://github.com/danielwegener/logback-kafka-appender.
>
> It has a fallback appender which might address the issue of Kafka being
> un-available.
>
>
> On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker <
> sschlansker@opentable.com> wrote:
>
>> Here’s my attempt at a Logback version, should be fairly easily ported:
>>
>> https://github.com/opentable/otj-logging/blob/master/kafka/src/main/java/com/opentable/logging/KafkaAppender.java
>>
>> On Feb 22, 2015, at 1:36 PM, Scott Chapman <sc...@woofplanet.com> wrote:
>>
>>> I am just starting to use it and could use a little guidance. I was able
>> to
>>> get it working with 0.8.2 but am not clear on best practices for using
>> it.
>>>
>>> Anyway willing to help me out a bit? Got a few questions, like how to
>>> protect applications from when kafka is down or unreachable.
>>>
>>> It seems like a great idea for being able to get logs from existing
>>> applications to be collected by kafka.
>>>
>>> Thanks in advance!
>>
>>
Re: Anyone using log4j Appender for Kafka?
Posted by anthony musyoki <an...@gmail.com>.
Theres also another one here.
https://github.com/danielwegener/logback-kafka-appender.
It has a fallback appender which might address the issue of Kafka being
un-available.
On Mon, Feb 23, 2015 at 9:45 AM, Steven Schlansker <
sschlansker@opentable.com> wrote:
> Here’s my attempt at a Logback version, should be fairly easily ported:
>
> https://github.com/opentable/otj-logging/blob/master/kafka/src/main/java/com/opentable/logging/KafkaAppender.java
>
> On Feb 22, 2015, at 1:36 PM, Scott Chapman <sc...@woofplanet.com> wrote:
>
> > I am just starting to use it and could use a little guidance. I was able
> to
> > get it working with 0.8.2 but am not clear on best practices for using
> it.
> >
> > Anyway willing to help me out a bit? Got a few questions, like how to
> > protect applications from when kafka is down or unreachable.
> >
> > It seems like a great idea for being able to get logs from existing
> > applications to be collected by kafka.
> >
> > Thanks in advance!
>
>
Re: Anyone using log4j Appender for Kafka?
Posted by Steven Schlansker <ss...@opentable.com>.
Here’s my attempt at a Logback version, should be fairly easily ported:
https://github.com/opentable/otj-logging/blob/master/kafka/src/main/java/com/opentable/logging/KafkaAppender.java
On Feb 22, 2015, at 1:36 PM, Scott Chapman <sc...@woofplanet.com> wrote:
> I am just starting to use it and could use a little guidance. I was able to
> get it working with 0.8.2 but am not clear on best practices for using it.
>
> Anyway willing to help me out a bit? Got a few questions, like how to
> protect applications from when kafka is down or unreachable.
>
> It seems like a great idea for being able to get logs from existing
> applications to be collected by kafka.
>
> Thanks in advance!