You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by navneet sharma <na...@gmail.com> on 2012/04/04 11:20:20 UTC
Dynamic weblog processing
Hi,
I am new to kafka. If i have an application which is generating logs and i
need to read and do some CEP on these logs.
For CEP, we are planning to use Esper and for reading weblogs, we are
examining Kafka. I have few questions:
1) What will be a producer in this case. Will it be an external
entity(standalone program) which will do file I/o and send weblogs to kafka
broker?
2) Or will it be part of application itself, so that it can read logs in
parallel with the logging utility?
Any pointers will be helpful...
Thanks,
Navneet
Re: Dynamic weblog processing
Posted by Jun Rao <ju...@gmail.com>.
Navneet,
Either way is fine. At LinkedIn, each app directly writes log data to a
kafka cluster using the producer api.
Thanks,
Jun
On Wed, Apr 4, 2012 at 2:20 AM, navneet sharma
<na...@gmail.com>wrote:
> Hi,
>
> I am new to kafka. If i have an application which is generating logs and i
> need to read and do some CEP on these logs.
>
> For CEP, we are planning to use Esper and for reading weblogs, we are
> examining Kafka. I have few questions:
> 1) What will be a producer in this case. Will it be an external
> entity(standalone program) which will do file I/o and send weblogs to kafka
> broker?
> 2) Or will it be part of application itself, so that it can read logs in
> parallel with the logging utility?
>
> Any pointers will be helpful...
>
> Thanks,
> Navneet
>
Re: Dynamic weblog processing
Posted by navneet sharma <na...@gmail.com>.
Thanks a lot for your responses.
I think i can start with prducer-in-app approach.
I saw the logger option as well- Kafka appender. Will it be more faster and
efficient? Anybody tried that?
On Fri, Apr 6, 2012 at 12:58 AM, Felix GV <fe...@mate1inc.com> wrote:
> As Hisham mentioned, what I've been working on is your option #2, and that
> can be done by using the Kafka APIs...
>
> Currently, the easiest way to get this up and running quickly would
> probably be your option #1, using the kafka-console-producer that was added
> by the kind Kafka folks in KAFKA-130 :)
>
> (It's available in 0.7)
>
> --
> Felix
>
>
>
> On Wed, Apr 4, 2012 at 9:56 PM, Hisham Mardam-Bey <hisham@mate1inc.com
> >wrote:
>
> > On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
> > <na...@gmail.com> wrote:
> > > Hi,
> > >
> > > I am new to kafka. If i have an application which is generating logs
> and
> > i
> > > need to read and do some CEP on these logs.
> > >
> > > For CEP, we are planning to use Esper and for reading weblogs, we are
> > > examining Kafka. I have few questions:
> > > 1) What will be a producer in this case. Will it be an external
> > > entity(standalone program) which will do file I/o and send weblogs to
> > kafka
> > > broker?
> > > 2) Or will it be part of application itself, so that it can read logs
> in
> > > parallel with the logging utility?
> > >
> > > Any pointers will be helpful...
> > >
> > > Thanks,
> > > Navneet
> >
> > Navneet,
> >
> > We're currently doing that for our Tomcat servers. We've built an
> > access log valve (inspired by the one bundled with Tomcat that writes
> > log files to disk) that will format logs and push them into Kafka. We
> > then pull the logs and push them into HBase for analysis. FelixGV (on
> > this list) is implementing the code if you have questions for him.
> >
> > Hope this helps,
> >
> > hisham.
> >
> > --
> > Hisham Mardam Bey
> >
> > A: Because it messes up the order in which people normally read text.
> > Q: Why is top-posting such a bad thing?
> > A: Top-posting.
> > Q: What is the most annoying thing in e-mail?
> >
> > -=[ Codito Ergo Sum ]=-
> >
>
Re: Dynamic weblog processing
Posted by Felix GV <fe...@mate1inc.com>.
As Hisham mentioned, what I've been working on is your option #2, and that
can be done by using the Kafka APIs...
Currently, the easiest way to get this up and running quickly would
probably be your option #1, using the kafka-console-producer that was added
by the kind Kafka folks in KAFKA-130 :)
(It's available in 0.7)
--
Felix
On Wed, Apr 4, 2012 at 9:56 PM, Hisham Mardam-Bey <hi...@mate1inc.com>wrote:
> On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
> <na...@gmail.com> wrote:
> > Hi,
> >
> > I am new to kafka. If i have an application which is generating logs and
> i
> > need to read and do some CEP on these logs.
> >
> > For CEP, we are planning to use Esper and for reading weblogs, we are
> > examining Kafka. I have few questions:
> > 1) What will be a producer in this case. Will it be an external
> > entity(standalone program) which will do file I/o and send weblogs to
> kafka
> > broker?
> > 2) Or will it be part of application itself, so that it can read logs in
> > parallel with the logging utility?
> >
> > Any pointers will be helpful...
> >
> > Thanks,
> > Navneet
>
> Navneet,
>
> We're currently doing that for our Tomcat servers. We've built an
> access log valve (inspired by the one bundled with Tomcat that writes
> log files to disk) that will format logs and push them into Kafka. We
> then pull the logs and push them into HBase for analysis. FelixGV (on
> this list) is implementing the code if you have questions for him.
>
> Hope this helps,
>
> hisham.
>
> --
> Hisham Mardam Bey
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>
Re: Dynamic weblog processing
Posted by Hisham Mardam-Bey <hi...@mate1inc.com>.
On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
<na...@gmail.com> wrote:
> Hi,
>
> I am new to kafka. If i have an application which is generating logs and i
> need to read and do some CEP on these logs.
>
> For CEP, we are planning to use Esper and for reading weblogs, we are
> examining Kafka. I have few questions:
> 1) What will be a producer in this case. Will it be an external
> entity(standalone program) which will do file I/o and send weblogs to kafka
> broker?
> 2) Or will it be part of application itself, so that it can read logs in
> parallel with the logging utility?
>
> Any pointers will be helpful...
>
> Thanks,
> Navneet
Navneet,
We're currently doing that for our Tomcat servers. We've built an
access log valve (inspired by the one bundled with Tomcat that writes
log files to disk) that will format logs and push them into Kafka. We
then pull the logs and push them into HBase for analysis. FelixGV (on
this list) is implementing the code if you have questions for him.
Hope this helps,
hisham.
--
Hisham Mardam Bey
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?
-=[ Codito Ergo Sum ]=-