You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Felix Giguere Villegas <fe...@mate1inc.com> on 2011/09/01 21:20:58 UTC
Tail or stdin producer?
Hi,
We are currently evaluating Kafka, so I'm trying to get one of our simple
use case working. I have read the design paper and most of everything else I
could find about Kafka, but I'm not sure how to set up a producer that tails
files or that reads from STDIN.
Do I have to write custom code just for that? It seems like a pretty
simple/common use case, so I would think it already exists out of the box.
I guess I could pipe a tail command into bin/kafka-producer-shell.sh ??
This kind of stuff is relatively easy to set up with Flume, but we like the
persistence and replay capabilities of Kafka, which is why we are trying it
out.
Thanks in advance for helping out a newb ;) !
--
Felix
Re: Tail or stdin producer?
Posted by Joel Koshy <jj...@gmail.com>.
Thanks for filing that. Piping tail into producer-shell will also
work, although the producer-shell is only on a per-broker basis. It
would be good to have a high-level producer shell.
Joel
On Thu, Sep 1, 2011 at 1:22 PM, Felix Giguere Villegas
<fe...@mate1inc.com> wrote:
> Hi :)
>
> Thanks for the reply :) . I have created
> KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130>
> .
>
> I'm not sure I have respected the proper standard for JIRA issues. Please
> let me know if I haven't.
>
> --
> Felix
>
>
>
> On Thu, Sep 1, 2011 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
>
>> Hi, Felix,
>>
>> Currently, we don't have a util to pipe data from a file to a producer. I
>> agree that it's a very convenient tool to have. Could you open a jira for
>> that?
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Sep 1, 2011 at 12:20 PM, Felix Giguere Villegas <
>> felix.giguere@mate1inc.com> wrote:
>>
>> > Hi,
>> >
>> > We are currently evaluating Kafka, so I'm trying to get one of our simple
>> > use case working. I have read the design paper and most of everything
>> else
>> > I
>> > could find about Kafka, but I'm not sure how to set up a producer that
>> > tails
>> > files or that reads from STDIN.
>> >
>> > Do I have to write custom code just for that? It seems like a pretty
>> > simple/common use case, so I would think it already exists out of the
>> box.
>> >
>> > I guess I could pipe a tail command into bin/kafka-producer-shell.sh ??
>> >
>> > This kind of stuff is relatively easy to set up with Flume, but we like
>> the
>> > persistence and replay capabilities of Kafka, which is why we are trying
>> it
>> > out.
>> >
>> > Thanks in advance for helping out a newb ;) !
>> >
>> > --
>> > Felix
>> >
>>
>
Re: Tail or stdin producer?
Posted by Bao Thai Ngo <ba...@gmail.com>.
Thanks for the reply Joel. Please see my answer below
On Tue, Oct 4, 2011 at 1:00 AM, Joel Koshy <jj...@gmail.com> wrote:
> > I am totally new to Kafka and we are evaluating Kafka like Felix. I'm
> > looking for a utility to pipe data from a file (or multiple files) to a
> > producer (like command tail -f in linux). This is an our typical
> use-case. I
> > saw the patch (KAFKA-130) you provided but not sure if this will work
> well
> > without loosing data if the file (or multiple files) will be rotated.
>
> Would "tail -F --retry" work for you? Also, if these are log files and
> you are using log4j, then kafka has a log4j appender that you can use
> to produce messages.
>
>
I believe tail would work but will not be convenient when people have
windows box (not always able to use similar utility for windows).
Also, log files are produced from variety of services including non-java
applications
> >
> > I would care about plugin that will such as filter message at producer;
> > modify message body in order to mark some critical kind of information,
> etc.
> > It would be good if you provide documentation about how to write a plugin
> > for Kafka.
>
> You can use async producer callback handlers to do this sort of
> filtering. I don't see an example in the distribution, but it is
> pretty straightforward: see
>
> https://svn.apache.org/repos/asf/incubator/kafka/trunk/core/src/main/scala/kafka/javaapi/producer/async/CallbackHandler.java
> (The kafka log4j producer currently uses the syncproducer, so you
> won't be able to use callback handlers with it).
>
> great ! I will look at this.
BTW, do you know is there any documentation like user guide or something
similar? Quickstart from Kafka does not help much.
> Thanks,
>
> Joel
>
> Thanks,
~Thai
> >
> > On Sun, Oct 2, 2011 at 9:51 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> Currently all the patch does is take input from stdin. Can you help me
> >> understand what you are looking for?
> >>
> >> What kind of plugin are you trying to write?
> >>
> >> -Jay
> >>
> >>
> >> On Thu, Sep 29, 2011 at 10:41 PM, Bao Thai Ngo <baothaingo@gmail.com
> >wrote:
> >>
> >>> Jay,
> >>>
> >>> Does the patch already support log rotation (file rolling)?
> >>>
> >>> Also, I wonder if there is any documentation about how to write a
> plug-in
> >>> for Kafka.
> >>>
> >>> Thanks,
> >>> ~Thai
> >>>
> >>> On Mon, Sep 12, 2011 at 10:17 AM, Jay Kreps <ja...@gmail.com>
> wrote:
> >>>
> >>> > I added a patch for this:
> >>> > https://issues.apache.org/jira/browse/KAFKA-130
> >>> >
> >>> > Felix, if you are still interested in this, let me know if it works
> for
> >>> > you.
> >>> >
> >>> > -Jay
> >>> >
> >>> > On Thu, Sep 1, 2011 at 1:22 PM, Felix Giguere Villegas <
> >>> > felix.giguere@mate1inc.com> wrote:
> >>> >
> >>> > > Hi :)
> >>> > >
> >>> > > Thanks for the reply :) . I have created
> >>> > > KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130>
> >>> > > .
> >>> > >
> >>> > > I'm not sure I have respected the proper standard for JIRA issues.
> >>> Please
> >>> > > let me know if I haven't.
> >>> > >
> >>> > > --
> >>> > > Felix
>
Re: Tail or stdin producer?
Posted by Joel Koshy <jj...@gmail.com>.
> I am totally new to Kafka and we are evaluating Kafka like Felix. I'm
> looking for a utility to pipe data from a file (or multiple files) to a
> producer (like command tail -f in linux). This is an our typical use-case. I
> saw the patch (KAFKA-130) you provided but not sure if this will work well
> without loosing data if the file (or multiple files) will be rotated.
Would "tail -F --retry" work for you? Also, if these are log files and
you are using log4j, then kafka has a log4j appender that you can use
to produce messages.
>
> I would care about plugin that will such as filter message at producer;
> modify message body in order to mark some critical kind of information, etc.
> It would be good if you provide documentation about how to write a plugin
> for Kafka.
You can use async producer callback handlers to do this sort of
filtering. I don't see an example in the distribution, but it is
pretty straightforward: see
https://svn.apache.org/repos/asf/incubator/kafka/trunk/core/src/main/scala/kafka/javaapi/producer/async/CallbackHandler.java
(The kafka log4j producer currently uses the syncproducer, so you
won't be able to use callback handlers with it).
Thanks,
Joel
>
> On Sun, Oct 2, 2011 at 9:51 PM, Jay Kreps <ja...@gmail.com> wrote:
>
>> Hi,
>>
>> Currently all the patch does is take input from stdin. Can you help me
>> understand what you are looking for?
>>
>> What kind of plugin are you trying to write?
>>
>> -Jay
>>
>>
>> On Thu, Sep 29, 2011 at 10:41 PM, Bao Thai Ngo <ba...@gmail.com>wrote:
>>
>>> Jay,
>>>
>>> Does the patch already support log rotation (file rolling)?
>>>
>>> Also, I wonder if there is any documentation about how to write a plug-in
>>> for Kafka.
>>>
>>> Thanks,
>>> ~Thai
>>>
>>> On Mon, Sep 12, 2011 at 10:17 AM, Jay Kreps <ja...@gmail.com> wrote:
>>>
>>> > I added a patch for this:
>>> > https://issues.apache.org/jira/browse/KAFKA-130
>>> >
>>> > Felix, if you are still interested in this, let me know if it works for
>>> > you.
>>> >
>>> > -Jay
>>> >
>>> > On Thu, Sep 1, 2011 at 1:22 PM, Felix Giguere Villegas <
>>> > felix.giguere@mate1inc.com> wrote:
>>> >
>>> > > Hi :)
>>> > >
>>> > > Thanks for the reply :) . I have created
>>> > > KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130>
>>> > > .
>>> > >
>>> > > I'm not sure I have respected the proper standard for JIRA issues.
>>> Please
>>> > > let me know if I haven't.
>>> > >
>>> > > --
>>> > > Felix
>>> > >
>>> > >
>>> > >
>>> > > On Thu, Sep 1, 2011 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
>>> > >
>>> > > > Hi, Felix,
>>> > > >
>>> > > > Currently, we don't have a util to pipe data from a file to a
>>> producer.
>>> > I
>>> > > > agree that it's a very convenient tool to have. Could you open a
>>> jira
>>> > for
>>> > > > that?
>>> > > >
>>> > > > Thanks,
>>> > > >
>>> > > > Jun
>>>
>>
>
Re: Tail or stdin producer?
Posted by Bao Thai Ngo <ba...@gmail.com>.
Hi,
I am totally new to Kafka and we are evaluating Kafka like Felix. I'm
looking for a utility to pipe data from a file (or multiple files) to a
producer (like command tail -f in linux). This is an our typical use-case. I
saw the patch (KAFKA-130) you provided but not sure if this will work well
without loosing data if the file (or multiple files) will be rotated.
I would care about plugin that will such as filter message at producer;
modify message body in order to mark some critical kind of information, etc.
It would be good if you provide documentation about how to write a plugin
for Kafka.
Thanks,
~Thai
On Sun, Oct 2, 2011 at 9:51 PM, Jay Kreps <ja...@gmail.com> wrote:
> Hi,
>
> Currently all the patch does is take input from stdin. Can you help me
> understand what you are looking for?
>
> What kind of plugin are you trying to write?
>
> -Jay
>
>
> On Thu, Sep 29, 2011 at 10:41 PM, Bao Thai Ngo <ba...@gmail.com>wrote:
>
>> Jay,
>>
>> Does the patch already support log rotation (file rolling)?
>>
>> Also, I wonder if there is any documentation about how to write a plug-in
>> for Kafka.
>>
>> Thanks,
>> ~Thai
>>
>> On Mon, Sep 12, 2011 at 10:17 AM, Jay Kreps <ja...@gmail.com> wrote:
>>
>> > I added a patch for this:
>> > https://issues.apache.org/jira/browse/KAFKA-130
>> >
>> > Felix, if you are still interested in this, let me know if it works for
>> > you.
>> >
>> > -Jay
>> >
>> > On Thu, Sep 1, 2011 at 1:22 PM, Felix Giguere Villegas <
>> > felix.giguere@mate1inc.com> wrote:
>> >
>> > > Hi :)
>> > >
>> > > Thanks for the reply :) . I have created
>> > > KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130>
>> > > .
>> > >
>> > > I'm not sure I have respected the proper standard for JIRA issues.
>> Please
>> > > let me know if I haven't.
>> > >
>> > > --
>> > > Felix
>> > >
>> > >
>> > >
>> > > On Thu, Sep 1, 2011 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
>> > >
>> > > > Hi, Felix,
>> > > >
>> > > > Currently, we don't have a util to pipe data from a file to a
>> producer.
>> > I
>> > > > agree that it's a very convenient tool to have. Could you open a
>> jira
>> > for
>> > > > that?
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Jun
>>
>
Re: Tail or stdin producer?
Posted by Jay Kreps <ja...@gmail.com>.
Hi,
Currently all the patch does is take input from stdin. Can you help me
understand what you are looking for?
What kind of plugin are you trying to write?
-Jay
On Thu, Sep 29, 2011 at 10:41 PM, Bao Thai Ngo <ba...@gmail.com> wrote:
> Jay,
>
> Does the patch already support log rotation (file rolling)?
>
> Also, I wonder if there is any documentation about how to write a plug-in
> for Kafka.
>
> Thanks,
> ~Thai
>
> On Mon, Sep 12, 2011 at 10:17 AM, Jay Kreps <ja...@gmail.com> wrote:
>
> > I added a patch for this:
> > https://issues.apache.org/jira/browse/KAFKA-130
> >
> > Felix, if you are still interested in this, let me know if it works for
> > you.
> >
> > -Jay
> >
> > On Thu, Sep 1, 2011 at 1:22 PM, Felix Giguere Villegas <
> > felix.giguere@mate1inc.com> wrote:
> >
> > > Hi :)
> > >
> > > Thanks for the reply :) . I have created
> > > KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130>
> > > .
> > >
> > > I'm not sure I have respected the proper standard for JIRA issues.
> Please
> > > let me know if I haven't.
> > >
> > > --
> > > Felix
> > >
> > >
> > >
> > > On Thu, Sep 1, 2011 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Hi, Felix,
> > > >
> > > > Currently, we don't have a util to pipe data from a file to a
> producer.
> > I
> > > > agree that it's a very convenient tool to have. Could you open a jira
> > for
> > > > that?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Sep 1, 2011 at 12:20 PM, Felix Giguere Villegas <
> > > > felix.giguere@mate1inc.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We are currently evaluating Kafka, so I'm trying to get one of our
> > > simple
> > > > > use case working. I have read the design paper and most of
> everything
> > > > else
> > > > > I
> > > > > could find about Kafka, but I'm not sure how to set up a producer
> > that
> > > > > tails
> > > > > files or that reads from STDIN.
> > > > >
> > > > > Do I have to write custom code just for that? It seems like a
> pretty
> > > > > simple/common use case, so I would think it already exists out of
> the
> > > > box.
> > > > >
> > > > > I guess I could pipe a tail command into
> bin/kafka-producer-shell.sh
> > ??
> > > > >
> > > > > This kind of stuff is relatively easy to set up with Flume, but we
> > like
> > > > the
> > > > > persistence and replay capabilities of Kafka, which is why we are
> > > trying
> > > > it
> > > > > out.
> > > > >
> > > > > Thanks in advance for helping out a newb ;) !
> > > > >
> > > > > --
> > > > > Felix
> > > > >
> > > >
> > >
> >
>
Re: Tail or stdin producer?
Posted by Bao Thai Ngo <ba...@gmail.com>.
Jay,
Does the patch already support log rotation (file rolling)?
Also, I wonder if there is any documentation about how to write a plug-in
for Kafka.
Thanks,
~Thai
On Mon, Sep 12, 2011 at 10:17 AM, Jay Kreps <ja...@gmail.com> wrote:
> I added a patch for this:
> https://issues.apache.org/jira/browse/KAFKA-130
>
> Felix, if you are still interested in this, let me know if it works for
> you.
>
> -Jay
>
> On Thu, Sep 1, 2011 at 1:22 PM, Felix Giguere Villegas <
> felix.giguere@mate1inc.com> wrote:
>
> > Hi :)
> >
> > Thanks for the reply :) . I have created
> > KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130>
> > .
> >
> > I'm not sure I have respected the proper standard for JIRA issues. Please
> > let me know if I haven't.
> >
> > --
> > Felix
> >
> >
> >
> > On Thu, Sep 1, 2011 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Hi, Felix,
> > >
> > > Currently, we don't have a util to pipe data from a file to a producer.
> I
> > > agree that it's a very convenient tool to have. Could you open a jira
> for
> > > that?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Sep 1, 2011 at 12:20 PM, Felix Giguere Villegas <
> > > felix.giguere@mate1inc.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > We are currently evaluating Kafka, so I'm trying to get one of our
> > simple
> > > > use case working. I have read the design paper and most of everything
> > > else
> > > > I
> > > > could find about Kafka, but I'm not sure how to set up a producer
> that
> > > > tails
> > > > files or that reads from STDIN.
> > > >
> > > > Do I have to write custom code just for that? It seems like a pretty
> > > > simple/common use case, so I would think it already exists out of the
> > > box.
> > > >
> > > > I guess I could pipe a tail command into bin/kafka-producer-shell.sh
> ??
> > > >
> > > > This kind of stuff is relatively easy to set up with Flume, but we
> like
> > > the
> > > > persistence and replay capabilities of Kafka, which is why we are
> > trying
> > > it
> > > > out.
> > > >
> > > > Thanks in advance for helping out a newb ;) !
> > > >
> > > > --
> > > > Felix
> > > >
> > >
> >
>
Re: Tail or stdin producer?
Posted by Jay Kreps <ja...@gmail.com>.
I added a patch for this:
https://issues.apache.org/jira/browse/KAFKA-130
Felix, if you are still interested in this, let me know if it works for you.
-Jay
On Thu, Sep 1, 2011 at 1:22 PM, Felix Giguere Villegas <
felix.giguere@mate1inc.com> wrote:
> Hi :)
>
> Thanks for the reply :) . I have created
> KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130>
> .
>
> I'm not sure I have respected the proper standard for JIRA issues. Please
> let me know if I haven't.
>
> --
> Felix
>
>
>
> On Thu, Sep 1, 2011 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Hi, Felix,
> >
> > Currently, we don't have a util to pipe data from a file to a producer. I
> > agree that it's a very convenient tool to have. Could you open a jira for
> > that?
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Sep 1, 2011 at 12:20 PM, Felix Giguere Villegas <
> > felix.giguere@mate1inc.com> wrote:
> >
> > > Hi,
> > >
> > > We are currently evaluating Kafka, so I'm trying to get one of our
> simple
> > > use case working. I have read the design paper and most of everything
> > else
> > > I
> > > could find about Kafka, but I'm not sure how to set up a producer that
> > > tails
> > > files or that reads from STDIN.
> > >
> > > Do I have to write custom code just for that? It seems like a pretty
> > > simple/common use case, so I would think it already exists out of the
> > box.
> > >
> > > I guess I could pipe a tail command into bin/kafka-producer-shell.sh ??
> > >
> > > This kind of stuff is relatively easy to set up with Flume, but we like
> > the
> > > persistence and replay capabilities of Kafka, which is why we are
> trying
> > it
> > > out.
> > >
> > > Thanks in advance for helping out a newb ;) !
> > >
> > > --
> > > Felix
> > >
> >
>
Re: Tail or stdin producer?
Posted by Felix Giguere Villegas <fe...@mate1inc.com>.
Hi :)
Thanks for the reply :) . I have created
KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130>
.
I'm not sure I have respected the proper standard for JIRA issues. Please
let me know if I haven't.
--
Felix
On Thu, Sep 1, 2011 at 4:04 PM, Jun Rao <ju...@gmail.com> wrote:
> Hi, Felix,
>
> Currently, we don't have a util to pipe data from a file to a producer. I
> agree that it's a very convenient tool to have. Could you open a jira for
> that?
>
> Thanks,
>
> Jun
>
> On Thu, Sep 1, 2011 at 12:20 PM, Felix Giguere Villegas <
> felix.giguere@mate1inc.com> wrote:
>
> > Hi,
> >
> > We are currently evaluating Kafka, so I'm trying to get one of our simple
> > use case working. I have read the design paper and most of everything
> else
> > I
> > could find about Kafka, but I'm not sure how to set up a producer that
> > tails
> > files or that reads from STDIN.
> >
> > Do I have to write custom code just for that? It seems like a pretty
> > simple/common use case, so I would think it already exists out of the
> box.
> >
> > I guess I could pipe a tail command into bin/kafka-producer-shell.sh ??
> >
> > This kind of stuff is relatively easy to set up with Flume, but we like
> the
> > persistence and replay capabilities of Kafka, which is why we are trying
> it
> > out.
> >
> > Thanks in advance for helping out a newb ;) !
> >
> > --
> > Felix
> >
>
Re: Tail or stdin producer?
Posted by Jun Rao <ju...@gmail.com>.
Hi, Felix,
Currently, we don't have a util to pipe data from a file to a producer. I
agree that it's a very convenient tool to have. Could you open a jira for
that?
Thanks,
Jun
On Thu, Sep 1, 2011 at 12:20 PM, Felix Giguere Villegas <
felix.giguere@mate1inc.com> wrote:
> Hi,
>
> We are currently evaluating Kafka, so I'm trying to get one of our simple
> use case working. I have read the design paper and most of everything else
> I
> could find about Kafka, but I'm not sure how to set up a producer that
> tails
> files or that reads from STDIN.
>
> Do I have to write custom code just for that? It seems like a pretty
> simple/common use case, so I would think it already exists out of the box.
>
> I guess I could pipe a tail command into bin/kafka-producer-shell.sh ??
>
> This kind of stuff is relatively easy to set up with Flume, but we like the
> persistence and replay capabilities of Kafka, which is why we are trying it
> out.
>
> Thanks in advance for helping out a newb ;) !
>
> --
> Felix
>