You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Neelesh <ne...@gmail.com> on 2012/05/19 00:21:28 UTC
Evaluating Kafka for use at my company
Hi,
I am intrigued by the design of Kafka and pushing for using it in our
org. We have many use cases where kafka fits the bill. Is there a timeframe
you are looking at for Kafka 0.8 to be released?I am especially interested
in long polling, (KAFKA-48) and thinking of writing a more sophisticated
PHP client which can do producer side load balancing, and possible
something like IronCount for the consumer side.
Thanks!
-neelesh
Re: Evaluating Kafka for use at my company
Posted by Joel Koshy <jj...@gmail.com>.
That is correct - as I mentioned it does not provide any ability to react
to producer failures which can happen if you bounce the source brokers or
if a source broker dies before flushing to disk.
Thanks,
Joel
On Tue, May 22, 2012 at 4:48 PM, Neelesh <ne...@gmail.com> wrote:
> Thanks for the superb document Joel. I may not have communicated very well,
> but even with mirrormaker, the mirroring is achieved through a consumer.
> This assumes that the message is persisted on the source Kafka broker. If
> the source broker dies before flushing the messages to disk, won't we lose
> those messages? I am sure I'm missing something here.
>
> Thanks!
> -neelesh
>
> On Tue, May 22, 2012 at 10:46 AM, Joel Koshy <jj...@gmail.com> wrote:
>
> > Thanks! In the meanwhile, do you think simply modifying the producer to
> > > log the same message to multiple brokers (which are not part of a
> > cluster)
> > > is a reasonably good solution to reduce the probability of lost
> > messages? I
> > > have seen the mirroring documentation, but it sounded like the
> mirroring
> > > consumer was embedded inside the broker, and hence a dead broker will
> > mean
> > > losing unflushed messages.
> > >
> >
> > The embedded consumer approach is being deprecated. I added a new
> mirroring
> > doc outlining the new approach which uses a separate tool to do
> mirroring.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)
> >
> > However, mirroring is orthogonal to your use case. i.e., it does not
> quite
> > address the ability to react to producer failures (e.g., during a bounce
> of
> > a broker) which will be available in 0.8.
> >
> > Thanks,
> >
> > Joel
> >
> >
> > On Fri, May 18, 2012 at 3:26 PM, Neha Narkhede <neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > Hi Neelesh,
> > > >
> > > > Thank you for your interest in Kafka !
> > > >
> > > > We are shooting for end of summer this year for the 0.8 release. You
> > can
> > > > track the progress of this feature here -
> > > > https://issues.apache.org/jira/browse/KAFKA-50
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Fri, May 18, 2012 at 3:21 PM, Neelesh <ne...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > > I am intrigued by the design of Kafka and pushing for using it in
> > our
> > > > > org. We have many use cases where kafka fits the bill. Is there a
> > > > timeframe
> > > > > you are looking at for Kafka 0.8 to be released?I am especially
> > > > interested
> > > > > in long polling, (KAFKA-48) and thinking of writing a more
> > > sophisticated
> > > > > PHP client which can do producer side load balancing, and possible
> > > > > something like IronCount for the consumer side.
> > > > >
> > > > > Thanks!
> > > > > -neelesh
> > > > >
> > > >
> > >
> >
>
Re: Evaluating Kafka for use at my company
Posted by Neelesh <ne...@gmail.com>.
Thanks for the superb document Joel. I may not have communicated very well,
but even with mirrormaker, the mirroring is achieved through a consumer.
This assumes that the message is persisted on the source Kafka broker. If
the source broker dies before flushing the messages to disk, won't we lose
those messages? I am sure I'm missing something here.
Thanks!
-neelesh
On Tue, May 22, 2012 at 10:46 AM, Joel Koshy <jj...@gmail.com> wrote:
> Thanks! In the meanwhile, do you think simply modifying the producer to
> > log the same message to multiple brokers (which are not part of a
> cluster)
> > is a reasonably good solution to reduce the probability of lost
> messages? I
> > have seen the mirroring documentation, but it sounded like the mirroring
> > consumer was embedded inside the broker, and hence a dead broker will
> mean
> > losing unflushed messages.
> >
>
> The embedded consumer approach is being deprecated. I added a new mirroring
> doc outlining the new approach which uses a separate tool to do mirroring.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)
>
> However, mirroring is orthogonal to your use case. i.e., it does not quite
> address the ability to react to producer failures (e.g., during a bounce of
> a broker) which will be available in 0.8.
>
> Thanks,
>
> Joel
>
>
> On Fri, May 18, 2012 at 3:26 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Hi Neelesh,
> > >
> > > Thank you for your interest in Kafka !
> > >
> > > We are shooting for end of summer this year for the 0.8 release. You
> can
> > > track the progress of this feature here -
> > > https://issues.apache.org/jira/browse/KAFKA-50
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Fri, May 18, 2012 at 3:21 PM, Neelesh <ne...@gmail.com> wrote:
> > >
> > > > Hi,
> > > > I am intrigued by the design of Kafka and pushing for using it in
> our
> > > > org. We have many use cases where kafka fits the bill. Is there a
> > > timeframe
> > > > you are looking at for Kafka 0.8 to be released?I am especially
> > > interested
> > > > in long polling, (KAFKA-48) and thinking of writing a more
> > sophisticated
> > > > PHP client which can do producer side load balancing, and possible
> > > > something like IronCount for the consumer side.
> > > >
> > > > Thanks!
> > > > -neelesh
> > > >
> > >
> >
>
Re: Evaluating Kafka for use at my company
Posted by Joel Koshy <jj...@gmail.com>.
Thanks! In the meanwhile, do you think simply modifying the producer to
> log the same message to multiple brokers (which are not part of a cluster)
> is a reasonably good solution to reduce the probability of lost messages? I
> have seen the mirroring documentation, but it sounded like the mirroring
> consumer was embedded inside the broker, and hence a dead broker will mean
> losing unflushed messages.
>
The embedded consumer approach is being deprecated. I added a new mirroring
doc outlining the new approach which uses a separate tool to do mirroring.
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)
However, mirroring is orthogonal to your use case. i.e., it does not quite
address the ability to react to producer failures (e.g., during a bounce of
a broker) which will be available in 0.8.
Thanks,
Joel
On Fri, May 18, 2012 at 3:26 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Hi Neelesh,
> >
> > Thank you for your interest in Kafka !
> >
> > We are shooting for end of summer this year for the 0.8 release. You can
> > track the progress of this feature here -
> > https://issues.apache.org/jira/browse/KAFKA-50
> >
> > Thanks,
> > Neha
> >
> >
> > On Fri, May 18, 2012 at 3:21 PM, Neelesh <ne...@gmail.com> wrote:
> >
> > > Hi,
> > > I am intrigued by the design of Kafka and pushing for using it in our
> > > org. We have many use cases where kafka fits the bill. Is there a
> > timeframe
> > > you are looking at for Kafka 0.8 to be released?I am especially
> > interested
> > > in long polling, (KAFKA-48) and thinking of writing a more
> sophisticated
> > > PHP client which can do producer side load balancing, and possible
> > > something like IronCount for the consumer side.
> > >
> > > Thanks!
> > > -neelesh
> > >
> >
>
Re: Evaluating Kafka for use at my company
Posted by Neelesh <ne...@gmail.com>.
Thanks! In the meanwhile, do you think simply modifying the producer to
log the same message to multiple brokers (which are not part of a cluster)
is a reasonably good solution to reduce the probability of lost messages? I
have seen the mirroring documentation, but it sounded like the mirroring
consumer was embedded inside the broker, and hence a dead broker will mean
losing unflushed messages.
Thank you!
-neelesh
On Fri, May 18, 2012 at 3:26 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Hi Neelesh,
>
> Thank you for your interest in Kafka !
>
> We are shooting for end of summer this year for the 0.8 release. You can
> track the progress of this feature here -
> https://issues.apache.org/jira/browse/KAFKA-50
>
> Thanks,
> Neha
>
>
> On Fri, May 18, 2012 at 3:21 PM, Neelesh <ne...@gmail.com> wrote:
>
> > Hi,
> > I am intrigued by the design of Kafka and pushing for using it in our
> > org. We have many use cases where kafka fits the bill. Is there a
> timeframe
> > you are looking at for Kafka 0.8 to be released?I am especially
> interested
> > in long polling, (KAFKA-48) and thinking of writing a more sophisticated
> > PHP client which can do producer side load balancing, and possible
> > something like IronCount for the consumer side.
> >
> > Thanks!
> > -neelesh
> >
>
Re: Evaluating Kafka for use at my company
Posted by Neha Narkhede <ne...@gmail.com>.
Hi Neelesh,
Thank you for your interest in Kafka !
We are shooting for end of summer this year for the 0.8 release. You can
track the progress of this feature here -
https://issues.apache.org/jira/browse/KAFKA-50
Thanks,
Neha
On Fri, May 18, 2012 at 3:21 PM, Neelesh <ne...@gmail.com> wrote:
> Hi,
> I am intrigued by the design of Kafka and pushing for using it in our
> org. We have many use cases where kafka fits the bill. Is there a timeframe
> you are looking at for Kafka 0.8 to be released?I am especially interested
> in long polling, (KAFKA-48) and thinking of writing a more sophisticated
> PHP client which can do producer side load balancing, and possible
> something like IronCount for the consumer side.
>
> Thanks!
> -neelesh
>