Why Apache Kafka is bettern than any other Message System?

[Lee, "Yeon Ok (이연옥)" <ye...@ebay.com>]

Hi, all.
I just got curiosity why Apache Kafka is better than any other Message System in terms of throughput, and durability.
What’s the fact to let Kafka have better performance?
I read some documents and researched, however, I am not quite sure about that..
And Zookeeper in Apache Kafka seems like a SPOF, Doesn’t it?
How Kafka be implemented in real world web-site?

Re: Why Apache Kafka is bettern than any other Message System?

[Roger Hoover <ro...@gmail.com>]

I'll give you my take in case it helps.

Kafka achieves great throughput + durability because

1) The broker does minimal work
    a) Routing is done by producers
    b) State management is done by consumers
  Other messaging brokers typically have to keep track of which messages
have been dispatched to which consumers and whether they've been
acknowledged yet or not.  Then they have to deal with retying failed
message attempts, etc.
2) Clients can batch and compress messages
3) Because of #1, broker reads and writes to files are sequential and make
great use of OS caching

It took a little while to get my head around this and see how clever the
design is.  Since the routing is done at the producer, you might worry
about unbalanced queuing in partitions.  Imagine a case where you have two
partitions for a topic and one consumer for each partition.  If one of the
consumers is slow and one is fast, the fast one may sit idle while messages
pile up for the slow one.  Here's why I think this isn't a big issue in
practice.  Most of the time your consumers will be io-bound and depend on a
shared resource such as a database or Hadoop cluster.  If that's the case,
all the consumers will tend to have to same latency in processing messages
such that routing at the producer works just fine.

HTH,

Roger


On Wed, Oct 30, 2013 at 9:51 PM, Neha Narkhede <ne...@gmail.com>wrote:

> You can find Kafka use cases described here -
> http://kafka.apache.org/documentation.html#uses
>
> Also, the design document is a good read to understand why Kafka is more
> efficient - http://kafka.apache.org/documentation.html#design
>
> Thanks,
> Neha
>
>
> On Wed, Oct 30, 2013 at 8:24 PM, Philip O'Toole <ph...@loggly.com> wrote:
>
> > On Wed, Oct 30, 2013 at 8:13 PM, Lee, Yeon Ok (이연옥) <yeonolee@ebay.com
> > >wrote:
> >
> > > Hi, all.
> > > I just got curiosity why Apache Kafka is better than any other Message
> > > System in terms of throughput, and durability.
> > >
> >
> > Because it's brilliant, that's why. :-)
> >
> >
> > > What’s the fact to let Kafka have better performance?
> > >
> >
> > Because it doesn't track consumer state, not in 0.72 anyway. And it makes
> > very efficient use of disks.
> >
> >
> > > I read some documents and researched, however, I am not quite sure
> about
> > > that..
> > > And Zookeeper in Apache Kafka seems like a SPOF, Doesn’t it?
> > >
> >
> > Kafka can be configured to run without Zookeeper, if you are that
> > concerned, at the cost of more rigid system.
> >
> >
> > > How Kafka be implemented in real world web-site?
> > >
> >
> > Depends what you want to do.  Whether you need to use it is the real
> > question.
> >
> > Philip
> >
>

Re: Why Apache Kafka is bettern than any other Message System?

[Neha Narkhede <ne...@gmail.com>]

You can find Kafka use cases described here -
http://kafka.apache.org/documentation.html#uses

Also, the design document is a good read to understand why Kafka is more
efficient - http://kafka.apache.org/documentation.html#design

Thanks,
Neha


On Wed, Oct 30, 2013 at 8:24 PM, Philip O'Toole <ph...@loggly.com> wrote:

> On Wed, Oct 30, 2013 at 8:13 PM, Lee, Yeon Ok (이연옥) <yeonolee@ebay.com
> >wrote:
>
> > Hi, all.
> > I just got curiosity why Apache Kafka is better than any other Message
> > System in terms of throughput, and durability.
> >
>
> Because it's brilliant, that's why. :-)
>
>
> > What’s the fact to let Kafka have better performance?
> >
>
> Because it doesn't track consumer state, not in 0.72 anyway. And it makes
> very efficient use of disks.
>
>
> > I read some documents and researched, however, I am not quite sure about
> > that..
> > And Zookeeper in Apache Kafka seems like a SPOF, Doesn’t it?
> >
>
> Kafka can be configured to run without Zookeeper, if you are that
> concerned, at the cost of more rigid system.
>
>
> > How Kafka be implemented in real world web-site?
> >
>
> Depends what you want to do.  Whether you need to use it is the real
> question.
>
> Philip
>

Re: Why Apache Kafka is bettern than any other Message System?

[Philip O'Toole <ph...@loggly.com>]

On Wed, Oct 30, 2013 at 8:13 PM, Lee, Yeon Ok (이연옥) <ye...@ebay.com>wrote:

> Hi, all.
> I just got curiosity why Apache Kafka is better than any other Message
> System in terms of throughput, and durability.
>

Because it's brilliant, that's why. :-)


> What’s the fact to let Kafka have better performance?
>

Because it doesn't track consumer state, not in 0.72 anyway. And it makes
very efficient use of disks.


> I read some documents and researched, however, I am not quite sure about
> that..
> And Zookeeper in Apache Kafka seems like a SPOF, Doesn’t it?
>

Kafka can be configured to run without Zookeeper, if you are that
concerned, at the cost of more rigid system.


> How Kafka be implemented in real world web-site?
>

Depends what you want to do.  Whether you need to use it is the real
question.

Philip