Our use case and am I right in my definition of Throughput?

["Kishore V. Kopalle" <ki...@greenmedsoft.com>]

Hello All,

Our use case is to display certain aggregates on GUI from a live stream of
data coming in at more than 100k messages/sec. Will I be able to use Kafka
for handling at least 100k messages/sec and send it to Twitter Storm for
aggregate calculations is the question I have. I already know that Storm
can handle more than 100k messages/sec.

I have seen slides presented at Apache Con 2011 North America on Kafka and
the Volume handled on slide 12 is mentioned as 150k events/sec. Can we take
this as a throughput possible with Kafka at 150k msgs/sec? I am assuming
Throughput definition as not the messages/sec sent at producer or
messages/sec received at the consumer, but the number of messages per
second that can be sent over the wire between consumer and producer. With
such a definition I am hardly getting 1 message/sec as per my earlier
mails. I am trying to measure it by calculating the time it takes for a
single message to go between producer and consumer. If the throughput,
according to my definition, is 150k messages/sec, the time taken for a
single message should have been 1/(150k) seconds.


Regards,
Kishore

Re: Our use case and am I right in my definition of Throughput?

[Neha Narkhede <ne...@gmail.com>]

Another parameter that you want to tune if you want higher throughput in
0.7 is the broker side flush interval. If you set it to something high,
like 50K or 100K, you can maximize the throughput achieved by your producer.

Thanks,
Neha


On Fri, May 17, 2013 at 8:11 AM, Jun Rao <ju...@gmail.com> wrote:

> You earlier email seems to focus on latency, not throughput. Typically, you
> can either optimize for latency or throughput, but not both. If you want
> higher throughput, you should consider using the async mode in the producer
> with a larger batch size (e.g., 1000 messages). Using more instances of
> producer also increases the overall throughput.
>
> Thanks,
>
> Jun
>
>
> On Fri, May 17, 2013 at 12:26 AM, Kishore V. Kopalle <
> kishore@greenmedsoft.com> wrote:
>
> > Hello All,
> >
> > Our use case is to display certain aggregates on GUI from a live stream
> of
> > data coming in at more than 100k messages/sec. Will I be able to use
> Kafka
> > for handling at least 100k messages/sec and send it to Twitter Storm for
> > aggregate calculations is the question I have. I already know that Storm
> > can handle more than 100k messages/sec.
> >
> > I have seen slides presented at Apache Con 2011 North America on Kafka
> and
> > the Volume handled on slide 12 is mentioned as 150k events/sec. Can we
> take
> > this as a throughput possible with Kafka at 150k msgs/sec? I am assuming
> > Throughput definition as not the messages/sec sent at producer or
> > messages/sec received at the consumer, but the number of messages per
> > second that can be sent over the wire between consumer and producer. With
> > such a definition I am hardly getting 1 message/sec as per my earlier
> > mails. I am trying to measure it by calculating the time it takes for a
> > single message to go between producer and consumer. If the throughput,
> > according to my definition, is 150k messages/sec, the time taken for a
> > single message should have been 1/(150k) seconds.
> >
> >
> > Regards,
> > Kishore
> >
>

Re: Our use case and am I right in my definition of Throughput?

[Jun Rao <ju...@gmail.com>]

You earlier email seems to focus on latency, not throughput. Typically, you
can either optimize for latency or throughput, but not both. If you want
higher throughput, you should consider using the async mode in the producer
with a larger batch size (e.g., 1000 messages). Using more instances of
producer also increases the overall throughput.

Thanks,

Jun


On Fri, May 17, 2013 at 12:26 AM, Kishore V. Kopalle <
kishore@greenmedsoft.com> wrote:

> Hello All,
>
> Our use case is to display certain aggregates on GUI from a live stream of
> data coming in at more than 100k messages/sec. Will I be able to use Kafka
> for handling at least 100k messages/sec and send it to Twitter Storm for
> aggregate calculations is the question I have. I already know that Storm
> can handle more than 100k messages/sec.
>
> I have seen slides presented at Apache Con 2011 North America on Kafka and
> the Volume handled on slide 12 is mentioned as 150k events/sec. Can we take
> this as a throughput possible with Kafka at 150k msgs/sec? I am assuming
> Throughput definition as not the messages/sec sent at producer or
> messages/sec received at the consumer, but the number of messages per
> second that can be sent over the wire between consumer and producer. With
> such a definition I am hardly getting 1 message/sec as per my earlier
> mails. I am trying to measure it by calculating the time it takes for a
> single message to go between producer and consumer. If the throughput,
> according to my definition, is 150k messages/sec, the time taken for a
> single message should have been 1/(150k) seconds.
>
>
> Regards,
> Kishore
>