You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Magnus Edenhill <ma...@edenhill.se> on 2013/11/22 12:58:27 UTC
librdkafka 0.8.0 released
This announces the 0.8.0 release of librdkafka - The Apache Kafka client C
library - now with 0.8 protocol support.
Features:
* Producer (~800K msgs/s)
* Consumer (~3M msgs/s)
* Compression (Snappy, gzip)
* Proper failover and leader re-election support - no message is ever lost.
* Configuration properties compatible with official Apache Kafka.
* Stabilized ABI-safe API
* Mainline Debian package submitted
* Production quality
Home:
https://github.com/edenhill/librdkafka
Introduction and performance numbers:
https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md
Have fun.
Regards,
Magnus
P.S.
Check out Wikimedia Foundation's varnishkafka daemon for a use case -
varnish log forwarding over Kafka:
https://github.com/wikimedia/varnishkafka
Re: librdkafka 0.8.0 released
Posted by Jun Rao <ju...@gmail.com>.
Thanks. The results make sense. Higher consistency (ack=-1 and ack=2)
typically means longer latency.
Do those number match our java producers?
Thanks,
Jun
On Mon, Nov 25, 2013 at 1:16 PM, Magnus Edenhill <ma...@edenhill.se> wrote:
> Producing to one partition, no replication, required.acks = 0:
> % 1000000 messages and 100000000 bytes produced in 1215ms: 822383 msgs/s
> and 82.24 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, no replication, required.acks = -1:
> % 1000000 messages and 100000000 bytes produced in 1422ms: 703091 msgs/s
> and 70.31 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, no replication, required.acks = 1:
> % 1000000 messages and 100000000 bytes produced in 1295ms: 771881 msgs/s
> and 77.19 Mb/s, 0 messages failed, no compression
>
>
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = 0:
> % 1000000 messages and 100000000 bytes produced in 1354ms: 738483 msgs/s
> and 73.85 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = -1:
> % 1000000 messages and 100000000 bytes produced in 3698ms: 270396 msgs/s
> and 27.04 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = 1:
> % 1000000 messages and 100000000 bytes produced in 1360ms: 735224 msgs/s
> and 73.52 Mb/s, 0 messages failed, no compression
>
> Producing to one partition, replication factor 2, 2 brokers ISR,
> required.acks = 2:
> % 1000000 messages and 100000000 bytes produced in 3568ms: 280241 msgs/s
> and 28.02 Mb/s, 0 messages failed, no compression
>
>
> These are the maximum values from a smaller number of naive tests.
>
> It would be interesting to see some numbers from relevant environments with
> proper hardware and networks.
> (rdkafka_performance -P -t <topic> -p <partition> -s <msgsize> -a
> <required.acks> -c 1000000 -q)
>
> Regards,
> Magnus
>
>
> 2013/11/25 Jun Rao <ju...@gmail.com>
>
> > Thanks for sharing the results. Was the topic created with replication
> > factor of 2? Could you test acks=-1 as well?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Nov 25, 2013 at 4:30 AM, Magnus Edenhill <ma...@edenhill.se>
> > wrote:
> >
> > > The following tests were using a single producer application
> > > (examples/rdkafka_performance):
> > >
> > > * Test1: 2 brokers, 2 partitions, required.acks=2, 100 byte messages:
> > > 850000 messages/second, 85 MB/second
> > >
> > > * Test2: 1 broker, 1 partition, required.acks=0, 100 byte messages:
> > 710000
> > > messages/second, 71 MB/second
> > >
> > > * Test3: 2 broker2, 2 partitions, required.acks=2, 100 byte messages,
> > > snappy compression: 300000 messages/second, 30 MB/second
> > >
> > > * Test4: 2 broker2, 2 partitions, required.acks=2, 100 byte messages,
> > gzip
> > > compression: 230000 messages/second, 23 MB/second
> > >
> > >
> > > log.flush broker configuration was increased to avoid the disk being
> the
> > > bottleneck.
> > >
> > >
> > > /Magnus
> > >
> > >
> > >
> > > 2013/11/24 Neha Narkhede <ne...@gmail.com>
> > >
> > > > So, a single producer'a throughput is 80 MB/s? That seems pretty
> high.
> > > What
> > > > was the number of acks setting? Thanks for sharing these numbers.
> > > >
> > > > On Sunday, November 24, 2013, Magnus Edenhill wrote:
> > > >
> > > > > Hi Neha,
> > > > >
> > > > > these tests were done using 100 byte messages. More information
> about
> > > the
> > > > > producer performance tests can be found here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > > > >
> > > > > The tests are indicative at best and in no way scientific, but I
> must
> > > say
> > > > > that the Kafka broker performance is impressive.
> > > > >
> > > > > Regards,
> > > > > Magnus
> > > > >
> > > > >
> > > > >
> > > > > 2013/11/22 Neha Narkhede <neha.narkhede@gmail.com <javascript:;>>
> > > > >
> > > > > > Thanks for sharing this! What is the message size for the
> > throughput
> > > > > > numbers stated below?
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > > On Nov 22, 2013 6:59 AM, "Magnus Edenhill" <magnus@edenhill.se
> > > > <javascript:;>>
> > > > > wrote:
> > > > > >
> > > > > > > This announces the 0.8.0 release of librdkafka - The Apache
> Kafka
> > > > > client
> > > > > > C
> > > > > > > library - now with 0.8 protocol support.
> > > > > > >
> > > > > > > Features:
> > > > > > > * Producer (~800K msgs/s)
> > > > > > > * Consumer (~3M msgs/s)
> > > > > > > * Compression (Snappy, gzip)
> > > > > > > * Proper failover and leader re-election support - no message
> is
> > > ever
> > > > > > lost.
> > > > > > > * Configuration properties compatible with official Apache
> Kafka.
> > > > > > > * Stabilized ABI-safe API
> > > > > > > * Mainline Debian package submitted
> > > > > > > * Production quality
> > > > > > >
> > > > > > >
> > > > > > > Home:
> > > > > > > https://github.com/edenhill/librdkafka
> > > > > > >
> > > > > > > Introduction and performance numbers:
> > > > > > >
> > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md
> > > > > > >
> > > > > > > Have fun.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Magnus
> > > > > > >
> > > > > > > P.S.
> > > > > > > Check out Wikimedia Foundation's varnishkafka daemon for a use
> > > case -
> > > > > > > varnish log forwarding over Kafka:
> > > > > > > https://github.com/wikimedia/varnishkafka
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: librdkafka 0.8.0 released
Posted by Magnus Edenhill <ma...@edenhill.se>.
Producing to one partition, no replication, required.acks = 0:
% 1000000 messages and 100000000 bytes produced in 1215ms: 822383 msgs/s
and 82.24 Mb/s, 0 messages failed, no compression
Producing to one partition, no replication, required.acks = -1:
% 1000000 messages and 100000000 bytes produced in 1422ms: 703091 msgs/s
and 70.31 Mb/s, 0 messages failed, no compression
Producing to one partition, no replication, required.acks = 1:
% 1000000 messages and 100000000 bytes produced in 1295ms: 771881 msgs/s
and 77.19 Mb/s, 0 messages failed, no compression
Producing to one partition, replication factor 2, 2 brokers ISR,
required.acks = 0:
% 1000000 messages and 100000000 bytes produced in 1354ms: 738483 msgs/s
and 73.85 Mb/s, 0 messages failed, no compression
Producing to one partition, replication factor 2, 2 brokers ISR,
required.acks = -1:
% 1000000 messages and 100000000 bytes produced in 3698ms: 270396 msgs/s
and 27.04 Mb/s, 0 messages failed, no compression
Producing to one partition, replication factor 2, 2 brokers ISR,
required.acks = 1:
% 1000000 messages and 100000000 bytes produced in 1360ms: 735224 msgs/s
and 73.52 Mb/s, 0 messages failed, no compression
Producing to one partition, replication factor 2, 2 brokers ISR,
required.acks = 2:
% 1000000 messages and 100000000 bytes produced in 3568ms: 280241 msgs/s
and 28.02 Mb/s, 0 messages failed, no compression
These are the maximum values from a smaller number of naive tests.
It would be interesting to see some numbers from relevant environments with
proper hardware and networks.
(rdkafka_performance -P -t <topic> -p <partition> -s <msgsize> -a
<required.acks> -c 1000000 -q)
Regards,
Magnus
2013/11/25 Jun Rao <ju...@gmail.com>
> Thanks for sharing the results. Was the topic created with replication
> factor of 2? Could you test acks=-1 as well?
>
> Thanks,
>
> Jun
>
>
> On Mon, Nov 25, 2013 at 4:30 AM, Magnus Edenhill <ma...@edenhill.se>
> wrote:
>
> > The following tests were using a single producer application
> > (examples/rdkafka_performance):
> >
> > * Test1: 2 brokers, 2 partitions, required.acks=2, 100 byte messages:
> > 850000 messages/second, 85 MB/second
> >
> > * Test2: 1 broker, 1 partition, required.acks=0, 100 byte messages:
> 710000
> > messages/second, 71 MB/second
> >
> > * Test3: 2 broker2, 2 partitions, required.acks=2, 100 byte messages,
> > snappy compression: 300000 messages/second, 30 MB/second
> >
> > * Test4: 2 broker2, 2 partitions, required.acks=2, 100 byte messages,
> gzip
> > compression: 230000 messages/second, 23 MB/second
> >
> >
> > log.flush broker configuration was increased to avoid the disk being the
> > bottleneck.
> >
> >
> > /Magnus
> >
> >
> >
> > 2013/11/24 Neha Narkhede <ne...@gmail.com>
> >
> > > So, a single producer'a throughput is 80 MB/s? That seems pretty high.
> > What
> > > was the number of acks setting? Thanks for sharing these numbers.
> > >
> > > On Sunday, November 24, 2013, Magnus Edenhill wrote:
> > >
> > > > Hi Neha,
> > > >
> > > > these tests were done using 100 byte messages. More information about
> > the
> > > > producer performance tests can be found here:
> > > >
> > > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > > >
> > > > The tests are indicative at best and in no way scientific, but I must
> > say
> > > > that the Kafka broker performance is impressive.
> > > >
> > > > Regards,
> > > > Magnus
> > > >
> > > >
> > > >
> > > > 2013/11/22 Neha Narkhede <neha.narkhede@gmail.com <javascript:;>>
> > > >
> > > > > Thanks for sharing this! What is the message size for the
> throughput
> > > > > numbers stated below?
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > > On Nov 22, 2013 6:59 AM, "Magnus Edenhill" <magnus@edenhill.se
> > > <javascript:;>>
> > > > wrote:
> > > > >
> > > > > > This announces the 0.8.0 release of librdkafka - The Apache Kafka
> > > > client
> > > > > C
> > > > > > library - now with 0.8 protocol support.
> > > > > >
> > > > > > Features:
> > > > > > * Producer (~800K msgs/s)
> > > > > > * Consumer (~3M msgs/s)
> > > > > > * Compression (Snappy, gzip)
> > > > > > * Proper failover and leader re-election support - no message is
> > ever
> > > > > lost.
> > > > > > * Configuration properties compatible with official Apache Kafka.
> > > > > > * Stabilized ABI-safe API
> > > > > > * Mainline Debian package submitted
> > > > > > * Production quality
> > > > > >
> > > > > >
> > > > > > Home:
> > > > > > https://github.com/edenhill/librdkafka
> > > > > >
> > > > > > Introduction and performance numbers:
> > > > > >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md
> > > > > >
> > > > > > Have fun.
> > > > > >
> > > > > > Regards,
> > > > > > Magnus
> > > > > >
> > > > > > P.S.
> > > > > > Check out Wikimedia Foundation's varnishkafka daemon for a use
> > case -
> > > > > > varnish log forwarding over Kafka:
> > > > > > https://github.com/wikimedia/varnishkafka
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: librdkafka 0.8.0 released
Posted by Jun Rao <ju...@gmail.com>.
Thanks for sharing the results. Was the topic created with replication
factor of 2? Could you test acks=-1 as well?
Thanks,
Jun
On Mon, Nov 25, 2013 at 4:30 AM, Magnus Edenhill <ma...@edenhill.se> wrote:
> The following tests were using a single producer application
> (examples/rdkafka_performance):
>
> * Test1: 2 brokers, 2 partitions, required.acks=2, 100 byte messages:
> 850000 messages/second, 85 MB/second
>
> * Test2: 1 broker, 1 partition, required.acks=0, 100 byte messages: 710000
> messages/second, 71 MB/second
>
> * Test3: 2 broker2, 2 partitions, required.acks=2, 100 byte messages,
> snappy compression: 300000 messages/second, 30 MB/second
>
> * Test4: 2 broker2, 2 partitions, required.acks=2, 100 byte messages, gzip
> compression: 230000 messages/second, 23 MB/second
>
>
> log.flush broker configuration was increased to avoid the disk being the
> bottleneck.
>
>
> /Magnus
>
>
>
> 2013/11/24 Neha Narkhede <ne...@gmail.com>
>
> > So, a single producer'a throughput is 80 MB/s? That seems pretty high.
> What
> > was the number of acks setting? Thanks for sharing these numbers.
> >
> > On Sunday, November 24, 2013, Magnus Edenhill wrote:
> >
> > > Hi Neha,
> > >
> > > these tests were done using 100 byte messages. More information about
> the
> > > producer performance tests can be found here:
> > >
> > >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> > >
> > > The tests are indicative at best and in no way scientific, but I must
> say
> > > that the Kafka broker performance is impressive.
> > >
> > > Regards,
> > > Magnus
> > >
> > >
> > >
> > > 2013/11/22 Neha Narkhede <neha.narkhede@gmail.com <javascript:;>>
> > >
> > > > Thanks for sharing this! What is the message size for the throughput
> > > > numbers stated below?
> > > >
> > > > Thanks,
> > > > Neha
> > > > On Nov 22, 2013 6:59 AM, "Magnus Edenhill" <magnus@edenhill.se
> > <javascript:;>>
> > > wrote:
> > > >
> > > > > This announces the 0.8.0 release of librdkafka - The Apache Kafka
> > > client
> > > > C
> > > > > library - now with 0.8 protocol support.
> > > > >
> > > > > Features:
> > > > > * Producer (~800K msgs/s)
> > > > > * Consumer (~3M msgs/s)
> > > > > * Compression (Snappy, gzip)
> > > > > * Proper failover and leader re-election support - no message is
> ever
> > > > lost.
> > > > > * Configuration properties compatible with official Apache Kafka.
> > > > > * Stabilized ABI-safe API
> > > > > * Mainline Debian package submitted
> > > > > * Production quality
> > > > >
> > > > >
> > > > > Home:
> > > > > https://github.com/edenhill/librdkafka
> > > > >
> > > > > Introduction and performance numbers:
> > > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md
> > > > >
> > > > > Have fun.
> > > > >
> > > > > Regards,
> > > > > Magnus
> > > > >
> > > > > P.S.
> > > > > Check out Wikimedia Foundation's varnishkafka daemon for a use
> case -
> > > > > varnish log forwarding over Kafka:
> > > > > https://github.com/wikimedia/varnishkafka
> > > > >
> > > >
> > >
> >
>
Re: librdkafka 0.8.0 released
Posted by Magnus Edenhill <ma...@edenhill.se>.
The following tests were using a single producer application
(examples/rdkafka_performance):
* Test1: 2 brokers, 2 partitions, required.acks=2, 100 byte messages:
850000 messages/second, 85 MB/second
* Test2: 1 broker, 1 partition, required.acks=0, 100 byte messages: 710000
messages/second, 71 MB/second
* Test3: 2 broker2, 2 partitions, required.acks=2, 100 byte messages,
snappy compression: 300000 messages/second, 30 MB/second
* Test4: 2 broker2, 2 partitions, required.acks=2, 100 byte messages, gzip
compression: 230000 messages/second, 23 MB/second
log.flush broker configuration was increased to avoid the disk being the
bottleneck.
/Magnus
2013/11/24 Neha Narkhede <ne...@gmail.com>
> So, a single producer'a throughput is 80 MB/s? That seems pretty high. What
> was the number of acks setting? Thanks for sharing these numbers.
>
> On Sunday, November 24, 2013, Magnus Edenhill wrote:
>
> > Hi Neha,
> >
> > these tests were done using 100 byte messages. More information about the
> > producer performance tests can be found here:
> >
> >
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
> >
> > The tests are indicative at best and in no way scientific, but I must say
> > that the Kafka broker performance is impressive.
> >
> > Regards,
> > Magnus
> >
> >
> >
> > 2013/11/22 Neha Narkhede <neha.narkhede@gmail.com <javascript:;>>
> >
> > > Thanks for sharing this! What is the message size for the throughput
> > > numbers stated below?
> > >
> > > Thanks,
> > > Neha
> > > On Nov 22, 2013 6:59 AM, "Magnus Edenhill" <magnus@edenhill.se
> <javascript:;>>
> > wrote:
> > >
> > > > This announces the 0.8.0 release of librdkafka - The Apache Kafka
> > client
> > > C
> > > > library - now with 0.8 protocol support.
> > > >
> > > > Features:
> > > > * Producer (~800K msgs/s)
> > > > * Consumer (~3M msgs/s)
> > > > * Compression (Snappy, gzip)
> > > > * Proper failover and leader re-election support - no message is ever
> > > lost.
> > > > * Configuration properties compatible with official Apache Kafka.
> > > > * Stabilized ABI-safe API
> > > > * Mainline Debian package submitted
> > > > * Production quality
> > > >
> > > >
> > > > Home:
> > > > https://github.com/edenhill/librdkafka
> > > >
> > > > Introduction and performance numbers:
> > > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md
> > > >
> > > > Have fun.
> > > >
> > > > Regards,
> > > > Magnus
> > > >
> > > > P.S.
> > > > Check out Wikimedia Foundation's varnishkafka daemon for a use case -
> > > > varnish log forwarding over Kafka:
> > > > https://github.com/wikimedia/varnishkafka
> > > >
> > >
> >
>
Re: librdkafka 0.8.0 released
Posted by Neha Narkhede <ne...@gmail.com>.
So, a single producer'a throughput is 80 MB/s? That seems pretty high. What
was the number of acks setting? Thanks for sharing these numbers.
On Sunday, November 24, 2013, Magnus Edenhill wrote:
> Hi Neha,
>
> these tests were done using 100 byte messages. More information about the
> producer performance tests can be found here:
>
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
>
> The tests are indicative at best and in no way scientific, but I must say
> that the Kafka broker performance is impressive.
>
> Regards,
> Magnus
>
>
>
> 2013/11/22 Neha Narkhede <neha.narkhede@gmail.com <javascript:;>>
>
> > Thanks for sharing this! What is the message size for the throughput
> > numbers stated below?
> >
> > Thanks,
> > Neha
> > On Nov 22, 2013 6:59 AM, "Magnus Edenhill" <magnus@edenhill.se<javascript:;>>
> wrote:
> >
> > > This announces the 0.8.0 release of librdkafka - The Apache Kafka
> client
> > C
> > > library - now with 0.8 protocol support.
> > >
> > > Features:
> > > * Producer (~800K msgs/s)
> > > * Consumer (~3M msgs/s)
> > > * Compression (Snappy, gzip)
> > > * Proper failover and leader re-election support - no message is ever
> > lost.
> > > * Configuration properties compatible with official Apache Kafka.
> > > * Stabilized ABI-safe API
> > > * Mainline Debian package submitted
> > > * Production quality
> > >
> > >
> > > Home:
> > > https://github.com/edenhill/librdkafka
> > >
> > > Introduction and performance numbers:
> > > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md
> > >
> > > Have fun.
> > >
> > > Regards,
> > > Magnus
> > >
> > > P.S.
> > > Check out Wikimedia Foundation's varnishkafka daemon for a use case -
> > > varnish log forwarding over Kafka:
> > > https://github.com/wikimedia/varnishkafka
> > >
> >
>
Re: librdkafka 0.8.0 released
Posted by Magnus Edenhill <ma...@edenhill.se>.
Hi Neha,
these tests were done using 100 byte messages. More information about the
producer performance tests can be found here:
https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#performance-numbers
The tests are indicative at best and in no way scientific, but I must say
that the Kafka broker performance is impressive.
Regards,
Magnus
2013/11/22 Neha Narkhede <ne...@gmail.com>
> Thanks for sharing this! What is the message size for the throughput
> numbers stated below?
>
> Thanks,
> Neha
> On Nov 22, 2013 6:59 AM, "Magnus Edenhill" <ma...@edenhill.se> wrote:
>
> > This announces the 0.8.0 release of librdkafka - The Apache Kafka client
> C
> > library - now with 0.8 protocol support.
> >
> > Features:
> > * Producer (~800K msgs/s)
> > * Consumer (~3M msgs/s)
> > * Compression (Snappy, gzip)
> > * Proper failover and leader re-election support - no message is ever
> lost.
> > * Configuration properties compatible with official Apache Kafka.
> > * Stabilized ABI-safe API
> > * Mainline Debian package submitted
> > * Production quality
> >
> >
> > Home:
> > https://github.com/edenhill/librdkafka
> >
> > Introduction and performance numbers:
> > https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md
> >
> > Have fun.
> >
> > Regards,
> > Magnus
> >
> > P.S.
> > Check out Wikimedia Foundation's varnishkafka daemon for a use case -
> > varnish log forwarding over Kafka:
> > https://github.com/wikimedia/varnishkafka
> >
>
Re: librdkafka 0.8.0 released
Posted by Neha Narkhede <ne...@gmail.com>.
Thanks for sharing this! What is the message size for the throughput
numbers stated below?
Thanks,
Neha
On Nov 22, 2013 6:59 AM, "Magnus Edenhill" <ma...@edenhill.se> wrote:
> This announces the 0.8.0 release of librdkafka - The Apache Kafka client C
> library - now with 0.8 protocol support.
>
> Features:
> * Producer (~800K msgs/s)
> * Consumer (~3M msgs/s)
> * Compression (Snappy, gzip)
> * Proper failover and leader re-election support - no message is ever lost.
> * Configuration properties compatible with official Apache Kafka.
> * Stabilized ABI-safe API
> * Mainline Debian package submitted
> * Production quality
>
>
> Home:
> https://github.com/edenhill/librdkafka
>
> Introduction and performance numbers:
> https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md
>
> Have fun.
>
> Regards,
> Magnus
>
> P.S.
> Check out Wikimedia Foundation's varnishkafka daemon for a use case -
> varnish log forwarding over Kafka:
> https://github.com/wikimedia/varnishkafka
>