You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Abhishek Choudhary <xa...@gmail.com> on 2018/11/16 09:00:32 UTC
Producer throughput with varying acks=0,1,-1
Hi,
I have been doing some performance tests with kafka cluster for my project.
I have a question regarding the send call and the 'acks' property of
producer. I observed below numbers with below invocation of send call. This
is a simple fire and forget call.
producer.send(record);
The topic has 5 partitions and I see below results with different acks
value and replication factor. The kafka cluster has 5 nodes running with
default values and using local disk
*acks Replication factor=1
Replication factor=3*
0 1330k msgs/sec
1260k msgs/sec
1 1220k msgs/sec
1200k msgs/sec
-1(all) 1220k msgs/sec
325k msgs/sec
As you can see as the acks value changes from 0 to all, the producer
throughput decreases. What I am not able to understand is that if the
producer send call is fire and forget in nature and producer is not waiting
for any acknowledgements then why does the producer throughput drops as we
move to stronger acks guarantees?
Any insights into how acks and producer send call works in Kakfa would be
greatly appreciated.
Thanks,
Abhishek
Re: Producer throughput with varying acks=0,1,-1
Posted by "Matthias J. Sax" <ma...@confluent.io>.
The producer has a config `max.in.flight.requests.per.connection` and
because of `ack=all`, even if you don't check the `Future`, a request
will be in-flight longer, because the leader needs to wait for the
followers to replicate the data before it can send the ack back.
Thus, I assume that you are limited by
`max.in.flight.request.per.connection` (you could try to increase it
though). However, if you don't check the `Future` from the producer,
setting ack=all is pointless anyway (and I would consider it a
miss-configuration).
-Matthias
On 11/18/18 8:41 PM, Srinivas, Kaushik (Nokia - IN/Bangalore) wrote:
> But if sends are not done in blocking way (with .get()) how does acks matter ?
>
> -----Original Message-----
> From: Matthias J. Sax <ma...@confluent.io>
> Sent: Saturday, November 17, 2018 12:15 AM
> To: users@kafka.apache.org
> Subject: Re: Producer throughput with varying acks=0,1,-1
>
> I you enable acks, it's not fire and forget any longer.
>
> -Matthias
>
> On 11/16/18 1:00 AM, Abhishek Choudhary wrote:
>> Hi,
>>
>> I have been doing some performance tests with kafka cluster for my project.
>> I have a question regarding the send call and the 'acks' property of
>> producer. I observed below numbers with below invocation of send call.
>> This is a simple fire and forget call.
>>
>> producer.send(record);
>>
>> The topic has 5 partitions and I see below results with different acks
>> value and replication factor. The kafka cluster has 5 nodes running
>> with default values and using local disk
>>
>> *acks Replication factor=1
>> Replication factor=3*
>> 0 1330k msgs/sec
>> 1260k msgs/sec
>> 1 1220k msgs/sec
>> 1200k msgs/sec
>> -1(all) 1220k msgs/sec
>> 325k msgs/sec
>>
>>
>> As you can see as the acks value changes from 0 to all, the producer
>> throughput decreases. What I am not able to understand is that if the
>> producer send call is fire and forget in nature and producer is not
>> waiting for any acknowledgements then why does the producer throughput
>> drops as we move to stronger acks guarantees?
>>
>> Any insights into how acks and producer send call works in Kakfa would
>> be greatly appreciated.
>>
>> Thanks,
>> Abhishek
>>
>
RE: Producer throughput with varying acks=0,1,-1
Posted by "Srinivas, Kaushik (Nokia - IN/Bangalore)" <ka...@nokia.com>.
But if sends are not done in blocking way (with .get()) how does acks matter ?
-----Original Message-----
From: Matthias J. Sax <ma...@confluent.io>
Sent: Saturday, November 17, 2018 12:15 AM
To: users@kafka.apache.org
Subject: Re: Producer throughput with varying acks=0,1,-1
I you enable acks, it's not fire and forget any longer.
-Matthias
On 11/16/18 1:00 AM, Abhishek Choudhary wrote:
> Hi,
>
> I have been doing some performance tests with kafka cluster for my project.
> I have a question regarding the send call and the 'acks' property of
> producer. I observed below numbers with below invocation of send call.
> This is a simple fire and forget call.
>
> producer.send(record);
>
> The topic has 5 partitions and I see below results with different acks
> value and replication factor. The kafka cluster has 5 nodes running
> with default values and using local disk
>
> *acks Replication factor=1
> Replication factor=3*
> 0 1330k msgs/sec
> 1260k msgs/sec
> 1 1220k msgs/sec
> 1200k msgs/sec
> -1(all) 1220k msgs/sec
> 325k msgs/sec
>
>
> As you can see as the acks value changes from 0 to all, the producer
> throughput decreases. What I am not able to understand is that if the
> producer send call is fire and forget in nature and producer is not
> waiting for any acknowledgements then why does the producer throughput
> drops as we move to stronger acks guarantees?
>
> Any insights into how acks and producer send call works in Kakfa would
> be greatly appreciated.
>
> Thanks,
> Abhishek
>
Re: Producer throughput with varying acks=0,1,-1
Posted by "Matthias J. Sax" <ma...@confluent.io>.
I you enable acks, it's not fire and forget any longer.
-Matthias
On 11/16/18 1:00 AM, Abhishek Choudhary wrote:
> Hi,
>
> I have been doing some performance tests with kafka cluster for my project.
> I have a question regarding the send call and the 'acks' property of
> producer. I observed below numbers with below invocation of send call. This
> is a simple fire and forget call.
>
> producer.send(record);
>
> The topic has 5 partitions and I see below results with different acks
> value and replication factor. The kafka cluster has 5 nodes running with
> default values and using local disk
>
> *acks Replication factor=1
> Replication factor=3*
> 0 1330k msgs/sec
> 1260k msgs/sec
> 1 1220k msgs/sec
> 1200k msgs/sec
> -1(all) 1220k msgs/sec
> 325k msgs/sec
>
>
> As you can see as the acks value changes from 0 to all, the producer
> throughput decreases. What I am not able to understand is that if the
> producer send call is fire and forget in nature and producer is not waiting
> for any acknowledgements then why does the producer throughput drops as we
> move to stronger acks guarantees?
>
> Any insights into how acks and producer send call works in Kakfa would be
> greatly appreciated.
>
> Thanks,
> Abhishek
>