You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Orelowitz, David" <da...@baml.com> on 2014/12/18 22:38:32 UTC
In Flight Requests
I notice that there is parameter max.in.flight.requests.per.connection in the ProducerConfig.java code that can be set. It is defaulted to 5.
This is not documented in the 0.8.2 documentation.
http://kafka.apache.org/082/documentation.html#newproducerconfigs
Is it something we should play with? Does anyone have any experience with the parameter?
Thanks,
David
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
RE: In Flight Requests
Posted by "Orelowitz, David" <da...@baml.com>.
Ray,
Thank you for the detailed explanation. I mistakenly thought this determined the number of messages that were outstanding - not the number of requests (batch of messages) that could be outstanding.
This makes sense.
Thank you.
-----Original Message-----
From: Jay Kreps [mailto:jay@confluent.io]
Sent: Thursday, December 18, 2014 5:04 PM
To: users@kafka.apache.org
Subject: Re: In Flight Requests
Hi David,
Each request sent to Kafka gets acknowledged. The protocol allows multiple requests to be sent on a connection without waiting on a connection. The number of requests currently awaiting acknowledgement is the in flight request count. By default once there are five unacknowledged requests on a connection the client will wait for one of these to get acknowledged before sending the next request to that node.
In general this is a fairly low-level performance tuning parameter and not one you likely need to monkey with unless you are trying to eek the last bit of performance out.
The impact is the following: sending more than one request at a time is good as you can have one request being sent while the previous one is being processed. So allowing more than one in-flight request makes sense. However allowing an unbounded number of in-flight requests is actually not a good thing--the reason is that the client will batch together messages that come in at the same time for the same topic/partition and these big batches are much more efficient to process than single message requests. This is one of the primary performance optimizations in the producer (see https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines).
So once we have a few requests in flight, it is best to stop sending more and more small requests (which are just going to queue up in the network stack anyway) and let the messages batch together into big, efficient batch requests.
The impact of this is that the more message acknowledgement falls behind, the more efficient the producer gets at creating big batch requests, and hence the better throughput gets. This means let's the producer dynamically batch for latency under load without any manually configured backoff.
In my testing this setting actually isn't very sensitive as long as there is some reasonable cap that forces the batching to kick in as requests back up.
We'll get the docs for this in with the final 0.8.2 release.
-Jay
On Thu, Dec 18, 2014 at 1:38 PM, Orelowitz, David <da...@baml.com>
wrote:
>
> I notice that there is parameter max.in.flight.requests.per.connection
> in the ProducerConfig.java code that can be set. It is defaulted to 5.
>
> This is not documented in the 0.8.2 documentation.
>
> http://kafka.apache.org/082/documentation.html#newproducerconfigs
>
> Is it something we should play with? Does anyone have any experience
> with the parameter?
>
> Thanks,
> David
>
>
> ----------------------------------------------------------------------
> This message, and any attachments, is for the intended recipient(s)
> only, may contain information that is privileged, confidential and/or
> proprietary and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the
> intended recipient, please delete this message.
>
----------------------------------------------------------------------
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.bankofamerica.com/emaildisclaimer. If you are not the intended recipient, please delete this message.
Re: In Flight Requests
Posted by Jay Kreps <ja...@confluent.io>.
Hi David,
Each request sent to Kafka gets acknowledged. The protocol allows multiple
requests to be sent on a connection without waiting on a connection. The
number of requests currently awaiting acknowledgement is the in flight
request count. By default once there are five unacknowledged requests on a
connection the client will wait for one of these to get acknowledged before
sending the next request to that node.
In general this is a fairly low-level performance tuning parameter and not
one you likely need to monkey with unless you are trying to eek the last
bit of performance out.
The impact is the following: sending more than one request at a time is
good as you can have one request being sent while the previous one is being
processed. So allowing more than one in-flight request makes sense. However
allowing an unbounded number of in-flight requests is actually not a good
thing--the reason is that the client will batch together messages that come
in at the same time for the same topic/partition and these big batches are
much more efficient to process than single message requests. This is one of
the primary performance optimizations in the producer (see
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines).
So once we have a few requests in flight, it is best to stop sending more
and more small requests (which are just going to queue up in the network
stack anyway) and let the messages batch together into big, efficient batch
requests.
The impact of this is that the more message acknowledgement falls behind,
the more efficient the producer gets at creating big batch requests, and
hence the better throughput gets. This means let's the producer dynamically
batch for latency under load without any manually configured backoff.
In my testing this setting actually isn't very sensitive as long as there
is some reasonable cap that forces the batching to kick in as requests back
up.
We'll get the docs for this in with the final 0.8.2 release.
-Jay
On Thu, Dec 18, 2014 at 1:38 PM, Orelowitz, David <da...@baml.com>
wrote:
>
> I notice that there is parameter max.in.flight.requests.per.connection in
> the ProducerConfig.java code that can be set. It is defaulted to 5.
>
> This is not documented in the 0.8.2 documentation.
>
> http://kafka.apache.org/082/documentation.html#newproducerconfigs
>
> Is it something we should play with? Does anyone have any experience with
> the parameter?
>
> Thanks,
> David
>
>
> ----------------------------------------------------------------------
> This message, and any attachments, is for the intended recipient(s) only,
> may contain information that is privileged, confidential and/or proprietary
> and subject to important terms and conditions available at
> http://www.bankofamerica.com/emaildisclaimer. If you are not the
> intended recipient, please delete this message.
>