Kafka - Is it possible to achieve low latency (below 5ms?)

[Pritam Kadam <kp...@thoughtworks.com>]

Hi,

I have been playing around with different kafka configurations at producer,
broker and consumer side.

But could not figure out a way to get predictable low latency below 5 to 10
ms.

*Use Case:*

   - Application has 100-200 producers, each producer is producing messages
   at different rates. Ex. 1 msg/sec, 10 msgs/sec, 100msgs/sec (100msg/sec is
   the max limit)
   - Each producer is producing messages on unique topic on a same
   partition so that ordering is maintained which is business requirement.
   - Every producer has corresponding consumers. Some producers might have
   1-10 consumers.

As I look at the above use case, I do not see I would be getting benefits
of batching which kafka is heavily depends on for getting maximum
throughput.

From whole system I expect max throughput of around 40 - 50k but we would
like to achieve latency below 5-10 ms.

Is it possible with kafka to get this numbers of latency and throughput?

Note: I tried different configs focussing on latency part, but could not
get latency below 150ms, for example

kafka-clients {
  linger.ms=0
  acks=0
  batch.size=1
}

akka.kafka.consumer {
  poll-interval = 0
  poll-timeout = 0
}


Thanks,
Pritam.

RE: [External] Kafka - Is it possible to achieve low latency (below 5ms?)

["Tauzell, Dave" <Da...@surescripts.com>]

What does the hardware side of your brokers look like - do you have enough memory to hold all pending messages in memory (i.e. before consumers get them).
At what rate are your clients trying to send messages?

-Dave

-----Original Message-----
From: Pritam Kadam [mailto:kpritam@thoughtworks.com]
Sent: Saturday, June 2, 2018 7:41 AM
To: users@kafka.apache.org
Subject: [External] Kafka - Is it possible to achieve low latency (below 5ms?)

Hi,

I have been playing around with different kafka configurations at producer, broker and consumer side.

But could not figure out a way to get predictable low latency below 5 to 10 ms.

*Use Case:*

   - Application has 100-200 producers, each producer is producing messages
   at different rates. Ex. 1 msg/sec, 10 msgs/sec, 100msgs/sec (100msg/sec is
   the max limit)
   - Each producer is producing messages on unique topic on a same
   partition so that ordering is maintained which is business requirement.
   - Every producer has corresponding consumers. Some producers might have
   1-10 consumers.

As I look at the above use case, I do not see I would be getting benefits of batching which kafka is heavily depends on for getting maximum throughput.

From whole system I expect max throughput of around 40 - 50k but we would like to achieve latency below 5-10 ms.

Is it possible with kafka to get this numbers of latency and throughput?

Note: I tried different configs focussing on latency part, but could not get latency below 150ms, for example

kafka-clients {
  linger.ms=0
  acks=0
  batch.size=1
}

akka.kafka.consumer {
  poll-interval = 0
  poll-timeout = 0
}


Thanks,
Pritam.
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.

Re: Kafka - Is it possible to achieve low latency (below 5ms?)

[Matt Howlett <ma...@confluent.io>]

produce latency or producer -> consumer latency?

150ms is very high. Have a look at the
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce broker
metric which is the total message produce time. This will also account for
most of the end-to-end latency the broker is responsible for, so it will
allow you to narrow down where the bottleneck is.

If your message throughput per client is very low, something that recently
caught me out is the bad interaction between TCP acks and the Nagle
algorithm (i.e. increasing throughput will actually help latency if you're
hitting this).

If your message rate is high, increasing batch size / linger time may
actually help average latency (at the expense of worse best case latency)
since there will be less requests, each of which has a small amount of
overhead.

In some tests i've been doing against confluent cloud (acks = all), I've
been able to get p50 end-to-end latencies of ~6ms / p99 latencies ~30ms (in
the best case scenario). That's probably an upper bound on what you can
expect to get. I would guess p50 latencies of around 15ms should be
relatively doable in practice.



On Sat, Jun 2, 2018 at 5:41 AM, Pritam Kadam <kp...@thoughtworks.com>
wrote:

> Hi,
>
> I have been playing around with different kafka configurations at producer,
> broker and consumer side.
>
> But could not figure out a way to get predictable low latency below 5 to 10
> ms.
>
> *Use Case:*
>
>    - Application has 100-200 producers, each producer is producing messages
>    at different rates. Ex. 1 msg/sec, 10 msgs/sec, 100msgs/sec (100msg/sec
> is
>    the max limit)
>    - Each producer is producing messages on unique topic on a same
>    partition so that ordering is maintained which is business requirement.
>    - Every producer has corresponding consumers. Some producers might have
>    1-10 consumers.
>
> As I look at the above use case, I do not see I would be getting benefits
> of batching which kafka is heavily depends on for getting maximum
> throughput.
>
> From whole system I expect max throughput of around 40 - 50k but we would
> like to achieve latency below 5-10 ms.
>
> Is it possible with kafka to get this numbers of latency and throughput?
>
> Note: I tried different configs focussing on latency part, but could not
> get latency below 150ms, for example
>
> kafka-clients {
>   linger.ms=0
>   acks=0
>   batch.size=1
> }
>
> akka.kafka.consumer {
>   poll-interval = 0
>   poll-timeout = 0
> }
>
>
> Thanks,
> Pritam.
>