RE: [External] Kafka - Is it possible to achieve low latency (below 5ms?)

["Tauzell, Dave" <Da...@surescripts.com>]

What does the hardware side of your brokers look like - do you have enough memory to hold all pending messages in memory (i.e. before consumers get them).
At what rate are your clients trying to send messages?

-Dave

-----Original Message-----
From: Pritam Kadam [mailto:kpritam@thoughtworks.com]
Sent: Saturday, June 2, 2018 7:41 AM
To: users@kafka.apache.org
Subject: [External] Kafka - Is it possible to achieve low latency (below 5ms?)

Hi,

I have been playing around with different kafka configurations at producer, broker and consumer side.

But could not figure out a way to get predictable low latency below 5 to 10 ms.

*Use Case:*

   - Application has 100-200 producers, each producer is producing messages
   at different rates. Ex. 1 msg/sec, 10 msgs/sec, 100msgs/sec (100msg/sec is
   the max limit)
   - Each producer is producing messages on unique topic on a same
   partition so that ordering is maintained which is business requirement.
   - Every producer has corresponding consumers. Some producers might have
   1-10 consumers.

As I look at the above use case, I do not see I would be getting benefits of batching which kafka is heavily depends on for getting maximum throughput.

From whole system I expect max throughput of around 40 - 50k but we would like to achieve latency below 5-10 ms.

Is it possible with kafka to get this numbers of latency and throughput?

Note: I tried different configs focussing on latency part, but could not get latency below 150ms, for example

kafka-clients {
  linger.ms=0
  acks=0
  batch.size=1
}

akka.kafka.consumer {
  poll-interval = 0
  poll-timeout = 0
}


Thanks,
Pritam.
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.