Kafka delay

[Patricia Callejo <pa...@imdea.org>]

Hi, 
we are developing a kafka environment in order to be able to process up hundreds of messages per second. The point is that if we send in different partitions a lot of messages, the time of the process is very good in ms, however if we send just a single message, the minimum time required is very high, at least 1 or 2 seconds, even more. 
Is there any configuration we are missing to avoid this minimum delay? or is the minimum time required by kafka for the whole process? 

Best Regards,
Patricia 

Re: Kafka delay

[Robert Quinlivan <rq...@signal.co>]

Can you clarify what you mean by "process time"? Is this the time it takes
to have a message sent from the producer appear in a consumer?

You may be seeing the result of batching. The producer batches messages so
if you are testing sending a single message at a time you will get
different performance characteristics than if you are sending many.

On Fri, Jan 13, 2017 at 2:36 AM, Patricia Callejo <
patricia.callejo@imdea.org> wrote:

> Hi,
> we are developing a kafka environment in order to be able to process up
> hundreds of messages per second. The point is that if we send in different
> partitions a lot of messages, the time of the process is very good in ms,
> however if we send just a single message, the minimum time required is very
> high, at least 1 or 2 seconds, even more.
> Is there any configuration we are missing to avoid this minimum delay? or
> is the minimum time required by kafka for the whole process?
>
> Best Regards,
> Patricia




-- 
Robert Quinlivan
Software Engineer, Signal

Re: Kafka delay

[Hans Jespersen <ha...@confluent.io>]

batch.size, linger.ms, max.in.flight.requests.per.connection, etc all
effect the trade off between latency and throughput. Your throughput is
very low so you should be able to tune for much better latency. Very easy
to get 1-2 ms of latency out of Kafka.

Here is a video on Tuning Kafka for low latency guaranteed messaging --
Jiangjie (Becket) Qin (LinkedIn)
<https://www.youtube.com/watch?v=oQe7PpDDdzA> which might give you much
better context and understanding of what these parameters mean and how they
work.

-hans

/**
 * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
 * hans@confluent.io (650)924-2670
 */

On Fri, Jan 13, 2017 at 12:36 AM, Patricia Callejo <
patricia.callejo@imdea.org> wrote:

> Hi,
> we are developing a kafka environment in order to be able to process up
> hundreds of messages per second. The point is that if we send in different
> partitions a lot of messages, the time of the process is very good in ms,
> however if we send just a single message, the minimum time required is very
> high, at least 1 or 2 seconds, even more.
> Is there any configuration we are missing to avoid this minimum delay? or
> is the minimum time required by kafka for the whole process?
>
> Best Regards,
> Patricia