You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Simon Cooper <si...@featurespace.co.uk> on 2017/05/22 15:04:54 UTC

Getting the consumer to operate deterministically with small messages

Hi,

I'm having significant problems getting the kafka consumer to operate deterministically with small message numbers & sizes (this is for local testing).

I'm controlling the offset manually, and using manual partition/topic assignment. I've set auto commit off, and set fetch.min.bytes to 1.

However, with 2 small (4-byte) messages in the queue, calling poll(0) on the consumer does not return them - it returns an empty list. poll(1) doesn't always return the messages either (sometimes it does, sometimes it doesn't).

How do I get the consumer to operate entirely deterministically - so that when I call poll(n) it is guaranteed to return all the messages available in the queue at that point, regardless of how small they are or how few messages there are in the queue, without blocking?

Thanks,
SimonC

RE: Getting the consumer to operate deterministically with small messages

Posted by Simon Cooper <si...@featurespace.co.uk>.
Forgot to mention - this is on 0.9. We can't upgrade to 0.10 yet, as we haven't upgraded our brokers.

-----Original Message-----
From: Simon Cooper [mailto:simon.cooper@featurespace.co.uk] 
Sent: 22 May 2017 16:05
To: users@kafka.apache.org
Subject: Getting the consumer to operate deterministically with small messages

Hi,

I'm having significant problems getting the kafka consumer to operate deterministically with small message numbers & sizes (this is for local testing).

I'm controlling the offset manually, and using manual partition/topic assignment. I've set auto commit off, and set fetch.min.bytes to 1.

However, with 2 small (4-byte) messages in the queue, calling poll(0) on the consumer does not return them - it returns an empty list. poll(1) doesn't always return the messages either (sometimes it does, sometimes it doesn't).

How do I get the consumer to operate entirely deterministically - so that when I call poll(n) it is guaranteed to return all the messages available in the queue at that point, regardless of how small they are or how few messages there are in the queue, without blocking?

Thanks,
SimonC