KafkaConsumer poll(timeout) doesn't seem to work as expected

[Josh Goodrich <ga...@gmail.com>]

The poll(timeout) method of the Java KafkaConsumer API doesn’t behave the
way you would think.  If you create a new Consumer with a groupId that has
been seen before, even if there are new events in the topic if you issue a
poll(0) it never returns any records.  I find I have to put in a loop of 2
poll(500) calls where the first call pretty much always returns nothing,
but the second will return some records (though not always all available).
Why does this happen and what’s the solution?



Josh

Re: KafkaConsumer poll(timeout) doesn't seem to work as expected

[Ewen Cheslack-Postava <ew...@confluent.io>]

Hey Josh,

There's no guarantee that a poll() will return data. It might send a
request, but if it takes longer than the timeout to retrieve some data from
the brokers, then the call will have no data to return. This doesn't
indicate anything is wrong, just that no data has been returned yet.

poll(0), in particular, is *very* likely to return no data. Unless a
prefetch for data was sent and returned, it is very unlikely any data will
be available to return to the consumer. With 500ms timeouts, it seems more
likely that data will be returned, but it still isn't a guarantee.

-Ewen

On Wed, Jul 13, 2016 at 2:37 PM, Josh Goodrich <ga...@gmail.com> wrote:

> The poll(timeout) method of the Java KafkaConsumer API doesn’t behave the
> way you would think.  If you create a new Consumer with a groupId that has
> been seen before, even if there are new events in the topic if you issue a
> poll(0) it never returns any records.  I find I have to put in a loop of 2
> poll(500) calls where the first call pretty much always returns nothing,
> but the second will return some records (though not always all available).
> Why does this happen and what’s the solution?
>
>
>
> Josh
>



-- 
Thanks,
Ewen