enable.auto.commit semantics

[Peter Levart <pe...@gmail.com>]

Hello,

I'm using Java KafkaConsumer and I'm wondering what is the guarantee of 
consuming messages when enable.auto.commit is set to true.

What is the earliest time the offsets of messages returned from last 
poll() may be committed? Immediately after returning from poll() or upon 
next call to poll()?

If it is the former, am I right to assume that messages may get lost in 
the event of consumer crash?

If that is the case then the only way to guarantee that no message will 
get lost is to commit consumed offsets manually after processing the 
messages returned from poll(), right?

Thanks, Peter

Re: enable.auto.commit semantics

[Pere Urbón Bayes <pe...@gmail.com>]

Hi,
 from the confluent docs,

> Offset Management: The two main settings affecting offset management are
whether auto-commit is enabled and the offset reset policy. First, if you
set enable.auto.commit (which is the default), then the consumer will
automatically commit offsets periodically at the interval set by
auto.commit.interval.ms. The default is 5 seconds.

As you can see
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L161

the coordinator is the responsible of committing the offsets every interval.

this is
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L358
as well another point when the offsets gets committed.

From your message:

> If that is the case then the only way to guarantee that no message will
get lost is to commit consumed offsets manually after processing the
messages returned from poll(), right?

messages in the consumers don't get lost, you mean you don't process them
twice, right?

Cheers

Missatge de Peter Levart <pe...@gmail.com> del dia dj., 31 de gen.
2019 a les 23:23:

> Hello,
>
> I'm using Java KafkaConsumer and I'm wondering what is the guarantee of
> consuming messages when enable.auto.commit is set to true.
>
> What is the earliest time the offsets of messages returned from last
> poll() may be committed? Immediately after returning from poll() or upon
> next call to poll()?
>
> If it is the former, am I right to assume that messages may get lost in
> the event of consumer crash?
>
> If that is the case then the only way to guarantee that no message will
> get lost is to commit consumed offsets manually after processing the
> messages returned from poll(), right?
>
> Thanks, Peter
>
>

-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/