kafka manual commit vs auto commit

[Shantanu Deshmukh <sh...@gmail.com>]

Hello everyone,

We have a 3 broker Kafka 0.10.1.0 cluster in production environment. Lately
we are seeing a lot of "auto commit failed because poll() spend too much
time processing" warning messages. Also, due to such events there is
constant fear of duplicate messages and the same does happen. To tackle
this, I have tried a number of configuration combinations. I have set
max.poll.records to as low as 5. Increased max.poll.interval.ms to 10
minutes. increased session.timeout.ms to 5 minutes etc. However, this has
not helped. So, this has got me wondering shall we go after manual commit
of offsets?

Our consumer process has one topic which has produce rate of about 15k
events per second. It takes about 25ms to process one event of that topic
and we have 5 consumers for that topic. If I do manual commit of every
single event that we process what cost will it incur on host machine or
network etc? Is there a better optimized method of manual commit? Or better
yet, how to avoid "auto commit failed" error?

*Thanks & Regards,*
*Shantanu Deshmukh*

Re: kafka manual commit vs auto commit

[Shantanu Deshmukh <sh...@gmail.com>]

So here, when I received a message I run some business logic on it and try
to send some email. Now sometimes we have a promotional campaign running
millions of emails need to be delivered. For such numerous events is manual
commit good? Will it generate too much network activity if I commit a
single message every time?

On Thu, May 24, 2018 at 6:20 PM M. Manna <ma...@gmail.com> wrote:

> Manual commit is important where event consumption eventually leads to some
> post-processing/database update/state change for your application. Without
> doing all those, you cannot truly say that you have "Received" the message.
> "Receiving" is interpreted differently and it's up to your target
> application.
>
> Jason Gustafson has written some descriptive post regarding transactions. I
> don't know if you can extrapolate your changes based on those:
>
> 1) https://www.confluent.io/blog/transactions-apache-kafka/
> 2)
>
> https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/
>
> ALso, please don't forget to read Javadoc on KafkaConsumer.java
>
> Regards,
>
> On 24 May 2018 at 13:29, Shantanu Deshmukh <sh...@gmail.com> wrote:
>
> > Hello everyone,
> >
> > We have a 3 broker Kafka 0.10.1.0 cluster in production environment.
> Lately
> > we are seeing a lot of "auto commit failed because poll() spend too much
> > time processing" warning messages. Also, due to such events there is
> > constant fear of duplicate messages and the same does happen. To tackle
> > this, I have tried a number of configuration combinations. I have set
> > max.poll.records to as low as 5. Increased max.poll.interval.ms to 10
> > minutes. increased session.timeout.ms to 5 minutes etc. However, this
> has
> > not helped. So, this has got me wondering shall we go after manual commit
> > of offsets?
> >
> > Our consumer process has one topic which has produce rate of about 15k
> > events per second. It takes about 25ms to process one event of that topic
> > and we have 5 consumers for that topic. If I do manual commit of every
> > single event that we process what cost will it incur on host machine or
> > network etc? Is there a better optimized method of manual commit? Or
> better
> > yet, how to avoid "auto commit failed" error?
> >
> > *Thanks & Regards,*
> > *Shantanu Deshmukh*
> >
>

Re: kafka manual commit vs auto commit

["M. Manna" <ma...@gmail.com>]

Manual commit is important where event consumption eventually leads to some
post-processing/database update/state change for your application. Without
doing all those, you cannot truly say that you have "Received" the message.
"Receiving" is interpreted differently and it's up to your target
application.

Jason Gustafson has written some descriptive post regarding transactions. I
don't know if you can extrapolate your changes based on those:

1) https://www.confluent.io/blog/transactions-apache-kafka/
2)
https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/

ALso, please don't forget to read Javadoc on KafkaConsumer.java

Regards,

On 24 May 2018 at 13:29, Shantanu Deshmukh <sh...@gmail.com> wrote:

> Hello everyone,
>
> We have a 3 broker Kafka 0.10.1.0 cluster in production environment. Lately
> we are seeing a lot of "auto commit failed because poll() spend too much
> time processing" warning messages. Also, due to such events there is
> constant fear of duplicate messages and the same does happen. To tackle
> this, I have tried a number of configuration combinations. I have set
> max.poll.records to as low as 5. Increased max.poll.interval.ms to 10
> minutes. increased session.timeout.ms to 5 minutes etc. However, this has
> not helped. So, this has got me wondering shall we go after manual commit
> of offsets?
>
> Our consumer process has one topic which has produce rate of about 15k
> events per second. It takes about 25ms to process one event of that topic
> and we have 5 consumers for that topic. If I do manual commit of every
> single event that we process what cost will it incur on host machine or
> network etc? Is there a better optimized method of manual commit? Or better
> yet, how to avoid "auto commit failed" error?
>
> *Thanks & Regards,*
> *Shantanu Deshmukh*
>