You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Bhavesh Mistry <mi...@gmail.com> on 2014/10/28 19:44:18 UTC

High Level Consumer and Close with Auto Commit On

Hi Kafka Team,

What is expected behavior when you close *ConsumerConnector* and auto
commit is on ?  Basically, when auto commit interval is set to 5 seconds
and shutdown is called (before 5 seconds elapses) does ConsumerConnector
commit the offset of message consumed by (next()) method or consumer will
get duplicate messages when it comes online after restart ?

ConsumerConnector.shutdown();

Thanks,

Bhavesh

Re: High Level Consumer and Close with Auto Commit On

Posted by Bhavesh Mistry <mi...@gmail.com>.
HI Gwen,

Thanks for info appreciate. Sometime, I have observed duplicated events...
so that is why I ask.

Thanks,

Bhavesh

On Tue, Oct 28, 2014 at 11:51 AM, Gwen Shapira <gs...@cloudera.com>
wrote:

> High level consumer commits before shutting down.
>
> If you'll look at ZookeeperConsumerConnector.scala (currently the only
> implementation of ConsumerConnector) you'll see shutdown() includes
> the following:
>
>           if (config.autoCommitEnable)
>             commitOffsets()
>
> Gwen
>
> On Tue, Oct 28, 2014 at 11:44 AM, Bhavesh Mistry
> <mi...@gmail.com> wrote:
> > Hi Kafka Team,
> >
> > What is expected behavior when you close *ConsumerConnector* and auto
> > commit is on ?  Basically, when auto commit interval is set to 5 seconds
> > and shutdown is called (before 5 seconds elapses) does ConsumerConnector
> > commit the offset of message consumed by (next()) method or consumer will
> > get duplicate messages when it comes online after restart ?
> >
> > ConsumerConnector.shutdown();
> >
> > Thanks,
> >
> > Bhavesh
>

Re: High Level Consumer and Close with Auto Commit On

Posted by Gwen Shapira <gs...@cloudera.com>.
High level consumer commits before shutting down.

If you'll look at ZookeeperConsumerConnector.scala (currently the only
implementation of ConsumerConnector) you'll see shutdown() includes
the following:

          if (config.autoCommitEnable)
            commitOffsets()

Gwen

On Tue, Oct 28, 2014 at 11:44 AM, Bhavesh Mistry
<mi...@gmail.com> wrote:
> Hi Kafka Team,
>
> What is expected behavior when you close *ConsumerConnector* and auto
> commit is on ?  Basically, when auto commit interval is set to 5 seconds
> and shutdown is called (before 5 seconds elapses) does ConsumerConnector
> commit the offset of message consumed by (next()) method or consumer will
> get duplicate messages when it comes online after restart ?
>
> ConsumerConnector.shutdown();
>
> Thanks,
>
> Bhavesh

Re: High Level Consumer and Close with Auto Commit On

Posted by Gwen Shapira <gs...@cloudera.com>.
High level consumer commits before shutting down.

If you'll look at ZookeeperConsumerConnector.scala (currently the only
implementation of ConsumerConnector) you'll see shutdown() includes
the following:

          if (config.autoCommitEnable)
            commitOffsets()

Gwen

On Tue, Oct 28, 2014 at 11:44 AM, Bhavesh Mistry
<mi...@gmail.com> wrote:
> Hi Kafka Team,
>
> What is expected behavior when you close *ConsumerConnector* and auto
> commit is on ?  Basically, when auto commit interval is set to 5 seconds
> and shutdown is called (before 5 seconds elapses) does ConsumerConnector
> commit the offset of message consumed by (next()) method or consumer will
> get duplicate messages when it comes online after restart ?
>
> ConsumerConnector.shutdown();
>
> Thanks,
>
> Bhavesh