Re: Delay in Zookeeper Offset updates

[nitin sharma <ku...@gmail.com>]

--re-initiating this thread

thanks Todd for your response but i guess you got the understanding of my
situation wrong..

a) I dont think problem is with my consumer setup.. I have
"auto.commit.enable=true" and "auto.commit.interval.ms" set to 500.

b) When you run ConsumerOffsetChecker command, you get "Depth" as one of
the column in output..  i am monitoring how frequently does the value in
this column change... As it should not be dependent on my Consumer's
consumption rate but on how frequently the offset is updated by
zookeeper...

i am seeing at least 5 second delay in getting values updated under "Depth"
column...

Any suggestion ?

Regards,
Nitin Kumar Sharma.


On Thu, Sep 17, 2015 at 4:25 PM, Todd Palino <tp...@gmail.com> wrote:

> Consumer offsets in Zookeeper are not handled by the Kafka brokers at all -
> the consumer writes those directly to Zookeeper. Most likely, what you are
> seeing is the interval over which the consumer is committing offsets.
> Assuming that you are using the auto.commit.enable setting (it defaults to
> true, so unless you have changed that in the consumer configuration it will
> apply here), you should check the setting of auto.commit.interval for your
> consumer. I believe the default is still 60 seconds (or a config value of
> 60000). If you're seeing the values change in ZK every 5-6 seconds, then it
> sounds like it is set much lower.
>
> The setting for the offset commit interval is entirely up to you. The lower
> the value, the more often offsets are checkpointed to Zookeeper. However
> that means that your consumer is writing to Zookeeper more often, and that
> has performance impacts. In an environment like ours, where there is a
> large number of consumers for a given cluster, we actually had to increase
> the interval to 10 minutes for quite some time because the load on
> Zookeeper was just too high. Once we moved the ZK transaction logs to SSDs,
> we were able to drop that back to 60 seconds.
>
> -Todd
>
>
>
> On Thu, Sep 17, 2015 at 1:14 PM, nitin sharma <kumarsharma.nitin@gmail.com
> >
> wrote:
>
> > anyone faced this issue?
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
> >
> > On Wed, Sep 2, 2015 at 5:32 PM, nitin sharma <
> kumarsharma.nitin@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > i have run into a weird issue with my Kafka setup.
> > >
> > > I see that it takes around 5-6 sec for Zookeeper to update the offset
> for
> > > Kafka topics.
> > >
> > > I am running "ConsumerOffsetChecker" tool to see that lag and what i
> > found
> > > that even when my consumer is not up, it takes 5-6 sec for Zookeeper to
> > > show the updated Offset .
> > >
> > > Is this behavior can be fixed? I have tried adding following parameters
> > in
> > > my Kafka server.property files but no luck.
> > >
> > > log.flush.interval.messages=1
> > > log.default.flush.scheduler.interval.ms=500
> > >
> > >
> > > ConsumerOffsetChecker:
> > >  bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect
> > > localhost:2181 --group perfConsumer
> > >
> > >
> > > Regards,
> > > Nitin Kumar Sharma.
> > >
> > >
> >
>