You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Pablo Barrera González <pa...@gmail.com> on 2013/08/22 16:41:37 UTC

Broker does not publish offset in zookeeper

Hello

I am trying to monitor our kafka cluster. For that I use a zookeeper client
that reads the information of all the consumer and a jmx agent that
connects to the brokers. I try to look for discrepancies between the data
being produced and the data being consumed.

I dislike the idea of using two different connect types (jmx and zookeeper)
for this so I wonder why broker does not publish the offset into zookeeper?
I don't want the last offset, just and update every X seconds or minutes.
Is there any reason for not doing that?

Other option is to use only jmx connections but then discovering the
consumers is impossible.

Thanks

Pablo

Re: Broker does not publish offset in zookeeper

Posted by Jun Rao <ju...@gmail.com>.
Yes, that's 0.8 specific. You can find the docs in
http://kafka.apache.org/documentation.html#monitoring

Thanks,

Jun


On Thu, Aug 22, 2013 at 8:42 AM, Pablo Barrera González <
pablo.barrera@gmail.com> wrote:

> Is that 0.8 specific?
>
> Sorry, I can't find the documentation you mention (there are no references
> to the variable maxLag).
>
> I still find more problems in having a dual connection (jmx and zookeeper)
> instead of only one (zookeeper).
>
>
>
>
> 2013/8/22 Jun Rao <ju...@gmail.com>
>
> > Are you trying to monitor the lag in the consumer? If so, you can use the
> > maxLag/minFetch JMX described in
> > http://kafka.apache.org/documentation.html#monitoring
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Aug 22, 2013 at 7:41 AM, Pablo Barrera González <
> > pablo.barrera@gmail.com> wrote:
> >
> > > Hello
> > >
> > > I am trying to monitor our kafka cluster. For that I use a zookeeper
> > client
> > > that reads the information of all the consumer and a jmx agent that
> > > connects to the brokers. I try to look for discrepancies between the
> data
> > > being produced and the data being consumed.
> > >
> > > I dislike the idea of using two different connect types (jmx and
> > zookeeper)
> > > for this so I wonder why broker does not publish the offset into
> > zookeeper?
> > > I don't want the last offset, just and update every X seconds or
> minutes.
> > > Is there any reason for not doing that?
> > >
> > > Other option is to use only jmx connections but then discovering the
> > > consumers is impossible.
> > >
> > > Thanks
> > >
> > > Pablo
> > >
> >
>

Re: Broker does not publish offset in zookeeper

Posted by Pablo Barrera González <pa...@gmail.com>.
Is that 0.8 specific?

Sorry, I can't find the documentation you mention (there are no references
to the variable maxLag).

I still find more problems in having a dual connection (jmx and zookeeper)
instead of only one (zookeeper).




2013/8/22 Jun Rao <ju...@gmail.com>

> Are you trying to monitor the lag in the consumer? If so, you can use the
> maxLag/minFetch JMX described in
> http://kafka.apache.org/documentation.html#monitoring
>
> Thanks,
>
> Jun
>
>
> On Thu, Aug 22, 2013 at 7:41 AM, Pablo Barrera González <
> pablo.barrera@gmail.com> wrote:
>
> > Hello
> >
> > I am trying to monitor our kafka cluster. For that I use a zookeeper
> client
> > that reads the information of all the consumer and a jmx agent that
> > connects to the brokers. I try to look for discrepancies between the data
> > being produced and the data being consumed.
> >
> > I dislike the idea of using two different connect types (jmx and
> zookeeper)
> > for this so I wonder why broker does not publish the offset into
> zookeeper?
> > I don't want the last offset, just and update every X seconds or minutes.
> > Is there any reason for not doing that?
> >
> > Other option is to use only jmx connections but then discovering the
> > consumers is impossible.
> >
> > Thanks
> >
> > Pablo
> >
>

Re: Broker does not publish offset in zookeeper

Posted by Jun Rao <ju...@gmail.com>.
Are you trying to monitor the lag in the consumer? If so, you can use the
maxLag/minFetch JMX described in
http://kafka.apache.org/documentation.html#monitoring

Thanks,

Jun


On Thu, Aug 22, 2013 at 7:41 AM, Pablo Barrera González <
pablo.barrera@gmail.com> wrote:

> Hello
>
> I am trying to monitor our kafka cluster. For that I use a zookeeper client
> that reads the information of all the consumer and a jmx agent that
> connects to the brokers. I try to look for discrepancies between the data
> being produced and the data being consumed.
>
> I dislike the idea of using two different connect types (jmx and zookeeper)
> for this so I wonder why broker does not publish the offset into zookeeper?
> I don't want the last offset, just and update every X seconds or minutes.
> Is there any reason for not doing that?
>
> Other option is to use only jmx connections but then discovering the
> consumers is impossible.
>
> Thanks
>
> Pablo
>