You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Harshvardhan Chauhan <ha...@gumgum.com> on 2014/03/28 19:54:28 UTC
Java API to monitor Consumer Offset and Lag
Hi,
I am trying to write a groovy script to get consumer offsets and lag for
our kafka cluster.
Is there a Java API to get this information ?
Thanks & regards,
--
*Harshvardhan Chauhan* | Big Data Engineer
*GumGum* <http://www.gumgum.com/> | *Ads that stick*
310-260-9666 | harsh@gumgum.com
RE: Java API to monitor Consumer Offset and Lag
Posted by "Seshadri, Balaji" <Ba...@dish.com>.
I don't know if we have Java API for this.
But if you dig deeper on consumer offset checker it just makes query from Zookeeper.
Kafka.utils.ZkUtils does all the calls for reading data from Zookeeper.
-----Original Message-----
From: Harshvardhan Chauhan [mailto:harsh@gumgum.com]
Sent: Friday, March 28, 2014 2:44 PM
To: users@kafka.apache.org
Subject: Re: Java API to monitor Consumer Offset and Lag
I am aware of that but I should have put my question more accurately.
I want to automate monitoring consumer offset and lag and wanted to know if there is a Java API for that.
The ConsumerOffsetChecker has a main method and prints the information on console, I want to capture the data and push it to a monitoring server in a cleaner way other than capturing ConsumerOffsetChecker console output.
Regards,
Harsh
On Fri, Mar 28, 2014 at 1:41 PM, Seshadri, Balaji
<Ba...@dish.com>wrote:
> This is Scala API should be possible to call it from Groovy.
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com]
> Sent: Friday, March 28, 2014 1:57 PM
> To: 'users@kafka.apache.org'
> Subject: RE: Java API to monitor Consumer Offset and Lag
>
> kafka.tools.ConsumerOffsetChecker
>
> -----Original Message-----
> From: Harshvardhan Chauhan [mailto:harsh@gumgum.com]
> Sent: Friday, March 28, 2014 12:54 PM
> To: users@kafka.apache.org
> Subject: Java API to monitor Consumer Offset and Lag
>
> Hi,
>
> I am trying to write a groovy script to get consumer offsets and lag
> for our kafka cluster.
> Is there a Java API to get this information ?
>
> Thanks & regards,
> --
> *Harshvardhan Chauhan* | Big Data Engineer
> *GumGum* <http://www.gumgum.com/> | *Ads that stick*
> 310-260-9666 | harsh@gumgum.com
>
--
*Harshvardhan Chauhan* | Big Data Engineer
*GumGum* <http://www.gumgum.com/> | *Ads that stick*
310-260-9666 | harsh@gumgum.com
Re: Java API to monitor Consumer Offset and Lag
Posted by Edward Capriolo <ed...@gmail.com>.
We have a nagios check that runs the command line lag check and determines
if any partition has a lag greater then X. Its about a 2 hour
solutiontocode up right but worth every penny.
On Fri, Mar 28, 2014 at 7:45 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi Harsh,
>
> Are you trying to monitor lag?
> SPM for Kafka has this in one of the graphs. We get this data from JMX.
> There are other tools that were mentioned here a couple of weeks
> ago....sorry, can't recall their names now, check
> http://search-hadoop.com/kafka , that get this info from ZK and graph it.
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Fri, Mar 28, 2014 at 4:43 PM, Harshvardhan Chauhan <harsh@gumgum.com
> >wrote:
>
> > I am aware of that but I should have put my question more accurately.
> >
> > I want to automate monitoring consumer offset and lag and wanted to know
> if
> > there is a Java API for that.
> > The ConsumerOffsetChecker has a main method and prints the information on
> > console, I want to capture the data and push it to a monitoring server
> in a
> > cleaner way other
> > than capturing ConsumerOffsetChecker console output.
> >
> > Regards,
> > Harsh
> >
> >
> > On Fri, Mar 28, 2014 at 1:41 PM, Seshadri, Balaji
> > <Ba...@dish.com>wrote:
> >
> > > This is Scala API should be possible to call it from Groovy.
> > >
> > > -----Original Message-----
> > > From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com]
> > > Sent: Friday, March 28, 2014 1:57 PM
> > > To: 'users@kafka.apache.org'
> > > Subject: RE: Java API to monitor Consumer Offset and Lag
> > >
> > > kafka.tools.ConsumerOffsetChecker
> > >
> > > -----Original Message-----
> > > From: Harshvardhan Chauhan [mailto:harsh@gumgum.com]
> > > Sent: Friday, March 28, 2014 12:54 PM
> > > To: users@kafka.apache.org
> > > Subject: Java API to monitor Consumer Offset and Lag
> > >
> > > Hi,
> > >
> > > I am trying to write a groovy script to get consumer offsets and lag
> for
> > > our kafka cluster.
> > > Is there a Java API to get this information ?
> > >
> > > Thanks & regards,
> > > --
> > > *Harshvardhan Chauhan* | Big Data Engineer
> > > *GumGum* <http://www.gumgum.com/> | *Ads that stick*
> > > 310-260-9666 | harsh@gumgum.com
> > >
> >
> >
> >
> > --
> > *Harshvardhan Chauhan* | Big Data Engineer
> > *GumGum* <http://www.gumgum.com/> | *Ads that stick*
> > 310-260-9666 | harsh@gumgum.com
> >
>
Re: Java API to monitor Consumer Offset and Lag
Posted by Otis Gospodnetic <ot...@gmail.com>.
This SPM: http://sematext.com/spm/
We have classic threshold alerts on this, but also
algolerts<http://blog.sematext.com/2013/10/15/introducing-algolerts-anomaly-detection-alerts/>to
detect any sudden anomalies.
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Sat, Mar 29, 2014 at 1:00 AM, Steve Morin <st...@stevemorin.com> wrote:
> Otis,
> What's SPM that your referencing?
> -Steve
>
>
> On Fri, Mar 28, 2014 at 4:45 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi Harsh,
> >
> > Are you trying to monitor lag?
> > SPM for Kafka has this in one of the graphs. We get this data from JMX.
> > There are other tools that were mentioned here a couple of weeks
> > ago....sorry, can't recall their names now, check
> > http://search-hadoop.com/kafka , that get this info from ZK and graph
> it.
> >
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Fri, Mar 28, 2014 at 4:43 PM, Harshvardhan Chauhan <harsh@gumgum.com
> > >wrote:
> >
> > > I am aware of that but I should have put my question more accurately.
> > >
> > > I want to automate monitoring consumer offset and lag and wanted to
> know
> > if
> > > there is a Java API for that.
> > > The ConsumerOffsetChecker has a main method and prints the information
> on
> > > console, I want to capture the data and push it to a monitoring server
> > in a
> > > cleaner way other
> > > than capturing ConsumerOffsetChecker console output.
> > >
> > > Regards,
> > > Harsh
> > >
> > >
> > > On Fri, Mar 28, 2014 at 1:41 PM, Seshadri, Balaji
> > > <Ba...@dish.com>wrote:
> > >
> > > > This is Scala API should be possible to call it from Groovy.
> > > >
> > > > -----Original Message-----
> > > > From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com]
> > > > Sent: Friday, March 28, 2014 1:57 PM
> > > > To: 'users@kafka.apache.org'
> > > > Subject: RE: Java API to monitor Consumer Offset and Lag
> > > >
> > > > kafka.tools.ConsumerOffsetChecker
> > > >
> > > > -----Original Message-----
> > > > From: Harshvardhan Chauhan [mailto:harsh@gumgum.com]
> > > > Sent: Friday, March 28, 2014 12:54 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Java API to monitor Consumer Offset and Lag
> > > >
> > > > Hi,
> > > >
> > > > I am trying to write a groovy script to get consumer offsets and lag
> > for
> > > > our kafka cluster.
> > > > Is there a Java API to get this information ?
> > > >
> > > > Thanks & regards,
> > > > --
> > > > *Harshvardhan Chauhan* | Big Data Engineer
> > > > *GumGum* <http://www.gumgum.com/> | *Ads that stick*
> > > > 310-260-9666 | harsh@gumgum.com
> > > >
> > >
> > >
> > >
> > > --
> > > *Harshvardhan Chauhan* | Big Data Engineer
> > > *GumGum* <http://www.gumgum.com/> | *Ads that stick*
> > > 310-260-9666 | harsh@gumgum.com
> > >
> >
>
Re: Java API to monitor Consumer Offset and Lag
Posted by Steve Morin <st...@stevemorin.com>.
Otis,
What's SPM that your referencing?
-Steve
On Fri, Mar 28, 2014 at 4:45 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi Harsh,
>
> Are you trying to monitor lag?
> SPM for Kafka has this in one of the graphs. We get this data from JMX.
> There are other tools that were mentioned here a couple of weeks
> ago....sorry, can't recall their names now, check
> http://search-hadoop.com/kafka , that get this info from ZK and graph it.
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Fri, Mar 28, 2014 at 4:43 PM, Harshvardhan Chauhan <harsh@gumgum.com
> >wrote:
>
> > I am aware of that but I should have put my question more accurately.
> >
> > I want to automate monitoring consumer offset and lag and wanted to know
> if
> > there is a Java API for that.
> > The ConsumerOffsetChecker has a main method and prints the information on
> > console, I want to capture the data and push it to a monitoring server
> in a
> > cleaner way other
> > than capturing ConsumerOffsetChecker console output.
> >
> > Regards,
> > Harsh
> >
> >
> > On Fri, Mar 28, 2014 at 1:41 PM, Seshadri, Balaji
> > <Ba...@dish.com>wrote:
> >
> > > This is Scala API should be possible to call it from Groovy.
> > >
> > > -----Original Message-----
> > > From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com]
> > > Sent: Friday, March 28, 2014 1:57 PM
> > > To: 'users@kafka.apache.org'
> > > Subject: RE: Java API to monitor Consumer Offset and Lag
> > >
> > > kafka.tools.ConsumerOffsetChecker
> > >
> > > -----Original Message-----
> > > From: Harshvardhan Chauhan [mailto:harsh@gumgum.com]
> > > Sent: Friday, March 28, 2014 12:54 PM
> > > To: users@kafka.apache.org
> > > Subject: Java API to monitor Consumer Offset and Lag
> > >
> > > Hi,
> > >
> > > I am trying to write a groovy script to get consumer offsets and lag
> for
> > > our kafka cluster.
> > > Is there a Java API to get this information ?
> > >
> > > Thanks & regards,
> > > --
> > > *Harshvardhan Chauhan* | Big Data Engineer
> > > *GumGum* <http://www.gumgum.com/> | *Ads that stick*
> > > 310-260-9666 | harsh@gumgum.com
> > >
> >
> >
> >
> > --
> > *Harshvardhan Chauhan* | Big Data Engineer
> > *GumGum* <http://www.gumgum.com/> | *Ads that stick*
> > 310-260-9666 | harsh@gumgum.com
> >
>
Re: Java API to monitor Consumer Offset and Lag
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Harsh,
Are you trying to monitor lag?
SPM for Kafka has this in one of the graphs. We get this data from JMX.
There are other tools that were mentioned here a couple of weeks
ago....sorry, can't recall their names now, check
http://search-hadoop.com/kafka , that get this info from ZK and graph it.
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Fri, Mar 28, 2014 at 4:43 PM, Harshvardhan Chauhan <ha...@gumgum.com>wrote:
> I am aware of that but I should have put my question more accurately.
>
> I want to automate monitoring consumer offset and lag and wanted to know if
> there is a Java API for that.
> The ConsumerOffsetChecker has a main method and prints the information on
> console, I want to capture the data and push it to a monitoring server in a
> cleaner way other
> than capturing ConsumerOffsetChecker console output.
>
> Regards,
> Harsh
>
>
> On Fri, Mar 28, 2014 at 1:41 PM, Seshadri, Balaji
> <Ba...@dish.com>wrote:
>
> > This is Scala API should be possible to call it from Groovy.
> >
> > -----Original Message-----
> > From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com]
> > Sent: Friday, March 28, 2014 1:57 PM
> > To: 'users@kafka.apache.org'
> > Subject: RE: Java API to monitor Consumer Offset and Lag
> >
> > kafka.tools.ConsumerOffsetChecker
> >
> > -----Original Message-----
> > From: Harshvardhan Chauhan [mailto:harsh@gumgum.com]
> > Sent: Friday, March 28, 2014 12:54 PM
> > To: users@kafka.apache.org
> > Subject: Java API to monitor Consumer Offset and Lag
> >
> > Hi,
> >
> > I am trying to write a groovy script to get consumer offsets and lag for
> > our kafka cluster.
> > Is there a Java API to get this information ?
> >
> > Thanks & regards,
> > --
> > *Harshvardhan Chauhan* | Big Data Engineer
> > *GumGum* <http://www.gumgum.com/> | *Ads that stick*
> > 310-260-9666 | harsh@gumgum.com
> >
>
>
>
> --
> *Harshvardhan Chauhan* | Big Data Engineer
> *GumGum* <http://www.gumgum.com/> | *Ads that stick*
> 310-260-9666 | harsh@gumgum.com
>
Re: Java API to monitor Consumer Offset and Lag
Posted by Harshvardhan Chauhan <ha...@gumgum.com>.
I am aware of that but I should have put my question more accurately.
I want to automate monitoring consumer offset and lag and wanted to know if
there is a Java API for that.
The ConsumerOffsetChecker has a main method and prints the information on
console, I want to capture the data and push it to a monitoring server in a
cleaner way other
than capturing ConsumerOffsetChecker console output.
Regards,
Harsh
On Fri, Mar 28, 2014 at 1:41 PM, Seshadri, Balaji
<Ba...@dish.com>wrote:
> This is Scala API should be possible to call it from Groovy.
>
> -----Original Message-----
> From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com]
> Sent: Friday, March 28, 2014 1:57 PM
> To: 'users@kafka.apache.org'
> Subject: RE: Java API to monitor Consumer Offset and Lag
>
> kafka.tools.ConsumerOffsetChecker
>
> -----Original Message-----
> From: Harshvardhan Chauhan [mailto:harsh@gumgum.com]
> Sent: Friday, March 28, 2014 12:54 PM
> To: users@kafka.apache.org
> Subject: Java API to monitor Consumer Offset and Lag
>
> Hi,
>
> I am trying to write a groovy script to get consumer offsets and lag for
> our kafka cluster.
> Is there a Java API to get this information ?
>
> Thanks & regards,
> --
> *Harshvardhan Chauhan* | Big Data Engineer
> *GumGum* <http://www.gumgum.com/> | *Ads that stick*
> 310-260-9666 | harsh@gumgum.com
>
--
*Harshvardhan Chauhan* | Big Data Engineer
*GumGum* <http://www.gumgum.com/> | *Ads that stick*
310-260-9666 | harsh@gumgum.com
RE: Java API to monitor Consumer Offset and Lag
Posted by "Seshadri, Balaji" <Ba...@dish.com>.
This is Scala API should be possible to call it from Groovy.
-----Original Message-----
From: Seshadri, Balaji [mailto:Balaji.Seshadri@dish.com]
Sent: Friday, March 28, 2014 1:57 PM
To: 'users@kafka.apache.org'
Subject: RE: Java API to monitor Consumer Offset and Lag
kafka.tools.ConsumerOffsetChecker
-----Original Message-----
From: Harshvardhan Chauhan [mailto:harsh@gumgum.com]
Sent: Friday, March 28, 2014 12:54 PM
To: users@kafka.apache.org
Subject: Java API to monitor Consumer Offset and Lag
Hi,
I am trying to write a groovy script to get consumer offsets and lag for our kafka cluster.
Is there a Java API to get this information ?
Thanks & regards,
--
*Harshvardhan Chauhan* | Big Data Engineer
*GumGum* <http://www.gumgum.com/> | *Ads that stick*
310-260-9666 | harsh@gumgum.com
RE: Java API to monitor Consumer Offset and Lag
Posted by "Seshadri, Balaji" <Ba...@dish.com>.
kafka.tools.ConsumerOffsetChecker
-----Original Message-----
From: Harshvardhan Chauhan [mailto:harsh@gumgum.com]
Sent: Friday, March 28, 2014 12:54 PM
To: users@kafka.apache.org
Subject: Java API to monitor Consumer Offset and Lag
Hi,
I am trying to write a groovy script to get consumer offsets and lag for our kafka cluster.
Is there a Java API to get this information ?
Thanks & regards,
--
*Harshvardhan Chauhan* | Big Data Engineer
*GumGum* <http://www.gumgum.com/> | *Ads that stick*
310-260-9666 | harsh@gumgum.com