You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by François Langelier <f....@gmail.com> on 2014/09/17 21:55:22 UTC
Disactivating Yammer Metrics Monitoring
Hi all!
We are using yammer metrics to monitor some parts of our system.
Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more data
getting in our graphite server and from what I saw, it looks like it all
come from our producers.
>From what i understand, since we already use graphite, our graphiteReporter
is enable in our main web site and our kafka producers are having fun using
it too to monitor in graphite.
The problem is that right now kafka is hammering of graphite server and we
have difficulty to saw our monitored data...
Is there a way to deactivate the monitoring of our kafka producers?
François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
Re: Disactivating Yammer Metrics Monitoring
Posted by François Langelier <f....@gmail.com>.
Good to know!
Thanks Jason, I'll look at it ASAP! :)
François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Membre Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
On Thu, Nov 6, 2014 at 12:30 PM, Jason Rosenberg <jb...@squareup.com> wrote:
> Hi Francois,
>
> We had the exact same problem. We embed Kafka in our service container,
> and we use yammer metrics to see data about the whole app (e.g. kafka, the
> jvm, the service container wrapping it). However, as you observed, by
> default, kafka produces an insane amount of metrics. So what we did, is
> using the yammer library, you can disable specific metrics by removing
> metrics from the yammer MetricsRegistry, which you can access from guice
> (if you are using guice). I implemented the MetricsRegistryListener, and
> added the ability to remove metric names by regex, so I can still have some
> metrics show up (like the simple 'AllTopic' counts for messages/bytes sent
> from the producer), but block everything else that's per topic, etc....
>
> Jason
>
> On Fri, Sep 19, 2014 at 11:34 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > Hi,
> >
> > I don't have any source or configs handy to check things, but you are
> > saying you've configured Kafka to use GraphiteReporter, right? So why
> not
> > remove that config, so metrics stop being sent to Graphite if your
> Graphite
> > setup is suffering? If you do that and you still want to see your Kafka
> > metrics, you can always use SPM <http://sematext.com/spm/> for Kafka
> > (though some of the graphs will be empty until we get KAFKA-1481, or
> > something else that improves metrics, in). If you just want to use it
> > temporarily, just use the free 30-day trial version until you beef up
> your
> > Graphite setup.
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Fri, Sep 19, 2014 at 10:08 AM, François Langelier <
> > f.langelier@gmail.com>
> > wrote:
> >
> > > Hi Daniel,
> > >
> > > Thank you for your answer.
> > >
> > > It's possible that I didn't understood something, if so correct me
> > please.
> > >
> > > From what I understood, from the kafka doc #monitoring
> > > <http://kafka.apache.org/documentation.html#monitoring>, kafka use
> > Yammer
> > > Metrics for monitoring the servers (the brokers) and the clients
> > (producers
> > > and consumers).
> > >
> > > Our web site is also using Yammer Metrics and push that to our Graphite
> > > server and our web site also produce message in kafka.
> > > From what I read, the Yammer Metrics GraphiteReporter is a kind of
> > > Singleton, once I Enable it, it is working for all the process. (But I
> > > might be wrong here...)
> > >
> > > We recently upgrade kafka from 0.7.2 to 0.8.1.1 and since the upgrade,
> > > kafka is monitoring in our Graphite Server and is hammering it, so we
> > > aren't able to use it because we always get timeout...
> > >
> > > SO, I was wondering if there is a way to disable the kafka monitoring
> to
> > > our Graphite server.
> > >
> > > We are using the code in the tag 0.8.1.1 on github, so if the
> > kafka-ganglia
> > > isn't in the tag, we aren't using it :)
> > >
> > >
> > > François Langelier
> > > Étudiant en génie Logiciel - École de Technologie Supérieure
> > > <http://www.etsmtl.ca/>
> > > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > > VP-Communication - CS Games <http://csgames.org> 2014
> > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > > Comité Organisateur Olympiades ÉTS 2012
> > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> > >
> > > On Wed, Sep 17, 2014 at 11:05 PM, Daniel Compton <
> desk@danielcompton.net
> > >
> > > wrote:
> > >
> > > > Hi Francois
> > > >
> > > > I didn't quite understand how you've set up your metrics reporting.
> Are
> > > you
> > > > using the https://github.com/criteo/kafka-ganglia metrics reporter?
> If
> > > so
> > > > then you should be able to adjust the config to exclude the metrics
> you
> > > > don't want, with kafka.ganglia.metrics.exclude.regex.
> > > >
> > > >
> > > > On 18 September 2014 07:55, François Langelier <
> f.langelier@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all!
> > > > >
> > > > > We are using yammer metrics to monitor some parts of our system.
> > > > >
> > > > > Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more
> data
> > > > > getting in our graphite server and from what I saw, it looks like
> it
> > > all
> > > > > come from our producers.
> > > > >
> > > > > From what i understand, since we already use graphite, our
> > > > graphiteReporter
> > > > > is enable in our main web site and our kafka producers are having
> fun
> > > > using
> > > > > it too to monitor in graphite.
> > > > >
> > > > > The problem is that right now kafka is hammering of graphite server
> > and
> > > > we
> > > > > have difficulty to saw our monitored data...
> > > > >
> > > > > Is there a way to deactivate the monitoring of our kafka producers?
> > > > >
> > > > >
> > > > > François Langelier
> > > > > Étudiant en génie Logiciel - École de Technologie Supérieure
> > > > > <http://www.etsmtl.ca/>
> > > > > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > > > > VP-Communication - CS Games <http://csgames.org> 2014
> > > > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > > > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > > > > Comité Organisateur Olympiades ÉTS 2012
> > > > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> > > > >
> > > >
> > >
> >
>
Re: Disactivating Yammer Metrics Monitoring
Posted by Jason Rosenberg <jb...@squareup.com>.
Hi Francois,
We had the exact same problem. We embed Kafka in our service container,
and we use yammer metrics to see data about the whole app (e.g. kafka, the
jvm, the service container wrapping it). However, as you observed, by
default, kafka produces an insane amount of metrics. So what we did, is
using the yammer library, you can disable specific metrics by removing
metrics from the yammer MetricsRegistry, which you can access from guice
(if you are using guice). I implemented the MetricsRegistryListener, and
added the ability to remove metric names by regex, so I can still have some
metrics show up (like the simple 'AllTopic' counts for messages/bytes sent
from the producer), but block everything else that's per topic, etc....
Jason
On Fri, Sep 19, 2014 at 11:34 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> I don't have any source or configs handy to check things, but you are
> saying you've configured Kafka to use GraphiteReporter, right? So why not
> remove that config, so metrics stop being sent to Graphite if your Graphite
> setup is suffering? If you do that and you still want to see your Kafka
> metrics, you can always use SPM <http://sematext.com/spm/> for Kafka
> (though some of the graphs will be empty until we get KAFKA-1481, or
> something else that improves metrics, in). If you just want to use it
> temporarily, just use the free 30-day trial version until you beef up your
> Graphite setup.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Fri, Sep 19, 2014 at 10:08 AM, François Langelier <
> f.langelier@gmail.com>
> wrote:
>
> > Hi Daniel,
> >
> > Thank you for your answer.
> >
> > It's possible that I didn't understood something, if so correct me
> please.
> >
> > From what I understood, from the kafka doc #monitoring
> > <http://kafka.apache.org/documentation.html#monitoring>, kafka use
> Yammer
> > Metrics for monitoring the servers (the brokers) and the clients
> (producers
> > and consumers).
> >
> > Our web site is also using Yammer Metrics and push that to our Graphite
> > server and our web site also produce message in kafka.
> > From what I read, the Yammer Metrics GraphiteReporter is a kind of
> > Singleton, once I Enable it, it is working for all the process. (But I
> > might be wrong here...)
> >
> > We recently upgrade kafka from 0.7.2 to 0.8.1.1 and since the upgrade,
> > kafka is monitoring in our Graphite Server and is hammering it, so we
> > aren't able to use it because we always get timeout...
> >
> > SO, I was wondering if there is a way to disable the kafka monitoring to
> > our Graphite server.
> >
> > We are using the code in the tag 0.8.1.1 on github, so if the
> kafka-ganglia
> > isn't in the tag, we aren't using it :)
> >
> >
> > François Langelier
> > Étudiant en génie Logiciel - École de Technologie Supérieure
> > <http://www.etsmtl.ca/>
> > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > VP-Communication - CS Games <http://csgames.org> 2014
> > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > Comité Organisateur Olympiades ÉTS 2012
> > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> >
> > On Wed, Sep 17, 2014 at 11:05 PM, Daniel Compton <desk@danielcompton.net
> >
> > wrote:
> >
> > > Hi Francois
> > >
> > > I didn't quite understand how you've set up your metrics reporting. Are
> > you
> > > using the https://github.com/criteo/kafka-ganglia metrics reporter? If
> > so
> > > then you should be able to adjust the config to exclude the metrics you
> > > don't want, with kafka.ganglia.metrics.exclude.regex.
> > >
> > >
> > > On 18 September 2014 07:55, François Langelier <f....@gmail.com>
> > > wrote:
> > >
> > > > Hi all!
> > > >
> > > > We are using yammer metrics to monitor some parts of our system.
> > > >
> > > > Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more data
> > > > getting in our graphite server and from what I saw, it looks like it
> > all
> > > > come from our producers.
> > > >
> > > > From what i understand, since we already use graphite, our
> > > graphiteReporter
> > > > is enable in our main web site and our kafka producers are having fun
> > > using
> > > > it too to monitor in graphite.
> > > >
> > > > The problem is that right now kafka is hammering of graphite server
> and
> > > we
> > > > have difficulty to saw our monitored data...
> > > >
> > > > Is there a way to deactivate the monitoring of our kafka producers?
> > > >
> > > >
> > > > François Langelier
> > > > Étudiant en génie Logiciel - École de Technologie Supérieure
> > > > <http://www.etsmtl.ca/>
> > > > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > > > VP-Communication - CS Games <http://csgames.org> 2014
> > > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > > > Comité Organisateur Olympiades ÉTS 2012
> > > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> > > >
> > >
> >
>
Re: Disactivating Yammer Metrics Monitoring
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
I don't have any source or configs handy to check things, but you are
saying you've configured Kafka to use GraphiteReporter, right? So why not
remove that config, so metrics stop being sent to Graphite if your Graphite
setup is suffering? If you do that and you still want to see your Kafka
metrics, you can always use SPM <http://sematext.com/spm/> for Kafka
(though some of the graphs will be empty until we get KAFKA-1481, or
something else that improves metrics, in). If you just want to use it
temporarily, just use the free 30-day trial version until you beef up your
Graphite setup.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Fri, Sep 19, 2014 at 10:08 AM, François Langelier <f....@gmail.com>
wrote:
> Hi Daniel,
>
> Thank you for your answer.
>
> It's possible that I didn't understood something, if so correct me please.
>
> From what I understood, from the kafka doc #monitoring
> <http://kafka.apache.org/documentation.html#monitoring>, kafka use Yammer
> Metrics for monitoring the servers (the brokers) and the clients (producers
> and consumers).
>
> Our web site is also using Yammer Metrics and push that to our Graphite
> server and our web site also produce message in kafka.
> From what I read, the Yammer Metrics GraphiteReporter is a kind of
> Singleton, once I Enable it, it is working for all the process. (But I
> might be wrong here...)
>
> We recently upgrade kafka from 0.7.2 to 0.8.1.1 and since the upgrade,
> kafka is monitoring in our Graphite Server and is hammering it, so we
> aren't able to use it because we always get timeout...
>
> SO, I was wondering if there is a way to disable the kafka monitoring to
> our Graphite server.
>
> We are using the code in the tag 0.8.1.1 on github, so if the kafka-ganglia
> isn't in the tag, we aren't using it :)
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> <http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
> On Wed, Sep 17, 2014 at 11:05 PM, Daniel Compton <de...@danielcompton.net>
> wrote:
>
> > Hi Francois
> >
> > I didn't quite understand how you've set up your metrics reporting. Are
> you
> > using the https://github.com/criteo/kafka-ganglia metrics reporter? If
> so
> > then you should be able to adjust the config to exclude the metrics you
> > don't want, with kafka.ganglia.metrics.exclude.regex.
> >
> >
> > On 18 September 2014 07:55, François Langelier <f....@gmail.com>
> > wrote:
> >
> > > Hi all!
> > >
> > > We are using yammer metrics to monitor some parts of our system.
> > >
> > > Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more data
> > > getting in our graphite server and from what I saw, it looks like it
> all
> > > come from our producers.
> > >
> > > From what i understand, since we already use graphite, our
> > graphiteReporter
> > > is enable in our main web site and our kafka producers are having fun
> > using
> > > it too to monitor in graphite.
> > >
> > > The problem is that right now kafka is hammering of graphite server and
> > we
> > > have difficulty to saw our monitored data...
> > >
> > > Is there a way to deactivate the monitoring of our kafka producers?
> > >
> > >
> > > François Langelier
> > > Étudiant en génie Logiciel - École de Technologie Supérieure
> > > <http://www.etsmtl.ca/>
> > > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > > VP-Communication - CS Games <http://csgames.org> 2014
> > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > > Comité Organisateur Olympiades ÉTS 2012
> > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> > >
> >
>
Re: Disactivating Yammer Metrics Monitoring
Posted by François Langelier <f....@gmail.com>.
Hi Daniel,
Thank you for your answer.
It's possible that I didn't understood something, if so correct me please.
>From what I understood, from the kafka doc #monitoring
<http://kafka.apache.org/documentation.html#monitoring>, kafka use Yammer
Metrics for monitoring the servers (the brokers) and the clients (producers
and consumers).
Our web site is also using Yammer Metrics and push that to our Graphite
server and our web site also produce message in kafka.
>From what I read, the Yammer Metrics GraphiteReporter is a kind of
Singleton, once I Enable it, it is working for all the process. (But I
might be wrong here...)
We recently upgrade kafka from 0.7.2 to 0.8.1.1 and since the upgrade,
kafka is monitoring in our Graphite Server and is hammering it, so we
aren't able to use it because we always get timeout...
SO, I was wondering if there is a way to disable the kafka monitoring to
our Graphite server.
We are using the code in the tag 0.8.1.1 on github, so if the kafka-ganglia
isn't in the tag, we aren't using it :)
François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
On Wed, Sep 17, 2014 at 11:05 PM, Daniel Compton <de...@danielcompton.net>
wrote:
> Hi Francois
>
> I didn't quite understand how you've set up your metrics reporting. Are you
> using the https://github.com/criteo/kafka-ganglia metrics reporter? If so
> then you should be able to adjust the config to exclude the metrics you
> don't want, with kafka.ganglia.metrics.exclude.regex.
>
>
> On 18 September 2014 07:55, François Langelier <f....@gmail.com>
> wrote:
>
> > Hi all!
> >
> > We are using yammer metrics to monitor some parts of our system.
> >
> > Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more data
> > getting in our graphite server and from what I saw, it looks like it all
> > come from our producers.
> >
> > From what i understand, since we already use graphite, our
> graphiteReporter
> > is enable in our main web site and our kafka producers are having fun
> using
> > it too to monitor in graphite.
> >
> > The problem is that right now kafka is hammering of graphite server and
> we
> > have difficulty to saw our monitored data...
> >
> > Is there a way to deactivate the monitoring of our kafka producers?
> >
> >
> > François Langelier
> > Étudiant en génie Logiciel - École de Technologie Supérieure
> > <http://www.etsmtl.ca/>
> > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > VP-Communication - CS Games <http://csgames.org> 2014
> > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> > Comité Organisateur Olympiades ÉTS 2012
> > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> >
>
Re: Disactivating Yammer Metrics Monitoring
Posted by Daniel Compton <de...@danielcompton.net>.
Hi Francois
I didn't quite understand how you've set up your metrics reporting. Are you
using the https://github.com/criteo/kafka-ganglia metrics reporter? If so
then you should be able to adjust the config to exclude the metrics you
don't want, with kafka.ganglia.metrics.exclude.regex.
On 18 September 2014 07:55, François Langelier <f....@gmail.com>
wrote:
> Hi all!
>
> We are using yammer metrics to monitor some parts of our system.
>
> Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more data
> getting in our graphite server and from what I saw, it looks like it all
> come from our producers.
>
> From what i understand, since we already use graphite, our graphiteReporter
> is enable in our main web site and our kafka producers are having fun using
> it too to monitor in graphite.
>
> The problem is that right now kafka is hammering of graphite server and we
> have difficulty to saw our monitored data...
>
> Is there a way to deactivate the monitoring of our kafka producers?
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> <http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>