You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Adrien Ruffie <ad...@gmail.com> on 2018/07/09 09:12:45 UTC
Monitoring Kafka
Hello Kafka Users,
I want to monitor our Kafka cluster correctly. I have read several articles
on "how to monitor Kafka" but I have the impression that every company is
doing a bit of a thing (rearranging them in his own way).
What the really thing I need to monitor, verify and set notifications
sending when a threshold is reached ? (meaningful alerting).
Is there a no commercial application (not paying) for this ?
For example where we can see the ratio of actual bytes on disk compared to
the configured maximum bytes for the topic, or log flush time, consumer
lagging, under replicated partitions.
And a way to set several notifications/alerts on:
- when ysed swap is > 128mb
- when disk is > 85% used
- IsrShinksPerSec|IsrExpandsPerSec != 0
- UnderReplicatedPartition > 0
- NetworkProcessorAvgIdlePercent < 0.3
I take all the ideas that could allow us to take care of our cluster.
Best regards.
Adrien
RE: Monitoring Kafka
Posted by adrien ruffie <ad...@hotmail.fr>.
Thank it is very useful for me, because I already have a kibana and grafana installed.
I will try it.
Thank again Daniel
Adrien
________________________________
De : Daniel Aparecido Martins Rosa <da...@daniel.eti.br>
Envoyé : mardi 10 juillet 2018 14:47:06
À : users@kafka.apache.org
Objet : Re: Monitoring Kafka
Hi Adrien,
Take a look at this post that I wrote. Maybe can guide you.
Enjoy,
https://medium.com/@danielmrosa/monitoring-kafka-b97d2d5a5434
2018-07-09 12:09 GMT-03:00 adrien ruffie <ad...@hotmail.fr>:
> Great ! Thank a lot Daniel !
>
> I will try it.
>
> Best Regards
>
> Adrien
> ________________________________
> De : Daniel Hanley <da...@confluent.io>
> Envoyé : lundi 9 juillet 2018 16:23:36
> À : users@kafka.apache.org
> Objet : Re: Monitoring Kafka
>
> Hi Adrien
> You could take some ideas from: https://github.com/framiere/
> monitoring-demo
>
> Alternatively, Confluent provide a very powerful Control Center for
> monitoring and managing Kafka
>
> (disclaimer, I work for Confluent!)
>
> Best Regards
>
> Dan
>
> On Mon, Jul 9, 2018 at 2:12 AM, Adrien Ruffie <ad...@gmail.com> wrote:
>
> > Hello Kafka Users,
> >
> > I want to monitor our Kafka cluster correctly. I have read several
> articles
> > on "how to monitor Kafka" but I have the impression that every company is
> > doing a bit of a thing (rearranging them in his own way).
> > What the really thing I need to monitor, verify and set notifications
> > sending when a threshold is reached ? (meaningful alerting).
> > Is there a no commercial application (not paying) for this ?
> > For example where we can see the ratio of actual bytes on disk compared
> to
> > the configured maximum bytes for the topic, or log flush time, consumer
> > lagging, under replicated partitions.
> > And a way to set several notifications/alerts on:
> > - when ysed swap is > 128mb
> > - when disk is > 85% used
> > - IsrShinksPerSec|IsrExpandsPerSec != 0
> > - UnderReplicatedPartition > 0
> > - NetworkProcessorAvgIdlePercent < 0.3
> >
> > I take all the ideas that could allow us to take care of our cluster.
> >
> > Best regards.
> >
> > Adrien
> >
>
Re: Monitoring Kafka
Posted by Daniel Aparecido Martins Rosa <da...@daniel.eti.br>.
Hi Adrien,
Take a look at this post that I wrote. Maybe can guide you.
Enjoy,
https://medium.com/@danielmrosa/monitoring-kafka-b97d2d5a5434
2018-07-09 12:09 GMT-03:00 adrien ruffie <ad...@hotmail.fr>:
> Great ! Thank a lot Daniel !
>
> I will try it.
>
> Best Regards
>
> Adrien
> ________________________________
> De : Daniel Hanley <da...@confluent.io>
> Envoyé : lundi 9 juillet 2018 16:23:36
> À : users@kafka.apache.org
> Objet : Re: Monitoring Kafka
>
> Hi Adrien
> You could take some ideas from: https://github.com/framiere/
> monitoring-demo
>
> Alternatively, Confluent provide a very powerful Control Center for
> monitoring and managing Kafka
>
> (disclaimer, I work for Confluent!)
>
> Best Regards
>
> Dan
>
> On Mon, Jul 9, 2018 at 2:12 AM, Adrien Ruffie <ad...@gmail.com> wrote:
>
> > Hello Kafka Users,
> >
> > I want to monitor our Kafka cluster correctly. I have read several
> articles
> > on "how to monitor Kafka" but I have the impression that every company is
> > doing a bit of a thing (rearranging them in his own way).
> > What the really thing I need to monitor, verify and set notifications
> > sending when a threshold is reached ? (meaningful alerting).
> > Is there a no commercial application (not paying) for this ?
> > For example where we can see the ratio of actual bytes on disk compared
> to
> > the configured maximum bytes for the topic, or log flush time, consumer
> > lagging, under replicated partitions.
> > And a way to set several notifications/alerts on:
> > - when ysed swap is > 128mb
> > - when disk is > 85% used
> > - IsrShinksPerSec|IsrExpandsPerSec != 0
> > - UnderReplicatedPartition > 0
> > - NetworkProcessorAvgIdlePercent < 0.3
> >
> > I take all the ideas that could allow us to take care of our cluster.
> >
> > Best regards.
> >
> > Adrien
> >
>
RE: Monitoring Kafka
Posted by adrien ruffie <ad...@hotmail.fr>.
Great ! Thank a lot Daniel !
I will try it.
Best Regards
Adrien
________________________________
De : Daniel Hanley <da...@confluent.io>
Envoyé : lundi 9 juillet 2018 16:23:36
À : users@kafka.apache.org
Objet : Re: Monitoring Kafka
Hi Adrien
You could take some ideas from: https://github.com/framiere/monitoring-demo
Alternatively, Confluent provide a very powerful Control Center for
monitoring and managing Kafka
(disclaimer, I work for Confluent!)
Best Regards
Dan
On Mon, Jul 9, 2018 at 2:12 AM, Adrien Ruffie <ad...@gmail.com> wrote:
> Hello Kafka Users,
>
> I want to monitor our Kafka cluster correctly. I have read several articles
> on "how to monitor Kafka" but I have the impression that every company is
> doing a bit of a thing (rearranging them in his own way).
> What the really thing I need to monitor, verify and set notifications
> sending when a threshold is reached ? (meaningful alerting).
> Is there a no commercial application (not paying) for this ?
> For example where we can see the ratio of actual bytes on disk compared to
> the configured maximum bytes for the topic, or log flush time, consumer
> lagging, under replicated partitions.
> And a way to set several notifications/alerts on:
> - when ysed swap is > 128mb
> - when disk is > 85% used
> - IsrShinksPerSec|IsrExpandsPerSec != 0
> - UnderReplicatedPartition > 0
> - NetworkProcessorAvgIdlePercent < 0.3
>
> I take all the ideas that could allow us to take care of our cluster.
>
> Best regards.
>
> Adrien
>
Re: Monitoring Kafka
Posted by Daniel Hanley <da...@confluent.io>.
Hi Adrien
You could take some ideas from: https://github.com/framiere/monitoring-demo
Alternatively, Confluent provide a very powerful Control Center for
monitoring and managing Kafka
(disclaimer, I work for Confluent!)
Best Regards
Dan
On Mon, Jul 9, 2018 at 2:12 AM, Adrien Ruffie <ad...@gmail.com> wrote:
> Hello Kafka Users,
>
> I want to monitor our Kafka cluster correctly. I have read several articles
> on "how to monitor Kafka" but I have the impression that every company is
> doing a bit of a thing (rearranging them in his own way).
> What the really thing I need to monitor, verify and set notifications
> sending when a threshold is reached ? (meaningful alerting).
> Is there a no commercial application (not paying) for this ?
> For example where we can see the ratio of actual bytes on disk compared to
> the configured maximum bytes for the topic, or log flush time, consumer
> lagging, under replicated partitions.
> And a way to set several notifications/alerts on:
> - when ysed swap is > 128mb
> - when disk is > 85% used
> - IsrShinksPerSec|IsrExpandsPerSec != 0
> - UnderReplicatedPartition > 0
> - NetworkProcessorAvgIdlePercent < 0.3
>
> I take all the ideas that could allow us to take care of our cluster.
>
> Best regards.
>
> Adrien
>