You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by "M. Manna" <ma...@gmail.com> on 2019/09/26 14:01:33 UTC

Monitoring Broker/Prod/Cons on Kubernetes

Hello,

Has anyone got any experience in using monitoring tool (e.g. Prometheus,
DataDog, or custom) for Kafka on K8s?

Cruise-control doesn't come with official docker image (Neither does Kafka
Manager). I was just curious to know if people usually monitor this as a
service through external providers, or using custom-made tools.

Thanks,

Re: Monitoring Broker/Prod/Cons on Kubernetes

Posted by Otis Gospodnetić <ot...@gmail.com>.

Hi,

I missed the original email, but I know Sematext can monitor your
containerized/k8sized Kafka.
Maybe of interest: https://sematext.com/blog/monitoring-kafka-with-sematext/

Otis



On Sun, Sep 29, 2019 at 6:26 AM Pere Urbón Bayes <pe...@gmail.com>
wrote:

> Hi,
>   personally speaking my usual monitoring of choice for k8s is Prometheus +
> Grafana, but end of the day most of the monitoring solutions do a good job,
> so I would mostly ask what do you know the most. For this environments I
> would be more interested on proper container management for kafka and be
> sure that the pods are running in the best way possible.
>
> -- Pere
>
> Missatge de M. Manna <ma...@gmail.com> del dia dj., 26 de set. 2019 a
> les 16:01:
>
> > Hello,
> >
> > Has anyone got any experience in using monitoring tool (e.g. Prometheus,
> > DataDog, or custom) for Kafka on K8s?
> >
> > Cruise-control doesn't come with official docker image (Neither does
> Kafka
> > Manager). I was just curious to know if people usually monitor this as a
> > service through external providers, or using custom-made tools.
> >
> > Thanks,
> >
>
>
> --
> Pere Urbon-Bayes
> Software Architect
> http://www.purbon.com
> https://twitter.com/purbon
> https://www.linkedin.com/in/purbon/
>

Re: Monitoring Broker/Prod/Cons on Kubernetes

Posted by "M. Manna" <ma...@gmail.com>.

Hey thanks for your response.

I was thinking along the same lines. Cruise Control comes with good set of
monitoring functions, but not ready for K8s. You would need to deploy that
as a container (no official image yet). Do you know if any k8s flavour has
been made available yet ?


Prometheus/grafana and DataDog are something we’re considering.

Thanks,

On Sun, 29 Sep 2019 at 11:26, Pere Urbón Bayes <pe...@gmail.com> wrote:

> Hi,
>   personally speaking my usual monitoring of choice for k8s is Prometheus +
> Grafana, but end of the day most of the monitoring solutions do a good job,
> so I would mostly ask what do you know the most. For this environments I
> would be more interested on proper container management for kafka and be
> sure that the pods are running in the best way possible.
>
> -- Pere
>
> Missatge de M. Manna <ma...@gmail.com> del dia dj., 26 de set. 2019 a
> les 16:01:
>
> > Hello,
> >
> > Has anyone got any experience in using monitoring tool (e.g. Prometheus,
> > DataDog, or custom) for Kafka on K8s?
> >
> > Cruise-control doesn't come with official docker image (Neither does
> Kafka
> > Manager). I was just curious to know if people usually monitor this as a
> > service through external providers, or using custom-made tools.
> >
> > Thanks,
> >
>
>
> --
> Pere Urbon-Bayes
> Software Architect
> http://www.purbon.com
> https://twitter.com/purbon
> https://www.linkedin.com/in/purbon/
>

Re: Monitoring Broker/Prod/Cons on Kubernetes

Posted by Pere Urbón Bayes <pe...@gmail.com>.

Hi,
  personally speaking my usual monitoring of choice for k8s is Prometheus +
Grafana, but end of the day most of the monitoring solutions do a good job,
so I would mostly ask what do you know the most. For this environments I
would be more interested on proper container management for kafka and be
sure that the pods are running in the best way possible.

-- Pere

Missatge de M. Manna <ma...@gmail.com> del dia dj., 26 de set. 2019 a
les 16:01:

> Hello,
>
> Has anyone got any experience in using monitoring tool (e.g. Prometheus,
> DataDog, or custom) for Kafka on K8s?
>
> Cruise-control doesn't come with official docker image (Neither does Kafka
> Manager). I was just curious to know if people usually monitor this as a
> service through external providers, or using custom-made tools.
>
> Thanks,
>


-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/

Re: Monitoring Broker/Prod/Cons on Kubernetes

Posted by Liam Clarke <li...@adscale.co.nz>.

I'm using the Prometheus jmx_exporter to export key metrics from the
brokers. Takes a bit of fiddling to get the regexes right, but works well.

On Fri, 27 Sep. 2019, 2:02 am M. Manna, <ma...@gmail.com> wrote:

> Hello,
>
> Has anyone got any experience in using monitoring tool (e.g. Prometheus,
> DataDog, or custom) for Kafka on K8s?
>
> Cruise-control doesn't come with official docker image (Neither does Kafka
> Manager). I was just curious to know if people usually monitor this as a
> service through external providers, or using custom-made tools.
>
> Thanks,
>