You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Tommy Becker <to...@tivo.com> on 2016/09/12 13:03:57 UTC
How do you monitor Samza?
One of the challenges we've faced with deploying Samza is effectively monitoring it. We currently use the MetricsSnapshotReporter and consume the resulting stream pulling out specific values that we want to monitor, but this approach isn't scalable. We're exploring pumping the metrics into an ELK stack, but the format isn't ideal (many keys contain dots, which ES doesn't like). Additionally, many of the keys are "dynamic" in that the names are determined by runtime configuration, which makes pulling specific values out a bit of a pain. I'd love to get some information on how other folks are monitoring their Samza jobs.
--
Tommy Becker
Senior Software Engineer
Digitalsmiths
A TiVo Company
www.digitalsmiths.com<http://www.digitalsmiths.com>
tobecker@tivo.com<ma...@tivo.com>
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
Re: How do you monitor Samza?
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Thanks for sharing Nicolas, This is super-useful.
On Mon, Sep 12, 2016 at 10:56 AM, Nicolas Maquet <nm...@gmail.com> wrote:
> Hey Tommy,
>
> At Movio, we feed all of our Samza metrics into Prometheus and use Grafana
> to create visual dashboards. We configure all of our jobs to produce
> metrics into a single topic and we have a monitoring tool that consumes
> that topic and exposes the data to Prometheus (
> https://github.com/movio/samza-prometheus-exporter).
>
> We've been using this setup in production for 6 months now and it's been
> working really well for us.
>
> Nick
>
>
> On 13 Sep 2016 1:04 a.m., "Tommy Becker" <to...@tivo.com> wrote:
>
> > One of the challenges we've faced with deploying Samza is effectively
> > monitoring it. We currently use the MetricsSnapshotReporter and consume
> the
> > resulting stream pulling out specific values that we want to monitor, but
> > this approach isn't scalable. We're exploring pumping the metrics into an
> > ELK stack, but the format isn't ideal (many keys contain dots, which ES
> > doesn't like). Additionally, many of the keys are "dynamic" in that the
> > names are determined by runtime configuration, which makes pulling
> specific
> > values out a bit of a pain. I'd love to get some information on how other
> > folks are monitoring their Samza jobs.
> >
> > --
> > Tommy Becker
> > Senior Software Engineer
> >
> > Digitalsmiths
> > A TiVo Company
> >
> > www.digitalsmiths.com<http://www.digitalsmiths.com>
> > tobecker@tivo.com<ma...@tivo.com>
> >
> > ________________________________
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
Re: How do you monitor Samza?
Posted by Nicolas Maquet <nm...@gmail.com>.
Hey Tommy,
At Movio, we feed all of our Samza metrics into Prometheus and use Grafana
to create visual dashboards. We configure all of our jobs to produce
metrics into a single topic and we have a monitoring tool that consumes
that topic and exposes the data to Prometheus (
https://github.com/movio/samza-prometheus-exporter).
We've been using this setup in production for 6 months now and it's been
working really well for us.
Nick
On 13 Sep 2016 1:04 a.m., "Tommy Becker" <to...@tivo.com> wrote:
> One of the challenges we've faced with deploying Samza is effectively
> monitoring it. We currently use the MetricsSnapshotReporter and consume the
> resulting stream pulling out specific values that we want to monitor, but
> this approach isn't scalable. We're exploring pumping the metrics into an
> ELK stack, but the format isn't ideal (many keys contain dots, which ES
> doesn't like). Additionally, many of the keys are "dynamic" in that the
> names are determined by runtime configuration, which makes pulling specific
> values out a bit of a pain. I'd love to get some information on how other
> folks are monitoring their Samza jobs.
>
> --
> Tommy Becker
> Senior Software Engineer
>
> Digitalsmiths
> A TiVo Company
>
> www.digitalsmiths.com<http://www.digitalsmiths.com>
> tobecker@tivo.com<ma...@tivo.com>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>