Monitoring with Riemann

[Paul Ingles <pa...@forward.co.uk>]

Hi,

Earlier today I finished work on a library that adds an additional metrics
reporter (https://github.com/pingles/kafka-riemann-reporter). I did it
because although we were writing the CSV metrics we had nothing to display
the data.

It pushes metrics to Riemann (http://riemann.io) a very cool tool for
aggregating and processing event streams. We've experimented with it a
little in the past but this is the first production integration we've had.
Streams are processed by Riemann and then plotted by Librato (
http://librato.com).

Installing requires building the project and copying the JAR to Kafka's
./libs dir and then putting the following in the server.properties:

kafka.metrics.reporters=org.pingles.kafka.KafkaRiemannReporter
kafka.riemann.metrics.reporter.enabled=true
kafka.riemann.metrics.reporter.publisher=riemann
kafka.riemann.metrics.reporter.publisher.host=127.0.0.1
kafka.riemann.metrics.reporter.publisher.port=5555

It lets you push directly from the broker to a Riemann service. However, we
actually push messages (the Riemann events- protocol buffer messages) onto
a Kafka topic to be later consumed by a bridged and forwarded to Riemann.
This has the added benefit of allowing metrics to be flowed into our
existing Redshift data load pipeline as well as being monitored in realtime.

It's pretty early days but I deployed it to our small Kafka cluster on EC2
earlier and have been watching the graphs all day. Although we were writing
metrics to CSVs before we had nothing to visualise it, this makes it pretty
easy to pump the broker metrics out.

I thought I'd mention it here in case other people are in a similar
position to us and wanted an easy (alebit still a little experimental :)
way to get the metrics out of Kafka.

I'd really appreciate some feedback, and of course I'd be delighted if
other people use it and help fix my mistakes :)

At the moment it's built against an earlier 0.8 that we're running in
production (9bd2a) but I'm keen to start working (and deploying) an
officially released build.

Paul

Re: Monitoring with Riemann

[Jun Rao <ju...@gmail.com>]

Paul,

Thanks for sharing.

Jun


On Thu, Jul 18, 2013 at 2:00 PM, Paul Ingles <pa...@forward.co.uk> wrote:

> Hi,
>
> Earlier today I finished work on a library that adds an additional metrics
> reporter (https://github.com/pingles/kafka-riemann-reporter). I did it
> because although we were writing the CSV metrics we had nothing to display
> the data.
>
> It pushes metrics to Riemann (http://riemann.io) a very cool tool for
> aggregating and processing event streams. We've experimented with it a
> little in the past but this is the first production integration we've had.
> Streams are processed by Riemann and then plotted by Librato (
> http://librato.com).
>
> Installing requires building the project and copying the JAR to Kafka's
> ./libs dir and then putting the following in the server.properties:
>
> kafka.metrics.reporters=org.pingles.kafka.KafkaRiemannReporter
> kafka.riemann.metrics.reporter.enabled=true
> kafka.riemann.metrics.reporter.publisher=riemann
> kafka.riemann.metrics.reporter.publisher.host=127.0.0.1
> kafka.riemann.metrics.reporter.publisher.port=5555
>
> It lets you push directly from the broker to a Riemann service. However, we
> actually push messages (the Riemann events- protocol buffer messages) onto
> a Kafka topic to be later consumed by a bridged and forwarded to Riemann.
> This has the added benefit of allowing metrics to be flowed into our
> existing Redshift data load pipeline as well as being monitored in
> realtime.
>
> It's pretty early days but I deployed it to our small Kafka cluster on EC2
> earlier and have been watching the graphs all day. Although we were writing
> metrics to CSVs before we had nothing to visualise it, this makes it pretty
> easy to pump the broker metrics out.
>
> I thought I'd mention it here in case other people are in a similar
> position to us and wanted an easy (alebit still a little experimental :)
> way to get the metrics out of Kafka.
>
> I'd really appreciate some feedback, and of course I'd be delighted if
> other people use it and help fix my mistakes :)
>
> At the moment it's built against an earlier 0.8 that we're running in
> production (9bd2a) but I'm keen to start working (and deploying) an
> officially released build.
>
> Paul
>