Announce: Capillary, a monitor for Kafka 0.8 spout storm topologies

[Cory Watson <gp...@keen.io>]

Hello all!

If you use Storm and Kafka 0.8 together I may have an interesting project
for you.  Recently at Keen IO <https://keen.io/> we upgraded from Kafka 0.7
to 0.8 and needed to replace the features of stormkafkamon
<https://github.com/otoolep/stormkafkamon> for monitoring spout offsets
since it only worked with Kafka 0.7. So we created Capillary
<https://github.com/keenlabs/capillary>!

Given a bit of information
<https://github.com/keenlabs/capillary#configuration> about your Zookeeper
setup for Storm and Kafka, Capillary is a web application that
automatically discovers topologies running Kafka 0.8 spouts and lets you
click on them to see per-partition offsets
<https://raw.githubusercontent.com/keenlabs/capillary/master/shot.png>.
It's JVM based and shouldn't require anything but Java where you run it. We
decided this was a useful difference over the
install-on-each-host-with-virtualenv model used by stormkafkamon.

In addition to the helpful UI there is an API that you might use for
collecting this information and sending it to your stats/observability
systems for historical or alerting purposes. There's even an example python
script
<https://github.com/keenlabs/capillary/blob/master/stats-to-datadog.py> we
use for reporting to Datadog <http://www.datadoghq.com/>.  It's very likely
we'll move this to a scheduled thread inside Capillary using metrics
<http://metrics.codahale.com/> and allow you to hook up your own reporter.

I hope other uses of Kafka + Storm find this useful. Feel free to open an
issue if you have issues setting it up or configuring it, or drop PRs if
you have additional functionality. Thanks!

(I cross posted this to the Kafka list as well!)

-- 
Cory Watson
Principal Infrastructure Engineer // Keen IO