You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Cory Watson <gp...@keen.io> on 2014/08/20 18:32:05 UTC
Announce: Capillary, a monitor for Kafka 0.8 spout topologies
Hello all!
If you use Storm and Kafka 0.8 together I may have an interesting project
for you. Recently at Keen IO <https://keen.io/> we upgraded from Kafka 0.7
to 0.8 and needed to replace the features of stormkafkamon
<https://github.com/otoolep/stormkafkamon> for monitoring spout offsets
since it only worked with Kafka 0.7. So we created Capillary
<https://github.com/keenlabs/capillary>!
Given a bit of information
<https://github.com/keenlabs/capillary#configuration> about your Zookeeper
setup for Storm and Kafka, Capillary is a web application that
automatically discovers topologies running Kafka 0.8 spouts and lets you
click on them to see per-partition offsets
<https://raw.githubusercontent.com/keenlabs/capillary/master/shot.png>.
It's JVM based and shouldn't require anything but Java where you run it. We
decided this was a useful difference over the
install-on-each-host-with-virtualenv model used by stormkafkamon.
In addition to the helpful UI there is an API that you might use for
collecting this information and sending it to your stats/observability
systems for historical or alerting purposes. There's even an example python
script
<https://github.com/keenlabs/capillary/blob/master/stats-to-datadog.py> we
use for reporting to Datadog <http://www.datadoghq.com/>. It's very likely
we'll move this to a scheduled thread inside Capillary using metrics
<http://metrics.codahale.com/> and allow you to hook up your own reporter.
I hope other uses of Kafka + Storm find this useful. Feel free to open an
issue if you have issues setting it up or configuring it, or drop PRs if
you have additional functionality. Thanks!
(I'll be cross-posting this to the Storm list also!)
--
Cory Watson
Principal Infrastructure Engineer // Keen IO
Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies --
and the upgrade to 0.8
Posted by Cory Watson <gp...@keen.io>.
Sure!
A couple things pushed us in to the upgrade. Our machines were very small
in terms of storage, we really wanted replication and we weren't really
monitoring our current machines well since JMX was turned off (and without
replication/leadership a restart was hard).
The first wrinkle was that Kafka 0.8 is basically a whole new system. Since
the only real way to upgrade is to set up a new cluster things were
obviously harder than install & reboot. :)
- We had to read a lot of the Storm Spout code to understand how things
work with leadership and replication. We've not yet updated Storm and had
to use storm-kafka-0.8-plus
<https://github.com/wurstmeister/storm-kafka-0.8-plus> since it's not in
our dist.
- It's really important that you test your producers and consumers
with the failure of a leader. The spout seems to throw an uncaught
exception and crash, but restarts. We at least know this is the behavior
now and can watch for it and test newer versions of storm.
- Replication was slower, but well worth it for us. We also had to
switch from Python's samsa <https://pypi.python.org/pypi/samsa> to
kafka-python <https://github.com/mumrah/kafka-python>.
- It's important you grok the sync/async and cluster/local commit
modes and make deliberate decisions and load test them in your
environment.
- Our use of samsa was async but I'm not sure anyone knew that. Our
write times went from ~ 0.5ms to 35ms on average.
- It's important to verify that your producers are making proper use of
partitioning so that you don't end up with hot spots in partitions!!
- We deployed with a few producers with a KeyedProducer but set no
key. This causes it to pick two partitions only. :)
- The information in this post
<http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/>
was very helpful.
- The quoted bean names in JMX are really weird and the structure of the
metrics is really a pain to fetch with JMX tools. Ugh.
The actual process for us was to set up new Kafka 0.8 machines where we
made good decisions about replication factors and partitioning. We wrote up
a runbook of tasks to perform and it ended up being something like "tear
everything down in this order and stand it back up in Kafka 0.8 after the
old stuff drains."
It went off without a hitch until the latency of writes cross-DC combined
with an accidental code change in the producer. I accidentally started
posting "batch" writes to our API individually instead of sending them in
one write to Kafka. Once we fixed that things were effectively perfect.
On Wed, Aug 20, 2014 at 10:17 AM, Philip O'Toole <ph...@yahoo.com>
wrote:
> Any thoughts on upgrading from 0.7 to 0.8 you'd like to share with the
> community? How did it go? How has it been?
>
> Philip
>
> -----------------------------------------
> http://www.philipotoole.com
>
>
> On Wednesday, August 20, 2014 10:13 AM, Cory Watson <gp...@keen.io>
> wrote:
>
>
> :P Thanks Philip!
>
>
> On Wed, Aug 20, 2014 at 10:04 AM, Philip O'Toole <
> philip.otoole@yahoo.com.invalid> wrote:
>
> Nice work. That tool I put together was getting a bit old. :-)
>
>
> I updated the Kafka "ecosystem" page with details of both tools.
>
> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
>
> Philip
>
>
>
> -----------------------------------------
> http://www.philipotoole.com
>
>
> On Wednesday, August 20, 2014 9:32 AM, Cory Watson <gp...@keen.io> wrote:
>
>
>
> Hello all!
>
> If you use Storm and Kafka 0.8 together I may have an interesting project
> for you. Recently at Keen IO <https://keen.io/> we upgraded from Kafka
> 0.7
> to 0.8 and needed to replace the features of stormkafkamon
> <https://github.com/otoolep/stormkafkamon> for monitoring spout offsets
> since it only worked with Kafka 0.7. So we created Capillary
> <https://github.com/keenlabs/capillary>!
>
> Given a bit of information
> <https://github.com/keenlabs/capillary#configuration> about your Zookeeper
> setup for Storm and Kafka, Capillary is a web application that
> automatically discovers topologies running Kafka 0.8 spouts and lets you
> click on them to see per-partition offsets
> <https://raw.githubusercontent.com/keenlabs/capillary/master/shot.png>.
> It's JVM based and shouldn't require anything but Java where you run it. We
> decided this was a useful difference over the
> install-on-each-host-with-virtualenv model used by stormkafkamon.
>
> In addition to the helpful UI there is an API that you might use for
> collecting this information and sending it to your stats/observability
> systems for historical or alerting purposes. There's even an example python
> script
> <https://github.com/keenlabs/capillary/blob/master/stats-to-datadog.py> we
> use for reporting to Datadog <http://www.datadoghq.com/>. It's very
> likely
> we'll move this to a scheduled thread inside Capillary using metrics
> <http://metrics.codahale.com/> and allow you to hook up your own reporter.
>
> I hope other uses of Kafka + Storm find this useful. Feel free to open an
> issue if you have issues setting it up or configuring it, or drop PRs if
> you have additional functionality. Thanks!
>
> (I'll be cross-posting this to the Storm list also!)
>
> --
> Cory Watson
> Principal Infrastructure Engineer // Keen IO
>
>
>
>
> --
> Cory Watson
> Principal Infrastructure Engineer // Keen IO
>
>
>
--
Cory Watson
Principal Infrastructure Engineer // Keen IO
Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies -- and the upgrade to 0.8
Posted by Philip O'Toole <ph...@yahoo.com.INVALID>.
Any thoughts on upgrading from 0.7 to 0.8 you'd like to share with the community? How did it go? How has it been?
Philip
-----------------------------------------
http://www.philipotoole.com
On Wednesday, August 20, 2014 10:13 AM, Cory Watson <gp...@keen.io> wrote:
:P Thanks Philip!
On Wed, Aug 20, 2014 at 10:04 AM, Philip O'Toole <ph...@yahoo.com.invalid> wrote:
Nice work. That tool I put together was getting a bit old. :-)
>
>
>I updated the Kafka "ecosystem" page with details of both tools.
>
>https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
>
>Philip
>
>
>
>-----------------------------------------
>http://www.philipotoole.com
>
>
>
>On Wednesday, August 20, 2014 9:32 AM, Cory Watson <gp...@keen.io> wrote:
>
>
>
>Hello all!
>
>If you use Storm and Kafka 0.8 together I may have an interesting project
>for you. Recently at Keen IO <https://keen.io/> we upgraded from Kafka 0.7
>
>to 0.8 and needed to replace the features of stormkafkamon
><https://github.com/otoolep/stormkafkamon> for monitoring spout offsets
>
>since it only worked with Kafka 0.7. So we created Capillary
><https://github.com/keenlabs/capillary>!
>
>
>Given a bit of information
><https://github.com/keenlabs/capillary#configuration> about your Zookeeper
>
>setup for Storm and Kafka, Capillary is a web application that
>automatically discovers topologies running Kafka 0.8 spouts and lets you
>click on them to see per-partition offsets
><https://raw.githubusercontent.com/keenlabs/capillary/master/shot.png>.
>
>It's JVM based and shouldn't require anything but Java where you run it. We
>decided this was a useful difference over the
>install-on-each-host-with-virtualenv model used by stormkafkamon.
>
>In addition to the helpful UI there is an API that you might use for
>collecting this information and sending it to your stats/observability
>systems for historical or alerting purposes. There's even an example python
>script
><https://github.com/keenlabs/capillary/blob/master/stats-to-datadog.py> we
>use for reporting to Datadog <http://www.datadoghq.com/>. It's very likely
>
>we'll move this to a scheduled thread inside Capillary using metrics
><http://metrics.codahale.com/> and allow you to hook up your own reporter.
>
>
>I hope other uses of Kafka + Storm find this useful. Feel free to open an
>issue if you have issues setting it up or configuring it, or drop PRs if
>you have additional functionality. Thanks!
>
>(I'll be cross-posting this to the Storm list also!)
>
>--
>Cory Watson
>Principal Infrastructure Engineer // Keen IO
--
Cory Watson
Principal Infrastructure Engineer // Keen IO
Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies
Posted by Cory Watson <gp...@keen.io>.
:P Thanks Philip!
On Wed, Aug 20, 2014 at 10:04 AM, Philip O'Toole <
philip.otoole@yahoo.com.invalid> wrote:
> Nice work. That tool I put together was getting a bit old. :-)
>
>
> I updated the Kafka "ecosystem" page with details of both tools.
>
> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
>
> Philip
>
>
>
> -----------------------------------------
> http://www.philipotoole.com
>
>
> On Wednesday, August 20, 2014 9:32 AM, Cory Watson <gp...@keen.io> wrote:
>
>
>
> Hello all!
>
> If you use Storm and Kafka 0.8 together I may have an interesting project
> for you. Recently at Keen IO <https://keen.io/> we upgraded from Kafka
> 0.7
> to 0.8 and needed to replace the features of stormkafkamon
> <https://github.com/otoolep/stormkafkamon> for monitoring spout offsets
> since it only worked with Kafka 0.7. So we created Capillary
> <https://github.com/keenlabs/capillary>!
>
> Given a bit of information
> <https://github.com/keenlabs/capillary#configuration> about your Zookeeper
> setup for Storm and Kafka, Capillary is a web application that
> automatically discovers topologies running Kafka 0.8 spouts and lets you
> click on them to see per-partition offsets
> <https://raw.githubusercontent.com/keenlabs/capillary/master/shot.png>.
> It's JVM based and shouldn't require anything but Java where you run it. We
> decided this was a useful difference over the
> install-on-each-host-with-virtualenv model used by stormkafkamon.
>
> In addition to the helpful UI there is an API that you might use for
> collecting this information and sending it to your stats/observability
> systems for historical or alerting purposes. There's even an example python
> script
> <https://github.com/keenlabs/capillary/blob/master/stats-to-datadog.py> we
> use for reporting to Datadog <http://www.datadoghq.com/>. It's very
> likely
> we'll move this to a scheduled thread inside Capillary using metrics
> <http://metrics.codahale.com/> and allow you to hook up your own reporter.
>
> I hope other uses of Kafka + Storm find this useful. Feel free to open an
> issue if you have issues setting it up or configuring it, or drop PRs if
> you have additional functionality. Thanks!
>
> (I'll be cross-posting this to the Storm list also!)
>
> --
> Cory Watson
> Principal Infrastructure Engineer // Keen IO
>
--
Cory Watson
Principal Infrastructure Engineer // Keen IO
Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies
Posted by Philip O'Toole <ph...@yahoo.com.INVALID>.
Nice work. That tool I put together was getting a bit old. :-)
I updated the Kafka "ecosystem" page with details of both tools.
https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
Philip
-----------------------------------------
http://www.philipotoole.com
On Wednesday, August 20, 2014 9:32 AM, Cory Watson <gp...@keen.io> wrote:
Hello all!
If you use Storm and Kafka 0.8 together I may have an interesting project
for you. Recently at Keen IO <https://keen.io/> we upgraded from Kafka 0.7
to 0.8 and needed to replace the features of stormkafkamon
<https://github.com/otoolep/stormkafkamon> for monitoring spout offsets
since it only worked with Kafka 0.7. So we created Capillary
<https://github.com/keenlabs/capillary>!
Given a bit of information
<https://github.com/keenlabs/capillary#configuration> about your Zookeeper
setup for Storm and Kafka, Capillary is a web application that
automatically discovers topologies running Kafka 0.8 spouts and lets you
click on them to see per-partition offsets
<https://raw.githubusercontent.com/keenlabs/capillary/master/shot.png>.
It's JVM based and shouldn't require anything but Java where you run it. We
decided this was a useful difference over the
install-on-each-host-with-virtualenv model used by stormkafkamon.
In addition to the helpful UI there is an API that you might use for
collecting this information and sending it to your stats/observability
systems for historical or alerting purposes. There's even an example python
script
<https://github.com/keenlabs/capillary/blob/master/stats-to-datadog.py> we
use for reporting to Datadog <http://www.datadoghq.com/>. It's very likely
we'll move this to a scheduled thread inside Capillary using metrics
<http://metrics.codahale.com/> and allow you to hook up your own reporter.
I hope other uses of Kafka + Storm find this useful. Feel free to open an
issue if you have issues setting it up or configuring it, or drop PRs if
you have additional functionality. Thanks!
(I'll be cross-posting this to the Storm list also!)
--
Cory Watson
Principal Infrastructure Engineer // Keen IO