You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Andrew Otto <ot...@wikimedia.org> on 2013/11/06 15:15:50 UTC

Kafka 0.8 jmxtrans + puppet

Hi,

I just got jmxtrans set up with Kafka 0.8 over at Wikimedia.  We're using puppet, and the https://github.com/wikimedia/puppet-kafka module to install and set up Kafka.  This module now comes with a kafka::server::jmxtrans class which will automatically set up jmxtrans with some nice Kafka queries.

If you aren't using puppet, you can use the rendered example jmxtrans JSON  config here:

https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.json.md

The exact queries will probably change as we get more operational experience and think of more things we want to monitor.

Hope this is useful to someone!

-Andrew Otto



Re: Kafka 0.8 jmxtrans + puppet

Posted by Andrew Otto <ot...@wikimedia.org>.
We will be using Kafka to transport access logs from frontend cache servers to HDFS.  But!  We aren't doing this in production yet, so I wouldn't quite yet feel comfortable saying we are powered by Kafka.  I hope to get this into production more in a few weeks*, and at that time we'll write a nice little blogpost describing the whole thing.

-Andrew

*a few weeks could really mean anything in busy nonprofit time :)




On Nov 6, 2013, at 9:37 AM, Joe Stein <jo...@stealth.ly> wrote:

> Yes, thank you Andrew this is great stuff.
> 
> May I ask how Wikimedia is using Kafka so I can update the powered by
> https://cwiki.apache.org/confluence/display/KAFKA/Powered+By
> 
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
> 
> 
> On Wed, Nov 6, 2013 at 9:29 AM, Neha Narkhede <ne...@gmail.com>wrote:
> 
>> Cool, thanks for sharing this!
>> 
>> -Neha
>> On Nov 6, 2013 6:16 AM, "Andrew Otto" <ot...@wikimedia.org> wrote:
>> 
>>> Hi,
>>> 
>>> I just got jmxtrans set up with Kafka 0.8 over at Wikimedia.  We're using
>>> puppet, and the https://github.com/wikimedia/puppet-kafka module to
>>> install and set up Kafka.  This module now comes with a
>>> kafka::server::jmxtrans class which will automatically set up jmxtrans
>> with
>>> some nice Kafka queries.
>>> 
>>> If you aren't using puppet, you can use the rendered example jmxtrans
>> JSON
>>> config here:
>>> 
>>> 
>>> 
>> https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.json.md
>>> 
>>> The exact queries will probably change as we get more operational
>>> experience and think of more things we want to monitor.
>>> 
>>> Hope this is useful to someone!
>>> 
>>> -Andrew Otto
>>> 
>>> 
>>> 
>> 


Re: Kafka 0.8 jmxtrans + puppet

Posted by Joe Stein <jo...@stealth.ly>.
Yes, thank you Andrew this is great stuff.

May I ask how Wikimedia is using Kafka so I can update the powered by
https://cwiki.apache.org/confluence/display/KAFKA/Powered+By

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/


On Wed, Nov 6, 2013 at 9:29 AM, Neha Narkhede <ne...@gmail.com>wrote:

> Cool, thanks for sharing this!
>
> -Neha
> On Nov 6, 2013 6:16 AM, "Andrew Otto" <ot...@wikimedia.org> wrote:
>
> > Hi,
> >
> > I just got jmxtrans set up with Kafka 0.8 over at Wikimedia.  We're using
> > puppet, and the https://github.com/wikimedia/puppet-kafka module to
> > install and set up Kafka.  This module now comes with a
> > kafka::server::jmxtrans class which will automatically set up jmxtrans
> with
> > some nice Kafka queries.
> >
> > If you aren't using puppet, you can use the rendered example jmxtrans
> JSON
> >  config here:
> >
> >
> >
> https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.json.md
> >
> > The exact queries will probably change as we get more operational
> > experience and think of more things we want to monitor.
> >
> > Hope this is useful to someone!
> >
> > -Andrew Otto
> >
> >
> >
>

Re: Kafka 0.8 jmxtrans + puppet

Posted by Neha Narkhede <ne...@gmail.com>.
Cool, thanks for sharing this!

-Neha
On Nov 6, 2013 6:16 AM, "Andrew Otto" <ot...@wikimedia.org> wrote:

> Hi,
>
> I just got jmxtrans set up with Kafka 0.8 over at Wikimedia.  We're using
> puppet, and the https://github.com/wikimedia/puppet-kafka module to
> install and set up Kafka.  This module now comes with a
> kafka::server::jmxtrans class which will automatically set up jmxtrans with
> some nice Kafka queries.
>
> If you aren't using puppet, you can use the rendered example jmxtrans JSON
>  config here:
>
>
> https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.json.md
>
> The exact queries will probably change as we get more operational
> experience and think of more things we want to monitor.
>
> Hope this is useful to someone!
>
> -Andrew Otto
>
>
>