You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Pierre Andrews <pi...@quantifind.com> on 2014/03/07 19:49:27 UTC
Anouncing Kafka Offset Monitor 0.1
Hello everyone,
at Quantifind, we are big users of Kafka and we like it a lot!
In a few use cases, we had to figure out if a queue was growing and how its
consumers were behaving. There are a few command-line tools to try to
figure out what's going on, but it's not always easy to debug and to see
what has happened while everyone was sleeping.
To be able to monitor our kafka queues and consumers, we thus developed a
tiny web app that could tell us the log size of each topic in the brokers
and the offsets of each consumers. That's very similar to what the kafka
ConsumerOffsetChecker tool* is doing, but instead of having a one off
snapshot, our app keeps an history and displays a nice graph of what's
going on.
You can find screenshots and more details here:
http://quantifind.github.io/KafkaOffsetMonitor/
the code is on github:
https://github.com/quantifind/KafkaOffsetMonitor
If you have kafka 0.8 setup, it's very easy to use:
1- download the current jar
http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
2- run it, pointing at your kafka brokers:
java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
com.quantifind.kafka.offsetapp.OffsetGetterWeb \
--zk zk-server1,zk-server2 \
--port 8080 \
--refresh 10.seconds \
--retain 2.days
3- open your browser and point it to localhost:8080
You can run it locally or host it on a server if you prefer.
It's all open source and we'll be happy to receive issue report and pull
requests.
I hope that you like it and that it can find some uses in the rest of the
Kafka community.
Best
Pierre
PS: we are aware of https://github.com/claudemamo/kafka-web-console but are
currently offering slightly different features. Hopefully we can merge the
projects in the future.
* here:
https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Claude Mamo <cl...@gmail.com>.
Agreed, sent you an email.
Caude
On Fri, Mar 7, 2014 at 4:55 PM, Pierre Andrews <pi...@quantifind.com>wrote:
> Claude, we should join forces ;)
>
>
> On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo <cl...@gmail.com> wrote:
>
> > Awesome!!! ;-)
> >
> > Claude
> >
> >
> > On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews <pierre@quantifind.com
> > >wrote:
> >
> > > Great! Thanks!
> > >
> > >
> > > On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <ja...@gmail.com> wrote:
> > >
> > > > This is really useful! I added it to the ecosystem page:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> > > >
> > > > -Jay
> > > >
> > > >
> > > > On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews <
> pierre@quantifind.com
> > > > >wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > at Quantifind, we are big users of Kafka and we like it a lot!
> > > > > In a few use cases, we had to figure out if a queue was growing and
> > how
> > > > its
> > > > > consumers were behaving. There are a few command-line tools to try
> to
> > > > > figure out what's going on, but it's not always easy to debug and
> to
> > > see
> > > > > what has happened while everyone was sleeping.
> > > > >
> > > > > To be able to monitor our kafka queues and consumers, we thus
> > > developed a
> > > > > tiny web app that could tell us the log size of each topic in the
> > > brokers
> > > > > and the offsets of each consumers. That's very similar to what the
> > > kafka
> > > > > ConsumerOffsetChecker tool* is doing, but instead of having a one
> off
> > > > > snapshot, our app keeps an history and displays a nice graph of
> > what's
> > > > > going on.
> > > > >
> > > > > You can find screenshots and more details here:
> > > > > http://quantifind.github.io/KafkaOffsetMonitor/
> > > > >
> > > > > the code is on github:
> > > > > https://github.com/quantifind/KafkaOffsetMonitor
> > > > >
> > > > > If you have kafka 0.8 setup, it's very easy to use:
> > > > >
> > > > > 1- download the current jar
> > > > >
> > > > >
> > > >
> > >
> >
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> > > > > 2- run it, pointing at your kafka brokers:
> > > > > java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> > > > > com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> > > > > --zk zk-server1,zk-server2 \
> > > > > --port 8080 \
> > > > > --refresh 10.seconds \
> > > > > --retain 2.days
> > > > > 3- open your browser and point it to localhost:8080
> > > > >
> > > > > You can run it locally or host it on a server if you prefer.
> > > > >
> > > > > It's all open source and we'll be happy to receive issue report and
> > > pull
> > > > > requests.
> > > > >
> > > > > I hope that you like it and that it can find some uses in the rest
> of
> > > the
> > > > > Kafka community.
> > > > >
> > > > > Best
> > > > >
> > > > > Pierre
> > > > >
> > > > > PS: we are aware of
> > https://github.com/claudemamo/kafka-web-consolebut
> > > > > are
> > > > > currently offering slightly different features. Hopefully we can
> > merge
> > > > the
> > > > > projects in the future.
> > > > >
> > > > > * here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> > > > >
> > > >
> > >
> >
>
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Pierre Andrews <pi...@quantifind.com>.
Hi Patricio,
thank you!
I don't know how kafka-spout works. Right now, the monitor uses the value
defined in kafka.utils.ZkUtils.ConsumersPath to grab the list of consumers
in ZK. Again, that could be a parameter to the app somewhere to be more
flexible and deal with separate zk path.
P
On Sat, Mar 8, 2014 at 12:42 AM, Patricio Echagüe <pa...@gmail.com>
wrote:
>
> Great work!
>
> In addition to Dan's question, does it work with storm kafka-spout which
> uses a separate zk path?
>
> Sent from my Nexus 4.
> On Mar 7, 2014 7:18 PM, "Dan Hoffman" <ho...@gmail.com> wrote:
>
> > Hi - am I right in that for this tool to be effective, consumers must be
> > using the high level consumer or otherwise keeping their offsets in
> > zookeeper? Is there any way to track performance without that?
> >
> >
> > On Fri, Mar 7, 2014 at 3:08 PM, Steve Morin <st...@gmail.com>
wrote:
> >
> > > Very nice
> > >
> > > > On Mar 7, 2014, at 11:55, Pierre Andrews <pi...@quantifind.com>
> > wrote:
> > > >
> > > > Claude, we should join forces ;)
> > > >
> > > >
> > > >> On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo <cl...@gmail.com>
> > > wrote:
> > > >>
> > > >> Awesome!!! ;-)
> > > >>
> > > >> Claude
> > > >>
> > > >>
> > > >> On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews <
pierre@quantifind.com
> > > >>> wrote:
> > > >>
> > > >>> Great! Thanks!
> > > >>>
> > > >>>
> > > >>>> On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <ja...@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>> This is really useful! I added it to the ecosystem page:
> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> > > >>>>
> > > >>>> -Jay
> > > >>>>
> > > >>>>
> > > >>>> On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews <
> > > pierre@quantifind.com
> > > >>>>> wrote:
> > > >>>>
> > > >>>>> Hello everyone,
> > > >>>>>
> > > >>>>> at Quantifind, we are big users of Kafka and we like it a lot!
> > > >>>>> In a few use cases, we had to figure out if a queue was growing
and
> > > >> how
> > > >>>> its
> > > >>>>> consumers were behaving. There are a few command-line tools to
try
> > to
> > > >>>>> figure out what's going on, but it's not always easy to debug
and
> > to
> > > >>> see
> > > >>>>> what has happened while everyone was sleeping.
> > > >>>>>
> > > >>>>> To be able to monitor our kafka queues and consumers, we thus
> > > >>> developed a
> > > >>>>> tiny web app that could tell us the log size of each topic in
the
> > > >>> brokers
> > > >>>>> and the offsets of each consumers. That's very similar to what
the
> > > >>> kafka
> > > >>>>> ConsumerOffsetChecker tool* is doing, but instead of having a
one
> > off
> > > >>>>> snapshot, our app keeps an history and displays a nice graph of
> > > >> what's
> > > >>>>> going on.
> > > >>>>>
> > > >>>>> You can find screenshots and more details here:
> > > >>>>> http://quantifind.github.io/KafkaOffsetMonitor/
> > > >>>>>
> > > >>>>> the code is on github:
> > > >>>>> https://github.com/quantifind/KafkaOffsetMonitor
> > > >>>>>
> > > >>>>> If you have kafka 0.8 setup, it's very easy to use:
> > > >>>>>
> > > >>>>> 1- download the current jar
> > > >>
> > >
> >
http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> > > >>>>> 2- run it, pointing at your kafka brokers:
> > > >>>>> java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> > > >>>>> com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> > > >>>>> --zk zk-server1,zk-server2 \
> > > >>>>> --port 8080 \
> > > >>>>> --refresh 10.seconds \
> > > >>>>> --retain 2.days
> > > >>>>> 3- open your browser and point it to localhost:8080
> > > >>>>>
> > > >>>>> You can run it locally or host it on a server if you prefer.
> > > >>>>>
> > > >>>>> It's all open source and we'll be happy to receive issue report
and
> > > >>> pull
> > > >>>>> requests.
> > > >>>>>
> > > >>>>> I hope that you like it and that it can find some uses in the
rest
> > of
> > > >>> the
> > > >>>>> Kafka community.
> > > >>>>>
> > > >>>>> Best
> > > >>>>>
> > > >>>>> Pierre
> > > >>>>>
> > > >>>>> PS: we are aware of
> > > >> https://github.com/claudemamo/kafka-web-consolebut
> > > >>>>> are
> > > >>>>> currently offering slightly different features. Hopefully we can
> > > >> merge
> > > >>>> the
> > > >>>>> projects in the future.
> > > >>>>>
> > > >>>>> * here:
> > > >>
> > >
> >
https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> > > >>
> > >
> >
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Patricio Echagüe <pa...@gmail.com>.
Great work!
In addition to Dan's question, does it work with storm kafka-spout which
uses a separate zk path?
Sent from my Nexus 4.
On Mar 7, 2014 7:18 PM, "Dan Hoffman" <ho...@gmail.com> wrote:
> Hi - am I right in that for this tool to be effective, consumers must be
> using the high level consumer or otherwise keeping their offsets in
> zookeeper? Is there any way to track performance without that?
>
>
> On Fri, Mar 7, 2014 at 3:08 PM, Steve Morin <st...@gmail.com> wrote:
>
> > Very nice
> >
> > > On Mar 7, 2014, at 11:55, Pierre Andrews <pi...@quantifind.com>
> wrote:
> > >
> > > Claude, we should join forces ;)
> > >
> > >
> > >> On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo <cl...@gmail.com>
> > wrote:
> > >>
> > >> Awesome!!! ;-)
> > >>
> > >> Claude
> > >>
> > >>
> > >> On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews <pierre@quantifind.com
> > >>> wrote:
> > >>
> > >>> Great! Thanks!
> > >>>
> > >>>
> > >>>> On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <ja...@gmail.com>
> > wrote:
> > >>>>
> > >>>> This is really useful! I added it to the ecosystem page:
> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> > >>>>
> > >>>> -Jay
> > >>>>
> > >>>>
> > >>>> On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews <
> > pierre@quantifind.com
> > >>>>> wrote:
> > >>>>
> > >>>>> Hello everyone,
> > >>>>>
> > >>>>> at Quantifind, we are big users of Kafka and we like it a lot!
> > >>>>> In a few use cases, we had to figure out if a queue was growing and
> > >> how
> > >>>> its
> > >>>>> consumers were behaving. There are a few command-line tools to try
> to
> > >>>>> figure out what's going on, but it's not always easy to debug and
> to
> > >>> see
> > >>>>> what has happened while everyone was sleeping.
> > >>>>>
> > >>>>> To be able to monitor our kafka queues and consumers, we thus
> > >>> developed a
> > >>>>> tiny web app that could tell us the log size of each topic in the
> > >>> brokers
> > >>>>> and the offsets of each consumers. That's very similar to what the
> > >>> kafka
> > >>>>> ConsumerOffsetChecker tool* is doing, but instead of having a one
> off
> > >>>>> snapshot, our app keeps an history and displays a nice graph of
> > >> what's
> > >>>>> going on.
> > >>>>>
> > >>>>> You can find screenshots and more details here:
> > >>>>> http://quantifind.github.io/KafkaOffsetMonitor/
> > >>>>>
> > >>>>> the code is on github:
> > >>>>> https://github.com/quantifind/KafkaOffsetMonitor
> > >>>>>
> > >>>>> If you have kafka 0.8 setup, it's very easy to use:
> > >>>>>
> > >>>>> 1- download the current jar
> > >>
> >
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> > >>>>> 2- run it, pointing at your kafka brokers:
> > >>>>> java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> > >>>>> com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> > >>>>> --zk zk-server1,zk-server2 \
> > >>>>> --port 8080 \
> > >>>>> --refresh 10.seconds \
> > >>>>> --retain 2.days
> > >>>>> 3- open your browser and point it to localhost:8080
> > >>>>>
> > >>>>> You can run it locally or host it on a server if you prefer.
> > >>>>>
> > >>>>> It's all open source and we'll be happy to receive issue report and
> > >>> pull
> > >>>>> requests.
> > >>>>>
> > >>>>> I hope that you like it and that it can find some uses in the rest
> of
> > >>> the
> > >>>>> Kafka community.
> > >>>>>
> > >>>>> Best
> > >>>>>
> > >>>>> Pierre
> > >>>>>
> > >>>>> PS: we are aware of
> > >> https://github.com/claudemamo/kafka-web-consolebut
> > >>>>> are
> > >>>>> currently offering slightly different features. Hopefully we can
> > >> merge
> > >>>> the
> > >>>>> projects in the future.
> > >>>>>
> > >>>>> * here:
> > >>
> >
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> > >>
> >
>
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Pierre Andrews <pi...@quantifind.com>.
On Sat, Mar 8, 2014 at 12:17 AM, Dan Hoffman <ho...@gmail.com> wrote:
> Hi - am I right in that for this tool to be effective, consumers must be
> using the high level consumer or otherwise keeping their offsets in
> zookeeper?
yes indeed, this will look into zookeeper for the consumers and their
offsets.
> Is there any way to track performance without that?
>
I have to say that we haven't been using any other type of consumers, so I
haven't explored how to track performance there. The code is pretty simple
and could be made modular to grab offsets and log sizes form other sources
I guess.
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Dan Hoffman <ho...@gmail.com>.
Hi - am I right in that for this tool to be effective, consumers must be
using the high level consumer or otherwise keeping their offsets in
zookeeper? Is there any way to track performance without that?
On Fri, Mar 7, 2014 at 3:08 PM, Steve Morin <st...@gmail.com> wrote:
> Very nice
>
> > On Mar 7, 2014, at 11:55, Pierre Andrews <pi...@quantifind.com> wrote:
> >
> > Claude, we should join forces ;)
> >
> >
> >> On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo <cl...@gmail.com>
> wrote:
> >>
> >> Awesome!!! ;-)
> >>
> >> Claude
> >>
> >>
> >> On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews <pierre@quantifind.com
> >>> wrote:
> >>
> >>> Great! Thanks!
> >>>
> >>>
> >>>> On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <ja...@gmail.com>
> wrote:
> >>>>
> >>>> This is really useful! I added it to the ecosystem page:
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> >>>>
> >>>> -Jay
> >>>>
> >>>>
> >>>> On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews <
> pierre@quantifind.com
> >>>>> wrote:
> >>>>
> >>>>> Hello everyone,
> >>>>>
> >>>>> at Quantifind, we are big users of Kafka and we like it a lot!
> >>>>> In a few use cases, we had to figure out if a queue was growing and
> >> how
> >>>> its
> >>>>> consumers were behaving. There are a few command-line tools to try to
> >>>>> figure out what's going on, but it's not always easy to debug and to
> >>> see
> >>>>> what has happened while everyone was sleeping.
> >>>>>
> >>>>> To be able to monitor our kafka queues and consumers, we thus
> >>> developed a
> >>>>> tiny web app that could tell us the log size of each topic in the
> >>> brokers
> >>>>> and the offsets of each consumers. That's very similar to what the
> >>> kafka
> >>>>> ConsumerOffsetChecker tool* is doing, but instead of having a one off
> >>>>> snapshot, our app keeps an history and displays a nice graph of
> >> what's
> >>>>> going on.
> >>>>>
> >>>>> You can find screenshots and more details here:
> >>>>> http://quantifind.github.io/KafkaOffsetMonitor/
> >>>>>
> >>>>> the code is on github:
> >>>>> https://github.com/quantifind/KafkaOffsetMonitor
> >>>>>
> >>>>> If you have kafka 0.8 setup, it's very easy to use:
> >>>>>
> >>>>> 1- download the current jar
> >>
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> >>>>> 2- run it, pointing at your kafka brokers:
> >>>>> java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> >>>>> com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> >>>>> --zk zk-server1,zk-server2 \
> >>>>> --port 8080 \
> >>>>> --refresh 10.seconds \
> >>>>> --retain 2.days
> >>>>> 3- open your browser and point it to localhost:8080
> >>>>>
> >>>>> You can run it locally or host it on a server if you prefer.
> >>>>>
> >>>>> It's all open source and we'll be happy to receive issue report and
> >>> pull
> >>>>> requests.
> >>>>>
> >>>>> I hope that you like it and that it can find some uses in the rest of
> >>> the
> >>>>> Kafka community.
> >>>>>
> >>>>> Best
> >>>>>
> >>>>> Pierre
> >>>>>
> >>>>> PS: we are aware of
> >> https://github.com/claudemamo/kafka-web-consolebut
> >>>>> are
> >>>>> currently offering slightly different features. Hopefully we can
> >> merge
> >>>> the
> >>>>> projects in the future.
> >>>>>
> >>>>> * here:
> >>
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> >>
>
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Steve Morin <st...@gmail.com>.
Very nice
> On Mar 7, 2014, at 11:55, Pierre Andrews <pi...@quantifind.com> wrote:
>
> Claude, we should join forces ;)
>
>
>> On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo <cl...@gmail.com> wrote:
>>
>> Awesome!!! ;-)
>>
>> Claude
>>
>>
>> On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews <pierre@quantifind.com
>>> wrote:
>>
>>> Great! Thanks!
>>>
>>>
>>>> On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <ja...@gmail.com> wrote:
>>>>
>>>> This is really useful! I added it to the ecosystem page:
>>>> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
>>>>
>>>> -Jay
>>>>
>>>>
>>>> On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews <pierre@quantifind.com
>>>>> wrote:
>>>>
>>>>> Hello everyone,
>>>>>
>>>>> at Quantifind, we are big users of Kafka and we like it a lot!
>>>>> In a few use cases, we had to figure out if a queue was growing and
>> how
>>>> its
>>>>> consumers were behaving. There are a few command-line tools to try to
>>>>> figure out what's going on, but it's not always easy to debug and to
>>> see
>>>>> what has happened while everyone was sleeping.
>>>>>
>>>>> To be able to monitor our kafka queues and consumers, we thus
>>> developed a
>>>>> tiny web app that could tell us the log size of each topic in the
>>> brokers
>>>>> and the offsets of each consumers. That's very similar to what the
>>> kafka
>>>>> ConsumerOffsetChecker tool* is doing, but instead of having a one off
>>>>> snapshot, our app keeps an history and displays a nice graph of
>> what's
>>>>> going on.
>>>>>
>>>>> You can find screenshots and more details here:
>>>>> http://quantifind.github.io/KafkaOffsetMonitor/
>>>>>
>>>>> the code is on github:
>>>>> https://github.com/quantifind/KafkaOffsetMonitor
>>>>>
>>>>> If you have kafka 0.8 setup, it's very easy to use:
>>>>>
>>>>> 1- download the current jar
>> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
>>>>> 2- run it, pointing at your kafka brokers:
>>>>> java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
>>>>> com.quantifind.kafka.offsetapp.OffsetGetterWeb \
>>>>> --zk zk-server1,zk-server2 \
>>>>> --port 8080 \
>>>>> --refresh 10.seconds \
>>>>> --retain 2.days
>>>>> 3- open your browser and point it to localhost:8080
>>>>>
>>>>> You can run it locally or host it on a server if you prefer.
>>>>>
>>>>> It's all open source and we'll be happy to receive issue report and
>>> pull
>>>>> requests.
>>>>>
>>>>> I hope that you like it and that it can find some uses in the rest of
>>> the
>>>>> Kafka community.
>>>>>
>>>>> Best
>>>>>
>>>>> Pierre
>>>>>
>>>>> PS: we are aware of
>> https://github.com/claudemamo/kafka-web-consolebut
>>>>> are
>>>>> currently offering slightly different features. Hopefully we can
>> merge
>>>> the
>>>>> projects in the future.
>>>>>
>>>>> * here:
>> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
>>
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Pierre Andrews <pi...@quantifind.com>.
Claude, we should join forces ;)
On Fri, Mar 7, 2014 at 4:45 PM, Claude Mamo <cl...@gmail.com> wrote:
> Awesome!!! ;-)
>
> Claude
>
>
> On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews <pierre@quantifind.com
> >wrote:
>
> > Great! Thanks!
> >
> >
> > On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> > > This is really useful! I added it to the ecosystem page:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> > >
> > > -Jay
> > >
> > >
> > > On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews <pierre@quantifind.com
> > > >wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > at Quantifind, we are big users of Kafka and we like it a lot!
> > > > In a few use cases, we had to figure out if a queue was growing and
> how
> > > its
> > > > consumers were behaving. There are a few command-line tools to try to
> > > > figure out what's going on, but it's not always easy to debug and to
> > see
> > > > what has happened while everyone was sleeping.
> > > >
> > > > To be able to monitor our kafka queues and consumers, we thus
> > developed a
> > > > tiny web app that could tell us the log size of each topic in the
> > brokers
> > > > and the offsets of each consumers. That's very similar to what the
> > kafka
> > > > ConsumerOffsetChecker tool* is doing, but instead of having a one off
> > > > snapshot, our app keeps an history and displays a nice graph of
> what's
> > > > going on.
> > > >
> > > > You can find screenshots and more details here:
> > > > http://quantifind.github.io/KafkaOffsetMonitor/
> > > >
> > > > the code is on github:
> > > > https://github.com/quantifind/KafkaOffsetMonitor
> > > >
> > > > If you have kafka 0.8 setup, it's very easy to use:
> > > >
> > > > 1- download the current jar
> > > >
> > > >
> > >
> >
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> > > > 2- run it, pointing at your kafka brokers:
> > > > java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> > > > com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> > > > --zk zk-server1,zk-server2 \
> > > > --port 8080 \
> > > > --refresh 10.seconds \
> > > > --retain 2.days
> > > > 3- open your browser and point it to localhost:8080
> > > >
> > > > You can run it locally or host it on a server if you prefer.
> > > >
> > > > It's all open source and we'll be happy to receive issue report and
> > pull
> > > > requests.
> > > >
> > > > I hope that you like it and that it can find some uses in the rest of
> > the
> > > > Kafka community.
> > > >
> > > > Best
> > > >
> > > > Pierre
> > > >
> > > > PS: we are aware of
> https://github.com/claudemamo/kafka-web-consolebut
> > > > are
> > > > currently offering slightly different features. Hopefully we can
> merge
> > > the
> > > > projects in the future.
> > > >
> > > > * here:
> > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> > > >
> > >
> >
>
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Claude Mamo <cl...@gmail.com>.
Awesome!!! ;-)
Claude
On Fri, Mar 7, 2014 at 4:03 PM, Pierre Andrews <pi...@quantifind.com>wrote:
> Great! Thanks!
>
>
> On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <ja...@gmail.com> wrote:
>
> > This is really useful! I added it to the ecosystem page:
> > https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
> >
> > -Jay
> >
> >
> > On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews <pierre@quantifind.com
> > >wrote:
> >
> > > Hello everyone,
> > >
> > > at Quantifind, we are big users of Kafka and we like it a lot!
> > > In a few use cases, we had to figure out if a queue was growing and how
> > its
> > > consumers were behaving. There are a few command-line tools to try to
> > > figure out what's going on, but it's not always easy to debug and to
> see
> > > what has happened while everyone was sleeping.
> > >
> > > To be able to monitor our kafka queues and consumers, we thus
> developed a
> > > tiny web app that could tell us the log size of each topic in the
> brokers
> > > and the offsets of each consumers. That's very similar to what the
> kafka
> > > ConsumerOffsetChecker tool* is doing, but instead of having a one off
> > > snapshot, our app keeps an history and displays a nice graph of what's
> > > going on.
> > >
> > > You can find screenshots and more details here:
> > > http://quantifind.github.io/KafkaOffsetMonitor/
> > >
> > > the code is on github:
> > > https://github.com/quantifind/KafkaOffsetMonitor
> > >
> > > If you have kafka 0.8 setup, it's very easy to use:
> > >
> > > 1- download the current jar
> > >
> > >
> >
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> > > 2- run it, pointing at your kafka brokers:
> > > java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> > > com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> > > --zk zk-server1,zk-server2 \
> > > --port 8080 \
> > > --refresh 10.seconds \
> > > --retain 2.days
> > > 3- open your browser and point it to localhost:8080
> > >
> > > You can run it locally or host it on a server if you prefer.
> > >
> > > It's all open source and we'll be happy to receive issue report and
> pull
> > > requests.
> > >
> > > I hope that you like it and that it can find some uses in the rest of
> the
> > > Kafka community.
> > >
> > > Best
> > >
> > > Pierre
> > >
> > > PS: we are aware of https://github.com/claudemamo/kafka-web-consolebut
> > > are
> > > currently offering slightly different features. Hopefully we can merge
> > the
> > > projects in the future.
> > >
> > > * here:
> > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> > >
> >
>
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Pierre Andrews <pi...@quantifind.com>.
Great! Thanks!
On Fri, Mar 7, 2014 at 3:59 PM, Jay Kreps <ja...@gmail.com> wrote:
> This is really useful! I added it to the ecosystem page:
> https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
>
> -Jay
>
>
> On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews <pierre@quantifind.com
> >wrote:
>
> > Hello everyone,
> >
> > at Quantifind, we are big users of Kafka and we like it a lot!
> > In a few use cases, we had to figure out if a queue was growing and how
> its
> > consumers were behaving. There are a few command-line tools to try to
> > figure out what's going on, but it's not always easy to debug and to see
> > what has happened while everyone was sleeping.
> >
> > To be able to monitor our kafka queues and consumers, we thus developed a
> > tiny web app that could tell us the log size of each topic in the brokers
> > and the offsets of each consumers. That's very similar to what the kafka
> > ConsumerOffsetChecker tool* is doing, but instead of having a one off
> > snapshot, our app keeps an history and displays a nice graph of what's
> > going on.
> >
> > You can find screenshots and more details here:
> > http://quantifind.github.io/KafkaOffsetMonitor/
> >
> > the code is on github:
> > https://github.com/quantifind/KafkaOffsetMonitor
> >
> > If you have kafka 0.8 setup, it's very easy to use:
> >
> > 1- download the current jar
> >
> >
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> > 2- run it, pointing at your kafka brokers:
> > java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> > com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> > --zk zk-server1,zk-server2 \
> > --port 8080 \
> > --refresh 10.seconds \
> > --retain 2.days
> > 3- open your browser and point it to localhost:8080
> >
> > You can run it locally or host it on a server if you prefer.
> >
> > It's all open source and we'll be happy to receive issue report and pull
> > requests.
> >
> > I hope that you like it and that it can find some uses in the rest of the
> > Kafka community.
> >
> > Best
> >
> > Pierre
> >
> > PS: we are aware of https://github.com/claudemamo/kafka-web-console but
> > are
> > currently offering slightly different features. Hopefully we can merge
> the
> > projects in the future.
> >
> > * here:
> >
> >
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> >
>
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Jay Kreps <ja...@gmail.com>.
This is really useful! I added it to the ecosystem page:
https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem
-Jay
On Fri, Mar 7, 2014 at 10:49 AM, Pierre Andrews <pi...@quantifind.com>wrote:
> Hello everyone,
>
> at Quantifind, we are big users of Kafka and we like it a lot!
> In a few use cases, we had to figure out if a queue was growing and how its
> consumers were behaving. There are a few command-line tools to try to
> figure out what's going on, but it's not always easy to debug and to see
> what has happened while everyone was sleeping.
>
> To be able to monitor our kafka queues and consumers, we thus developed a
> tiny web app that could tell us the log size of each topic in the brokers
> and the offsets of each consumers. That's very similar to what the kafka
> ConsumerOffsetChecker tool* is doing, but instead of having a one off
> snapshot, our app keeps an history and displays a nice graph of what's
> going on.
>
> You can find screenshots and more details here:
> http://quantifind.github.io/KafkaOffsetMonitor/
>
> the code is on github:
> https://github.com/quantifind/KafkaOffsetMonitor
>
> If you have kafka 0.8 setup, it's very easy to use:
>
> 1- download the current jar
>
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> 2- run it, pointing at your kafka brokers:
> java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> --zk zk-server1,zk-server2 \
> --port 8080 \
> --refresh 10.seconds \
> --retain 2.days
> 3- open your browser and point it to localhost:8080
>
> You can run it locally or host it on a server if you prefer.
>
> It's all open source and we'll be happy to receive issue report and pull
> requests.
>
> I hope that you like it and that it can find some uses in the rest of the
> Kafka community.
>
> Best
>
> Pierre
>
> PS: we are aware of https://github.com/claudemamo/kafka-web-console but
> are
> currently offering slightly different features. Hopefully we can merge the
> projects in the future.
>
> * here:
>
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
>
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Azuryy Yu <az...@gmail.com>.
hi,
Kafka web console will be included in the next release.
On Mar 8, 2014 2:49 AM, "Pierre Andrews" <pi...@quantifind.com> wrote:
> Hello everyone,
>
> at Quantifind, we are big users of Kafka and we like it a lot!
> In a few use cases, we had to figure out if a queue was growing and how its
> consumers were behaving. There are a few command-line tools to try to
> figure out what's going on, but it's not always easy to debug and to see
> what has happened while everyone was sleeping.
>
> To be able to monitor our kafka queues and consumers, we thus developed a
> tiny web app that could tell us the log size of each topic in the brokers
> and the offsets of each consumers. That's very similar to what the kafka
> ConsumerOffsetChecker tool* is doing, but instead of having a one off
> snapshot, our app keeps an history and displays a nice graph of what's
> going on.
>
> You can find screenshots and more details here:
> http://quantifind.github.io/KafkaOffsetMonitor/
>
> the code is on github:
> https://github.com/quantifind/KafkaOffsetMonitor
>
> If you have kafka 0.8 setup, it's very easy to use:
>
> 1- download the current jar
>
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> 2- run it, pointing at your kafka brokers:
> java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> --zk zk-server1,zk-server2 \
> --port 8080 \
> --refresh 10.seconds \
> --retain 2.days
> 3- open your browser and point it to localhost:8080
>
> You can run it locally or host it on a server if you prefer.
>
> It's all open source and we'll be happy to receive issue report and pull
> requests.
>
> I hope that you like it and that it can find some uses in the rest of the
> Kafka community.
>
> Best
>
> Pierre
>
> PS: we are aware of https://github.com/claudemamo/kafka-web-console but
> are
> currently offering slightly different features. Hopefully we can merge the
> projects in the future.
>
> * here:
>
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
>
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Otis Gospodnetic <ot...@gmail.com>.
Interesting. We were able to get this from JMX for Kafka 0.7.2 - here's a
snapshot for one of our Kafka clusters:
https://apps.sematext.com/spm-reports/s/93EtvhnOz0
Is getting this from ZK instead of JMX better?
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Sat, Mar 8, 2014 at 7:33 AM, Pierre Andrews <pi...@quantifind.com>wrote:
> It's been developed with 0.8 in mind. Giving a quick look at how 0.7 stores
> data in ZK, it looks like it's slightly different and might not work
> straight away.
>
> If you are intersted, have a look at:
>
> https://github.com/quantifind/KafkaOffsetMonitor/blob/master/src/main/scala/com/quantifind/kafka/OffsetGetter.scala
>
> This is where the data is read from ZK and Kafka. If you have a 0.7 setup,
> you might be able to adapt it to get the right offsets, you would probably
> not have to change anything else in the code.
>
> P
>
>
> On Sat, Mar 8, 2014 at 2:48 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com
> > wrote:
>
> > Hi,
> >
> > Does this work with Kafka 0.7.x or does Kafka 0.7.x not expose the info
> > needed for computing the lag?
> >
> > Thanks,
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Fri, Mar 7, 2014 at 1:49 PM, Pierre Andrews <pierre@quantifind.com
> > >wrote:
> >
> > > Hello everyone,
> > >
> > > at Quantifind, we are big users of Kafka and we like it a lot!
> > > In a few use cases, we had to figure out if a queue was growing and how
> > its
> > > consumers were behaving. There are a few command-line tools to try to
> > > figure out what's going on, but it's not always easy to debug and to
> see
> > > what has happened while everyone was sleeping.
> > >
> > > To be able to monitor our kafka queues and consumers, we thus
> developed a
> > > tiny web app that could tell us the log size of each topic in the
> brokers
> > > and the offsets of each consumers. That's very similar to what the
> kafka
> > > ConsumerOffsetChecker tool* is doing, but instead of having a one off
> > > snapshot, our app keeps an history and displays a nice graph of what's
> > > going on.
> > >
> > > You can find screenshots and more details here:
> > > http://quantifind.github.io/KafkaOffsetMonitor/
> > >
> > > the code is on github:
> > > https://github.com/quantifind/KafkaOffsetMonitor
> > >
> > > If you have kafka 0.8 setup, it's very easy to use:
> > >
> > > 1- download the current jar
> > >
> > >
> >
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> > > 2- run it, pointing at your kafka brokers:
> > > java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> > > com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> > > --zk zk-server1,zk-server2 \
> > > --port 8080 \
> > > --refresh 10.seconds \
> > > --retain 2.days
> > > 3- open your browser and point it to localhost:8080
> > >
> > > You can run it locally or host it on a server if you prefer.
> > >
> > > It's all open source and we'll be happy to receive issue report and
> pull
> > > requests.
> > >
> > > I hope that you like it and that it can find some uses in the rest of
> the
> > > Kafka community.
> > >
> > > Best
> > >
> > > Pierre
> > >
> > > PS: we are aware of https://github.com/claudemamo/kafka-web-consolebut
> > > are
> > > currently offering slightly different features. Hopefully we can merge
> > the
> > > projects in the future.
> > >
> > > * here:
> > >
> > >
> >
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> > >
> >
>
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Pierre Andrews <pi...@quantifind.com>.
It's been developed with 0.8 in mind. Giving a quick look at how 0.7 stores
data in ZK, it looks like it's slightly different and might not work
straight away.
If you are intersted, have a look at:
https://github.com/quantifind/KafkaOffsetMonitor/blob/master/src/main/scala/com/quantifind/kafka/OffsetGetter.scala
This is where the data is read from ZK and Kafka. If you have a 0.7 setup,
you might be able to adapt it to get the right offsets, you would probably
not have to change anything else in the code.
P
On Sat, Mar 8, 2014 at 2:48 AM, Otis Gospodnetic <otis.gospodnetic@gmail.com
> wrote:
> Hi,
>
> Does this work with Kafka 0.7.x or does Kafka 0.7.x not expose the info
> needed for computing the lag?
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Fri, Mar 7, 2014 at 1:49 PM, Pierre Andrews <pierre@quantifind.com
> >wrote:
>
> > Hello everyone,
> >
> > at Quantifind, we are big users of Kafka and we like it a lot!
> > In a few use cases, we had to figure out if a queue was growing and how
> its
> > consumers were behaving. There are a few command-line tools to try to
> > figure out what's going on, but it's not always easy to debug and to see
> > what has happened while everyone was sleeping.
> >
> > To be able to monitor our kafka queues and consumers, we thus developed a
> > tiny web app that could tell us the log size of each topic in the brokers
> > and the offsets of each consumers. That's very similar to what the kafka
> > ConsumerOffsetChecker tool* is doing, but instead of having a one off
> > snapshot, our app keeps an history and displays a nice graph of what's
> > going on.
> >
> > You can find screenshots and more details here:
> > http://quantifind.github.io/KafkaOffsetMonitor/
> >
> > the code is on github:
> > https://github.com/quantifind/KafkaOffsetMonitor
> >
> > If you have kafka 0.8 setup, it's very easy to use:
> >
> > 1- download the current jar
> >
> >
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> > 2- run it, pointing at your kafka brokers:
> > java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> > com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> > --zk zk-server1,zk-server2 \
> > --port 8080 \
> > --refresh 10.seconds \
> > --retain 2.days
> > 3- open your browser and point it to localhost:8080
> >
> > You can run it locally or host it on a server if you prefer.
> >
> > It's all open source and we'll be happy to receive issue report and pull
> > requests.
> >
> > I hope that you like it and that it can find some uses in the rest of the
> > Kafka community.
> >
> > Best
> >
> > Pierre
> >
> > PS: we are aware of https://github.com/claudemamo/kafka-web-console but
> > are
> > currently offering slightly different features. Hopefully we can merge
> the
> > projects in the future.
> >
> > * here:
> >
> >
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
> >
>
Re: Anouncing Kafka Offset Monitor 0.1
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
Does this work with Kafka 0.7.x or does Kafka 0.7.x not expose the info
needed for computing the lag?
Thanks,
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Fri, Mar 7, 2014 at 1:49 PM, Pierre Andrews <pi...@quantifind.com>wrote:
> Hello everyone,
>
> at Quantifind, we are big users of Kafka and we like it a lot!
> In a few use cases, we had to figure out if a queue was growing and how its
> consumers were behaving. There are a few command-line tools to try to
> figure out what's going on, but it's not always easy to debug and to see
> what has happened while everyone was sleeping.
>
> To be able to monitor our kafka queues and consumers, we thus developed a
> tiny web app that could tell us the log size of each topic in the brokers
> and the offsets of each consumers. That's very similar to what the kafka
> ConsumerOffsetChecker tool* is doing, but instead of having a one off
> snapshot, our app keeps an history and displays a nice graph of what's
> going on.
>
> You can find screenshots and more details here:
> http://quantifind.github.io/KafkaOffsetMonitor/
>
> the code is on github:
> https://github.com/quantifind/KafkaOffsetMonitor
>
> If you have kafka 0.8 setup, it's very easy to use:
>
> 1- download the current jar
>
> http://quantifind.github.io/KafkaOffsetMonitor/dist/KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar
> 2- run it, pointing at your kafka brokers:
> java -cp KafkaOffsetMonitor-assembly-0.1.0-SNAPSHOT.jar \
> com.quantifind.kafka.offsetapp.OffsetGetterWeb \
> --zk zk-server1,zk-server2 \
> --port 8080 \
> --refresh 10.seconds \
> --retain 2.days
> 3- open your browser and point it to localhost:8080
>
> You can run it locally or host it on a server if you prefer.
>
> It's all open source and we'll be happy to receive issue report and pull
> requests.
>
> I hope that you like it and that it can find some uses in the rest of the
> Kafka community.
>
> Best
>
> Pierre
>
> PS: we are aware of https://github.com/claudemamo/kafka-web-console but
> are
> currently offering slightly different features. Hopefully we can merge the
> projects in the future.
>
> * here:
>
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
>