You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sa Li <sa...@gmail.com> on 2015/01/08 20:08:32 UTC
kafka monitoring
Hello, All
I understand many of you are using jmxtrans along with graphite/ganglia to
pull out metrics, according to https://kafka.apache.org/081/ops.html, it
says "The easiest way to see the available metrics to fire up jconsole and
point it at a running kafka client or server; this will all browsing all
metrics with JMX. .."
I tried to fire up a jconsole on windows attempting to access our dev and
production cluster which are running good,
here is the main node of my dev:
10.100.75.128, broker port:9092, zk port:2181
Jconsole shows:
New Connection
Remote Process:
Usage: <hostname>:<port> OR service:jmx:<protocol>:<sap>
Username: Password:
Sorry about my naive, I tried connect base on above ip just can't be
connected, do I need to do something in dev server to be able to make it
work?
thanks
--
Alec Li
Re: kafka monitoring
Posted by Rajasekar Elango <re...@salesforce.com>.
Hi Sa Li,
You need to set environment variable $JMX_PORT to enable jmx while starting
kafka. See to kafka-run-class.sh on how it is used. Then you can connect
to <hostname>:<jmxport> using Jconsole.
Thanks,
Raja.
On Thu, Jan 8, 2015 at 2:08 PM, Sa Li <sa...@gmail.com> wrote:
> Hello, All
>
> I understand many of you are using jmxtrans along with graphite/ganglia to
> pull out metrics, according to https://kafka.apache.org/081/ops.html, it
> says "The easiest way to see the available metrics to fire up jconsole and
> point it at a running kafka client or server; this will all browsing all
> metrics with JMX. .."
>
> I tried to fire up a jconsole on windows attempting to access our dev and
> production cluster which are running good,
> here is the main node of my dev:
> 10.100.75.128, broker port:9092, zk port:2181
>
> Jconsole shows:
>
> New Connection
> Remote Process:
>
> Usage: <hostname>:<port> OR service:jmx:<protocol>:<sap>
> Username: Password:
>
> Sorry about my naive, I tried connect base on above ip just can't be
> connected, do I need to do something in dev server to be able to make it
> work?
>
> thanks
>
> --
>
> Alec Li
>
--
Thanks,
Raja.
Re: kafka monitoring
Posted by Vladimir Tretyakov <vl...@sematext.com>.
Hi Sa Li,
For the monitoring piece there is SPM - see
*http://blog.sematext.com/2015/02/10/kafka-0-8-2-monitoring/
<http://blog.sematext.com/2015/02/10/kafka-0-8-2-monitoring/>*.
Demo https://apps.sematext.com/demo (just select 'SPM.Prod.Kafka' system
after you login as DEMO user)
It will monitor your ZooKeeper, too. Kafka uses ZK, so you may want
to
keep an eye on it as well.
On Fri, Jan 9, 2015 at 12:39 AM, Sa Li <sa...@gmail.com> wrote:
> In addition, I found all the attributes in jconsole MBeans are cool, but
> not being graphed, so again, if I want to view the real-time graphing,
> jmxtrans + graphite is the solution?
>
> thanks
>
> AL
>
> On Thu, Jan 8, 2015 at 1:35 PM, Sa Li <sa...@gmail.com> wrote:
>
> > Thank you very much for all the reply, I am able to connect jconsole now,
> > by set env JMX_PORT=9999 start server. However, when I connect it I found
> > there is a port conflict with the kafka-run-class.sh,
> >
> > Error: Exception thrown by the agent : java.rmi.server.ExportException:
> > Port already in use: 9999; nested exception is:
> > java.net.BindException: Address already in use
> >
> > I can of course to reset the JMX_PORT number other than 9999, but I am
> > curious, do I have to ?
> >
> > thanks
> >
> > AL
> >
> > On Thu, Jan 8, 2015 at 11:57 AM, Gene Robichaux <
> Gene.Robichaux@match.com>
> > wrote:
> >
> >> Is there a firewall between your DEV and PROD environments? If so you
> >> will need to open access on all ports, not just JMX port.
> >>
> >> It gets complicated with JMX.....
> >>
> >> Gene Robichaux
> >> Manager, Database Operations
> >> Match.com
> >> 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225
> >>
> >> -----Original Message-----
> >> From: Sa Li [mailto:salicn@gmail.com]
> >> Sent: Thursday, January 08, 2015 1:09 PM
> >> To: users@kafka.apache.org
> >> Subject: kafka monitoring
> >>
> >> Hello, All
> >>
> >> I understand many of you are using jmxtrans along with graphite/ganglia
> >> to pull out metrics, according to https://kafka.apache.org/081/ops.html
> ,
> >> it says "The easiest way to see the available metrics to fire up
> jconsole
> >> and point it at a running kafka client or server; this will all browsing
> >> all metrics with JMX. .."
> >>
> >> I tried to fire up a jconsole on windows attempting to access our dev
> and
> >> production cluster which are running good, here is the main node of my
> dev:
> >> 10.100.75.128, broker port:9092, zk port:2181
> >>
> >> Jconsole shows:
> >>
> >> New Connection
> >> Remote Process:
> >>
> >> Usage: <hostname>:<port> OR service:jmx:<protocol>:<sap>
> >> Username: Password:
> >>
> >> Sorry about my naive, I tried connect base on above ip just can't be
> >> connected, do I need to do something in dev server to be able to make it
> >> work?
> >>
> >> thanks
> >>
> >> --
> >>
> >> Alec Li
> >>
> >
> >
> >
> > --
> >
> > Alec Li
> >
>
>
>
> --
>
> Alec Li
>
Re: kafka monitoring
Posted by Sa Li <sa...@gmail.com>.
In addition, I found all the attributes in jconsole MBeans are cool, but
not being graphed, so again, if I want to view the real-time graphing,
jmxtrans + graphite is the solution?
thanks
AL
On Thu, Jan 8, 2015 at 1:35 PM, Sa Li <sa...@gmail.com> wrote:
> Thank you very much for all the reply, I am able to connect jconsole now,
> by set env JMX_PORT=9999 start server. However, when I connect it I found
> there is a port conflict with the kafka-run-class.sh,
>
> Error: Exception thrown by the agent : java.rmi.server.ExportException:
> Port already in use: 9999; nested exception is:
> java.net.BindException: Address already in use
>
> I can of course to reset the JMX_PORT number other than 9999, but I am
> curious, do I have to ?
>
> thanks
>
> AL
>
> On Thu, Jan 8, 2015 at 11:57 AM, Gene Robichaux <Ge...@match.com>
> wrote:
>
>> Is there a firewall between your DEV and PROD environments? If so you
>> will need to open access on all ports, not just JMX port.
>>
>> It gets complicated with JMX.....
>>
>> Gene Robichaux
>> Manager, Database Operations
>> Match.com
>> 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225
>>
>> -----Original Message-----
>> From: Sa Li [mailto:salicn@gmail.com]
>> Sent: Thursday, January 08, 2015 1:09 PM
>> To: users@kafka.apache.org
>> Subject: kafka monitoring
>>
>> Hello, All
>>
>> I understand many of you are using jmxtrans along with graphite/ganglia
>> to pull out metrics, according to https://kafka.apache.org/081/ops.html,
>> it says "The easiest way to see the available metrics to fire up jconsole
>> and point it at a running kafka client or server; this will all browsing
>> all metrics with JMX. .."
>>
>> I tried to fire up a jconsole on windows attempting to access our dev and
>> production cluster which are running good, here is the main node of my dev:
>> 10.100.75.128, broker port:9092, zk port:2181
>>
>> Jconsole shows:
>>
>> New Connection
>> Remote Process:
>>
>> Usage: <hostname>:<port> OR service:jmx:<protocol>:<sap>
>> Username: Password:
>>
>> Sorry about my naive, I tried connect base on above ip just can't be
>> connected, do I need to do something in dev server to be able to make it
>> work?
>>
>> thanks
>>
>> --
>>
>> Alec Li
>>
>
>
>
> --
>
> Alec Li
>
--
Alec Li
Re: kafka monitoring
Posted by Sa Li <sa...@gmail.com>.
Thank you very much for all the reply, I am able to connect jconsole now,
by set env JMX_PORT=9999 start server. However, when I connect it I found
there is a port conflict with the kafka-run-class.sh,
Error: Exception thrown by the agent : java.rmi.server.ExportException:
Port already in use: 9999; nested exception is:
java.net.BindException: Address already in use
I can of course to reset the JMX_PORT number other than 9999, but I am
curious, do I have to ?
thanks
AL
On Thu, Jan 8, 2015 at 11:57 AM, Gene Robichaux <Ge...@match.com>
wrote:
> Is there a firewall between your DEV and PROD environments? If so you will
> need to open access on all ports, not just JMX port.
>
> It gets complicated with JMX.....
>
> Gene Robichaux
> Manager, Database Operations
> Match.com
> 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225
>
> -----Original Message-----
> From: Sa Li [mailto:salicn@gmail.com]
> Sent: Thursday, January 08, 2015 1:09 PM
> To: users@kafka.apache.org
> Subject: kafka monitoring
>
> Hello, All
>
> I understand many of you are using jmxtrans along with graphite/ganglia to
> pull out metrics, according to https://kafka.apache.org/081/ops.html, it
> says "The easiest way to see the available metrics to fire up jconsole and
> point it at a running kafka client or server; this will all browsing all
> metrics with JMX. .."
>
> I tried to fire up a jconsole on windows attempting to access our dev and
> production cluster which are running good, here is the main node of my dev:
> 10.100.75.128, broker port:9092, zk port:2181
>
> Jconsole shows:
>
> New Connection
> Remote Process:
>
> Usage: <hostname>:<port> OR service:jmx:<protocol>:<sap>
> Username: Password:
>
> Sorry about my naive, I tried connect base on above ip just can't be
> connected, do I need to do something in dev server to be able to make it
> work?
>
> thanks
>
> --
>
> Alec Li
>
--
Alec Li
RE: kafka monitoring
Posted by Gene Robichaux <Ge...@match.com>.
Is there a firewall between your DEV and PROD environments? If so you will need to open access on all ports, not just JMX port.
It gets complicated with JMX.....
Gene Robichaux
Manager, Database Operations
Match.com
8300 Douglas Avenue I Suite 800 I Dallas, TX 75225
-----Original Message-----
From: Sa Li [mailto:salicn@gmail.com]
Sent: Thursday, January 08, 2015 1:09 PM
To: users@kafka.apache.org
Subject: kafka monitoring
Hello, All
I understand many of you are using jmxtrans along with graphite/ganglia to pull out metrics, according to https://kafka.apache.org/081/ops.html, it says "The easiest way to see the available metrics to fire up jconsole and point it at a running kafka client or server; this will all browsing all metrics with JMX. .."
I tried to fire up a jconsole on windows attempting to access our dev and production cluster which are running good, here is the main node of my dev:
10.100.75.128, broker port:9092, zk port:2181
Jconsole shows:
New Connection
Remote Process:
Usage: <hostname>:<port> OR service:jmx:<protocol>:<sap>
Username: Password:
Sorry about my naive, I tried connect base on above ip just can't be connected, do I need to do something in dev server to be able to make it work?
thanks
--
Alec Li
Re: kafka monitoring
Posted by Joe Stein <jo...@stealth.ly>.
You need to export the JMX_PORT for kafka to use on startup before starting
up
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Thu, Jan 8, 2015 at 2:08 PM, Sa Li <sa...@gmail.com> wrote:
> Hello, All
>
> I understand many of you are using jmxtrans along with graphite/ganglia to
> pull out metrics, according to https://kafka.apache.org/081/ops.html, it
> says "The easiest way to see the available metrics to fire up jconsole and
> point it at a running kafka client or server; this will all browsing all
> metrics with JMX. .."
>
> I tried to fire up a jconsole on windows attempting to access our dev and
> production cluster which are running good,
> here is the main node of my dev:
> 10.100.75.128, broker port:9092, zk port:2181
>
> Jconsole shows:
>
> New Connection
> Remote Process:
>
> Usage: <hostname>:<port> OR service:jmx:<protocol>:<sap>
> Username: Password:
>
> Sorry about my naive, I tried connect base on above ip just can't be
> connected, do I need to do something in dev server to be able to make it
> work?
>
> thanks
>
> --
>
> Alec Li
>
Re: kafka monitoring
Posted by Manikumar Reddy <ku...@nmsworks.co.in>.
Hi,
you need to set jmx remote port.
you can set this by executing below line in terminal and start server.
(or) add below line to kafka-run-class.sh and start server.
export JMX_PORT=9999 (jmx remote port)
and connect jconsole by giving brokerip:9999
On Fri, Jan 9, 2015 at 12:38 AM, Sa Li <sa...@gmail.com> wrote:
> Hello, All
>
> I understand many of you are using jmxtrans along with graphite/ganglia to
> pull out metrics, according to https://kafka.apache.org/081/ops.html, it
> says "The easiest way to see the available metrics to fire up jconsole and
> point it at a running kafka client or server; this will all browsing all
> metrics with JMX. .."
>
> I tried to fire up a jconsole on windows attempting to access our dev and
> production cluster which are running good,
> here is the main node of my dev:
> 10.100.75.128, broker port:9092, zk port:2181
>
> Jconsole shows:
>
> New Connection
> Remote Process:
>
> Usage: <hostname>:<port> OR service:jmx:<protocol>:<sap>
> Username: Password:
>
> Sorry about my naive, I tried connect base on above ip just can't be
> connected, do I need to do something in dev server to be able to make it
> work?
>
> thanks
>
> --
>
> Alec Li
>