You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by aaron morton <aa...@thelastpickle.com> on 2010/09/11 01:29:32 UTC
Monitoring with Cacti
Am going through the rather painful process of trying to monitor cassandra using Cacti (it's what we use at work). At the moment it feels like a losing battle :)
Does anyone know of some cacti resources for monitoring the JVM or Cassandra metrics other than...
mysql-cacti-templates
http://code.google.com/p/mysql-cacti-templates/
- provides templates and data sources that require ssh and can monitor JVM heap and a few things.
Cassandra-cacti-m6
http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
Coded for version 0.6* , have made some changes to stop it looking for stats that no longer exist. Missing some metrics I think but it's probably the best bet so far. If I get it working I'll contribute it back to them. Most of the problems were probably down the how much effort it takes to setup cacti.
jmxterm
http://www.cyclopsgroup.org/projects/jmxterm/
Allows for command line access to JMX. I started down the path of writing a cacti data source to use this just to see how it worked. Looks like a lot of work.
Thanks for any advice.
Aaron
Re: Monitoring with Cacti
Posted by hoivan <ih...@evidentsoftware.com>.
Just as an FYI, Evident ClearStone supports monitoring of Cassandra clusters.
Our product supports various NoSQL products (including Cassandra) and other
distributed caching technologies.
The product aggregates metrics (from JMX) across all the nodes in the
cluster. We also monitor JVM stats like CPU, heap, and GC. The user
interface is an Adobe Flex application. We have also include some of the
nodetool operations within the product to allow you to point an click on a
node to run some of the nodetool functions. There's many more features in
the product such as alerting and historical reporting.
You can find more information about Evident ClearStone here:
http://www.evidentsoftware.com/products/clearstone-for-cassandra
Free downloads are available here: http://www.evidentsoftware.com/download
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Monitoring-with-Cacti-tp5520342p5698737.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Monitoring with Cacti
Posted by Ranjib Dey <ra...@thoughtworks.com>.
I use nagios + nrpe + some custom scripts to monitor our cassandra/hadoop
nodes. Given our long time comfortability with nagios, i didn't find any
major gotchas ..
regards
ranjib
On Mon, Sep 13, 2010 at 2:10 AM, Aaron Morton <aa...@thelastpickle.com>wrote:
> This is my first encounter with cacti, and it's feels a lot like having a
> cactus violently inserted in me :) Hopefully this week I can get back to it
> with a clearer head, part of my annoyance was probably trying to rush it
> through on a Friday and it's somewhat taxing configuration.
>
> Over the weekend I was thinking about going with some python (our in
> house favorite) in front of the jmxterm jar.
>
> I'll also try to learn a bit more about cacti, it cannot be as hard as it
> seemed on Friday.
>
> I'll email you out of the list this week if I make some progress.
>
> Aaron
>
>
> On 11 Sep, 2010,at 03:31 PM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
> On Fri, Sep 10, 2010 at 7:29 PM, aaron morton <aa...@thelastpickle.com>
> wrote:
> > Am going through the rather painful process of trying to monitor
> cassandra using Cacti (it's what we use at work). At the moment it feels
> like a losing battle :)
> >
> > Does anyone know of some cacti resources for monitoring the JVM or
> Cassandra metrics other than...
> >
> > mysql-cacti-templates
> > http://code.google.com/p/mysql-cacti-templates/
> > - provides templates and data sources that require ssh and can monitor
> JVM heap and a few things.
> >
> > Cassandra-cacti-m6
> > http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
> > Coded for version 0.6* , have made some changes to stop it looking for
> stats that no longer exist. Missing some metrics I think but it's probably
> the best bet so far. If I get it working I'll contribute it back to them
> Most of the problems were probably down the how much effort it takes to
> setup cacti.
> >
> > jmxterm
> > http://www.cyclopsgroup.org/projects/jmxterm/
> > Allows for command line access to JMX. I started down the path of writing
> a cacti data source to use this just to see how it worked. Looks like a lot
> of work.
> >
> > Thanks for any advice.
> > Aaron
> >
> >
>
> Setting up cacti is easy, the second time, and third time :)
> As for cassandra-cacti-m6 (i am the author). Unfortunately, I have
> been fighting the jmx switcharo battle for about 3 years now
> hadoop/hbase/cassandra/hornetq/vserver
>
> In a nutshell there is ALWAYS work involved. First, is because as you
> noticed attributes change/remove/add/renamed. Second it takes a human
> to logically group things together. For example, if you have two items
> "cache hits" and "cache misses". You really do not want two separate
> graphs that will scale independently. You want one slick stack graph,
> with nice colors, and you want a CDEF to calculate the cache hit
> percentage by dividing one into the other and show that at the bottom.
>
> If you want to have a 7.0 branch to cassandra-cacti-m6 I would love
> the help. We are not on 7.0 yet so I have not had the time just to go
> out and make graphs for a version we are not using yet :) but if you
> come up with patches they are happily accepted.
>
> Edward
>
>
Re: RE: Monitoring with Cacti
Posted by Aaron Morton <aa...@thelastpickle.com>.
Will take another look to see what I can "borrow".
Thanks
Aaron
On 14 Sep, 2010,at 03:53 AM, Jeremiah Jordan <JE...@morningstar.com> wrote:
Have you checked out the munin scripts? If you want a starting point for something that pulls data out of jmx and prints it out (so you can catch and graph it). It is very easy to setup this to get any jmx value you want.
http://github.com/jamesgolick/cassandra-munin-plugins
From: Aaron Morton [mailto:aaron@thelastpickle.com]
Sent: Sunday, September 12, 2010 3:40 PM
To: user@cassandra.apacheorg
Cc: user@cassandra.apache.org
Subject: Re: Monitoring with Cacti
This is my first encounter with cacti, and it's feels a lot like having a cactus violently inserted in me :) Hopefully this week I can get back to it with a clearer head, part of my annoyance was probably trying to rush it through on a Friday and it's somewhat taxing configuration.
Over the weekend I was thinking about going with some python (our in house favorite) in front of the jmxterm jar.
I'll also try to learn a bit more about cacti, it cannot be as hard as it seemed on Friday.
I'll email you out of the list this week if I make some progress.
Aaron
On 11 Sep, 2010,at 03:31 PM, Edward Capriolo <ed...@gmail.com> wrote:
On Fri, Sep 10, 2010 at 7:29 PM, aaron morton <aa...@thelastpickle.com> wrote:
> Am going through the rather painful process of trying to monitor cassandra using Cacti (it's what we use at work). At the moment it feels like a losing battle :)
>
> Does anyone know of some cacti resources for monitoring the JVM or Cassandra metrics other than...
>
> mysql-cacti-templates
> http://code.google.com/p/mysql-cacti-templates/
> - provides templates and data sources that require ssh and can monitor JVM heap and a few things.
>
> Cassandra-cacti-m6
> http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
> Coded for version 0.6* , have made some changes to stop it looking for stats that no longer exist. Missing some metrics I think but it's probably the best bet so far. If I get it working I'll contribute it back to them Most of the problems were probably down the how much effort it takes to setup cacti.
>
> jmxterm
> http://www.cyclopsgroup.org/projects/jmxterm/
> Allows for command line access to JMX. I started down the path of writing a cacti data source to use this just to see how it worked. Looks like a lot of work.
>
> Thanks for any advice.
> Aaron
>
>
Setting up cacti is easy, the second time, and third time :)
As for cassandra-cacti-m6 (i am the author). Unfortunately, I have
been fighting the jmx switcharo battle for about 3 years now
hadoop/hbase/cassandra/hornetq/vserver
In a nutshell there is ALWAYS work involved. First, is because as you
noticed attributes change/remove/add/renamed. Second it takes a human
to logically group things together. For example, if you have two items
"cache hits" and "cache misses". You really do not want two separate
graphs that will scale independently. You want one slick stack graph,
with nice colors, and you want a CDEF to calculate the cache hit
percentage by dividing one into the other and show that at the bottom.
If you want to have a 7.0 branch to cassandra-cacti-m6 I would love
the help. We are not on 7.0 yet so I have not had the time just to go
out and make graphs for a version we are not using yet :) but if you
come up with patches they are happily accepted.
Edward
RE: Monitoring with Cacti
Posted by Jeremiah Jordan <JE...@morningstar.com>.
Have you checked out the munin scripts? If you want a starting point
for something that pulls data out of jmx and prints it out (so you can
catch and graph it). It is very easy to setup this to get any jmx value
you want.
http://github.com/jamesgolick/cassandra-munin-plugins
________________________________
From: Aaron Morton [mailto:aaron@thelastpickle.com]
Sent: Sunday, September 12, 2010 3:40 PM
To: user@cassandra.apache.org
Cc: user@cassandra.apache.org
Subject: Re: Monitoring with Cacti
This is my first encounter with cacti, and it's feels a lot like having
a cactus violently inserted in me :) Hopefully this week I can get back
to it with a clearer head, part of my annoyance was probably trying to
rush it through on a Friday and it's somewhat taxing configuration.
Over the weekend I was thinking about going with some python (our in
house favorite) in front of the jmxterm jar.
I'll also try to learn a bit more about cacti, it cannot be as hard as
it seemed on Friday.
I'll email you out of the list this week if I make some progress.
Aaron
On 11 Sep, 2010,at 03:31 PM, Edward Capriolo <ed...@gmail.com>
wrote:
On Fri, Sep 10, 2010 at 7:29 PM, aaron morton
<aa...@thelastpickle.com> wrote:
> Am going through the rather painful process of trying to
monitor cassandra using Cacti (it's what we use at work). At the moment
it feels like a losing battle :)
>
> Does anyone know of some cacti resources for monitoring the
JVM or Cassandra metrics other than...
>
> mysql-cacti-templates
> http://code.google.com/p/mysql-cacti-templates/
> - provides templates and data sources that require ssh and can
monitor JVM heap and a few things.
>
> Cassandra-cacti-m6
> http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
> Coded for version 0.6* , have made some changes to stop it
looking for stats that no longer exist. Missing some metrics I think but
it's probably the best bet so far. If I get it working I'll contribute
it back to them Most of the problems were probably down the how much
effort it takes to setup cacti.
>
> jmxterm
> http://www.cyclopsgroup.org/projects/jmxterm/
> Allows for command line access to JMX. I started down the path
of writing a cacti data source to use this just to see how it worked.
Looks like a lot of work.
>
> Thanks for any advice.
> Aaron
>
>
Setting up cacti is easy, the second time, and third time :)
As for cassandra-cacti-m6 (i am the author). Unfortunately, I
have
been fighting the jmx switcharo battle for about 3 years now
hadoop/hbase/cassandra/hornetq/vserver
In a nutshell there is ALWAYS work involved. First, is because
as you
noticed attributes change/remove/add/renamed. Second it takes a
human
to logically group things together. For example, if you have two
items
"cache hits" and "cache misses". You really do not want two
separate
graphs that will scale independently. You want one slick stack
graph,
with nice colors, and you want a CDEF to calculate the cache hit
percentage by dividing one into the other and show that at the
bottom.
If you want to have a 7.0 branch to cassandra-cacti-m6 I would
love
the help. We are not on 7.0 yet so I have not had the time just
to go
out and make graphs for a version we are not using yet :) but if
you
come up with patches they are happily accepted.
Edward
Re: Monitoring with Cacti
Posted by Dan Di Spaltro <da...@gmail.com>.
Cloudkick does monitor JMX now. That + custom alerts is pretty powerful.
I work for Cloudkick, btw
On Sun, Sep 12, 2010 at 9:39 AM, Dave Viner <da...@pobox.com> wrote:
> I haven't tried cacti, but I'm using CloudKick as an external service for
> monitoring Cassandra. It's super easy to get setup. Happy to share my
> setup if that'd help.
>
> It doesn't currently monitor JMX information, but it does offer some basic
> checks like thread pool and column family stats -
> https://support.cloudkick.com/Cassandra_Checks.
>
> Dave Viner
>
>
> On Fri, Sep 10, 2010 at 8:31 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> On Fri, Sep 10, 2010 at 7:29 PM, aaron morton <aa...@thelastpickle.com>
>> wrote:
>> > Am going through the rather painful process of trying to monitor
>> cassandra using Cacti (it's what we use at work). At the moment it feels
>> like a losing battle :)
>> >
>> > Does anyone know of some cacti resources for monitoring the JVM or
>> Cassandra metrics other than...
>> >
>> > mysql-cacti-templates
>> > http://code.google.com/p/mysql-cacti-templates/
>> > - provides templates and data sources that require ssh and can monitor
>> JVM heap and a few things.
>> >
>> > Cassandra-cacti-m6
>> > http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
>> > Coded for version 0.6* , have made some changes to stop it looking for
>> stats that no longer exist. Missing some metrics I think but it's probably
>> the best bet so far. If I get it working I'll contribute it back to them.
>> Most of the problems were probably down the how much effort it takes to
>> setup cacti.
>> >
>> > jmxterm
>> > http://www.cyclopsgroup.org/projects/jmxterm/
>> > Allows for command line access to JMX. I started down the path of
>> writing a cacti data source to use this just to see how it worked. Looks
>> like a lot of work.
>> >
>> > Thanks for any advice.
>> > Aaron
>> >
>> >
>>
>> Setting up cacti is easy, the second time, and third time :)
>> As for cassandra-cacti-m6 (i am the author). Unfortunately, I have
>> been fighting the jmx switcharo battle for about 3 years now
>> hadoop/hbase/cassandra/hornetq/vserver
>>
>> In a nutshell there is ALWAYS work involved. First, is because as you
>> noticed attributes change/remove/add/renamed. Second it takes a human
>> to logically group things together. For example, if you have two items
>> "cache hits" and "cache misses". You really do not want two separate
>> graphs that will scale independently. You want one slick stack graph,
>> with nice colors, and you want a CDEF to calculate the cache hit
>> percentage by dividing one into the other and show that at the bottom.
>>
>> If you want to have a 7.0 branch to cassandra-cacti-m6 I would love
>> the help. We are not on 7.0 yet so I have not had the time just to go
>> out and make graphs for a version we are not using yet :) but if you
>> come up with patches they are happily accepted.
>>
>> Edward
>>
>
>
--
Dan Di Spaltro
Re: Monitoring with Cacti
Posted by Aaron Morton <aa...@thelastpickle.com>.
Thanks but this is an internal only application.
Cheers
Aaron
On 13 Sep, 2010,at 04:39 AM, Dave Viner <da...@pobox.com> wrote:
I haven't tried cacti, but I'm using CloudKick as an external service for monitoring Cassandra. It's super easy to get setup. Happy to share my setup if that'd help.
It doesn't currently monitor JMX information, but it does offer some basic checks like thread pool and column family stats - https://support.cloudkick.com/Cassandra_Checks.
Dave Viner
On Fri, Sep 10, 2010 at 8:31 PM, Edward Capriolo <ed...@gmail.com> wrote:
On Fri, Sep 10, 2010 at 7:29 PM, aaron morton <aa...@thelastpickle.com> wrote:
> Am going through the rather painful process of trying to monitor cassandra using Cacti (it's what we use at work). At the moment it feels like a losing battle :)
>
> Does anyone know of some cacti resources for monitoring the JVM or Cassandra metrics other than...
>
> mysql-cacti-templates
> http://code.google.com/p/mysql-cacti-templates/
> - provides templates and data sources that require ssh and can monitor JVM heap and a few things.
>
> Cassandra-cacti-m6
> http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
> Coded for version 0.6* , have made some changes to stop it looking for stats that no longer exist. Missing some metrics I think but it's probably the best bet so far. If I get it working I'll contribute it back to them. Most of the problems were probably down the how much effort it takes to setup cacti.
>
> jmxterm
> http://www.cyclopsgroup.org/projects/jmxterm/
> Allows for command line access to JMX. I started down the path of writing a cacti data source to use this just to see how it worked. Looks like a lot of work.
>
> Thanks for any advice.
> Aaron
>
>
Setting up cacti is easy, the second time, and third time :)
As for cassandra-cacti-m6 (i am the author). Unfortunately, I have
been fighting the jmx switcharo battle for about 3 years now
hadoop/hbase/cassandra/hornetq/vserver
In a nutshell there is ALWAYS work involved. First, is because as you
noticed attributes change/remove/add/renamed. Second it takes a human
to logically group things together. For example, if you have two items
"cache hits" and "cache misses". You really do not want two separate
graphs that will scale independently. You want one slick stack graph,
with nice colors, and you want a CDEF to calculate the cache hit
percentage by dividing one into the other and show that at the bottom.
If you want to have a 7.0 branch to cassandra-cacti-m6 I would love
the help. We are not on 7.0 yet so I have not had the time just to go
out and make graphs for a version we are not using yet :) but if you
come up with patches they are happily accepted.
Edward
Re: Monitoring with Cacti
Posted by Dave Viner <da...@pobox.com>.
I haven't tried cacti, but I'm using CloudKick as an external service for
monitoring Cassandra. It's super easy to get setup. Happy to share my
setup if that'd help.
It doesn't currently monitor JMX information, but it does offer some basic
checks like thread pool and column family stats -
https://support.cloudkick.com/Cassandra_Checks.
Dave Viner
On Fri, Sep 10, 2010 at 8:31 PM, Edward Capriolo <ed...@gmail.com>wrote:
> On Fri, Sep 10, 2010 at 7:29 PM, aaron morton <aa...@thelastpickle.com>
> wrote:
> > Am going through the rather painful process of trying to monitor
> cassandra using Cacti (it's what we use at work). At the moment it feels
> like a losing battle :)
> >
> > Does anyone know of some cacti resources for monitoring the JVM or
> Cassandra metrics other than...
> >
> > mysql-cacti-templates
> > http://code.google.com/p/mysql-cacti-templates/
> > - provides templates and data sources that require ssh and can monitor
> JVM heap and a few things.
> >
> > Cassandra-cacti-m6
> > http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
> > Coded for version 0.6* , have made some changes to stop it looking for
> stats that no longer exist. Missing some metrics I think but it's probably
> the best bet so far. If I get it working I'll contribute it back to them.
> Most of the problems were probably down the how much effort it takes to
> setup cacti.
> >
> > jmxterm
> > http://www.cyclopsgroup.org/projects/jmxterm/
> > Allows for command line access to JMX. I started down the path of writing
> a cacti data source to use this just to see how it worked. Looks like a lot
> of work.
> >
> > Thanks for any advice.
> > Aaron
> >
> >
>
> Setting up cacti is easy, the second time, and third time :)
> As for cassandra-cacti-m6 (i am the author). Unfortunately, I have
> been fighting the jmx switcharo battle for about 3 years now
> hadoop/hbase/cassandra/hornetq/vserver
>
> In a nutshell there is ALWAYS work involved. First, is because as you
> noticed attributes change/remove/add/renamed. Second it takes a human
> to logically group things together. For example, if you have two items
> "cache hits" and "cache misses". You really do not want two separate
> graphs that will scale independently. You want one slick stack graph,
> with nice colors, and you want a CDEF to calculate the cache hit
> percentage by dividing one into the other and show that at the bottom.
>
> If you want to have a 7.0 branch to cassandra-cacti-m6 I would love
> the help. We are not on 7.0 yet so I have not had the time just to go
> out and make graphs for a version we are not using yet :) but if you
> come up with patches they are happily accepted.
>
> Edward
>
Re: Monitoring with Cacti
Posted by Aaron Morton <aa...@thelastpickle.com>.
This is my first encounter with cacti, and it's feels a lot like having a cactus violently inserted in me :) Hopefully this week I can get back to it with a clearer head, part of my annoyance was probably trying to rush it through on a Friday and it's somewhat taxing configuration.
Over the weekend I was thinking about going with some python (our in house favorite) in front of the jmxterm jar.
I'll also try to learn a bit more about cacti, it cannot be as hard as it seemed on Friday.
I'll email you out of the list this week if I make some progress.
Aaron
On 11 Sep, 2010,at 03:31 PM, Edward Capriolo <ed...@gmailcom> wrote:
On Fri, Sep 10, 2010 at 7:29 PM, aaron morton <aa...@thelastpickle.com> wrote:
> Am going through the rather painful process of trying to monitor cassandra using Cacti (it's what we use at work). At the moment it feels like a losing battle :)
>
> Does anyone know of some cacti resources for monitoring the JVM or Cassandra metrics other than...
>
> mysql-cacti-templates
> http://code.google.com/p/mysql-cacti-templates/
> - provides templates and data sources that require ssh and can monitor JVM heap and a few things.
>
> Cassandra-cacti-m6
> http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
> Coded for version 06* , have made some changes to stop it looking for stats that no longer exist. Missing some metrics I think but it's probably the best bet so far. If I get it working I'll contribute it back to them. Most of the problems were probably down the how much effort it takes to setup cacti.
>
> jmxterm
> http://www.cyclopsgroup.org/projects/jmxterm/
> Allows for command line access to JMX. I started down the path of writing a cacti data source to use this just to see how it worked. Looks like a lot of work.
>
> Thanks for any advice.
> Aaron
>
>
Setting up cacti is easy, the second time, and third time :)
As for cassandra-cacti-m6 (i am the author). Unfortunately, I have
been fighting the jmx switcharo battle for about 3 years now
hadoop/hbase/cassandra/hornetq/vserver
In a nutshell there is ALWAYS work involved. First, is because as you
noticed attributes change/remove/add/renamed. Second it takes a human
to logically group things together. For example, if you have two items
"cache hits" and "cache misses". You really do not want two separate
graphs that will scale independently. You want one slick stack graph,
with nice colors, and you want a CDEF to calculate the cache hit
percentage by dividing one into the other and show that at the bottom.
If you want to have a 7.0 branch to cassandra-cacti-m6 I would love
the help. We are not on 70 yet so I have not had the time just to go
out and make graphs for a version we are not using yet :) but if you
come up with patches they are happily accepted.
Edward
Re: Monitoring with Cacti
Posted by Edward Capriolo <ed...@gmail.com>.
On Fri, Sep 10, 2010 at 7:29 PM, aaron morton <aa...@thelastpickle.com> wrote:
> Am going through the rather painful process of trying to monitor cassandra using Cacti (it's what we use at work). At the moment it feels like a losing battle :)
>
> Does anyone know of some cacti resources for monitoring the JVM or Cassandra metrics other than...
>
> mysql-cacti-templates
> http://code.google.com/p/mysql-cacti-templates/
> - provides templates and data sources that require ssh and can monitor JVM heap and a few things.
>
> Cassandra-cacti-m6
> http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
> Coded for version 0.6* , have made some changes to stop it looking for stats that no longer exist. Missing some metrics I think but it's probably the best bet so far. If I get it working I'll contribute it back to them. Most of the problems were probably down the how much effort it takes to setup cacti.
>
> jmxterm
> http://www.cyclopsgroup.org/projects/jmxterm/
> Allows for command line access to JMX. I started down the path of writing a cacti data source to use this just to see how it worked. Looks like a lot of work.
>
> Thanks for any advice.
> Aaron
>
>
Setting up cacti is easy, the second time, and third time :)
As for cassandra-cacti-m6 (i am the author). Unfortunately, I have
been fighting the jmx switcharo battle for about 3 years now
hadoop/hbase/cassandra/hornetq/vserver
In a nutshell there is ALWAYS work involved. First, is because as you
noticed attributes change/remove/add/renamed. Second it takes a human
to logically group things together. For example, if you have two items
"cache hits" and "cache misses". You really do not want two separate
graphs that will scale independently. You want one slick stack graph,
with nice colors, and you want a CDEF to calculate the cache hit
percentage by dividing one into the other and show that at the bottom.
If you want to have a 7.0 branch to cassandra-cacti-m6 I would love
the help. We are not on 7.0 yet so I have not had the time just to go
out and make graphs for a version we are not using yet :) but if you
come up with patches they are happily accepted.
Edward