You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by mcasandra <mo...@gmail.com> on 2011/03/24 19:44:33 UTC
Central monitoring of Cassandra cluster
Can someone share if they have centralized monitoring for all cassandra
servers. With many nodes it becomes difficult to monitor them individually
unless we can look at data in one place. I am looking at solutions where
this can be done. Looking at Cacti currently but not sure how to integrate
it with JMX.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Central-monitoring-of-Cassandra-cluster-tp6205275p6205275.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Central monitoring of Cassandra cluster
Posted by mcasandra <mo...@gmail.com>.
Thanks everyone this gives me a good head start.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Central-monitoring-of-Cassandra-cluster-tp6205275p6208331.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Central monitoring of Cassandra cluster
Posted by aaron morton <aa...@thelastpickle.com>.
http://www.datastax.com/products/opscenter
https://www.cloudkick.com/features/monitoring
Aaron
On 25 Mar 2011, at 08:36, Prasanna Jayapalan wrote:
> Forgot to mention, you can use the product for free upto 10 Cassandra
> nodes. We want every developer to use this as much as they want.
>
> Prasanna
>
> -----Original Message-----
> From: Prasanna Jayapalan [mailto:pjayapalan@evidentsoftware.com]
> Sent: Thursday, March 24, 2011 3:31 PM
> To: user@cassandra.apache.org
> Subject: RE: Central monitoring of Cassandra cluster
>
> We at Evident Software have ClearStone, a comprehensive JMX based
> monitoring solution for Cassandra (and other NoSQL technologies). You can
> get more details here
> http://www.evidentsoftware.com/products/clearstone-for-cassandra/. You
> should be able to connect to individual ndoes and also get an aggregated
> view of you whole cluster. Check out the heatmaps to get holistic view of
> all the nodes in your cluster
>
> Prasanna Jayapalan , Senior Solution Consultant , Evident Software
> -----Original Message-----
> From: mcasandra [mailto:mohitanchlia@gmail.com]
> Sent: Thursday, March 24, 2011 2:45 PM
> To: cassandra-user@incubator.apache.org
> Subject: Central monitoring of Cassandra cluster
>
> Can someone share if they have centralized monitoring for all cassandra
> servers. With many nodes it becomes difficult to monitor them individually
> unless we can look at data in one place. I am looking at solutions where
> this can be done. Looking at Cacti currently but not sure how to integrate
> it with JMX.
>
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Central-m
> onitoring-of-Cassandra-cluster-tp6205275p6205275.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
RE: Central monitoring of Cassandra cluster
Posted by Prasanna Jayapalan <pj...@evidentsoftware.com>.
Forgot to mention, you can use the product for free upto 10 Cassandra
nodes. We want every developer to use this as much as they want.
Prasanna
-----Original Message-----
From: Prasanna Jayapalan [mailto:pjayapalan@evidentsoftware.com]
Sent: Thursday, March 24, 2011 3:31 PM
To: user@cassandra.apache.org
Subject: RE: Central monitoring of Cassandra cluster
We at Evident Software have ClearStone, a comprehensive JMX based
monitoring solution for Cassandra (and other NoSQL technologies). You can
get more details here
http://www.evidentsoftware.com/products/clearstone-for-cassandra/. You
should be able to connect to individual ndoes and also get an aggregated
view of you whole cluster. Check out the heatmaps to get holistic view of
all the nodes in your cluster
Prasanna Jayapalan , Senior Solution Consultant , Evident Software
-----Original Message-----
From: mcasandra [mailto:mohitanchlia@gmail.com]
Sent: Thursday, March 24, 2011 2:45 PM
To: cassandra-user@incubator.apache.org
Subject: Central monitoring of Cassandra cluster
Can someone share if they have centralized monitoring for all cassandra
servers. With many nodes it becomes difficult to monitor them individually
unless we can look at data in one place. I am looking at solutions where
this can be done. Looking at Cacti currently but not sure how to integrate
it with JMX.
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Central-m
onitoring-of-Cassandra-cluster-tp6205275p6205275.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at
Nabble.com.
RE: Central monitoring of Cassandra cluster
Posted by Prasanna Jayapalan <pj...@evidentsoftware.com>.
We at Evident Software have ClearStone, a comprehensive JMX based
monitoring solution for Cassandra (and other NoSQL technologies). You can
get more details here
http://www.evidentsoftware.com/products/clearstone-for-cassandra/. You
should be able to connect to individual ndoes and also get an aggregated
view of you whole cluster. Check out the heatmaps to get holistic view of
all the nodes in your cluster
Prasanna Jayapalan , Senior Solution Consultant , Evident Software
-----Original Message-----
From: mcasandra [mailto:mohitanchlia@gmail.com]
Sent: Thursday, March 24, 2011 2:45 PM
To: cassandra-user@incubator.apache.org
Subject: Central monitoring of Cassandra cluster
Can someone share if they have centralized monitoring for all cassandra
servers. With many nodes it becomes difficult to monitor them individually
unless we can look at data in one place. I am looking at solutions where
this can be done. Looking at Cacti currently but not sure how to integrate
it with JMX.
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Central-m
onitoring-of-Cassandra-cluster-tp6205275p6205275.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at
Nabble.com.
RE: Central monitoring of Cassandra cluster
Posted by Gregory Szorc <gr...@xobni.com>.
We have Collectd (http://www.collectd.org/) monitoring Cassandra via its Java/JMX plugin. Collectd feeds data to a central Graphite/Carbon (http://graphite.wikidot.com/start) instance via https://github.com/indygreg/collectd-carbon . With Graphite, you can effortlessly utilize the web UI (or HTTP API) to build and save graph definitions that sum/display/etc related values over the whole cluster. You can also utilize Graphite's HTTP API to export raw data. Your monitoring infrastructure could then poll this for alerting.
I have a script that parses a storage-conf.xml file into a Collectd config snippet. But, I don't have that posted in public domain at the moment. In lieu of that, here are some samples that work with Cassandra 0.6.12:
Add the following to a Collectd types file:
cassandra_pool active:GAUGE:0:U, pending:GAUGE:0:U, completed:COUNTER:0:U
cassandra_stage active:GAUGE:0:2147483648, pending:GAUGE:0:2147483648, completed:COUNTER:0:U
cassandra_cf_cache rcnt_hit_rate:GAUGE:0:1, size:GAUGE:0:2147483648, capacity:GAUGE:0:2147483648, hits:COUNTER:0:U, requests:COUNTER:0:U
cassandra_cf_store pending_tasks:GAUGE:0:2147483648, min_row_size:GAUGE:0:U, max_row_size:GAUGE:0:U, mean_row_size:GAUGE:0:U, memtbl_col_cnt:GAUGE:0:U, memtbl_data_size:GAUGE:0:U, memtbl_switch_cnt:COUNTER:0:U, read_cnt:COUNTER:0:U, rcnt_rd_latency:GAUGE:0:2147483648, tot_rd_latency:COUNTER:0:U, write_cnt:COUNTER:0:U, rcnt_wr_latency:GAUGE:0:2147483648, tot_wr_latency:COUNTER:0:U, disk_used_total:GAUGE:0:U, disk_used_live:GAUGE:0:U, ss_table_count:GAUGE:0:1000000, bloom_false_pos:COUNTER:0:U, bloom_rcnt_f_ratio:GAUGE:0:1, bloom_false_ratio:GAUGE:0:1
cassandra_compaction_manager pending:GAUGE:0:U, bytes_in_progress:GAUGE:0:U, bytes_compacted:GAUGE:0:U
cassandra_storage_proxy rcnt_rd_latency:GAUGE:0:2147483648, tot_rd_latency:COUNTER:0:U, rcnt_wr_latency:GAUGE:0:2147483648, tot_wr_latency:COUNTER:0:U, read_operations:COUNTER:0:U, range_operations:COUNTER:0:U tot_rg_latency:COUNTER:0:U rcnt_rg_latency:GAUGE:0:2147483648 write_operations:COUNTER:0:U
(The weird names are due to a character length limitation in Collectd, which enforces the restrictions of RRD, since it uses that out of the box.)
And add the following to your Collectd config file:
<Plugin "java">
JVMArg "-Djava.class.path=/usr/share/collectd/java/collectd-api.jar:/usr/share/collectd/java/generic-jmx.jar"
LoadPlugin "org.collectd.java.GenericJMX"
<Plugin "GenericJMX">
<!-- this will read a stage mbean -->
<MBean "cassandra-row-read-stage">
ObjectName "org.apache.cassandra.concurrent:type=ROW-READ-STAGE"
InstancePrefix "cassandra_row_read_stage"
<Value>
Type "cassandra_stage"
Attribute "ActiveCount"
Attribute "PendingTasks"
Attribute "CompletedTasks"
</Value>
</MBean>
<!-- this will read a specific column family mbean -->
<MBean "cassandra-cf-foo">
ObjectName "org.apache.cassandra.db:columnfamily=Foo,keyspace=KeySpace,type=ColumnFamilyStores"
InstancePrefix "cassandra_cf_foo"
<Value>
Type "cassandra_cf_store"
Attribute "PendingTasks"
Attribute "MinRowCompactedSize"
Attribute "MaxRowCompactedSize"
Attribute "MeanRowCompactedSize"
Attribute "MemtableColumnsCount"
Attribute "MemtableDataSize"
Attribute "MemtableSwitchCount"
Attribute "ReadCount"
Attribute "RecentReadLatencyMicros"
Attribute "TotalReadLatencyMicros"
Attribute "WriteCount"
Attribute "RecentWriteLatencyMicros"
Attribute "TotalWriteLatencyMicros"
Attribute "TotalDiskSpaceUsed"
Attribute "LiveDiskSpaceUsed"
Attribute "LiveSSTableCount"
Attribute "BloomFilterFalsePositives"
Attribute "RecentBloomFilterFalseRatio"
Attribute "BloomFilterFalseRatio"
</Value>
</MBean>
<!-- this defines what to connect to and what to collect. I /think/ you can define multiple connections to monitor many Cassandra instances from one Collectd instance, but haven't tried this -->
<Connection>
Host "cassandra"
Collect "cassandra-row-read-stage"
Collect "cassandra-cf-foo"
ServiceURL "service:jmx:rmi:///jndi/rmi://localhost:8080/jmxrmi"
</Connection>
</Plugin>
</Plugin>
Greg
> -----Original Message-----
> From: mcasandra [mailto:mohitanchlia@gmail.com]
> Sent: Thursday, March 24, 2011 11:45 AM
> To: cassandra-user@incubator.apache.org
> Subject: Central monitoring of Cassandra cluster
>
> Can someone share if they have centralized monitoring for all cassandra
> servers. With many nodes it becomes difficult to monitor them individually
> unless we can look at data in one place. I am looking at solutions where this
> can be done. Looking at Cacti currently but not sure how to integrate it with
> JMX.
>
> --
> View this message in context: http://cassandra-user-incubator-apache-
> org.3065146.n2.nabble.com/Central-monitoring-of-Cassandra-cluster-
> tp6205275p6205275.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
Re: Central monitoring of Cassandra cluster
Posted by Jonathan Colby <jo...@gmail.com>.
Cacti and Munin are great for graphing, nagios is good for monitoring.
I wrote a very simple JMX proxy that you can send a request to and it retrieves the desired JMX beans.
there are jmx proxys out there if you don't want to write your own, for example http://code.google.com/p/polarrose-jmx-rest-bridge/
There is even a JMX proxy that integrates with nagios. I don't remember the name but google will help you.
On Mar 24, 2011, at 7:44 PM, mcasandra wrote:
> Can someone share if they have centralized monitoring for all cassandra
> servers. With many nodes it becomes difficult to monitor them individually
> unless we can look at data in one place. I am looking at solutions where
> this can be done. Looking at Cacti currently but not sure how to integrate
> it with JMX.
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Central-monitoring-of-Cassandra-cluster-tp6205275p6205275.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Central monitoring of Cassandra cluster
Posted by vineet daniel <vi...@gmail.com>.
...in process of using nagios to monitor three servers. Will post updates
shortly.
Regards
Vineet Daniel
Cell : +918106217121
Websites :
Blog <http://vineetdaniel.blogspot.com> |
Linkedin<http://in.linkedin.com/in/vineetdaniel>
| Twitter <https://twitter.com/vineetdaniel>
On Fri, Mar 25, 2011 at 12:14 AM, mcasandra <mo...@gmail.com> wrote:
> Can someone share if they have centralized monitoring for all cassandra
> servers. With many nodes it becomes difficult to monitor them individually
> unless we can look at data in one place. I am looking at solutions where
> this can be done. Looking at Cacti currently but not sure how to integrate
> it with JMX.
>
> --
> View this message in context:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Central-monitoring-of-Cassandra-cluster-tp6205275p6205275.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at
> Nabble.com.
>