You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2009/04/01 17:23:12 UTC
[jira] Created: (CASSANDRA-36) Log runtime stats for analysis by
ops
Log runtime stats for analysis by ops
-------------------------------------
Key: CASSANDRA-36
URL: https://issues.apache.org/jira/browse/CASSANDRA-36
Project: Cassandra
Issue Type: Improvement
Reporter: Jonathan Ellis
We need to log stats that indicate node / cluster health. Some that I can think of are:
- executor pool pending operation (queue) length [memtable per CF, memtablemanager, storageservice.bootstraper, storageservice.consistencymanager]
- memtable size (which is more useful: per CF or total?)
- sstable size, per CF
- number of unmerged SSTables, per CF
- size of sstable indexes (this is the other major semi-permanent memory chunk)
- writes, reads per second (throughput)
- average seconds per write / read (latency)
- percent of reads that have to hit a SSTable (we don't know if this is in the OS cache or not, so is this actually useful?)
- commitlog on-disk size (want to make sure these are getting cleaned out regularly)
Currently some of these are logged in an ad-hoc manner, e.g. time to read in ReadVerbHandler, but aggregation is not done and logs on a per-op basis are going to get quite spammy. I'd like one thread to be in charge of logging, to dump aggregate data (at level INFO) every minute or so.
Might also be nice to expose this on the web console.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-36) Log runtime stats for analysis by
ops
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708744#action_12708744 ]
Jonathan Ellis commented on CASSANDRA-36:
-----------------------------------------
we've made a lot of progress towards covering this with JMX but there are still a few ideas here worth addressing IMO.
> Log runtime stats for analysis by ops
> -------------------------------------
>
> Key: CASSANDRA-36
> URL: https://issues.apache.org/jira/browse/CASSANDRA-36
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Fix For: 0.4
>
>
> We need to log stats that indicate node / cluster health. Some that I can think of are:
> - executor pool pending operation (queue) length [memtable per CF, memtablemanager, storageservice.bootstraper, storageservice.consistencymanager]
> - memtable size (which is more useful: per CF or total?)
> - sstable size, per CF
> - number of unmerged SSTables, per CF
> - size of sstable indexes (this is the other major semi-permanent memory chunk)
> - writes, reads per second (throughput)
> - average seconds per write / read (latency)
> - percent of reads that have to hit a SSTable (we don't know if this is in the OS cache or not, so is this actually useful?)
> - commitlog on-disk size (want to make sure these are getting cleaned out regularly)
> Currently some of these are logged in an ad-hoc manner, e.g. time to read in ReadVerbHandler, but aggregation is not done and logs on a per-op basis are going to get quite spammy. I'd like one thread to be in charge of logging, to dump aggregate data (at level INFO) every minute or so.
> Might also be nice to expose this on the web console.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-36) Log runtime stats for analysis by
ops
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694581#action_12694581 ]
Jonathan Ellis commented on CASSANDRA-36:
-----------------------------------------
Todd points out https://issues.apache.org/jira/browse/HADOOP-5469.
Can we just log a huge json object with log4j, or are per-log-message size limits going to bite?
> Log runtime stats for analysis by ops
> -------------------------------------
>
> Key: CASSANDRA-36
> URL: https://issues.apache.org/jira/browse/CASSANDRA-36
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
>
> We need to log stats that indicate node / cluster health. Some that I can think of are:
> - executor pool pending operation (queue) length [memtable per CF, memtablemanager, storageservice.bootstraper, storageservice.consistencymanager]
> - memtable size (which is more useful: per CF or total?)
> - sstable size, per CF
> - number of unmerged SSTables, per CF
> - size of sstable indexes (this is the other major semi-permanent memory chunk)
> - writes, reads per second (throughput)
> - average seconds per write / read (latency)
> - percent of reads that have to hit a SSTable (we don't know if this is in the OS cache or not, so is this actually useful?)
> - commitlog on-disk size (want to make sure these are getting cleaned out regularly)
> Currently some of these are logged in an ad-hoc manner, e.g. time to read in ReadVerbHandler, but aggregation is not done and logs on a per-op basis are going to get quite spammy. I'd like one thread to be in charge of logging, to dump aggregate data (at level INFO) every minute or so.
> Might also be nice to expose this on the web console.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-36) Log runtime stats for analysis by
ops
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-36?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-36.
-------------------------------------
Resolution: Fixed
Assignee: Jonathan Ellis
The low-hanging fruit is done here. I'll create separate issues (like CASSANDRA-183) for related work.
> Log runtime stats for analysis by ops
> -------------------------------------
>
> Key: CASSANDRA-36
> URL: https://issues.apache.org/jira/browse/CASSANDRA-36
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4
>
>
> We need to log stats that indicate node / cluster health. Some that I can think of are:
> - executor pool pending operation (queue) length [memtable per CF, memtablemanager, storageservice.bootstraper, storageservice.consistencymanager]
> - memtable size (which is more useful: per CF or total?)
> - sstable size, per CF
> - number of unmerged SSTables, per CF
> - size of sstable indexes (this is the other major semi-permanent memory chunk)
> - writes, reads per second (throughput)
> - average seconds per write / read (latency)
> - percent of reads that have to hit a SSTable (we don't know if this is in the OS cache or not, so is this actually useful?)
> - commitlog on-disk size (want to make sure these are getting cleaned out regularly)
> Currently some of these are logged in an ad-hoc manner, e.g. time to read in ReadVerbHandler, but aggregation is not done and logs on a per-op basis are going to get quite spammy. I'd like one thread to be in charge of logging, to dump aggregate data (at level INFO) every minute or so.
> Might also be nice to expose this on the web console.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-36) Log runtime stats for analysis by
ops
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-36?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-36:
------------------------------------
Fix Version/s: 0.4
> Log runtime stats for analysis by ops
> -------------------------------------
>
> Key: CASSANDRA-36
> URL: https://issues.apache.org/jira/browse/CASSANDRA-36
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Fix For: 0.4
>
>
> We need to log stats that indicate node / cluster health. Some that I can think of are:
> - executor pool pending operation (queue) length [memtable per CF, memtablemanager, storageservice.bootstraper, storageservice.consistencymanager]
> - memtable size (which is more useful: per CF or total?)
> - sstable size, per CF
> - number of unmerged SSTables, per CF
> - size of sstable indexes (this is the other major semi-permanent memory chunk)
> - writes, reads per second (throughput)
> - average seconds per write / read (latency)
> - percent of reads that have to hit a SSTable (we don't know if this is in the OS cache or not, so is this actually useful?)
> - commitlog on-disk size (want to make sure these are getting cleaned out regularly)
> Currently some of these are logged in an ad-hoc manner, e.g. time to read in ReadVerbHandler, but aggregation is not done and logs on a per-op basis are going to get quite spammy. I'd like one thread to be in charge of logging, to dump aggregate data (at level INFO) every minute or so.
> Might also be nice to expose this on the web console.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-36) Log runtime stats for analysis by
ops
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-36?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12697825#action_12697825 ]
Jonathan Ellis commented on CASSANDRA-36:
-----------------------------------------
The hadoop guys are right; a pull interface (allowing the stats aggregator to request via http for example) is better than push (writing to logs and making the aggregator parse them out).
> Log runtime stats for analysis by ops
> -------------------------------------
>
> Key: CASSANDRA-36
> URL: https://issues.apache.org/jira/browse/CASSANDRA-36
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jonathan Ellis
> Fix For: 0.4
>
>
> We need to log stats that indicate node / cluster health. Some that I can think of are:
> - executor pool pending operation (queue) length [memtable per CF, memtablemanager, storageservice.bootstraper, storageservice.consistencymanager]
> - memtable size (which is more useful: per CF or total?)
> - sstable size, per CF
> - number of unmerged SSTables, per CF
> - size of sstable indexes (this is the other major semi-permanent memory chunk)
> - writes, reads per second (throughput)
> - average seconds per write / read (latency)
> - percent of reads that have to hit a SSTable (we don't know if this is in the OS cache or not, so is this actually useful?)
> - commitlog on-disk size (want to make sure these are getting cleaned out regularly)
> Currently some of these are logged in an ad-hoc manner, e.g. time to read in ReadVerbHandler, but aggregation is not done and logs on a per-op basis are going to get quite spammy. I'd like one thread to be in charge of logging, to dump aggregate data (at level INFO) every minute or so.
> Might also be nice to expose this on the web console.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-36) Log runtime stats for analysis by
ops
Posted by "Michael Greene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-36?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Greene updated CASSANDRA-36:
------------------------------------
Component/s: Tools
> Log runtime stats for analysis by ops
> -------------------------------------
>
> Key: CASSANDRA-36
> URL: https://issues.apache.org/jira/browse/CASSANDRA-36
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 0.4
>
>
> We need to log stats that indicate node / cluster health. Some that I can think of are:
> - executor pool pending operation (queue) length [memtable per CF, memtablemanager, storageservice.bootstraper, storageservice.consistencymanager]
> - memtable size (which is more useful: per CF or total?)
> - sstable size, per CF
> - number of unmerged SSTables, per CF
> - size of sstable indexes (this is the other major semi-permanent memory chunk)
> - writes, reads per second (throughput)
> - average seconds per write / read (latency)
> - percent of reads that have to hit a SSTable (we don't know if this is in the OS cache or not, so is this actually useful?)
> - commitlog on-disk size (want to make sure these are getting cleaned out regularly)
> Currently some of these are logged in an ad-hoc manner, e.g. time to read in ReadVerbHandler, but aggregation is not done and logs on a per-op basis are going to get quite spammy. I'd like one thread to be in charge of logging, to dump aggregate data (at level INFO) every minute or so.
> Might also be nice to expose this on the web console.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.