You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Anthony Molinaro <an...@alumni.caltech.edu> on 2010/05/20 21:33:29 UTC
Compaction JMX Stats?
Hi,
In the 0.5.x series there was a COMPACTION-POOL which kept track of
in process, pending and completed compactions. With 0.6.x this seems
to have vanished and instead we only have the CompactionManager PendingTasks
statistic. Is there also a completed tasks somewhere? Is there any
way to determine via nodetool if a compaction or anticompaction is occuring?
I'm trying to figure out the cause of random spikes in the Message Deserializer
Queue, and wanted to know if there was any correlation with compaction,
but with only PendingTasks, I would have to be sampling continuously to
catch it, as pending is mostly zero.
Are there other possible causes for a backup in Message Deserializer?
My traffic patterns are pretty constant, I don't see spikes in the
thread pool completed tasks, and often the rate is below the high water
mark for rates.
Thanks,
-Anthony
--
------------------------------------------------------------------------
Anthony Molinaro <an...@alumni.caltech.edu>
Re: Compaction JMX Stats?
Posted by Jonathan Ellis <jb...@gmail.com>.
they are null when there is no compaction in progress
On Fri, May 21, 2010 at 3:13 PM, Anthony Molinaro
<an...@alumni.caltech.edu> wrote:
> On Thu, May 20, 2010 at 02:11:23PM -0700, Jonathan Ellis wrote:
>> No, CM is not exposed to nodetool yet. (You should really be putting
>> metrics into a real monitoring system rather than relying on nodetool.
>> Some example munin plugins are at
>> http://github.com/jbellis/cassandra-munin-plugins, for instance.)
>>
>> CM also has BytesCompacted/BytesTotalInProgress.
>
> I actually do put metrics into a monitoring system, but often use nodetool
> to see what's happening as well. However, the system I use to poll jmx
> is not finding BytesCompacted/BytesTotalInProgress (I mentioned the system
> we used in a previous email with some screenshots). So I downloaded
> jmxquery.jar and ran it, which gave me this.
>
> % java -jar ./jmxquery.jar \
> --url=service:jmx:rmi:///jndi/rmi://localhost:32123/jmxrmi | \
> grep -i compaction
> org.apache.cassandra.db:type=CompactionManager%PendingTasks.value 0
> org.apache.cassandra.db:type=CompactionManager%MinimumCompactionThreshold.value
> 4
> org.apache.cassandra.db:type=CompactionManager%MaximumCompactionThreshold.value
> 32
> org.apache.cassandra.db:type=CompactionManager%ColumnFamilyInProgress.value null
> org.apache.cassandra.db:type=CompactionManager%BytesTotalInProgress.value null
> org.apache.cassandra.db:type=CompactionManager%BytesCompacted.value null
>
> Which explains why I never see it, our poller ignores 'null's
>
> So why are the Bytes* entries null? (I'm using 0.6.2 with the the first
> 4 bugs fixed 992, 1024 x 2, 1039). Is this something which only works
> with trunk?
>
>> Backup in deserialize is likely to be (a) the read or write stage
>> being full so deserialize in turn backs up. Anything that sucks up
>> enough CPU could also cause it, I suppose, but other than that
>> (generating lots of CPU load) there is nothing special about
>> compactions wrt deserialize.
>
> Is there some metric I could look at to determine if it's a? CPU is
> usually at most 20% on these boxes, so I assume it's a causing the backup.
>
> -Anthony
>
>> On Thu, May 20, 2010 at 12:33 PM, Anthony Molinaro
>> <an...@alumni.caltech.edu> wrote:
>> > Hi,
>> >
>> > In the 0.5.x series there was a COMPACTION-POOL which kept track of
>> > in process, pending and completed compactions. With 0.6.x this seems
>> > to have vanished and instead we only have the CompactionManager PendingTasks
>> > statistic. Is there also a completed tasks somewhere? Is there any
>> > way to determine via nodetool if a compaction or anticompaction is occuring?
>> > I'm trying to figure out the cause of random spikes in the Message Deserializer
>> > Queue, and wanted to know if there was any correlation with compaction,
>> > but with only PendingTasks, I would have to be sampling continuously to
>> > catch it, as pending is mostly zero.
>> >
>> > Are there other possible causes for a backup in Message Deserializer?
>> > My traffic patterns are pretty constant, I don't see spikes in the
>> > thread pool completed tasks, and often the rate is below the high water
>> > mark for rates.
>> >
>> > Thanks,
>> >
>> > -Anthony
>> >
>> > --
>> > ------------------------------------------------------------------------
>> > Anthony Molinaro <an...@alumni.caltech.edu>
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro <an...@alumni.caltech.edu>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Compaction JMX Stats?
Posted by Anthony Molinaro <an...@alumni.caltech.edu>.
On Thu, May 20, 2010 at 02:11:23PM -0700, Jonathan Ellis wrote:
> No, CM is not exposed to nodetool yet. (You should really be putting
> metrics into a real monitoring system rather than relying on nodetool.
> Some example munin plugins are at
> http://github.com/jbellis/cassandra-munin-plugins, for instance.)
>
> CM also has BytesCompacted/BytesTotalInProgress.
I actually do put metrics into a monitoring system, but often use nodetool
to see what's happening as well. However, the system I use to poll jmx
is not finding BytesCompacted/BytesTotalInProgress (I mentioned the system
we used in a previous email with some screenshots). So I downloaded
jmxquery.jar and ran it, which gave me this.
% java -jar ./jmxquery.jar \
--url=service:jmx:rmi:///jndi/rmi://localhost:32123/jmxrmi | \
grep -i compaction
org.apache.cassandra.db:type=CompactionManager%PendingTasks.value 0
org.apache.cassandra.db:type=CompactionManager%MinimumCompactionThreshold.value
4
org.apache.cassandra.db:type=CompactionManager%MaximumCompactionThreshold.value
32
org.apache.cassandra.db:type=CompactionManager%ColumnFamilyInProgress.value null
org.apache.cassandra.db:type=CompactionManager%BytesTotalInProgress.value null
org.apache.cassandra.db:type=CompactionManager%BytesCompacted.value null
Which explains why I never see it, our poller ignores 'null's
So why are the Bytes* entries null? (I'm using 0.6.2 with the the first
4 bugs fixed 992, 1024 x 2, 1039). Is this something which only works
with trunk?
> Backup in deserialize is likely to be (a) the read or write stage
> being full so deserialize in turn backs up. Anything that sucks up
> enough CPU could also cause it, I suppose, but other than that
> (generating lots of CPU load) there is nothing special about
> compactions wrt deserialize.
Is there some metric I could look at to determine if it's a? CPU is
usually at most 20% on these boxes, so I assume it's a causing the backup.
-Anthony
> On Thu, May 20, 2010 at 12:33 PM, Anthony Molinaro
> <an...@alumni.caltech.edu> wrote:
> > Hi,
> >
> > In the 0.5.x series there was a COMPACTION-POOL which kept track of
> > in process, pending and completed compactions. With 0.6.x this seems
> > to have vanished and instead we only have the CompactionManager PendingTasks
> > statistic. Is there also a completed tasks somewhere? Is there any
> > way to determine via nodetool if a compaction or anticompaction is occuring?
> > I'm trying to figure out the cause of random spikes in the Message Deserializer
> > Queue, and wanted to know if there was any correlation with compaction,
> > but with only PendingTasks, I would have to be sampling continuously to
> > catch it, as pending is mostly zero.
> >
> > Are there other possible causes for a backup in Message Deserializer?
> > My traffic patterns are pretty constant, I don't see spikes in the
> > thread pool completed tasks, and often the rate is below the high water
> > mark for rates.
> >
> > Thanks,
> >
> > -Anthony
> >
> > --
> > ------------------------------------------------------------------------
> > Anthony Molinaro <an...@alumni.caltech.edu>
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
--
------------------------------------------------------------------------
Anthony Molinaro <an...@alumni.caltech.edu>
Re: Compaction JMX Stats?
Posted by Jonathan Ellis <jb...@gmail.com>.
No, CM is not exposed to nodetool yet. (You should really be putting
metrics into a real monitoring system rather than relying on nodetool.
Some example munin plugins are at
http://github.com/jbellis/cassandra-munin-plugins, for instance.)
CM also has BytesCompacted/BytesTotalInProgress.
Backup in deserialize is likely to be (a) the read or write stage
being full so deserialize in turn backs up. Anything that sucks up
enough CPU could also cause it, I suppose, but other than that
(generating lots of CPU load) there is nothing special about
compactions wrt deserialize.
On Thu, May 20, 2010 at 12:33 PM, Anthony Molinaro
<an...@alumni.caltech.edu> wrote:
> Hi,
>
> In the 0.5.x series there was a COMPACTION-POOL which kept track of
> in process, pending and completed compactions. With 0.6.x this seems
> to have vanished and instead we only have the CompactionManager PendingTasks
> statistic. Is there also a completed tasks somewhere? Is there any
> way to determine via nodetool if a compaction or anticompaction is occuring?
> I'm trying to figure out the cause of random spikes in the Message Deserializer
> Queue, and wanted to know if there was any correlation with compaction,
> but with only PendingTasks, I would have to be sampling continuously to
> catch it, as pending is mostly zero.
>
> Are there other possible causes for a backup in Message Deserializer?
> My traffic patterns are pretty constant, I don't see spikes in the
> thread pool completed tasks, and often the rate is below the high water
> mark for rates.
>
> Thanks,
>
> -Anthony
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro <an...@alumni.caltech.edu>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com