Very large HintsColumnFamily

[Keith Wright <kw...@nanigans.com>]

Hi all,

    I am seeing a VERY large HintsColumnFamily (40+ GB) on one of my nodes (I have 2 DC with 3 nodes each with 2 RF).  Nodetool ring as a result reports load as being way higher for the one node (the delta being the size of the HintsColumnFamily).  This behavior seems to occur if I do a large amount of data loading using that node as the coordinator node.  I found a post related to this (http://mail-archives.apache.org/mod_mbox/cassandra-user/201203.mbox/%3C376CEC01195C894CB9F8A3C274029A96B471D61A@FISH-EX2K10-01.azaleos.net%3E) but wanted to see if there were better ways to handle it then the reset suggested as it seems somewhat risky.  Nodetool netstats never seems to show any streaming data.  With past nodes it seemed like the node eventually fixed itself.  Note that I have the OOTB gc_grace_seconds so perhaps I just need to wait 10 days before that runs again and the data gets deleted?  Is there a way to change gc_grace_seconds outside Cassandra.yaml and thus save myself a node restart?

Note that I am seeing severely degraded performance on this node when it attempts to compact the HintsColumnFamily to the point where I had to set setcompactionthroughput to 999 to ensure it doesn't run again (after which the node started serving requests much faster).

I appreciate the help!

Thanks

Re: Very large HintsColumnFamily

[Rob Coli <rc...@palominodb.com>]

Before we start.. what version of cassandra?

On Fri, Dec 21, 2012 at 4:25 PM, Keith Wright <kw...@nanigans.com> wrote:
> This behavior seems to occur if I do a large
> amount of data loading using that node as the coordinator node.

In general you want to use all nodes to coordinate, not a single one.

> Nodetool netstats never seems to show
> any streaming data.  With past nodes it seemed like the node eventually
> fixed itself.

That node is storing hints for other nodes it believes are or were at
some point in DOWN state. The first step to preventing this problem
from recurring is to understand why it believes/d other nodes are
down. My conjecture is that you are overloading the coordinating node
and/or other nodes with the large amount of write.

> Note that I am seeing severely degraded performance on this node when it
> attempts to compact the HintsColumnFamily to the point where I had to set
> setcompactionthroughput to 999 to ensure it doesn't run again (after which
> the node started serving requests much faster).

Depending on version, your 40gb of hints could be in one 40gb wide
row. Look at nodetool cfstats for HintsColumnFamily to determine if
this is the case.

Do you see "Timed out replaying hint" messages, or are the hints being
successfully delivered?

You have two broad options :

1) purge your hints and then either reload the data (if reloading it
will be idempotent) or "repair -pr" on every node in the cluster.
2) reduce load enough that hints will be successfully delivered,
reduce gc_grace_seconds on the hints cf to 0 and then do a major
compaction.

If I were you, I would probably do 1). The easiest way is to stop the
node and remove all sstables in the HintsColumnFamily.

=Rob

-- 
=Robert Coli
AIM&GTALK - rcoli@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb