hints system CF getting out of control

[Jeremy Hanna <je...@gmail.com>]

We're trying to bootstrap some new nodes and it appears when adding a new node that there is a lot of logging on hints being flushed and compacted.  It's been taking about 75 minutes thus far to bootstrap for only about 10 GB of data.  It's ballooned up to over 40 GB on the new node.  I do 'ls -l $CASSANDRA/data/system | grep -i hint | wc -l' and it's over 2000.

Any idea why this would be happening and why it's going crazy flushing these files?  It's also trying to keep up and compacting them as well.

Completed flushing $CASSANDRA/data/system/HintsColumnFamily-f-1740-Data.db

Version 0.7.8 (upgrading from 0.7.6 right now)
Number of nodes: 20

Re: hints system CF getting out of control

[Jeremy Hanna <je...@gmail.com>]

So if you kill a joining node, it will remain as a Fat Client (as of 0.7.x) in gossip.  That is innocuous on its own.  However if a node was decommissioned while that was joining, then the lingering Fat Client will also gossip about the node that was decommissioned.  Then it will try to hint for that decommissioned node.  That was the origin of our out of control hints.

So we killed the joining node, cleared it out to start fresh, and did a full cluster restart to clear out the gossip of the Fat Clients and the hints.  That worked.

On Aug 18, 2011, at 7:56 PM, Jonathan Ellis wrote:

> I would assume it's because it thinks some node is down and is
> creating hints for it.
> 
> On Thu, Aug 18, 2011 at 6:31 PM, Jeremy Hanna
> <je...@gmail.com> wrote:
>> We're trying to bootstrap some new nodes and it appears when adding a new node that there is a lot of logging on hints being flushed and compacted.  It's been taking about 75 minutes thus far to bootstrap for only about 10 GB of data.  It's ballooned up to over 40 GB on the new node.  I do 'ls -l $CASSANDRA/data/system | grep -i hint | wc -l' and it's over 2000.
>> 
>> Any idea why this would be happening and why it's going crazy flushing these files?  It's also trying to keep up and compacting them as well.
>> 
>> Completed flushing $CASSANDRA/data/system/HintsColumnFamily-f-1740-Data.db
>> 
>> Version 0.7.8 (upgrading from 0.7.6 right now)
>> Number of nodes: 20
> 
> 
> 
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com

Re: hints system CF getting out of control

[Jonathan Ellis <jb...@gmail.com>]

I would assume it's because it thinks some node is down and is
creating hints for it.

On Thu, Aug 18, 2011 at 6:31 PM, Jeremy Hanna
<je...@gmail.com> wrote:
> We're trying to bootstrap some new nodes and it appears when adding a new node that there is a lot of logging on hints being flushed and compacted.  It's been taking about 75 minutes thus far to bootstrap for only about 10 GB of data.  It's ballooned up to over 40 GB on the new node.  I do 'ls -l $CASSANDRA/data/system | grep -i hint | wc -l' and it's over 2000.
>
> Any idea why this would be happening and why it's going crazy flushing these files?  It's also trying to keep up and compacting them as well.
>
> Completed flushing $CASSANDRA/data/system/HintsColumnFamily-f-1740-Data.db
>
> Version 0.7.8 (upgrading from 0.7.6 right now)
> Number of nodes: 20



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com