Re: IllegalStateException when bootstrapping new nodes

[Cyril Scetbon <cy...@free.fr>]

We got issues with Hinted Handoff threads (2 actives) now which cause High CPU Load and a lot of garbage collection only on our new nodes. You can see that we have a lot of Hints on our new nodes (the 4 last ones)

node001
2.2M	/cassandra/data/system/hints/
node002
1.6M	/cassandra/data/system/hints/
node003
1.8M	/cassandra/data/system/hints/
node004
1.9M	/cassandra/data/system/hints/
node005
1.9M	/cassandra/data/system/hints/
node006
2.0M	/cassandra/data/system/hints/
node007
1.9M	/cassandra/data/system/hints/
node008
1.8M	/cassandra/data/system/hints/
node009
34G	/cassandra/data/system/hints/
node011
43G	/cassandra/data/system/hints/
node013
43G	/cassandra/data/system/hints/
node015
16G	/cassandra/data/system/hints/

I try to pause and resume hinted which made CPU return to normal load, but they are still there and CPU seems to get high again later with same consequences (lot of garbage collections) which seems to be again related to hints.

any idea ?
-- 
Cyril SCETBON

On 31 Oct 2013, at 23:35, Cyril Scetbon <cy...@free.fr> wrote:

> No we're using vnodes
> -- 
> Cyril SCETBON
> 
> On 30 Oct 2013, at 20:25, Robert Coli <rc...@eventbrite.com> wrote:
> 
>> On Wed, Oct 30, 2013 at 12:22 PM, Cyril Scetbon <cy...@free.fr> wrote:
>> FYI, we should upgrade to the last 1.2 version (1.2.11+) in January 2014. However, we would like to know if it's a known fixed bug or inform you about this issue if it's not.
>> 
>> Did you bootstrap multiple nodes into the same token range? That is unsupported...
>> 
>> What does "nodetool ring" say?
>> 
>> =Rob
>>  
>