memory problems still post- CASSANDRA-3492

[Mick Semb Wever <mc...@apache.org>]

I've got a following problem to CASSANDRA-3492, also related to
ridiculously high memory. 

After the fix yesterday for CASSANDRA-3492 I have that node in question
up and running. 

But another node (on the same machine but different cluster), even after
an upgrade to the staging 1.0.3 and a `nodetool scrub`, always soaks all
available memory (up to and plateau at 30G). In fact no cf there use
compression anymore. 

  It has been down for some days now while I was working on that other
node.
  After it has finished startup memory just keeps growing to 30G.
Although i don't see any OOM when Xmx is set lower the node basically
becomes unusable.
  I can see in tpstats 

"HintedHandoff  (active)1    (pending)2"

 and it just seems to stay like that. 

Is there a way to more closely monitor that active hinted handoff? 
Can one hinted handoff be responsible for such heap?

~mck


-- 
"Driving ambition is the last refuge of the failure." Oscar Wilde 

| http://semb.wever.org | http://sesat.no |
| http://tech.finn.no   | Java XSS Filter |

Re: memory problems still post- CASSANDRA-3492

[Radim Kolar <hs...@sendmail.cz>]

Dne 15.11.2011 22:04, Mick Semb Wever napsal(a):
> But another node (on the same machine but different cluster), even after
> an upgrade to the staging 1.0.3 and a `nodetool scrub`, always soaks all
> available memory (up to and plateau at 30G). In fact no cf there use
> compression anymore.
I had similar problem yesterday with running nodetool scrub on 1.0.3 
while i was trying to convert -g- tables to current format.  There is 
memory leak in scrub. I do not use compression either.

>   "HintedHandoff  (active)1    (pending)2" and it just seems to stay like that.
>
> Is there a way to more closely monitor that active hinted handoff?
you can count columns in system table holding hints but i got OOM 
everytime i tried.

> Can one hinted handoff be responsible for such heap?
no. it is scrub because heap increases after each sstable is processed.