experience with 1.0 branch

[Radim Kolar <hs...@sendmail.cz>]

1.0 branch is less stable then 0.8 for production. We discovered 
following problems:

1. memory leak in scrub (also reported on this list)

2. problem with saving key caches for super column family -  CASSANDRA-3511

3. in 1.0.3 some hints are stuck in system tables. Hints to other nodes 
seems to deliver fine, but node is probably writing also hints to itself 
and they get stuck there. I cleaned hints in entire cluster (stopped it) 
to be sure that no < 1.0 hints are there

4. memory leak. Node boots and work fine, but after some time (few 
hours) it is doing stuff like:

  WARN [ScheduledTasks:1] 2011-11-22 07:48:18,843 GCInspector.java (line 
146) Heap is 0.8598437159194559 full.  You may need to reduce memtable 
and/or cache sizes.  Cassandra will now flush up to the two largest 
memtables to free up memory.  Adjust flush_largest_memtables_at 
threshold in cassandra.yaml if you don't want Cassandra to do this 
automatically
  WARN [ScheduledTasks:1] 2011-11-22 07:48:18,844 StorageService.java 
(line 2511) Flushing CFS to relieve memory pressure
  INFO [ScheduledTasks:1] 2011-11-22 07:48:18,844 ColumnFamilyStore.java 
(line 688) Enqueuing flush 1034292896(3645/57056 serialized/live bytes, 
51 ops)
flushed table is very small. After node restart it is flushing CFs in 
10MB serialized chunks. It is not caches are too large problem because 
row caches are minimal (now are even reduced to 0.6 their sizes). It 
works with larger caches after node restart just fine.

It will OOM because situation is getting worse. I am now monitoring this 
issue more closely, collecting heap dumps.

799686 instances of class org.apache.cassandra.db.DecoratedKey
796969 instances of class org.apache.cassandra.dht.BigIntegerToken
612414 instances of class 
org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
444125 instances of class org.apache.cassandra.db.ExpiringColumn
187205 instances of class 
com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
187205 instances of class 
com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$WeightedValue
186347 instances of class org.apache.cassandra.utils.Pair
111813 instances of class 
org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
111813 instances of class org.apache.cassandra.db.ThreadSafeSortedColumns
110932 instances of class org.apache.cassandra.db.SuperColumn