When many sstables(thousand) in level 0 ，compaction very slow

[wateray <wa...@163.com>]

After I use sstableloader from a old cluster, the new cluster node's data size double. The new cluster has the same node with old cluster( 3 nodes).


The old cluster has 500G each node,after sstableloader completely, the new cluster data size nearly 1T each node.


The problem is that compaction(LCS) doesn't endless for over 10 days. It seems at level 0 ,it pick 32 sstables compaction into one large ones. And then pick this large sstable and other 31 original sstable compaction into   larger one. 


SStable Info:
SSTable count: 6965
SSTables in each level: [6931/4, 32/10, 0, 0, 0, 0, 0, 0, 0]


Compaction throgutput: 200MB/s


The compaction logs:
INFO  [CompactionExecutor:615] 2016-01-03 20:25:14,739 CompactionTask.java:274 - Compacted 32 sstables to [/disk/data3/cassandra/test/t_calllog-43f6c250aa0d11e59ac60b2534
4dc914/test-t_calllog-ka-11299,].  5,370,317,257 bytes to 5,370,305,557 (~99% of original) in 10,766,638ms = 0.475684MB/s.  219,175 total partitions merged to 219,175.  P
artition merge counts were {1:219175, }


SStable size:
-rw-r--r-- 1 storage storage 47033344647 Jan  2 11:11 test-t_calllog-ka-11233-Data.db
-rw-r--r-- 1 storage storage  5370290840 Jan  3 23:09 test-t_calllog-ka-11303-Data.db
-rw-r--r-- 2 storage storage  1461236940 Jan  4 15:18 test-t_calllog-tmp-ka-11329-Data.db
-rw-r--r-- 2 storage storage  1461236940 Jan  4 15:18 test-t_calllog-tmplink-ka-11329-Data.db
-rw-r--r-- 1 storage storage  1204113103 Jan  2 11:11 test-t_calllog-ka-11233-Index.db
-rw-r--r-- 1 storage storage   168593618 Dec 25 19:59 test-t_calllog-ka-10505-Data.db
-rw-r--r-- 1 storage storage   168592670 Dec 24 22:29 test-t_calllog-ka-2564-Data.db
-rw-r--r-- 1 storage storage   168591491 Dec 24 20:46 test-t_calllog-ka-1528-Data.db


Does it is not a good usecase using sstableloader to load large cluster? As which will create many uncompacted sstables.