cassandra massive write perfomance problem

[lebron james <le...@gmail.com>]

Please help me solve one problem I have server with 4 GB RAM and 2x 4 cores
CPU When i start do massive writes in cassandra all works fine. but after
couple hours with 10K inserts per second database grows up to 25+ GB
performance go down to 500 insert per seconds I find out this because
compacting operations is very slow and i dont understand why, i set 8
concurrent compacting threads but cassandra dont use 8 threads only 2 cores
are loaded.

Re: cassandra massive write perfomance problem

[aaron morton <aa...@thelastpickle.com>]

background http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts

Without more info my initial guess is some GC pressure and/or IO pressure from compaction. Check the logs for messages from the GCInspector or connect JConsole to the instance and take a look at the heap. Here is some info on looking at the IO stats http://spyced.blogspot.com/2010/01/linux-performance-basics.html
 
With regard to the 25+GB on disk, that all depends on how much data you are writing. Be aware that compacted files are not immediately deleted http://wiki.apache.org/cassandra/FAQ#cleaning_compacted_tables

You may also want to track things by looking at nodetool tpstats and cfstats (for latency).

Cheers
  
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 21 Jul 2011, at 21:49, lebron james wrote:

> Please help me solve one problem I have server with 4 GB RAM and 2x 4 cores CPU When i start do massive writes in cassandra all works fine. but after couple hours with 10K inserts per second database grows up to 25+ GB performance go down to 500 insert per seconds I find out this because compacting operations is very slow and i dont understand why, i set 8 concurrent compacting threads but cassandra dont use 8 threads only 2 cores are loaded.