Write performance tuning

[Ad...@thomsonreuters.com]

Hi All,
I need some help about the Cassandra 3 performance tuning. We are now upgrading from Cassandra 1 to Cassandra 3.7. After we've done all the development work and functional test, we found the writing performance is not good; the write time is about twice of the Cassandra 1. I want to know is this normal or we can do some tuning.

Some Information:


1. It is a six nodes cluster, two data centers, and three nodes for each datacenter. The consistency level we are using is LOCAL_QUORUM

2. Cassandra server Info

[Description: cid:image004.png@01D2306A.AA197F30]

3. Max Heap size is 12GB; new Heap size is 2GB.

4. I got the Cassandra server debug log from Cassandra server in attachment.
5. The result of cfstats of the largest table
Table: vsc_data
                Space used (live): 17695995770
                Space used (total): 17695995770
                Space used by snapshots (total): 0
                Off heap memory used (total): 19336644
                SSTable Compression Ratio: 0.3265535268789067
                Number of keys (estimate): 7894992
                Memtable cell count: 3369173
                Memtable data size: 257421448
                Memtable off heap memory used: 0
                Memtable switch count: 895
                Local read count: 27804356
                Local read latency: 2.066 ms
                Local write count: 141749394
                Local write latency: 0.311 ms
                Pending flushes: 0
                Bloom filter false positives: 10
                Bloom filter false ratio: 0.00000
                Bloom filter space used: 10787088
                Bloom filter off heap memory used: 10786896
                Index summary off heap memory used: 2032620
                Compression metadata off heap memory used: 6517128
                Compacted partition minimum bytes: 30
                Compacted partition maximum bytes: 74975550
                Compacted partition mean bytes: 6749
                Average live cells per slice (last five minutes): 4.383624640522535
                Maximum live cells per slice (last five minutes): 5722
                Average tombstones per slice (last five minutes): 2.5042559003034777
                Maximum tombstones per slice (last five minutes): 29521
                Dropped Mutations: 0


6. The result of cfhistograms of the largest table
Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count
                              (micros)          (micros)           (bytes)
50%             0.00             20.50             42.51               770                50
75%             1.00             51.01             73.46              1109                86
95%            17.00           1629.72           1629.72             29521              1916
98%            20.00           2816.16          25109.16             42510              3311
99%            24.00           4055.27          43388.63             51012              3973
Min             0.00              1.60              8.24                30                 0
Max            35.00        3449259.15        4139110.98          74975550           5839588



Any clue will be appreciated, thank you very much!
-------------------------------------------------------------------------------------------
Adeline Pan
Senior Software Engineer

Thomson Reuters
Phone: 62674654

Adeline.pan@thomsonreuters.com