[jira] [Created] (CASSANDRA-10229) Very bad read performance after a major compaction

["Alan Boudreault (JIRA)" <ji...@apache.org>]

Alan Boudreault created CASSANDRA-10229:
-------------------------------------------

             Summary: Very bad read performance after a major compaction
                 Key: CASSANDRA-10229
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10229
             Project: Cassandra
          Issue Type: Bug
            Reporter: Alan Boudreault
         Attachments: users-caching.yaml

I am trying to understand what I am seeing. My scenario is very basic, it's a simple users table with key cache and row cache disabled. I write 50M then read 5M random elements. The read performance is not that bad BEFORE a major compaction of the data. I see a ~3x performance regression AFTER I run a major compaction. 

Here's the read performance numbers for my scenario: 
{code}
3.0 before a major compaction (Key cache and row cache disabled)
================================================================

Results:
op rate                   : 9149 [read:9149]
partition rate            : 9149 [read:9149]
row rate                  : 9149 [read:9149]
latency mean              : 32.8 [read:32.8]
latency median            : 31.2 [read:31.2]
latency 95th percentile   : 47.2 [read:47.2]
latency 99th percentile   : 55.0 [read:55.0]
latency 99.9th percentile : 66.4 [read:66.4]
latency max               : 305.4 [read:305.4]
Total partitions          : 50000000 [read:50000000]
Total errors              : 0 [read:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 01:31:05
END

-rw-rw-r-- 1 aboudreault aboudreault  4.7G Aug 31 08:51 ma-1024-big-Data.db
-rw-rw-r-- 1 aboudreault aboudreault  4.9G Aug 31 09:08 ma-1077-big-Data.db

3.0 after a major compaction (Key cache and row cache disabled)
===============================================================

Results:
op rate                   : 3275 [read:3275]
partition rate            : 3275 [read:3275]
row rate                  : 3275 [read:3275]
latency mean              : 91.6 [read:91.6]
latency median            : 88.8 [read:88.8]
latency 95th percentile   : 107.2 [read:107.2]
latency 99th percentile   : 116.0 [read:116.0]
latency 99.9th percentile : 125.5 [read:125.5]
latency max               : 249.0 [read:249.0]
Total partitions          : 50000000 [read:50000000]
Total errors              : 0 [read:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 04:14:26
END

-rw-rw-r-- 1 aboudreault aboudreault 9.5G Aug 31 09:40 ma-1079-big-Data.db

2.1 before major compaction (Key cache and row cache disabled)
==============================================================

Results:
op rate                   : 21348 [read:21348]
partition rate            : 21348 [read:21348]
row rate                  : 21348 [read:21348]
latency mean              : 14.1 [read:14.1]
latency median            : 8.0 [read:8.0]
latency 95th percentile   : 38.5 [read:38.5]
latency 99th percentile   : 60.8 [read:60.8]
latency 99.9th percentile : 99.2 [read:99.2]
latency max               : 229.2 [read:229.2]
Total partitions          : 5000000 [read:5000000]
Total errors              : 0 [read:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:03:54
END

2.1 after major compaction (Key cache and row cache disabled)
=============================================================

Results:
op rate                   : 5262 [read:5262]
partition rate            : 5262 [read:5262]
row rate                  : 5262 [read:5262]
latency mean              : 57.0 [read:57.0]
latency median            : 55.5 [read:55.5]
latency 95th percentile   : 69.4 [read:69.4]
latency 99th percentile   : 83.3 [read:83.3]
latency 99.9th percentile : 197.4 [read:197.4]
latency max               : 1169.0 [read:1169.0]
Total partitions          : 5000000 [read:5000000]
Total errors              : 0 [read:0]
total gc count            : 0
total gc mb               : 0
total gc time (s)         : 0
avg gc time(ms)           : NaN
stdev gc time(ms)         : 0
Total operation time      : 00:15:50
END
{code}

I can reproduce that read performance regression on EC2 and locally. To reproduce:

1. Launch a 1 node cluster (2.1, 2.2 or 3.0)
2. Set the compaction thoughput at 0. (need a restart IIRC)
3. Write 50M elements (so we get the same sstable size for the test). The yaml profile is attached in this ticket.
{code}
cassandra-stress user profile=`pwd`/users-caching.yaml ops\(insert=1\) n=50M -rate threads=100
{code}
4. Wait for the auto-compaction to finish. You should get around 2-6 sstables when it's done.
5. Read 5M elements
{code}
cassandra-stress user profile=/path/to/users-caching.yaml ops\(read=1\) n=5M -rate threads=300
{code}
6. Restart C*, then start a major compaction and wait it's finish.
{code}
ccm stop && ccm start
ccm nodetool compact
{code}
7. Read 5M elements
{code}
cassandra-stress user profile=/path/to/users-caching.yaml ops\(read=1\) n=5M -rate threads=300
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)