You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Philippe <wa...@gmail.com> on 2011/06/05 00:34:25 UTC
Troubleshooting IO performance ?
Hello,
I am evaluating using cassandra and I'm running into some strange IO
behavior that I can't explain, I'd like some help/ideas to troubleshoot it.
I am running a 1 node cluster with a keyspace consisting of two columns
families, one of which has dozens of supercolumns itself containing dozens
of columns.
All in all, this is a couple gigabytes of data, 12GB on the hard drive.
The hardware is pretty good : 16GB memory + RAID-0 SSD drives with LVM and
an i5 processor (4 cores).
Keyspace: xxxxxxxxxxxxxxxxxxx
Read Count: 460754852
Read Latency: 1.108205793092766 ms.
Write Count: 30620665
Write Latency: 0.01411020877567486 ms.
Pending Tasks: 0
Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
SSTable count: 5
Space used (live): 548700725
Space used (total): 548700725
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 11
Read Count: 2891192
Read Latency: NaN ms.
Write Count: 3157547
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 367396
Key cache size: 367396
Key cache hit rate: NaN
Row cache capacity: 112683
Row cache size: 112683
Row cache hit rate: NaN
Compacted row minimum size: 125
Compacted row maximum size: 924
Compacted row mean size: 172
Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
SSTable count: 7
Space used (live): 8707538781
Space used (total): 8707538781
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 30
Read Count: 457863660
Read Latency: 2.381 ms.
Write Count: 27463118
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 4518387
Key cache size: 4518387
Key cache hit rate: 0.9247881700850826
Row cache capacity: 1349682
Row cache size: 1349682
Row cache hit rate: 0.39400533823415573
Compacted row minimum size: 125
Compacted row maximum size: 6866
Compacted row mean size: 165
My app makes a bunch of requests using a MultigetSuperSliceQuery for a set
of keys, typically a couple dozen at most. It also selects a subset of the
supercolumns. I am running 8 requests in parallel at most.
Two days, I ran a 1.5 hour process that basically read every key. The server
had no IOwaits and everything was humming along. However, right at the end
of the process, there was a huge spike in IOs. I didn't think much of it.
Today, after two days of inactivity, any query I run raises the IOs to 80%
utilization of the SSD drives even though I'm running the same query over
and over (no cache??)
Any ideas on how to troubleshoot this, or better, how to solve this ?
thanks
Philippe
Re: Troubleshooting IO performance ?
Posted by Philippe <wa...@gmail.com>.
very even
will answer aaron's email...
will upgrade to 0.8 too !
Le 7 juin 2011 13:09, "Terje Marthinussen" <tm...@gmail.com> a
écrit :
> If you run iostat without output every few second, is the I/O stable or do
> you see very uneven I/O?
>
> Regards,
> Terje
>
> On Tue, Jun 7, 2011 at 11:12 AM, aaron morton <aaron@thelastpickle.com
>wrote:
>
>> There is a big IO queue and reads are spending a lot of time in the
queue.
>>
>> Some more questions:
>> - what version are you on ?
>> - what is the concurrent_reads config setting ?
>> - what is nodetool tpstats showing during the slow down ?
>> - exactly how much data are you asking for ? how many rows and what sort
of
>> slice
>> - has their been a lot of deletes or TTL columns used ?
>>
>> Hope that helps.
>> Aaron
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 7 Jun 2011, at 10:09, Philippe wrote:
>>
>> Ok, here it goes again... No swapping at all...
>>
>> procs -----------memory---------- ---swap-- -----io---- -system--
>> ----cpu----
>> r b swpd free buff cache si so bi bo in cs us sy id
>> wa
>> 1 63 32044 88736 37996 7116524 0 0 227156 0 18314 5607 30 5
>> 11 53
>> 1 63 32044 90844 37996 7103904 0 0 233524 202 17418 4977 29 4
>> 9 58
>> 0 42 32044 91304 37996 7123884 0 0 249736 0 16197 5433 19 6
>> 3 72
>> 3 25 32044 89864 37996 7135980 0 0 223140 16 18135 7567 32 5
>> 11 52
>> 1 1 32044 88664 37996 7150728 0 0 229416 128 19168 7554 36 4
>> 10 51
>> 4 0 32044 89464 37996 7149428 0 0 213852 18 21041 8819 45 5
>> 12 38
>> 4 0 32044 90372 37996 7149432 0 0 233086 142 19909 7041 43 5
>> 10 41
>> 7 1 32044 89752 37996 7149520 0 0 206906 0 19350 6875 50 4
>> 11 35
>>
>> Lots and lots of disk activity
>> iostat -dmx 2
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
>> avgqu-sz await r_await w_await svctm %util
>> sda 52.50 0.00 7813.00 0.00 108.01 0.00 28.31
>> 117.15 14.89 14.89 0.00 0.11 83.00
>> sdb 56.00 0.00 7755.50 0.00 108.51 0.00 28.66
>> 118.67 15.18 15.18 0.00 0.11 82.80
>> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> md5 0.00 0.00 15796.50 0.00 219.21 0.00 28.42
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> dm-0 0.00 0.00 15796.50 0.00 219.21 0.00 28.42
>> 273.42 17.03 17.03 0.00 0.05 83.40
>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>>
>> More info :
>> - all the data directory containing the data I'm querying into is 9.7GB
>> and this is a server with 16GB
>> - I'm hitting the server with 6 concurrent multigetsuperslicequeries on
>> multiple keys, some of them can bring back quite a number of data
>> - I'm reading all the keys for one column, pretty much sequentially
>>
>> This is a query in a rollup table that was originally in MySQL and it
>> doesn't look like the performance to query by key is better. So I'm
betting
>> I'm doing something wrong here... but what ?
>>
>> Any ideas ?
>> Thanks
>>
>> 2011/6/6 Philippe <wa...@gmail.com>
>>
>>> hum..no, it wasn't swapping. cassandra was the only thing running on
that
>>> server
>>> and i was querying the same keys over and over
>>>
>>> i restarted Cassandra and doing the same thing, io is now down to zero
>>> while cpu is up which dosen't surprise me as much.
>>>
>>> I'll report if it happens again.
>>> Le 5 juin 2011 16:55, "Jonathan Ellis" <jb...@gmail.com> a écrit :
>>>
>>> > You may be swapping.
>>> >
>>> > http://spyced.blogspot.com/2010/01/linux-performance-basics.html
>>> > explains how to check this as well as how to see what threads are busy
>>> > in the Java process.
>>> >
>>> > On Sat, Jun 4, 2011 at 5:34 PM, Philippe <wa...@gmail.com> wrote:
>>> >> Hello,
>>> >> I am evaluating using cassandra and I'm running into some strange IO
>>> >> behavior that I can't explain, I'd like some help/ideas to
troubleshoot
>>> it.
>>> >> I am running a 1 node cluster with a keyspace consisting of two
columns
>>> >> families, one of which has dozens of supercolumns itself containing
>>> dozens
>>> >> of columns.
>>> >> All in all, this is a couple gigabytes of data, 12GB on the hard
drive.
>>> >> The hardware is pretty good : 16GB memory + RAID-0 SSD drives with
LVM
>>> and
>>> >> an i5 processor (4 cores).
>>> >> Keyspace: xxxxxxxxxxxxxxxxxxx
>>> >> Read Count: 460754852
>>> >> Read Latency: 1.108205793092766 ms.
>>> >> Write Count: 30620665
>>> >> Write Latency: 0.01411020877567486 ms.
>>> >> Pending Tasks: 0
>>> >> Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
>>> >> SSTable count: 5
>>> >> Space used (live): 548700725
>>> >> Space used (total): 548700725
>>> >> Memtable Columns Count: 0
>>> >> Memtable Data Size: 0
>>> >> Memtable Switch Count: 11
>>> >> Read Count: 2891192
>>> >> Read Latency: NaN ms.
>>> >> Write Count: 3157547
>>> >> Write Latency: NaN ms.
>>> >> Pending Tasks: 0
>>> >> Key cache capacity: 367396
>>> >> Key cache size: 367396
>>> >> Key cache hit rate: NaN
>>> >> Row cache capacity: 112683
>>> >> Row cache size: 112683
>>> >> Row cache hit rate: NaN
>>> >> Compacted row minimum size: 125
>>> >> Compacted row maximum size: 924
>>> >> Compacted row mean size: 172
>>> >> Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
>>> >> SSTable count: 7
>>> >> Space used (live): 8707538781
>>> >> Space used (total): 8707538781
>>> >> Memtable Columns Count: 0
>>> >> Memtable Data Size: 0
>>> >> Memtable Switch Count: 30
>>> >> Read Count: 457863660
>>> >> Read Latency: 2.381 ms.
>>> >> Write Count: 27463118
>>> >> Write Latency: NaN ms.
>>> >> Pending Tasks: 0
>>> >> Key cache capacity: 4518387
>>> >> Key cache size: 4518387
>>> >> Key cache hit rate: 0.9247881700850826
>>> >> Row cache capacity: 1349682
>>> >> Row cache size: 1349682
>>> >> Row cache hit rate: 0.39400533823415573
>>> >> Compacted row minimum size: 125
>>> >> Compacted row maximum size: 6866
>>> >> Compacted row mean size: 165
>>> >> My app makes a bunch of requests using a MultigetSuperSliceQuery for
a
>>> set
>>> >> of keys, typically a couple dozen at most. It also selects a subset
of
>>> the
>>> >> supercolumns. I am running 8 requests in parallel at most.
>>> >>
>>> >> Two days, I ran a 1.5 hour process that basically read every key. The
>>> server
>>> >> had no IOwaits and everything was humming along. However, right at
the
>>> end
>>> >> of the process, there was a huge spike in IOs. I didn't think much of
>>> it.
>>> >> Today, after two days of inactivity, any query I run raises the IOs
to
>>> 80%
>>> >> utilization of the SSD drives even though I'm running the same query
>>> over
>>> >> and over (no cache??)
>>> >> Any ideas on how to troubleshoot this, or better, how to solve this ?
>>> >> thanks
>>> >> Philippe
>>> >
>>> >
>>> >
>>> > --
>>> > Jonathan Ellis
>>> > Project Chair, Apache Cassandra
>>> > co-founder of DataStax, the source for professional Cassandra support
>>> > http://www.datastax.com
>>>
>>
>>
>>
Re: Troubleshooting IO performance ?
Posted by Terje Marthinussen <tm...@gmail.com>.
If you run iostat without output every few second, is the I/O stable or do
you see very uneven I/O?
Regards,
Terje
On Tue, Jun 7, 2011 at 11:12 AM, aaron morton <aa...@thelastpickle.com>wrote:
> There is a big IO queue and reads are spending a lot of time in the queue.
>
> Some more questions:
> - what version are you on ?
> - what is the concurrent_reads config setting ?
> - what is nodetool tpstats showing during the slow down ?
> - exactly how much data are you asking for ? how many rows and what sort of
> slice
> - has their been a lot of deletes or TTL columns used ?
>
> Hope that helps.
> Aaron
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 7 Jun 2011, at 10:09, Philippe wrote:
>
> Ok, here it goes again... No swapping at all...
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> 1 63 32044 88736 37996 7116524 0 0 227156 0 18314 5607 30 5
> 11 53
> 1 63 32044 90844 37996 7103904 0 0 233524 202 17418 4977 29 4
> 9 58
> 0 42 32044 91304 37996 7123884 0 0 249736 0 16197 5433 19 6
> 3 72
> 3 25 32044 89864 37996 7135980 0 0 223140 16 18135 7567 32 5
> 11 52
> 1 1 32044 88664 37996 7150728 0 0 229416 128 19168 7554 36 4
> 10 51
> 4 0 32044 89464 37996 7149428 0 0 213852 18 21041 8819 45 5
> 12 38
> 4 0 32044 90372 37996 7149432 0 0 233086 142 19909 7041 43 5
> 10 41
> 7 1 32044 89752 37996 7149520 0 0 206906 0 19350 6875 50 4
> 11 35
>
> Lots and lots of disk activity
> iostat -dmx 2
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sda 52.50 0.00 7813.00 0.00 108.01 0.00 28.31
> 117.15 14.89 14.89 0.00 0.11 83.00
> sdb 56.00 0.00 7755.50 0.00 108.51 0.00 28.66
> 118.67 15.18 15.18 0.00 0.11 82.80
> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> md5 0.00 0.00 15796.50 0.00 219.21 0.00 28.42
> 0.00 0.00 0.00 0.00 0.00 0.00
> dm-0 0.00 0.00 15796.50 0.00 219.21 0.00 28.42
> 273.42 17.03 17.03 0.00 0.05 83.40
> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> More info :
> - all the data directory containing the data I'm querying into is 9.7GB
> and this is a server with 16GB
> - I'm hitting the server with 6 concurrent multigetsuperslicequeries on
> multiple keys, some of them can bring back quite a number of data
> - I'm reading all the keys for one column, pretty much sequentially
>
> This is a query in a rollup table that was originally in MySQL and it
> doesn't look like the performance to query by key is better. So I'm betting
> I'm doing something wrong here... but what ?
>
> Any ideas ?
> Thanks
>
> 2011/6/6 Philippe <wa...@gmail.com>
>
>> hum..no, it wasn't swapping. cassandra was the only thing running on that
>> server
>> and i was querying the same keys over and over
>>
>> i restarted Cassandra and doing the same thing, io is now down to zero
>> while cpu is up which dosen't surprise me as much.
>>
>> I'll report if it happens again.
>> Le 5 juin 2011 16:55, "Jonathan Ellis" <jb...@gmail.com> a écrit :
>>
>> > You may be swapping.
>> >
>> > http://spyced.blogspot.com/2010/01/linux-performance-basics.html
>> > explains how to check this as well as how to see what threads are busy
>> > in the Java process.
>> >
>> > On Sat, Jun 4, 2011 at 5:34 PM, Philippe <wa...@gmail.com> wrote:
>> >> Hello,
>> >> I am evaluating using cassandra and I'm running into some strange IO
>> >> behavior that I can't explain, I'd like some help/ideas to troubleshoot
>> it.
>> >> I am running a 1 node cluster with a keyspace consisting of two columns
>> >> families, one of which has dozens of supercolumns itself containing
>> dozens
>> >> of columns.
>> >> All in all, this is a couple gigabytes of data, 12GB on the hard drive.
>> >> The hardware is pretty good : 16GB memory + RAID-0 SSD drives with LVM
>> and
>> >> an i5 processor (4 cores).
>> >> Keyspace: xxxxxxxxxxxxxxxxxxx
>> >> Read Count: 460754852
>> >> Read Latency: 1.108205793092766 ms.
>> >> Write Count: 30620665
>> >> Write Latency: 0.01411020877567486 ms.
>> >> Pending Tasks: 0
>> >> Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
>> >> SSTable count: 5
>> >> Space used (live): 548700725
>> >> Space used (total): 548700725
>> >> Memtable Columns Count: 0
>> >> Memtable Data Size: 0
>> >> Memtable Switch Count: 11
>> >> Read Count: 2891192
>> >> Read Latency: NaN ms.
>> >> Write Count: 3157547
>> >> Write Latency: NaN ms.
>> >> Pending Tasks: 0
>> >> Key cache capacity: 367396
>> >> Key cache size: 367396
>> >> Key cache hit rate: NaN
>> >> Row cache capacity: 112683
>> >> Row cache size: 112683
>> >> Row cache hit rate: NaN
>> >> Compacted row minimum size: 125
>> >> Compacted row maximum size: 924
>> >> Compacted row mean size: 172
>> >> Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
>> >> SSTable count: 7
>> >> Space used (live): 8707538781
>> >> Space used (total): 8707538781
>> >> Memtable Columns Count: 0
>> >> Memtable Data Size: 0
>> >> Memtable Switch Count: 30
>> >> Read Count: 457863660
>> >> Read Latency: 2.381 ms.
>> >> Write Count: 27463118
>> >> Write Latency: NaN ms.
>> >> Pending Tasks: 0
>> >> Key cache capacity: 4518387
>> >> Key cache size: 4518387
>> >> Key cache hit rate: 0.9247881700850826
>> >> Row cache capacity: 1349682
>> >> Row cache size: 1349682
>> >> Row cache hit rate: 0.39400533823415573
>> >> Compacted row minimum size: 125
>> >> Compacted row maximum size: 6866
>> >> Compacted row mean size: 165
>> >> My app makes a bunch of requests using a MultigetSuperSliceQuery for a
>> set
>> >> of keys, typically a couple dozen at most. It also selects a subset of
>> the
>> >> supercolumns. I am running 8 requests in parallel at most.
>> >>
>> >> Two days, I ran a 1.5 hour process that basically read every key. The
>> server
>> >> had no IOwaits and everything was humming along. However, right at the
>> end
>> >> of the process, there was a huge spike in IOs. I didn't think much of
>> it.
>> >> Today, after two days of inactivity, any query I run raises the IOs to
>> 80%
>> >> utilization of the SSD drives even though I'm running the same query
>> over
>> >> and over (no cache??)
>> >> Any ideas on how to troubleshoot this, or better, how to solve this ?
>> >> thanks
>> >> Philippe
>> >
>> >
>> >
>> > --
>> > Jonathan Ellis
>> > Project Chair, Apache Cassandra
>> > co-founder of DataStax, the source for professional Cassandra support
>> > http://www.datastax.com
>>
>
>
>
Re: Troubleshooting IO performance ?
Posted by aaron morton <aa...@thelastpickle.com>.
To reduce the number of SSTables increase the memtable_threshold for the CF.
The IO numbers may be because of compaction kicking in. The CompactionManager provides information via JMX on it's progress, or you can check the logs. You could increase the min_compaction_threshold for the CF or disable compaction if you want to during the bulk load. In your case it's probably a bad idea as in your case every write requires a read and compaction will help the read performance.
These numbers show that about 37GB of data was written to disk, but compaction has shrunk this down to about 7GB. If you are not doing deletes that you are doing a log of overwrites
> Space used (live): 8283980539
> Space used (total): 39972827945
I'm getting a bit confused about what the problem is. Is it increasing latency for read requests or the unexplained IO ?
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 12 Jun 2011, at 08:55, Philippe wrote:
> More info below
>
> I just loaded 4.8GB of similar data in another keyspace and ran the same process as in my previous tests but on that data.
> I started with three threads hitting cassandra. No I/O, hardly any CPU (15% on a 4 core server)
> After an hour or so, I raised it to 6 threads in parallel. Then to 9 threads in parallel.
>
> I never got any IO, in fact iostat showed me there wasn't any disk reads. I hardly saw the CPU elevate except at the end.
>
> The only difference between the two datasets is that the size of the other one is 8.4Gb. So the second one doesn't fit completely in memory.So my woes are related to how well cassandra is fetching the data in the SSTAbles right ?
>
>
> So what are my options ? My rows are very small at the moment (like well < 4 kBytes). Should I reduce the read buffer ? Should I reduce the number of SST tables ?
> I'm reloading the data from scratch using the incremental update I will be using in production. I'm hitting the cluster with 6 threads. Yes I should have decreased it but I was too optimistic and I dont want to stop it now. The data I'm loading is used for computing running averages (sum & total) and so every update requires a read.
>
> As soon as the data no longer fits in memory, I'm seeing huge amounts of io (almost 380MB/s reading) that I'd like to understand.
> My hypothesis is that because I am updating the same key so many times (dozens if not hundreds of times in some cases), the row is split across the SSTables and every read needs to go through all the SST tables.
> Unfortunately, at some point, cassandra compacted the keys from 5 tables to 3 and the throughput did not increase after that so I'm not even sure this makes sense.
> 1) Is there another explanation ? Can I do something about this ?
> 2) why is the readlatency displayed twice in "cfstats" and why does it differ ?
>
> Thanks
>
> vmstat
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 0 7 47724 94604 22556 7373252 0 0 391646 2152 10498 10297 6 6 26 62
> 0 6 47724 93736 22556 7372400 0 0 396774 0 10881 11177 5 6 29 60
> 2 5 47724 92496 22556 7374824 0 0 372406 15 11212 11149 8 7 25 59
> 0 5 47724 89520 22568 7378484 0 0 399730 526 10964 11975 6 7 24 63
> 0 7 47724 87908 22568 7379444 0 0 396216 0 10405 10880 5 7 22 66
>
> iostat -dmx 2
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> sdb 168.50 0.00 3152.00 0.50 185.49 0.25 120.66 54.86 17.39 17.39 4.00 0.31 96.80
> sda 178.50 0.50 3090.50 0.50 184.47 0.19 122.35 61.16 19.71 19.71 4.00 0.31 97.20
> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> md5 0.00 0.00 6612.50 1.50 372.82 0.44 115.58 0.00 0.00 0.00 0.00 0.00 0.00
> dm-0 0.00 0.00 6612.50 0.00 372.82 0.00 115.47 123.15 18.58 18.58 0.00 0.15 97.20
>
> cfstats
> Read Count: 88215069
> Read Latency: 1.821254258759351 ms.
> Write Count: 88215059
> Write Latency: 0.013311765885686253 ms.
> Pending Tasks: 0
> Column Family: PUBLIC_MONTHLY
> SSTable count: 3
> Space used (live): 8283980539
> Space used (total): 39972827945
> Memtable Columns Count: 449201
> Memtable Data Size: 21788245
> Memtable Switch Count: 72
> Read Count: 88215069
> Read Latency: 7.433 ms.
> Write Count: 88215069
> Write Latency: 0.016 ms.
> Pending Tasks: 0
> Key cache capacity: 2000000
> Key cache size: 2000000
> Key cache hit rate: 2.93152540950838E-4
> Row cache capacity: 50000
> Row cache size: 50000
> Row cache hit rate: 0.20131872518304333
> Compacted row minimum size: 125
> Compacted row maximum size: 17084
> Compacted row mean size: 176
>
> tpstats
> Pool Name Active Pending Completed
> ReadStage 6 7 88200770
> RequestResponseStage 0 0 0
> MutationStage 0 0 88200760
> ReadRepairStage 0 0 0
> GossipStage 0 0 0
> AntiEntropyStage 0 0 0
> MigrationStage 0 0 0
> MemtablePostFlusher 0 0 74
> StreamStage 0 0 0
> FlushWriter 0 0 74
> FILEUTILS-DELETE-POOL 0 0 358
> MiscStage 0 0 0
> FlushSorter 0 0 0
> InternalResponseStage 0 0 0
> HintedHandoff 0 0 0
>
>
> cfhistograms
> Offset SSTables Write Latency Read Latency Row Size Column Count
> 1 27628 0 0 0 29101531
> 2 12405 3 9 0 6677734
> 3 0 30 130 0 2477249
> 4 0 191 451 0 391129
> 5 0 414 990 0 51983
> 6 0 644 1587 0 4041
> 7 0 1131 1957 0 710
> 8 0 1328 2291 0 220
> 10 0 1663 5515 0 131
> 12 0 1349 6133 0 24
> 14 0 1557 6461 0 5
> 17 0 880 8911 0 0
> 20 0 594 6685 0 0
> 24 0 1099 5316 0 0
> 29 0 2044 2283 0 0
> 35 0 2901 474 0 0
> 42 0 2737 66 0 0
> 50 0 876 10 0 0
> 60 0 192 4 0 0
> 72 0 361 0 0 0
> 86 0 771 0 0 0
> 103 0 1486 0 0 0
> 124 0 2285 0 0 11
> 149 0 2618 0 24141265 0
> 179 0 2438 0 1791177 0
> 215 0 1909 0 5326712 0
> 258 0 1008 0 1841927 0
> 310 0 420 0 2298870 0
> 372 0 105 0 1234895 0
> 446 0 3 0 1036341 0
> 535 0 3 0 553275 0
> 642 0 2 0 273339 0
> 770 0 1 0 144826 0
> 924 0 1 0 48619 0
> 1109 0 1 0 10275 0
> 1331 0 1 0 2134 0
> 1597 5 0 637 0
> 1916 5 0 291 0
> 2299 10 0 115 0
> 2759 15 0 42 0
> 3311 51 0 16 0
> 3973 86 0 1 0
> 4768 100 0 0 0
> 5722 165 0 0 0
> 6866 255 0 0 0
> 8239 412 0 0 0
> 9887 556 0 0 0
> 11864 945 0 0 0
> 14237 1358 0 6 0
>
Re: Troubleshooting IO performance ?
Posted by Philippe <wa...@gmail.com>.
More info below
> I just loaded 4.8GB of similar data in another keyspace and ran the same
> process as in my previous tests but on that data.
> I started with three threads hitting cassandra. No I/O, hardly any CPU (15%
> on a 4 core server)
> After an hour or so, I raised it to 6 threads in parallel. Then to 9
> threads in parallel.
>
> I never got any IO, in fact iostat showed me there wasn't any disk reads. I
> hardly saw the CPU elevate except at the end.
>
> The only difference between the two datasets is that the size of the other
> one is 8.4Gb. So the second one doesn't fit completely in memory.So my woes
> are related to how well cassandra is fetching the data in the SSTAbles right
> ?
>
>
> So what are my options ? My rows are very small at the moment (like well <
> 4 kBytes). Should I reduce the read buffer ? Should I reduce the number of
> SST tables ?
>
I'm reloading the data from scratch using the incremental update I will be
using in production. I'm hitting the cluster with 6 threads. Yes I should
have decreased it but I was too optimistic and I dont want to stop it
now. The data I'm loading is used for computing running averages (sum &
total) and so every update requires a read.
As soon as the data no longer fits in memory, I'm seeing huge amounts of io
(almost 380MB/s reading) that I'd like to understand.
My hypothesis is that because I am updating the same key so many times
(dozens if not hundreds of times in some cases), the row is split across the
SSTables and every read needs to go through all the SST tables.
Unfortunately, at some point, cassandra compacted the keys from 5 tables to
3 and the throughput did not increase after that so I'm not even sure this
makes sense.
1) Is there another explanation ? Can I do something about this ?
2) why is the readlatency displayed twice in "cfstats" and why does it
differ ?
Thanks
vmstat
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
0 7 47724 94604 22556 7373252 0 0 391646 2152 10498 10297 6 6
26 62
0 6 47724 93736 22556 7372400 0 0 396774 0 10881 11177 5 6
29 60
2 5 47724 92496 22556 7374824 0 0 372406 15 11212 11149 8 7
25 59
0 5 47724 89520 22568 7378484 0 0 399730 526 10964 11975 6 7
24 63
0 7 47724 87908 22568 7379444 0 0 396216 0 10405 10880 5 7
22 66
iostat -dmx 2
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sdb 168.50 0.00 3152.00 0.50 185.49 0.25 120.66
54.86 17.39 17.39 4.00 0.31 96.80
sda 178.50 0.50 3090.50 0.50 184.47 0.19 122.35
61.16 19.71 19.71 4.00 0.31 97.20
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
md5 0.00 0.00 6612.50 1.50 372.82 0.44 115.58
0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 6612.50 0.00 372.82 0.00 115.47
123.15 18.58 18.58 0.00 0.15 97.20
cfstats
Read Count: 88215069
Read Latency: 1.821254258759351 ms.
Write Count: 88215059
Write Latency: 0.013311765885686253 ms.
Pending Tasks: 0
Column Family: PUBLIC_MONTHLY
SSTable count: 3
Space used (live): 8283980539
Space used (total): 39972827945
Memtable Columns Count: 449201
Memtable Data Size: 21788245
Memtable Switch Count: 72
Read Count: 88215069
Read Latency: 7.433 ms.
Write Count: 88215069
Write Latency: 0.016 ms.
Pending Tasks: 0
Key cache capacity: 2000000
Key cache size: 2000000
Key cache hit rate: 2.93152540950838E-4
Row cache capacity: 50000
Row cache size: 50000
Row cache hit rate: 0.20131872518304333
Compacted row minimum size: 125
Compacted row maximum size: 17084
Compacted row mean size: 176
tpstats
Pool Name Active Pending Completed
ReadStage 6 7 88200770
RequestResponseStage 0 0 0
MutationStage 0 0 88200760
ReadRepairStage 0 0 0
GossipStage 0 0 0
AntiEntropyStage 0 0 0
MigrationStage 0 0 0
MemtablePostFlusher 0 0 74
StreamStage 0 0 0
FlushWriter 0 0 74
FILEUTILS-DELETE-POOL 0 0 358
MiscStage 0 0 0
FlushSorter 0 0 0
InternalResponseStage 0 0 0
HintedHandoff 0 0 0
cfhistograms
Offset SSTables Write Latency Read Latency Row Size
Column Count
1 27628 0 0 0
29101531
2 12405 3 9 0
6677734
3 0 30 130 0
2477249
4 0 191 451 0
391129
5 0 414 990 0
51983
6 0 644 1587 0
4041
7 0 1131 1957 0
710
8 0 1328 2291 0
220
10 0 1663 5515 0
131
12 0 1349 6133 0
24
14 0 1557 6461 0
5
17 0 880 8911 0
0
20 0 594 6685 0
0
24 0 1099 5316 0
0
29 0 2044 2283 0
0
35 0 2901 474 0
0
42 0 2737 66 0
0
50 0 876 10 0
0
60 0 192 4 0
0
72 0 361 0 0
0
86 0 771 0 0
0
103 0 1486 0 0
0
124 0 2285 0 0
11
149 0 2618 0 24141265
0
179 0 2438 0 1791177
0
215 0 1909 0 5326712
0
258 0 1008 0 1841927
0
310 0 420 0 2298870
0
372 0 105 0 1234895
0
446 0 3 0 1036341
0
535 0 3 0 553275
0
642 0 2 0 273339
0
770 0 1 0 144826
0
924 0 1 0 48619
0
1109 0 1 0 10275
0
1331 0 1 0 2134
0
1597 5 0 637
0
1916 5 0 291
0
2299 10 0 115
0
2759 15 0 42
0
3311 51 0 16
0
3973 86 0 1
0
4768 100 0 0
0
5722 165 0 0
0
6866 255 0 0
0
8239 412 0 0
0
9887 556 0 0
0
11864 945 0 0
0
14237 1358 0 6
0
Re: Troubleshooting IO performance ?
Posted by Philippe <wa...@gmail.com>.
>
>
> I'd check you are reading the data you expect then wind back the number of
> requests and rows / columns requested. Get to a stable baseline and then add
> pressure to see when / how things go wrong.
>
I just loaded 4.8GB of similar data in another keyspace and ran the same
process as in my previous tests but on that data.
I started with three threads hitting cassandra. No I/O, hardly any CPU (15%
on a 4 core server)
After an hour or so, I raised it to 6 threads in parallel. Then to 9 threads
in parallel.
I never got any IO, in fact iostat showed me there wasn't any disk reads. I
hardly saw the CPU elevate except at the end.
The only difference between the two datasets is that the size of the other
one is 8.4Gb. So the second one doesn't fit completely in memory.So my woes
are related to how well cassandra is fetching the data in the SSTAbles right
?
So what are my options ? My rows are very small at the moment (like well < 4
kBytes). Should I reduce the read buffer ? Should I reduce the number of SST
tables ?
Thanks
Philippe
>
> Hope that helps.
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 8 Jun 2011, at 08:00, Philippe wrote:
>
> Aaron,
>
> - what version are you on ?
> 0.7.6-2
>
> - what is the concurrent_reads config setting ?
>>
> concurrent_reads: 64
> concurrent_writes: 64
>
> Givent that I've got 4 cores and SSD drives, I doubled the concurrent
> writes recommended.
> Given that I've RAID-0ed the SSD drive, I figured I could at least double
> for SSD and double for RAID-0 the recommended version.
> Wrong assumptions ?
>
> BTW, cassandra is running on an XFS filesystem over LVM over RAID-0
>
> - what is nodetool tpstats showing during the slow down ?
>>
> The only value that changes is the ReadStage line. Here's values from a
> sample every second
> Pool Name Active Pending Completed
> ReadStage 64 99303 463085056
> ReadStage 64 88430 463095929
> ReadStage 64 91937 463107782
>
> So basically, I'm flooding the system right ? For example 99303 means there
> are 99303 key reads pending, possibly from just a couple MultiSlice gets ?
>
>
>> - exactly how much data are you asking for ? how many rows and what sort
>> of slice
>>
> According to some munin monitoring, the server is cranking out to the
> client, over the network, 10Mbits/s = 1.25 Mbytes/s
>
> The same munin monitoring shows me 200Mbytes/s read from the disks. This is
> what is worrying me...
>
> - has their been a lot of deletes or TTL columns used ?
>>
> No deletes, only update, don't know if that counts as deletes though...
>
> This is going to be a read-heavy, update-heavy cluster.
> No TTL columns, no counter columns
>
> One question : when nodetool cfstats says the average read latency is 5ms,
> is that counted once the query is being executed or does that include the
> time spent "pending" ?
>
> Thanks
> Philippe
>
>>
>> Hope that helps.
>> Aaron
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 7 Jun 2011, at 10:09, Philippe wrote:
>>
>> Ok, here it goes again... No swapping at all...
>>
>> procs -----------memory---------- ---swap-- -----io---- -system--
>> ----cpu----
>> r b swpd free buff cache si so bi bo in cs us sy id
>> wa
>> 1 63 32044 88736 37996 7116524 0 0 227156 0 18314 5607 30 5
>> 11 53
>> 1 63 32044 90844 37996 7103904 0 0 233524 202 17418 4977 29 4
>> 9 58
>> 0 42 32044 91304 37996 7123884 0 0 249736 0 16197 5433 19 6
>> 3 72
>> 3 25 32044 89864 37996 7135980 0 0 223140 16 18135 7567 32 5
>> 11 52
>> 1 1 32044 88664 37996 7150728 0 0 229416 128 19168 7554 36 4
>> 10 51
>> 4 0 32044 89464 37996 7149428 0 0 213852 18 21041 8819 45 5
>> 12 38
>> 4 0 32044 90372 37996 7149432 0 0 233086 142 19909 7041 43 5
>> 10 41
>> 7 1 32044 89752 37996 7149520 0 0 206906 0 19350 6875 50 4
>> 11 35
>>
>> Lots and lots of disk activity
>> iostat -dmx 2
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
>> avgqu-sz await r_await w_await svctm %util
>> sda 52.50 0.00 7813.00 0.00 108.01 0.00 28.31
>> 117.15 14.89 14.89 0.00 0.11 83.00
>> sdb 56.00 0.00 7755.50 0.00 108.51 0.00 28.66
>> 118.67 15.18 15.18 0.00 0.11 82.80
>> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>> md5 0.00 0.00 15796.50 0.00 219.21 0.00
>> 28.42 0.00 0.00 0.00 0.00 0.00 0.00
>> dm-0 0.00 0.00 15796.50 0.00 219.21 0.00
>> 28.42 273.42 17.03 17.03 0.00 0.05 83.40
>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 0.00 0.00 0.00 0.00
>>
>> More info :
>> - all the data directory containing the data I'm querying into is 9.7GB
>> and this is a server with 16GB
>> - I'm hitting the server with 6 concurrent multigetsuperslicequeries on
>> multiple keys, some of them can bring back quite a number of data
>> - I'm reading all the keys for one column, pretty much sequentially
>>
>> This is a query in a rollup table that was originally in MySQL and it
>> doesn't look like the performance to query by key is better. So I'm betting
>> I'm doing something wrong here... but what ?
>>
>> Any ideas ?
>> Thanks
>>
>> 2011/6/6 Philippe <wa...@gmail.com>
>>
>>> hum..no, it wasn't swapping. cassandra was the only thing running on that
>>> server
>>> and i was querying the same keys over and over
>>>
>>> i restarted Cassandra and doing the same thing, io is now down to zero
>>> while cpu is up which dosen't surprise me as much.
>>>
>>> I'll report if it happens again.
>>> Le 5 juin 2011 16:55, "Jonathan Ellis" <jb...@gmail.com> a écrit :
>>>
>>> > You may be swapping.
>>> >
>>> > http://spyced.blogspot.com/2010/01/linux-performance-basics.html
>>> > explains how to check this as well as how to see what threads are busy
>>> > in the Java process.
>>> >
>>> > On Sat, Jun 4, 2011 at 5:34 PM, Philippe <wa...@gmail.com> wrote:
>>> >> Hello,
>>> >> I am evaluating using cassandra and I'm running into some strange IO
>>> >> behavior that I can't explain, I'd like some help/ideas to
>>> troubleshoot it.
>>> >> I am running a 1 node cluster with a keyspace consisting of two
>>> columns
>>> >> families, one of which has dozens of supercolumns itself containing
>>> dozens
>>> >> of columns.
>>> >> All in all, this is a couple gigabytes of data, 12GB on the hard
>>> drive.
>>> >> The hardware is pretty good : 16GB memory + RAID-0 SSD drives with LVM
>>> and
>>> >> an i5 processor (4 cores).
>>> >> Keyspace: xxxxxxxxxxxxxxxxxxx
>>> >> Read Count: 460754852
>>> >> Read Latency: 1.108205793092766 ms.
>>> >> Write Count: 30620665
>>> >> Write Latency: 0.01411020877567486 ms.
>>> >> Pending Tasks: 0
>>> >> Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
>>> >> SSTable count: 5
>>> >> Space used (live): 548700725
>>> >> Space used (total): 548700725
>>> >> Memtable Columns Count: 0
>>> >> Memtable Data Size: 0
>>> >> Memtable Switch Count: 11
>>> >> Read Count: 2891192
>>> >> Read Latency: NaN ms.
>>> >> Write Count: 3157547
>>> >> Write Latency: NaN ms.
>>> >> Pending Tasks: 0
>>> >> Key cache capacity: 367396
>>> >> Key cache size: 367396
>>> >> Key cache hit rate: NaN
>>> >> Row cache capacity: 112683
>>> >> Row cache size: 112683
>>> >> Row cache hit rate: NaN
>>> >> Compacted row minimum size: 125
>>> >> Compacted row maximum size: 924
>>> >> Compacted row mean size: 172
>>> >> Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
>>> >> SSTable count: 7
>>> >> Space used (live): 8707538781
>>> >> Space used (total): 8707538781
>>> >> Memtable Columns Count: 0
>>> >> Memtable Data Size: 0
>>> >> Memtable Switch Count: 30
>>> >> Read Count: 457863660
>>> >> Read Latency: 2.381 ms.
>>> >> Write Count: 27463118
>>> >> Write Latency: NaN ms.
>>> >> Pending Tasks: 0
>>> >> Key cache capacity: 4518387
>>> >> Key cache size: 4518387
>>> >> Key cache hit rate: 0.9247881700850826
>>> >> Row cache capacity: 1349682
>>> >> Row cache size: 1349682
>>> >> Row cache hit rate: 0.39400533823415573
>>> >> Compacted row minimum size: 125
>>> >> Compacted row maximum size: 6866
>>> >> Compacted row mean size: 165
>>> >> My app makes a bunch of requests using a MultigetSuperSliceQuery for a
>>> set
>>> >> of keys, typically a couple dozen at most. It also selects a subset of
>>> the
>>> >> supercolumns. I am running 8 requests in parallel at most.
>>> >>
>>> >> Two days, I ran a 1.5 hour process that basically read every key. The
>>> server
>>> >> had no IOwaits and everything was humming along. However, right at the
>>> end
>>> >> of the process, there was a huge spike in IOs. I didn't think much of
>>> it.
>>> >> Today, after two days of inactivity, any query I run raises the IOs to
>>> 80%
>>> >> utilization of the SSD drives even though I'm running the same query
>>> over
>>> >> and over (no cache??)
>>> >> Any ideas on how to troubleshoot this, or better, how to solve this ?
>>> >> thanks
>>> >> Philippe
>>> >
>>> >
>>> >
>>> > --
>>> > Jonathan Ellis
>>> > Project Chair, Apache Cassandra
>>> > co-founder of DataStax, the source for professional Cassandra support
>>> > http://www.datastax.com
>>>
>>
>>
>>
>
>
Re: Troubleshooting IO performance ?
Posted by aaron morton <aa...@thelastpickle.com>.
> So basically, I'm flooding the system right ? For example 99303 means there are 99303 key reads pending, possibly from just a couple MultiSlice gets ?
Yes and then some. Each row you ask for in a multiget turns into a single row request in the server. You are overloading the server.
> - exactly how much data are you asking for ? how many rows and what sort of slice
> According to some munin monitoring, the server is cranking out to the client, over the network, 10Mbits/s = 1.25 Mbytes/s
Was thinking in terms of rows, but thats irrelevant now. The answer is "a lot"
> BTW, cassandra is running on an XFS filesystem over LVM
Others know more about this than me.
> One question : when nodetool cfstats says the average read latency is 5ms, is that counted once the query is being executed or does that include the time spent "pending" ?
In the cf stats output the latency displayed under the Keyspace is the total latency for all CF's / the total read count. The latency displayed for the individual CF's is for the actual time taken getting the columns requested for a row. It's taking 5ms to read the data from disk and apply the filter.
I'd check you are reading the data you expect then wind back the number of requests and rows / columns requested. Get to a stable baseline and then add pressure to see when / how things go wrong.
Hope that helps.
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 8 Jun 2011, at 08:00, Philippe wrote:
> Aaron,
>
> - what version are you on ?
> 0.7.6-2
>
> - what is the concurrent_reads config setting ?
> concurrent_reads: 64
> concurrent_writes: 64
>
> Givent that I've got 4 cores and SSD drives, I doubled the concurrent writes recommended.
> Given that I've RAID-0ed the SSD drive, I figured I could at least double for SSD and double for RAID-0 the recommended version.
> Wrong assumptions ?
>
> BTW, cassandra is running on an XFS filesystem over LVM over RAID-0
>
> - what is nodetool tpstats showing during the slow down ?
> The only value that changes is the ReadStage line. Here's values from a sample every second
> Pool Name Active Pending Completed
> ReadStage 64 99303 463085056
> ReadStage 64 88430 463095929
> ReadStage 64 91937 463107782
>
> So basically, I'm flooding the system right ? For example 99303 means there are 99303 key reads pending, possibly from just a couple MultiSlice gets ?
>
> - exactly how much data are you asking for ? how many rows and what sort of slice
> According to some munin monitoring, the server is cranking out to the client, over the network, 10Mbits/s = 1.25 Mbytes/s
>
> The same munin monitoring shows me 200Mbytes/s read from the disks. This is what is worrying me...
>
> - has their been a lot of deletes or TTL columns used ?
> No deletes, only update, don't know if that counts as deletes though...
>
> This is going to be a read-heavy, update-heavy cluster.
> No TTL columns, no counter columns
>
> One question : when nodetool cfstats says the average read latency is 5ms, is that counted once the query is being executed or does that include the time spent "pending" ?
>
> Thanks
> Philippe
>
> Hope that helps.
> Aaron
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 7 Jun 2011, at 10:09, Philippe wrote:
>
>> Ok, here it goes again... No swapping at all...
>>
>> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
>> r b swpd free buff cache si so bi bo in cs us sy id wa
>> 1 63 32044 88736 37996 7116524 0 0 227156 0 18314 5607 30 5 11 53
>> 1 63 32044 90844 37996 7103904 0 0 233524 202 17418 4977 29 4 9 58
>> 0 42 32044 91304 37996 7123884 0 0 249736 0 16197 5433 19 6 3 72
>> 3 25 32044 89864 37996 7135980 0 0 223140 16 18135 7567 32 5 11 52
>> 1 1 32044 88664 37996 7150728 0 0 229416 128 19168 7554 36 4 10 51
>> 4 0 32044 89464 37996 7149428 0 0 213852 18 21041 8819 45 5 12 38
>> 4 0 32044 90372 37996 7149432 0 0 233086 142 19909 7041 43 5 10 41
>> 7 1 32044 89752 37996 7149520 0 0 206906 0 19350 6875 50 4 11 35
>>
>> Lots and lots of disk activity
>> iostat -dmx 2
>> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sda 52.50 0.00 7813.00 0.00 108.01 0.00 28.31 117.15 14.89 14.89 0.00 0.11 83.00
>> sdb 56.00 0.00 7755.50 0.00 108.51 0.00 28.66 118.67 15.18 15.18 0.00 0.11 82.80
>> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> md5 0.00 0.00 15796.50 0.00 219.21 0.00 28.42 0.00 0.00 0.00 0.00 0.00 0.00
>> dm-0 0.00 0.00 15796.50 0.00 219.21 0.00 28.42 273.42 17.03 17.03 0.00 0.05 83.40
>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>
>> More info :
>> - all the data directory containing the data I'm querying into is 9.7GB and this is a server with 16GB
>> - I'm hitting the server with 6 concurrent multigetsuperslicequeries on multiple keys, some of them can bring back quite a number of data
>> - I'm reading all the keys for one column, pretty much sequentially
>>
>> This is a query in a rollup table that was originally in MySQL and it doesn't look like the performance to query by key is better. So I'm betting I'm doing something wrong here... but what ?
>>
>> Any ideas ?
>> Thanks
>>
>> 2011/6/6 Philippe <wa...@gmail.com>
>> hum..no, it wasn't swapping. cassandra was the only thing running on that server
>> and i was querying the same keys over and over
>>
>> i restarted Cassandra and doing the same thing, io is now down to zero while cpu is up which dosen't surprise me as much.
>>
>> I'll report if it happens again.
>>
>> Le 5 juin 2011 16:55, "Jonathan Ellis" <jb...@gmail.com> a écrit :
>>
>> > You may be swapping.
>> >
>> > http://spyced.blogspot.com/2010/01/linux-performance-basics.html
>> > explains how to check this as well as how to see what threads are busy
>> > in the Java process.
>> >
>> > On Sat, Jun 4, 2011 at 5:34 PM, Philippe <wa...@gmail.com> wrote:
>> >> Hello,
>> >> I am evaluating using cassandra and I'm running into some strange IO
>> >> behavior that I can't explain, I'd like some help/ideas to troubleshoot it.
>> >> I am running a 1 node cluster with a keyspace consisting of two columns
>> >> families, one of which has dozens of supercolumns itself containing dozens
>> >> of columns.
>> >> All in all, this is a couple gigabytes of data, 12GB on the hard drive.
>> >> The hardware is pretty good : 16GB memory + RAID-0 SSD drives with LVM and
>> >> an i5 processor (4 cores).
>> >> Keyspace: xxxxxxxxxxxxxxxxxxx
>> >> Read Count: 460754852
>> >> Read Latency: 1.108205793092766 ms.
>> >> Write Count: 30620665
>> >> Write Latency: 0.01411020877567486 ms.
>> >> Pending Tasks: 0
>> >> Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
>> >> SSTable count: 5
>> >> Space used (live): 548700725
>> >> Space used (total): 548700725
>> >> Memtable Columns Count: 0
>> >> Memtable Data Size: 0
>> >> Memtable Switch Count: 11
>> >> Read Count: 2891192
>> >> Read Latency: NaN ms.
>> >> Write Count: 3157547
>> >> Write Latency: NaN ms.
>> >> Pending Tasks: 0
>> >> Key cache capacity: 367396
>> >> Key cache size: 367396
>> >> Key cache hit rate: NaN
>> >> Row cache capacity: 112683
>> >> Row cache size: 112683
>> >> Row cache hit rate: NaN
>> >> Compacted row minimum size: 125
>> >> Compacted row maximum size: 924
>> >> Compacted row mean size: 172
>> >> Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
>> >> SSTable count: 7
>> >> Space used (live): 8707538781
>> >> Space used (total): 8707538781
>> >> Memtable Columns Count: 0
>> >> Memtable Data Size: 0
>> >> Memtable Switch Count: 30
>> >> Read Count: 457863660
>> >> Read Latency: 2.381 ms.
>> >> Write Count: 27463118
>> >> Write Latency: NaN ms.
>> >> Pending Tasks: 0
>> >> Key cache capacity: 4518387
>> >> Key cache size: 4518387
>> >> Key cache hit rate: 0.9247881700850826
>> >> Row cache capacity: 1349682
>> >> Row cache size: 1349682
>> >> Row cache hit rate: 0.39400533823415573
>> >> Compacted row minimum size: 125
>> >> Compacted row maximum size: 6866
>> >> Compacted row mean size: 165
>> >> My app makes a bunch of requests using a MultigetSuperSliceQuery for a set
>> >> of keys, typically a couple dozen at most. It also selects a subset of the
>> >> supercolumns. I am running 8 requests in parallel at most.
>> >>
>> >> Two days, I ran a 1.5 hour process that basically read every key. The server
>> >> had no IOwaits and everything was humming along. However, right at the end
>> >> of the process, there was a huge spike in IOs. I didn't think much of it.
>> >> Today, after two days of inactivity, any query I run raises the IOs to 80%
>> >> utilization of the SSD drives even though I'm running the same query over
>> >> and over (no cache??)
>> >> Any ideas on how to troubleshoot this, or better, how to solve this ?
>> >> thanks
>> >> Philippe
>> >
>> >
>> >
>> > --
>> > Jonathan Ellis
>> > Project Chair, Apache Cassandra
>> > co-founder of DataStax, the source for professional Cassandra support
>> > http://www.datastax.com
>>
>
>
Re: Troubleshooting IO performance ?
Posted by Philippe <wa...@gmail.com>.
Aaron,
- what version are you on ?
0.7.6-2
- what is the concurrent_reads config setting ?
>
concurrent_reads: 64
concurrent_writes: 64
Givent that I've got 4 cores and SSD drives, I doubled the concurrent writes
recommended.
Given that I've RAID-0ed the SSD drive, I figured I could at least double
for SSD and double for RAID-0 the recommended version.
Wrong assumptions ?
BTW, cassandra is running on an XFS filesystem over LVM over RAID-0
- what is nodetool tpstats showing during the slow down ?
>
The only value that changes is the ReadStage line. Here's values from a
sample every second
Pool Name Active Pending Completed
ReadStage 64 99303 463085056
ReadStage 64 88430 463095929
ReadStage 64 91937 463107782
So basically, I'm flooding the system right ? For example 99303 means there
are 99303 key reads pending, possibly from just a couple MultiSlice gets ?
> - exactly how much data are you asking for ? how many rows and what sort of
> slice
>
According to some munin monitoring, the server is cranking out to the
client, over the network, 10Mbits/s = 1.25 Mbytes/s
The same munin monitoring shows me 200Mbytes/s read from the disks. This is
what is worrying me...
- has their been a lot of deletes or TTL columns used ?
>
No deletes, only update, don't know if that counts as deletes though...
This is going to be a read-heavy, update-heavy cluster.
No TTL columns, no counter columns
One question : when nodetool cfstats says the average read latency is 5ms,
is that counted once the query is being executed or does that include the
time spent "pending" ?
Thanks
Philippe
>
> Hope that helps.
> Aaron
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 7 Jun 2011, at 10:09, Philippe wrote:
>
> Ok, here it goes again... No swapping at all...
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id
> wa
> 1 63 32044 88736 37996 7116524 0 0 227156 0 18314 5607 30 5
> 11 53
> 1 63 32044 90844 37996 7103904 0 0 233524 202 17418 4977 29 4
> 9 58
> 0 42 32044 91304 37996 7123884 0 0 249736 0 16197 5433 19 6
> 3 72
> 3 25 32044 89864 37996 7135980 0 0 223140 16 18135 7567 32 5
> 11 52
> 1 1 32044 88664 37996 7150728 0 0 229416 128 19168 7554 36 4
> 10 51
> 4 0 32044 89464 37996 7149428 0 0 213852 18 21041 8819 45 5
> 12 38
> 4 0 32044 90372 37996 7149432 0 0 233086 142 19909 7041 43 5
> 10 41
> 7 1 32044 89752 37996 7149520 0 0 206906 0 19350 6875 50 4
> 11 35
>
> Lots and lots of disk activity
> iostat -dmx 2
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sda 52.50 0.00 7813.00 0.00 108.01 0.00 28.31
> 117.15 14.89 14.89 0.00 0.11 83.00
> sdb 56.00 0.00 7755.50 0.00 108.51 0.00 28.66
> 118.67 15.18 15.18 0.00 0.11 82.80
> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
> md5 0.00 0.00 15796.50 0.00 219.21 0.00 28.42
> 0.00 0.00 0.00 0.00 0.00 0.00
> dm-0 0.00 0.00 15796.50 0.00 219.21 0.00 28.42
> 273.42 17.03 17.03 0.00 0.05 83.40
> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00
>
> More info :
> - all the data directory containing the data I'm querying into is 9.7GB
> and this is a server with 16GB
> - I'm hitting the server with 6 concurrent multigetsuperslicequeries on
> multiple keys, some of them can bring back quite a number of data
> - I'm reading all the keys for one column, pretty much sequentially
>
> This is a query in a rollup table that was originally in MySQL and it
> doesn't look like the performance to query by key is better. So I'm betting
> I'm doing something wrong here... but what ?
>
> Any ideas ?
> Thanks
>
> 2011/6/6 Philippe <wa...@gmail.com>
>
>> hum..no, it wasn't swapping. cassandra was the only thing running on that
>> server
>> and i was querying the same keys over and over
>>
>> i restarted Cassandra and doing the same thing, io is now down to zero
>> while cpu is up which dosen't surprise me as much.
>>
>> I'll report if it happens again.
>> Le 5 juin 2011 16:55, "Jonathan Ellis" <jb...@gmail.com> a écrit :
>>
>> > You may be swapping.
>> >
>> > http://spyced.blogspot.com/2010/01/linux-performance-basics.html
>> > explains how to check this as well as how to see what threads are busy
>> > in the Java process.
>> >
>> > On Sat, Jun 4, 2011 at 5:34 PM, Philippe <wa...@gmail.com> wrote:
>> >> Hello,
>> >> I am evaluating using cassandra and I'm running into some strange IO
>> >> behavior that I can't explain, I'd like some help/ideas to troubleshoot
>> it.
>> >> I am running a 1 node cluster with a keyspace consisting of two columns
>> >> families, one of which has dozens of supercolumns itself containing
>> dozens
>> >> of columns.
>> >> All in all, this is a couple gigabytes of data, 12GB on the hard drive.
>> >> The hardware is pretty good : 16GB memory + RAID-0 SSD drives with LVM
>> and
>> >> an i5 processor (4 cores).
>> >> Keyspace: xxxxxxxxxxxxxxxxxxx
>> >> Read Count: 460754852
>> >> Read Latency: 1.108205793092766 ms.
>> >> Write Count: 30620665
>> >> Write Latency: 0.01411020877567486 ms.
>> >> Pending Tasks: 0
>> >> Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
>> >> SSTable count: 5
>> >> Space used (live): 548700725
>> >> Space used (total): 548700725
>> >> Memtable Columns Count: 0
>> >> Memtable Data Size: 0
>> >> Memtable Switch Count: 11
>> >> Read Count: 2891192
>> >> Read Latency: NaN ms.
>> >> Write Count: 3157547
>> >> Write Latency: NaN ms.
>> >> Pending Tasks: 0
>> >> Key cache capacity: 367396
>> >> Key cache size: 367396
>> >> Key cache hit rate: NaN
>> >> Row cache capacity: 112683
>> >> Row cache size: 112683
>> >> Row cache hit rate: NaN
>> >> Compacted row minimum size: 125
>> >> Compacted row maximum size: 924
>> >> Compacted row mean size: 172
>> >> Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
>> >> SSTable count: 7
>> >> Space used (live): 8707538781
>> >> Space used (total): 8707538781
>> >> Memtable Columns Count: 0
>> >> Memtable Data Size: 0
>> >> Memtable Switch Count: 30
>> >> Read Count: 457863660
>> >> Read Latency: 2.381 ms.
>> >> Write Count: 27463118
>> >> Write Latency: NaN ms.
>> >> Pending Tasks: 0
>> >> Key cache capacity: 4518387
>> >> Key cache size: 4518387
>> >> Key cache hit rate: 0.9247881700850826
>> >> Row cache capacity: 1349682
>> >> Row cache size: 1349682
>> >> Row cache hit rate: 0.39400533823415573
>> >> Compacted row minimum size: 125
>> >> Compacted row maximum size: 6866
>> >> Compacted row mean size: 165
>> >> My app makes a bunch of requests using a MultigetSuperSliceQuery for a
>> set
>> >> of keys, typically a couple dozen at most. It also selects a subset of
>> the
>> >> supercolumns. I am running 8 requests in parallel at most.
>> >>
>> >> Two days, I ran a 1.5 hour process that basically read every key. The
>> server
>> >> had no IOwaits and everything was humming along. However, right at the
>> end
>> >> of the process, there was a huge spike in IOs. I didn't think much of
>> it.
>> >> Today, after two days of inactivity, any query I run raises the IOs to
>> 80%
>> >> utilization of the SSD drives even though I'm running the same query
>> over
>> >> and over (no cache??)
>> >> Any ideas on how to troubleshoot this, or better, how to solve this ?
>> >> thanks
>> >> Philippe
>> >
>> >
>> >
>> > --
>> > Jonathan Ellis
>> > Project Chair, Apache Cassandra
>> > co-founder of DataStax, the source for professional Cassandra support
>> > http://www.datastax.com
>>
>
>
>
Re: Troubleshooting IO performance ?
Posted by aaron morton <aa...@thelastpickle.com>.
There is a big IO queue and reads are spending a lot of time in the queue.
Some more questions:
- what version are you on ?
- what is the concurrent_reads config setting ?
- what is nodetool tpstats showing during the slow down ?
- exactly how much data are you asking for ? how many rows and what sort of slice
- has their been a lot of deletes or TTL columns used ?
Hope that helps.
Aaron
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 7 Jun 2011, at 10:09, Philippe wrote:
> Ok, here it goes again... No swapping at all...
>
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 1 63 32044 88736 37996 7116524 0 0 227156 0 18314 5607 30 5 11 53
> 1 63 32044 90844 37996 7103904 0 0 233524 202 17418 4977 29 4 9 58
> 0 42 32044 91304 37996 7123884 0 0 249736 0 16197 5433 19 6 3 72
> 3 25 32044 89864 37996 7135980 0 0 223140 16 18135 7567 32 5 11 52
> 1 1 32044 88664 37996 7150728 0 0 229416 128 19168 7554 36 4 10 51
> 4 0 32044 89464 37996 7149428 0 0 213852 18 21041 8819 45 5 12 38
> 4 0 32044 90372 37996 7149432 0 0 233086 142 19909 7041 43 5 10 41
> 7 1 32044 89752 37996 7149520 0 0 206906 0 19350 6875 50 4 11 35
>
> Lots and lots of disk activity
> iostat -dmx 2
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> sda 52.50 0.00 7813.00 0.00 108.01 0.00 28.31 117.15 14.89 14.89 0.00 0.11 83.00
> sdb 56.00 0.00 7755.50 0.00 108.51 0.00 28.66 118.67 15.18 15.18 0.00 0.11 82.80
> md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> md5 0.00 0.00 15796.50 0.00 219.21 0.00 28.42 0.00 0.00 0.00 0.00 0.00 0.00
> dm-0 0.00 0.00 15796.50 0.00 219.21 0.00 28.42 273.42 17.03 17.03 0.00 0.05 83.40
> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>
> More info :
> - all the data directory containing the data I'm querying into is 9.7GB and this is a server with 16GB
> - I'm hitting the server with 6 concurrent multigetsuperslicequeries on multiple keys, some of them can bring back quite a number of data
> - I'm reading all the keys for one column, pretty much sequentially
>
> This is a query in a rollup table that was originally in MySQL and it doesn't look like the performance to query by key is better. So I'm betting I'm doing something wrong here... but what ?
>
> Any ideas ?
> Thanks
>
> 2011/6/6 Philippe <wa...@gmail.com>
> hum..no, it wasn't swapping. cassandra was the only thing running on that server
> and i was querying the same keys over and over
>
> i restarted Cassandra and doing the same thing, io is now down to zero while cpu is up which dosen't surprise me as much.
>
> I'll report if it happens again.
>
> Le 5 juin 2011 16:55, "Jonathan Ellis" <jb...@gmail.com> a écrit :
>
> > You may be swapping.
> >
> > http://spyced.blogspot.com/2010/01/linux-performance-basics.html
> > explains how to check this as well as how to see what threads are busy
> > in the Java process.
> >
> > On Sat, Jun 4, 2011 at 5:34 PM, Philippe <wa...@gmail.com> wrote:
> >> Hello,
> >> I am evaluating using cassandra and I'm running into some strange IO
> >> behavior that I can't explain, I'd like some help/ideas to troubleshoot it.
> >> I am running a 1 node cluster with a keyspace consisting of two columns
> >> families, one of which has dozens of supercolumns itself containing dozens
> >> of columns.
> >> All in all, this is a couple gigabytes of data, 12GB on the hard drive.
> >> The hardware is pretty good : 16GB memory + RAID-0 SSD drives with LVM and
> >> an i5 processor (4 cores).
> >> Keyspace: xxxxxxxxxxxxxxxxxxx
> >> Read Count: 460754852
> >> Read Latency: 1.108205793092766 ms.
> >> Write Count: 30620665
> >> Write Latency: 0.01411020877567486 ms.
> >> Pending Tasks: 0
> >> Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
> >> SSTable count: 5
> >> Space used (live): 548700725
> >> Space used (total): 548700725
> >> Memtable Columns Count: 0
> >> Memtable Data Size: 0
> >> Memtable Switch Count: 11
> >> Read Count: 2891192
> >> Read Latency: NaN ms.
> >> Write Count: 3157547
> >> Write Latency: NaN ms.
> >> Pending Tasks: 0
> >> Key cache capacity: 367396
> >> Key cache size: 367396
> >> Key cache hit rate: NaN
> >> Row cache capacity: 112683
> >> Row cache size: 112683
> >> Row cache hit rate: NaN
> >> Compacted row minimum size: 125
> >> Compacted row maximum size: 924
> >> Compacted row mean size: 172
> >> Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
> >> SSTable count: 7
> >> Space used (live): 8707538781
> >> Space used (total): 8707538781
> >> Memtable Columns Count: 0
> >> Memtable Data Size: 0
> >> Memtable Switch Count: 30
> >> Read Count: 457863660
> >> Read Latency: 2.381 ms.
> >> Write Count: 27463118
> >> Write Latency: NaN ms.
> >> Pending Tasks: 0
> >> Key cache capacity: 4518387
> >> Key cache size: 4518387
> >> Key cache hit rate: 0.9247881700850826
> >> Row cache capacity: 1349682
> >> Row cache size: 1349682
> >> Row cache hit rate: 0.39400533823415573
> >> Compacted row minimum size: 125
> >> Compacted row maximum size: 6866
> >> Compacted row mean size: 165
> >> My app makes a bunch of requests using a MultigetSuperSliceQuery for a set
> >> of keys, typically a couple dozen at most. It also selects a subset of the
> >> supercolumns. I am running 8 requests in parallel at most.
> >>
> >> Two days, I ran a 1.5 hour process that basically read every key. The server
> >> had no IOwaits and everything was humming along. However, right at the end
> >> of the process, there was a huge spike in IOs. I didn't think much of it.
> >> Today, after two days of inactivity, any query I run raises the IOs to 80%
> >> utilization of the SSD drives even though I'm running the same query over
> >> and over (no cache??)
> >> Any ideas on how to troubleshoot this, or better, how to solve this ?
> >> thanks
> >> Philippe
> >
> >
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder of DataStax, the source for professional Cassandra support
> > http://www.datastax.com
>
Re: Troubleshooting IO performance ?
Posted by Philippe <wa...@gmail.com>.
Ok, here it goes again... No swapping at all...
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
1 63 32044 88736 37996 7116524 0 0 227156 0 18314 5607 30 5
11 53
1 63 32044 90844 37996 7103904 0 0 233524 202 17418 4977 29 4
9 58
0 42 32044 91304 37996 7123884 0 0 249736 0 16197 5433 19 6
3 72
3 25 32044 89864 37996 7135980 0 0 223140 16 18135 7567 32 5
11 52
1 1 32044 88664 37996 7150728 0 0 229416 128 19168 7554 36 4
10 51
4 0 32044 89464 37996 7149428 0 0 213852 18 21041 8819 45 5
12 38
4 0 32044 90372 37996 7149432 0 0 233086 142 19909 7041 43 5
10 41
7 1 32044 89752 37996 7149520 0 0 206906 0 19350 6875 50 4
11 35
Lots and lots of disk activity
iostat -dmx 2
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await r_await w_await svctm %util
sda 52.50 0.00 7813.00 0.00 108.01 0.00 28.31
117.15 14.89 14.89 0.00 0.11 83.00
sdb 56.00 0.00 7755.50 0.00 108.51 0.00 28.66
118.67 15.18 15.18 0.00 0.11 82.80
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
md5 0.00 0.00 15796.50 0.00 219.21 0.00 28.42
0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 15796.50 0.00 219.21 0.00 28.42
273.42 17.03 17.03 0.00 0.05 83.40
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.00
More info :
- all the data directory containing the data I'm querying into is 9.7GB and
this is a server with 16GB
- I'm hitting the server with 6 concurrent multigetsuperslicequeries on
multiple keys, some of them can bring back quite a number of data
- I'm reading all the keys for one column, pretty much sequentially
This is a query in a rollup table that was originally in MySQL and it
doesn't look like the performance to query by key is better. So I'm betting
I'm doing something wrong here... but what ?
Any ideas ?
Thanks
2011/6/6 Philippe <wa...@gmail.com>
> hum..no, it wasn't swapping. cassandra was the only thing running on that
> server
> and i was querying the same keys over and over
>
> i restarted Cassandra and doing the same thing, io is now down to zero
> while cpu is up which dosen't surprise me as much.
>
> I'll report if it happens again.
> Le 5 juin 2011 16:55, "Jonathan Ellis" <jb...@gmail.com> a écrit :
>
> > You may be swapping.
> >
> > http://spyced.blogspot.com/2010/01/linux-performance-basics.html
> > explains how to check this as well as how to see what threads are busy
> > in the Java process.
> >
> > On Sat, Jun 4, 2011 at 5:34 PM, Philippe <wa...@gmail.com> wrote:
> >> Hello,
> >> I am evaluating using cassandra and I'm running into some strange IO
> >> behavior that I can't explain, I'd like some help/ideas to troubleshoot
> it.
> >> I am running a 1 node cluster with a keyspace consisting of two columns
> >> families, one of which has dozens of supercolumns itself containing
> dozens
> >> of columns.
> >> All in all, this is a couple gigabytes of data, 12GB on the hard drive.
> >> The hardware is pretty good : 16GB memory + RAID-0 SSD drives with LVM
> and
> >> an i5 processor (4 cores).
> >> Keyspace: xxxxxxxxxxxxxxxxxxx
> >> Read Count: 460754852
> >> Read Latency: 1.108205793092766 ms.
> >> Write Count: 30620665
> >> Write Latency: 0.01411020877567486 ms.
> >> Pending Tasks: 0
> >> Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
> >> SSTable count: 5
> >> Space used (live): 548700725
> >> Space used (total): 548700725
> >> Memtable Columns Count: 0
> >> Memtable Data Size: 0
> >> Memtable Switch Count: 11
> >> Read Count: 2891192
> >> Read Latency: NaN ms.
> >> Write Count: 3157547
> >> Write Latency: NaN ms.
> >> Pending Tasks: 0
> >> Key cache capacity: 367396
> >> Key cache size: 367396
> >> Key cache hit rate: NaN
> >> Row cache capacity: 112683
> >> Row cache size: 112683
> >> Row cache hit rate: NaN
> >> Compacted row minimum size: 125
> >> Compacted row maximum size: 924
> >> Compacted row mean size: 172
> >> Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
> >> SSTable count: 7
> >> Space used (live): 8707538781
> >> Space used (total): 8707538781
> >> Memtable Columns Count: 0
> >> Memtable Data Size: 0
> >> Memtable Switch Count: 30
> >> Read Count: 457863660
> >> Read Latency: 2.381 ms.
> >> Write Count: 27463118
> >> Write Latency: NaN ms.
> >> Pending Tasks: 0
> >> Key cache capacity: 4518387
> >> Key cache size: 4518387
> >> Key cache hit rate: 0.9247881700850826
> >> Row cache capacity: 1349682
> >> Row cache size: 1349682
> >> Row cache hit rate: 0.39400533823415573
> >> Compacted row minimum size: 125
> >> Compacted row maximum size: 6866
> >> Compacted row mean size: 165
> >> My app makes a bunch of requests using a MultigetSuperSliceQuery for a
> set
> >> of keys, typically a couple dozen at most. It also selects a subset of
> the
> >> supercolumns. I am running 8 requests in parallel at most.
> >>
> >> Two days, I ran a 1.5 hour process that basically read every key. The
> server
> >> had no IOwaits and everything was humming along. However, right at the
> end
> >> of the process, there was a huge spike in IOs. I didn't think much of
> it.
> >> Today, after two days of inactivity, any query I run raises the IOs to
> 80%
> >> utilization of the SSD drives even though I'm running the same query
> over
> >> and over (no cache??)
> >> Any ideas on how to troubleshoot this, or better, how to solve this ?
> >> thanks
> >> Philippe
> >
> >
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder of DataStax, the source for professional Cassandra support
> > http://www.datastax.com
>
Re: Troubleshooting IO performance ?
Posted by Philippe <wa...@gmail.com>.
hum..no, it wasn't swapping. cassandra was the only thing running on that
server
and i was querying the same keys over and over
i restarted Cassandra and doing the same thing, io is now down to zero while
cpu is up which dosen't surprise me as much.
I'll report if it happens again.
Le 5 juin 2011 16:55, "Jonathan Ellis" <jb...@gmail.com> a écrit :
> You may be swapping.
>
> http://spyced.blogspot.com/2010/01/linux-performance-basics.html
> explains how to check this as well as how to see what threads are busy
> in the Java process.
>
> On Sat, Jun 4, 2011 at 5:34 PM, Philippe <wa...@gmail.com> wrote:
>> Hello,
>> I am evaluating using cassandra and I'm running into some strange IO
>> behavior that I can't explain, I'd like some help/ideas to troubleshoot
it.
>> I am running a 1 node cluster with a keyspace consisting of two columns
>> families, one of which has dozens of supercolumns itself containing
dozens
>> of columns.
>> All in all, this is a couple gigabytes of data, 12GB on the hard drive.
>> The hardware is pretty good : 16GB memory + RAID-0 SSD drives with LVM
and
>> an i5 processor (4 cores).
>> Keyspace: xxxxxxxxxxxxxxxxxxx
>> Read Count: 460754852
>> Read Latency: 1.108205793092766 ms.
>> Write Count: 30620665
>> Write Latency: 0.01411020877567486 ms.
>> Pending Tasks: 0
>> Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
>> SSTable count: 5
>> Space used (live): 548700725
>> Space used (total): 548700725
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 11
>> Read Count: 2891192
>> Read Latency: NaN ms.
>> Write Count: 3157547
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 367396
>> Key cache size: 367396
>> Key cache hit rate: NaN
>> Row cache capacity: 112683
>> Row cache size: 112683
>> Row cache hit rate: NaN
>> Compacted row minimum size: 125
>> Compacted row maximum size: 924
>> Compacted row mean size: 172
>> Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
>> SSTable count: 7
>> Space used (live): 8707538781
>> Space used (total): 8707538781
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 30
>> Read Count: 457863660
>> Read Latency: 2.381 ms.
>> Write Count: 27463118
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 4518387
>> Key cache size: 4518387
>> Key cache hit rate: 0.9247881700850826
>> Row cache capacity: 1349682
>> Row cache size: 1349682
>> Row cache hit rate: 0.39400533823415573
>> Compacted row minimum size: 125
>> Compacted row maximum size: 6866
>> Compacted row mean size: 165
>> My app makes a bunch of requests using a MultigetSuperSliceQuery for a
set
>> of keys, typically a couple dozen at most. It also selects a subset of
the
>> supercolumns. I am running 8 requests in parallel at most.
>>
>> Two days, I ran a 1.5 hour process that basically read every key. The
server
>> had no IOwaits and everything was humming along. However, right at the
end
>> of the process, there was a huge spike in IOs. I didn't think much of it.
>> Today, after two days of inactivity, any query I run raises the IOs to
80%
>> utilization of the SSD drives even though I'm running the same query over
>> and over (no cache??)
>> Any ideas on how to troubleshoot this, or better, how to solve this ?
>> thanks
>> Philippe
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: Troubleshooting IO performance ?
Posted by Jonathan Ellis <jb...@gmail.com>.
You may be swapping.
http://spyced.blogspot.com/2010/01/linux-performance-basics.html
explains how to check this as well as how to see what threads are busy
in the Java process.
On Sat, Jun 4, 2011 at 5:34 PM, Philippe <wa...@gmail.com> wrote:
> Hello,
> I am evaluating using cassandra and I'm running into some strange IO
> behavior that I can't explain, I'd like some help/ideas to troubleshoot it.
> I am running a 1 node cluster with a keyspace consisting of two columns
> families, one of which has dozens of supercolumns itself containing dozens
> of columns.
> All in all, this is a couple gigabytes of data, 12GB on the hard drive.
> The hardware is pretty good : 16GB memory + RAID-0 SSD drives with LVM and
> an i5 processor (4 cores).
> Keyspace: xxxxxxxxxxxxxxxxxxx
> Read Count: 460754852
> Read Latency: 1.108205793092766 ms.
> Write Count: 30620665
> Write Latency: 0.01411020877567486 ms.
> Pending Tasks: 0
> Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
> SSTable count: 5
> Space used (live): 548700725
> Space used (total): 548700725
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 11
> Read Count: 2891192
> Read Latency: NaN ms.
> Write Count: 3157547
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 367396
> Key cache size: 367396
> Key cache hit rate: NaN
> Row cache capacity: 112683
> Row cache size: 112683
> Row cache hit rate: NaN
> Compacted row minimum size: 125
> Compacted row maximum size: 924
> Compacted row mean size: 172
> Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
> SSTable count: 7
> Space used (live): 8707538781
> Space used (total): 8707538781
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 30
> Read Count: 457863660
> Read Latency: 2.381 ms.
> Write Count: 27463118
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 4518387
> Key cache size: 4518387
> Key cache hit rate: 0.9247881700850826
> Row cache capacity: 1349682
> Row cache size: 1349682
> Row cache hit rate: 0.39400533823415573
> Compacted row minimum size: 125
> Compacted row maximum size: 6866
> Compacted row mean size: 165
> My app makes a bunch of requests using a MultigetSuperSliceQuery for a set
> of keys, typically a couple dozen at most. It also selects a subset of the
> supercolumns. I am running 8 requests in parallel at most.
>
> Two days, I ran a 1.5 hour process that basically read every key. The server
> had no IOwaits and everything was humming along. However, right at the end
> of the process, there was a huge spike in IOs. I didn't think much of it.
> Today, after two days of inactivity, any query I run raises the IOs to 80%
> utilization of the SSD drives even though I'm running the same query over
> and over (no cache??)
> Any ideas on how to troubleshoot this, or better, how to solve this ?
> thanks
> Philippe
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com