You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Stefan Reek <st...@unitedgames.com> on 2012/03/01 11:13:41 UTC
Re: Impact of old data on performance
Swap is disabled on the machines, so I'm sure the JVM is not swapping out.
Cassandra is the only process running on those machines, so network
contention shouldn't be an issues either.
Which leaves Garbage collection.
In the logs I can see that the Garbage Collection takes a long time, for
example:
GCInspector.java (line 130) GC for ConcurrentMarkSweep: 384113 ms,
-84703136 reclaimed leaving 1018203472 used; max is 17208836096
But I configured the Garbage Collector to use the ConcurrentMarkSweep
algorithm. As far as I understand this means that there will only be two
pauses during the compaction, the mark and the remark stage.
As I also have extra garbage collection logging enabled on one node, I
checked to see if the mark/remark times sometimes exceed the 10 second
RPC timeout and I found some logging statements like this:
422208.422: [CMS-concurrent-mark: 0.434/20.004 secs] [Times: user=13.40
sys=0.75, real=20.00 secs]
Does this mean that the mark session took over 10 seconds and all other
Cassandra threads were stopped during this
time? If so, that would explain why I sometimes get timeouts on requests
and also see Dropped Messages in my logs.
Cheers,
Stefan
On 02/29/2012 07:48 PM, Dan Retzlaff wrote:
> First to be clear, I'm not an expert but I suggested "cfstats" because
> it can show unhealthy signs. That said, yours looks okay to me: few
> live SSTables per column family, reasonable quantity of data... The
> next things I'd verify are that (1) the JVM isn't swapping out during
> these periods of bad performance, (2) GC completes in less than your
> RPC timeout (check the logs), and (3) your internode network
> connection isn't degraded by another application.
>
> The fact that *writes* are timing out is interesting, since existing
> data should have little to no effect on those.
>
> BTW, since ColumnFamily7 sees most of your action, you might consider
> whether a key/row cache is appropriate. By my count it has 2M rows and
> 7M reads since startup. But this fine-tuning is probably a distraction
> from whatever's causing your timeouts.
>
> Hope that helps,
> Dan
>
> On Wed, Feb 29, 2012 at 5:14 AM, Stefan Reek <stefan@unitedgames.com
> <ma...@unitedgames.com>> wrote:
>
> Hi Dan,
>
> Thanks for answering.
> I included the output of cfstats below.
> I hope you can say something about our problems with it.
>
> cheers,
>
> Stefan
>
> Keyspace: Keyspace1
> Read Count: 60703419
> Read Latency: 1.1790332096286043 ms.
> Write Count: 105871791
> Write Latency: 0.019847457393065166 ms.
> Pending Tasks: 0
> Column Family: ColumnFamily1
> SSTable count: 0
> Space used (live): 0
> Space used (total): 0
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 200000
> Key cache size: 0
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 0
> Compacted row maximum size: 0
> Compacted row mean size: 0
>
> Column Family: ColumnFamily2
> SSTable count: 2
> Space used (live): 482828
> Space used (total): 482828
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 3
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 9
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 200000
> Key cache size: 0
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 258
> Compacted row maximum size: 275
> Compacted row mean size: 266
>
> Column Family: ColumnFamily3
> SSTable count: 0
> Space used (live): 0
> Space used (total): 0
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 200000
> Key cache size: 0
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 0
> Compacted row maximum size: 0
> Compacted row mean size: 0
>
> Column Family: ColumnFamily4
> SSTable count: 1
> Space used (live): 19793722
> Space used (total): 19793722
> Memtable Columns Count: 306
> Memtable Data Size: 12731
> Memtable Switch Count: 3
> Read Count: 2309
> Read Latency: NaN ms.
> Write Count: 3234
> Write Latency: 0.021 ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 465813
> Compacted row maximum size: 4784014
> Compacted row mean size: 3295196
>
> Column Family: ColumnFamily5
> SSTable count: 6
> Space used (live): 364775507
> Space used (total): 364775507
> Memtable Columns Count: 14193
> Memtable Data Size: 552090
> Memtable Switch Count: 3
> Read Count: 211451
> Read Latency: NaN ms.
> Write Count: 26567
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 187
> Compacted row maximum size: 333233
> Compacted row mean size: 662
>
> Column Family: ColumnFamily6
> SSTable count: 0
> Space used (live): 0
> Space used (total): 0
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 0
> Compacted row maximum size: 0
> Compacted row mean size: 0
>
> Column Family: ColumnFamily7
> SSTable count: 12
> Space used (live): 6031951162
> Space used (total): 6031951162
> Memtable Columns Count: 480647
> Memtable Data Size: 29424092
> Memtable Switch Count: 43
> Read Count: 7506944
> Read Latency: 0.761 ms.
> Write Count: 3022099
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 239
> Compacted row maximum size: 93925
> Compacted row mean size: 2737
>
> Column Family: ColumnFamily8
> SSTable count: 1
> Space used (live): 241104
> Space used (total): 241104
> Memtable Columns Count: 2
> Memtable Data Size: 94
> Memtable Switch Count: 3
> Read Count: 685
> Read Latency: NaN ms.
> Write Count: 143
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 200000
> Key cache size: 254
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 254
> Compacted row maximum size: 479
> Compacted row mean size: 305
>
> Column Family: ColumnFamily9
> SSTable count: 0
> Space used (live): 0
> Space used (total): 0
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 200000
> Key cache size: 0
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 0
> Compacted row maximum size: 0
> Compacted row mean size: 0
>
> Column Family: ColumnFamily10
> SSTable count: 3
> Space used (live): 22874666
> Space used (total): 22874666
> Memtable Columns Count: 11886
> Memtable Data Size: 519166
> Memtable Switch Count: 3
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 73970
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 161
> Compacted row maximum size: 103903
> Compacted row mean size: 11504
>
> Column Family: ColumnFamily11
> SSTable count: 3
> Space used (live): 6533310
> Space used (total): 6533310
> Memtable Columns Count: 8
> Memtable Data Size: 334
> Memtable Switch Count: 3
> Read Count: 5431
> Read Latency: NaN ms.
> Write Count: 197
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 200000
> Key cache size: 653
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 262
> Compacted row maximum size: 17149
> Compacted row mean size: 359
>
> Column Family: ColumnFamily12
> SSTable count: 1
> Space used (live): 5504796
> Space used (total): 5504796
> Memtable Columns Count: 722
> Memtable Data Size: 5495928
> Memtable Switch Count: 3
> Read Count: 6413
> Read Latency: NaN ms.
> Write Count: 5784
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 167
> Compacted row maximum size: 5498089
> Compacted row mean size: 261974
>
> Column Family: ColumnFamily13
> SSTable count: 3
> Space used (live): 18208017
> Space used (total): 18208017
> Memtable Columns Count: 40
> Memtable Data Size: 1730
> Memtable Switch Count: 3
> Read Count: 4617
> Read Latency: NaN ms.
> Write Count: 132
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 200000
> Key cache size: 1604
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 512
> Compacted row maximum size: 1776
> Compacted row mean size: 683
>
> Column Family: ColumnFamily14
> SSTable count: 9
> Space used (live): 1631088415
> Space used (total): 1631088415
> Memtable Columns Count: 355843
> Memtable Data Size: 15750980
> Memtable Switch Count: 1142
> Read Count: 37874988
> Read Latency: 0.830 ms.
> Write Count: 96285506
> Write Latency: 0.032 ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 599
> Compacted row maximum size: 1124
> Compacted row mean size: 861
>
> Column Family: ColumnFamily15
> SSTable count: 2
> Space used (live): 69991756
> Space used (total): 69991756
> Memtable Columns Count: 315483
> Memtable Data Size: 14827701
> Memtable Switch Count: 65
> Read Count: 5653028
> Read Latency: 6.888 ms.
> Write Count: 47666
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 160
> Compacted row maximum size: 5683855
> Compacted row mean size: 35031
>
> Column Family: ColumnFamily16
> SSTable count: 1
> Space used (live): 44605875
> Space used (total): 44605875
> Memtable Columns Count: 20273
> Memtable Data Size: 645383
> Memtable Switch Count: 3
> Read Count: 373155
> Read Latency: 0.058 ms.
> Write Count: 199847
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 209
> Compacted row maximum size: 82030
> Compacted row mean size: 617
>
> Column Family: ColumnFamily17
> SSTable count: 1
> Space used (live): 5609
> Space used (total): 5609
> Memtable Columns Count: 166
> Memtable Data Size: 2075
> Memtable Switch Count: 3
> Read Count: 17630
> Read Latency: NaN ms.
> Write Count: 542
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 522
> Compacted row maximum size: 563
> Compacted row mean size: 545
>
> Column Family: ColumnFamily18
> SSTable count: 3
> Space used (live): 20705514
> Space used (total): 20705514
> Memtable Columns Count: 164
> Memtable Data Size: 5892
> Memtable Switch Count: 3
> Read Count: 5605
> Read Latency: NaN ms.
> Write Count: 318
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 200000
> Key cache size: 2314
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 521
> Compacted row maximum size: 1239
> Compacted row mean size: 604
>
> Column Family: ColumnFamily19
> SSTable count: 1
> Space used (live): 29516821
> Space used (total): 29516821
> Memtable Columns Count: 8402
> Memtable Data Size: 268574
> Memtable Switch Count: 3
> Read Count: 7587671
> Read Latency: 0.193 ms.
> Write Count: 72597
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 186
> Compacted row maximum size: 18443
> Compacted row mean size: 270
>
> Column Family: ColumnFamily20
> SSTable count: 1
> Space used (live): 711618
> Space used (total): 711618
> Memtable Columns Count: 16
> Memtable Data Size: 496
> Memtable Switch Count: 3
> Read Count: 454
> Read Latency: NaN ms.
> Write Count: 97
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 200000
> Key cache size: 385
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 238
> Compacted row maximum size: 426
> Compacted row mean size: 303
>
> Column Family: ColumnFamily21
> SSTable count: 6
> Space used (live): 499913576
> Space used (total): 499913576
> Memtable Columns Count: 79019
> Memtable Data Size: 4820159
> Memtable Switch Count: 11
> Read Count: 65780
> Read Latency: NaN ms.
> Write Count: 6129833
> Write Latency: 0.036 ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 278
> Compacted row maximum size: 288
> Compacted row mean size: 284
>
> Column Family: ColumnFamily22
> SSTable count: 3
> Space used (live): 378024
> Space used (total): 378024
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 724
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 0
> Compacted row maximum size: 0
> Compacted row mean size: 0
>
> Column Family: ColumnFamily23
> SSTable count: 3
> Space used (live): 11425
> Space used (total): 11425
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 1
> Read Count: 578
> Read Latency: NaN ms.
> Write Count: 1
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 0
> Compacted row maximum size: 0
> Compacted row mean size: 0
>
> Column Family: ColumnFamily24
> SSTable count: 2
> Space used (live): 20577518
> Space used (total): 20577518
> Memtable Columns Count: 306
> Memtable Data Size: 12731
> Memtable Switch Count: 3
> Read Count: 18
> Read Latency: NaN ms.
> Write Count: 3234
> Write Latency: 0.013 ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 465334
> Compacted row maximum size: 4552398
> Compacted row mean size: 2815873
>
> Column Family: ColumnFamily25
> SSTable count: 1
> Space used (live): 2888
> Space used (total): 2888
> Memtable Columns Count: 2
> Memtable Data Size: 68
> Memtable Switch Count: 3
> Read Count: 1385944
> Read Latency: NaN ms.
> Write Count: 19
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 223
> Compacted row maximum size: 234
> Compacted row mean size: 230
>
> Column Family: ColumnFamily26
> SSTable count: 0
> Space used (live): 0
> Space used (total): 0
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 200000
> Key cache size: 0
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 0
> Compacted row maximum size: 0
> Compacted row mean size: 0
>
> ----------------
> Keyspace: system
> Read Count: 14
> Read Latency: 123.1865 ms.
> Write Count: 6
> Write Latency: 0.11666666666666667 ms.
> Pending Tasks: 0
> Column Family: LocationInfo
> SSTable count: 1
> Space used (live): 2871
> Space used (total): 2871
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 2
> Read Count: 2
> Read Latency: NaN ms.
> Write Count: 6
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 1
> Key cache size: 1
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 203
> Compacted row maximum size: 554
> Compacted row mean size: 378
>
> Column Family: HintsColumnFamily
> SSTable count: 2
> Space used (live): 4876356
> Space used (total): 4876356
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 12
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Key cache capacity: 2
> Key cache size: 2
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 0
> Compacted row maximum size: 0
> Compacted row mean size: 0
>
>
>
> On 02/28/2012 06:51 PM, Dan Retzlaff wrote:
>> Hi Stefan. Can you share the output of nodetool cfstats?
>>
>> On Tue, Feb 28, 2012 at 1:50 AM, Stefan Reek
>> <stefan@unitedgames.com <ma...@unitedgames.com>> wrote:
>>
>> Hi All,
>>
>> We are running a 3-node cluster with Cassandra 0.6.13.
>> We are in the process of upgrading to 1.x, but can't do so
>> for a while because we can't take the cluster offline.
>> Until now 0.6.13 has run without problems, but lately we are
>> getting some performance issues.
>> We are getting timeouts on both reads and writes.
>> The cluster contains a lot of "old" data which is only read
>> very occasionally and not written to at all.
>> Is it possible for this data to negatively impact the
>> performance of the cluster?
>> If so, would it help to move this data to another
>> columnfamily/keyspace?
>>
>> Cheers,
>>
>> Stefan
>>
>>
>
>
Re: Impact of old data on performance
Posted by Stefan Reek <st...@unitedgames.com>.
Thanks for the help Dan.
Is there anyone else that can shed some light on this issue?
For example, what are the options for reducing the time taken by the
mark/remark pauses?
cheers,
Stefan
On 03/01/2012 06:51 PM, Dan Retzlaff wrote:
> I've never had to deal with GC tuning since our cluster has relatively
> few (but large) columns. So I'll leave further comment to others, but
> it sounds like you're on the right track.
>
> On Thu, Mar 1, 2012 at 2:13 AM, Stefan Reek <stefan@unitedgames.com
> <ma...@unitedgames.com>> wrote:
>
> Swap is disabled on the machines, so I'm sure the JVM is not
> swapping out.
> Cassandra is the only process running on those machines, so
> network contention shouldn't be an issues either.
> Which leaves Garbage collection.
>
> In the logs I can see that the Garbage Collection takes a long
> time, for example:
>
> GCInspector.java (line 130) GC for ConcurrentMarkSweep: 384113 ms,
> -84703136 reclaimed leaving 1018203472 used; max is 17208836096
> <tel:17208836096>
>
> But I configured the Garbage Collector to use the
> ConcurrentMarkSweep algorithm. As far as I understand this means
> that there will only be two pauses during the compaction, the mark
> and the remark stage.
>
> As I also have extra garbage collection logging enabled on one
> node, I checked to see if the mark/remark times sometimes exceed
> the 10 second RPC timeout and I found some logging statements like
> this:
>
> 422208.422: [CMS-concurrent-mark: 0.434/20.004 secs] [Times:
> user=13.40 sys=0.75, real=20.00 secs]
>
> Does this mean that the mark session took over 10 seconds and all
> other Cassandra threads were stopped during this
> time? If so, that would explain why I sometimes get timeouts on
> requests and also see Dropped Messages in my logs.
>
> Cheers,
>
> Stefan
>
>
>
> On 02/29/2012 07:48 PM, Dan Retzlaff wrote:
>> First to be clear, I'm not an expert but I suggested "cfstats"
>> because it can show unhealthy signs. That said, yours looks okay
>> to me: few live SSTables per column family, reasonable quantity
>> of data... The next things I'd verify are that (1) the JVM isn't
>> swapping out during these periods of bad performance, (2) GC
>> completes in less than your RPC timeout (check the logs), and (3)
>> your internode network connection isn't degraded by another
>> application.
>>
>> The fact that *writes* are timing out is interesting, since
>> existing data should have little to no effect on those.
>>
>> BTW, since ColumnFamily7 sees most of your action, you might
>> consider whether a key/row cache is appropriate. By my count it
>> has 2M rows and 7M reads since startup. But this fine-tuning is
>> probably a distraction from whatever's causing your timeouts.
>>
>> Hope that helps,
>> Dan
>>
>> On Wed, Feb 29, 2012 at 5:14 AM, Stefan Reek
>> <stefan@unitedgames.com <ma...@unitedgames.com>> wrote:
>>
>> Hi Dan,
>>
>> Thanks for answering.
>> I included the output of cfstats below.
>> I hope you can say something about our problems with it.
>>
>> cheers,
>>
>> Stefan
>>
>> Keyspace: Keyspace1
>> Read Count: 60703419
>> Read Latency: 1.1790332096286043 ms.
>> Write Count: 105871791
>> Write Latency: 0.019847457393065166 ms.
>> Pending Tasks: 0
>> Column Family: ColumnFamily1
>> SSTable count: 0
>> Space used (live): 0
>> Space used (total): 0
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 0
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily2
>> SSTable count: 2
>> Space used (live): 482828
>> Space used (total): 482828
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 3
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 9
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 0
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 258
>> Compacted row maximum size: 275
>> Compacted row mean size: 266
>>
>> Column Family: ColumnFamily3
>> SSTable count: 0
>> Space used (live): 0
>> Space used (total): 0
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 0
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily4
>> SSTable count: 1
>> Space used (live): 19793722
>> Space used (total): 19793722
>> Memtable Columns Count: 306
>> Memtable Data Size: 12731
>> Memtable Switch Count: 3
>> Read Count: 2309
>> Read Latency: NaN ms.
>> Write Count: 3234
>> Write Latency: 0.021 ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 465813
>> Compacted row maximum size: 4784014
>> Compacted row mean size: 3295196
>>
>> Column Family: ColumnFamily5
>> SSTable count: 6
>> Space used (live): 364775507
>> Space used (total): 364775507
>> Memtable Columns Count: 14193
>> Memtable Data Size: 552090
>> Memtable Switch Count: 3
>> Read Count: 211451
>> Read Latency: NaN ms.
>> Write Count: 26567
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 187
>> Compacted row maximum size: 333233
>> Compacted row mean size: 662
>>
>> Column Family: ColumnFamily6
>> SSTable count: 0
>> Space used (live): 0
>> Space used (total): 0
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily7
>> SSTable count: 12
>> Space used (live): 6031951162
>> Space used (total): 6031951162
>> Memtable Columns Count: 480647
>> Memtable Data Size: 29424092
>> Memtable Switch Count: 43
>> Read Count: 7506944
>> Read Latency: 0.761 ms.
>> Write Count: 3022099
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 239
>> Compacted row maximum size: 93925
>> Compacted row mean size: 2737
>>
>> Column Family: ColumnFamily8
>> SSTable count: 1
>> Space used (live): 241104
>> Space used (total): 241104
>> Memtable Columns Count: 2
>> Memtable Data Size: 94
>> Memtable Switch Count: 3
>> Read Count: 685
>> Read Latency: NaN ms.
>> Write Count: 143
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 254
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 254
>> Compacted row maximum size: 479
>> Compacted row mean size: 305
>>
>> Column Family: ColumnFamily9
>> SSTable count: 0
>> Space used (live): 0
>> Space used (total): 0
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 0
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily10
>> SSTable count: 3
>> Space used (live): 22874666
>> Space used (total): 22874666
>> Memtable Columns Count: 11886
>> Memtable Data Size: 519166
>> Memtable Switch Count: 3
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 73970
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 161
>> Compacted row maximum size: 103903
>> Compacted row mean size: 11504
>>
>> Column Family: ColumnFamily11
>> SSTable count: 3
>> Space used (live): 6533310
>> Space used (total): 6533310
>> Memtable Columns Count: 8
>> Memtable Data Size: 334
>> Memtable Switch Count: 3
>> Read Count: 5431
>> Read Latency: NaN ms.
>> Write Count: 197
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 653
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 262
>> Compacted row maximum size: 17149
>> Compacted row mean size: 359
>>
>> Column Family: ColumnFamily12
>> SSTable count: 1
>> Space used (live): 5504796
>> Space used (total): 5504796
>> Memtable Columns Count: 722
>> Memtable Data Size: 5495928
>> Memtable Switch Count: 3
>> Read Count: 6413
>> Read Latency: NaN ms.
>> Write Count: 5784
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 167
>> Compacted row maximum size: 5498089
>> Compacted row mean size: 261974
>>
>> Column Family: ColumnFamily13
>> SSTable count: 3
>> Space used (live): 18208017
>> Space used (total): 18208017
>> Memtable Columns Count: 40
>> Memtable Data Size: 1730
>> Memtable Switch Count: 3
>> Read Count: 4617
>> Read Latency: NaN ms.
>> Write Count: 132
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 1604
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 512
>> Compacted row maximum size: 1776
>> Compacted row mean size: 683
>>
>> Column Family: ColumnFamily14
>> SSTable count: 9
>> Space used (live): 1631088415
>> Space used (total): 1631088415
>> Memtable Columns Count: 355843
>> Memtable Data Size: 15750980
>> Memtable Switch Count: 1142
>> Read Count: 37874988
>> Read Latency: 0.830 ms.
>> Write Count: 96285506
>> Write Latency: 0.032 ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 599
>> Compacted row maximum size: 1124
>> Compacted row mean size: 861
>>
>> Column Family: ColumnFamily15
>> SSTable count: 2
>> Space used (live): 69991756
>> Space used (total): 69991756
>> Memtable Columns Count: 315483
>> Memtable Data Size: 14827701
>> Memtable Switch Count: 65
>> Read Count: 5653028
>> Read Latency: 6.888 ms.
>> Write Count: 47666
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 160
>> Compacted row maximum size: 5683855
>> Compacted row mean size: 35031
>>
>> Column Family: ColumnFamily16
>> SSTable count: 1
>> Space used (live): 44605875
>> Space used (total): 44605875
>> Memtable Columns Count: 20273
>> Memtable Data Size: 645383
>> Memtable Switch Count: 3
>> Read Count: 373155
>> Read Latency: 0.058 ms.
>> Write Count: 199847
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 209
>> Compacted row maximum size: 82030
>> Compacted row mean size: 617
>>
>> Column Family: ColumnFamily17
>> SSTable count: 1
>> Space used (live): 5609
>> Space used (total): 5609
>> Memtable Columns Count: 166
>> Memtable Data Size: 2075
>> Memtable Switch Count: 3
>> Read Count: 17630
>> Read Latency: NaN ms.
>> Write Count: 542
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 522
>> Compacted row maximum size: 563
>> Compacted row mean size: 545
>>
>> Column Family: ColumnFamily18
>> SSTable count: 3
>> Space used (live): 20705514
>> Space used (total): 20705514
>> Memtable Columns Count: 164
>> Memtable Data Size: 5892
>> Memtable Switch Count: 3
>> Read Count: 5605
>> Read Latency: NaN ms.
>> Write Count: 318
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 2314
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 521
>> Compacted row maximum size: 1239
>> Compacted row mean size: 604
>>
>> Column Family: ColumnFamily19
>> SSTable count: 1
>> Space used (live): 29516821
>> Space used (total): 29516821
>> Memtable Columns Count: 8402
>> Memtable Data Size: 268574
>> Memtable Switch Count: 3
>> Read Count: 7587671
>> Read Latency: 0.193 ms.
>> Write Count: 72597
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 186
>> Compacted row maximum size: 18443
>> Compacted row mean size: 270
>>
>> Column Family: ColumnFamily20
>> SSTable count: 1
>> Space used (live): 711618
>> Space used (total): 711618
>> Memtable Columns Count: 16
>> Memtable Data Size: 496
>> Memtable Switch Count: 3
>> Read Count: 454
>> Read Latency: NaN ms.
>> Write Count: 97
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 385
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 238
>> Compacted row maximum size: 426
>> Compacted row mean size: 303
>>
>> Column Family: ColumnFamily21
>> SSTable count: 6
>> Space used (live): 499913576
>> Space used (total): 499913576
>> Memtable Columns Count: 79019
>> Memtable Data Size: 4820159
>> Memtable Switch Count: 11
>> Read Count: 65780
>> Read Latency: NaN ms.
>> Write Count: 6129833
>> Write Latency: 0.036 ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 278
>> Compacted row maximum size: 288
>> Compacted row mean size: 284
>>
>> Column Family: ColumnFamily22
>> SSTable count: 3
>> Space used (live): 378024
>> Space used (total): 378024
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 724
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily23
>> SSTable count: 3
>> Space used (live): 11425
>> Space used (total): 11425
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 1
>> Read Count: 578
>> Read Latency: NaN ms.
>> Write Count: 1
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily24
>> SSTable count: 2
>> Space used (live): 20577518
>> Space used (total): 20577518
>> Memtable Columns Count: 306
>> Memtable Data Size: 12731
>> Memtable Switch Count: 3
>> Read Count: 18
>> Read Latency: NaN ms.
>> Write Count: 3234
>> Write Latency: 0.013 ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 465334
>> Compacted row maximum size: 4552398
>> Compacted row mean size: 2815873
>>
>> Column Family: ColumnFamily25
>> SSTable count: 1
>> Space used (live): 2888
>> Space used (total): 2888
>> Memtable Columns Count: 2
>> Memtable Data Size: 68
>> Memtable Switch Count: 3
>> Read Count: 1385944
>> Read Latency: NaN ms.
>> Write Count: 19
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 223
>> Compacted row maximum size: 234
>> Compacted row mean size: 230
>>
>> Column Family: ColumnFamily26
>> SSTable count: 0
>> Space used (live): 0
>> Space used (total): 0
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 0
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> ----------------
>> Keyspace: system
>> Read Count: 14
>> Read Latency: 123.1865 ms.
>> Write Count: 6
>> Write Latency: 0.11666666666666667 ms.
>> Pending Tasks: 0
>> Column Family: LocationInfo
>> SSTable count: 1
>> Space used (live): 2871
>> Space used (total): 2871
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 2
>> Read Count: 2
>> Read Latency: NaN ms.
>> Write Count: 6
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 1
>> Key cache size: 1
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 203
>> Compacted row maximum size: 554
>> Compacted row mean size: 378
>>
>> Column Family: HintsColumnFamily
>> SSTable count: 2
>> Space used (live): 4876356
>> Space used (total): 4876356
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 12
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 2
>> Key cache size: 2
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>>
>>
>> On 02/28/2012 06:51 PM, Dan Retzlaff wrote:
>>> Hi Stefan. Can you share the output of nodetool cfstats?
>>>
>>> On Tue, Feb 28, 2012 at 1:50 AM, Stefan Reek
>>> <stefan@unitedgames.com <ma...@unitedgames.com>> wrote:
>>>
>>> Hi All,
>>>
>>> We are running a 3-node cluster with Cassandra 0.6.13.
>>> We are in the process of upgrading to 1.x, but can't do
>>> so for a while because we can't take the cluster offline.
>>> Until now 0.6.13 has run without problems, but lately we
>>> are getting some performance issues.
>>> We are getting timeouts on both reads and writes.
>>> The cluster contains a lot of "old" data which is only
>>> read very occasionally and not written to at all.
>>> Is it possible for this data to negatively impact the
>>> performance of the cluster?
>>> If so, would it help to move this data to another
>>> columnfamily/keyspace?
>>>
>>> Cheers,
>>>
>>> Stefan
>>>
>>>
>>
>>
>
>
Re: Impact of old data on performance
Posted by Dan Retzlaff <dr...@gmail.com>.
I've never had to deal with GC tuning since our cluster has relatively few
(but large) columns. So I'll leave further comment to others, but it sounds
like you're on the right track.
On Thu, Mar 1, 2012 at 2:13 AM, Stefan Reek <st...@unitedgames.com> wrote:
> **
> Swap is disabled on the machines, so I'm sure the JVM is not swapping out.
> Cassandra is the only process running on those machines, so network
> contention shouldn't be an issues either.
> Which leaves Garbage collection.
>
> In the logs I can see that the Garbage Collection takes a long time, for
> example:
>
> GCInspector.java (line 130) GC for ConcurrentMarkSweep: 384113 ms,
> -84703136 reclaimed leaving 1018203472 used; max is 17208836096
>
> But I configured the Garbage Collector to use the ConcurrentMarkSweep
> algorithm. As far as I understand this means that there will only be two
> pauses during the compaction, the mark and the remark stage.
>
> As I also have extra garbage collection logging enabled on one node, I
> checked to see if the mark/remark times sometimes exceed the 10 second RPC
> timeout and I found some logging statements like this:
>
> 422208.422: [CMS-concurrent-mark: 0.434/20.004 secs] [Times: user=13.40
> sys=0.75, real=20.00 secs]
>
> Does this mean that the mark session took over 10 seconds and all other
> Cassandra threads were stopped during this
> time? If so, that would explain why I sometimes get timeouts on requests
> and also see Dropped Messages in my logs.
>
> Cheers,
>
> Stefan
>
>
>
> On 02/29/2012 07:48 PM, Dan Retzlaff wrote:
>
> First to be clear, I'm not an expert but I suggested "cfstats" because it
> can show unhealthy signs. That said, yours looks okay to me: few live
> SSTables per column family, reasonable quantity of data... The next things
> I'd verify are that (1) the JVM isn't swapping out during these periods of
> bad performance, (2) GC completes in less than your RPC timeout (check the
> logs), and (3) your internode network connection isn't degraded by another
> application.
>
> The fact that *writes* are timing out is interesting, since existing
> data should have little to no effect on those.
>
> BTW, since ColumnFamily7 sees most of your action, you might consider
> whether a key/row cache is appropriate. By my count it has 2M rows and 7M
> reads since startup. But this fine-tuning is probably a distraction from
> whatever's causing your timeouts.
>
> Hope that helps,
> Dan
>
> On Wed, Feb 29, 2012 at 5:14 AM, Stefan Reek <st...@unitedgames.com>wrote:
>
>> Hi Dan,
>>
>> Thanks for answering.
>> I included the output of cfstats below.
>> I hope you can say something about our problems with it.
>>
>> cheers,
>>
>> Stefan
>>
>> Keyspace: Keyspace1
>> Read Count: 60703419
>> Read Latency: 1.1790332096286043 ms.
>> Write Count: 105871791
>> Write Latency: 0.019847457393065166 ms.
>> Pending Tasks: 0
>> Column Family: ColumnFamily1
>> SSTable count: 0
>> Space used (live): 0
>> Space used (total): 0
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 0
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily2
>> SSTable count: 2
>> Space used (live): 482828
>> Space used (total): 482828
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 3
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 9
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 0
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 258
>> Compacted row maximum size: 275
>> Compacted row mean size: 266
>>
>> Column Family: ColumnFamily3
>> SSTable count: 0
>> Space used (live): 0
>> Space used (total): 0
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 0
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily4
>> SSTable count: 1
>> Space used (live): 19793722
>> Space used (total): 19793722
>> Memtable Columns Count: 306
>> Memtable Data Size: 12731
>> Memtable Switch Count: 3
>> Read Count: 2309
>> Read Latency: NaN ms.
>> Write Count: 3234
>> Write Latency: 0.021 ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 465813
>> Compacted row maximum size: 4784014
>> Compacted row mean size: 3295196
>>
>> Column Family: ColumnFamily5
>> SSTable count: 6
>> Space used (live): 364775507
>> Space used (total): 364775507
>> Memtable Columns Count: 14193
>> Memtable Data Size: 552090
>> Memtable Switch Count: 3
>> Read Count: 211451
>> Read Latency: NaN ms.
>> Write Count: 26567
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 187
>> Compacted row maximum size: 333233
>> Compacted row mean size: 662
>>
>> Column Family: ColumnFamily6
>> SSTable count: 0
>> Space used (live): 0
>> Space used (total): 0
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily7
>> SSTable count: 12
>> Space used (live): 6031951162
>> Space used (total): 6031951162
>> Memtable Columns Count: 480647
>> Memtable Data Size: 29424092
>> Memtable Switch Count: 43
>> Read Count: 7506944
>> Read Latency: 0.761 ms.
>> Write Count: 3022099
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 239
>> Compacted row maximum size: 93925
>> Compacted row mean size: 2737
>>
>> Column Family: ColumnFamily8
>> SSTable count: 1
>> Space used (live): 241104
>> Space used (total): 241104
>> Memtable Columns Count: 2
>> Memtable Data Size: 94
>> Memtable Switch Count: 3
>> Read Count: 685
>> Read Latency: NaN ms.
>> Write Count: 143
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 254
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 254
>> Compacted row maximum size: 479
>> Compacted row mean size: 305
>>
>> Column Family: ColumnFamily9
>> SSTable count: 0
>> Space used (live): 0
>> Space used (total): 0
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 0
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily10
>> SSTable count: 3
>> Space used (live): 22874666
>> Space used (total): 22874666
>> Memtable Columns Count: 11886
>> Memtable Data Size: 519166
>> Memtable Switch Count: 3
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 73970
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 161
>> Compacted row maximum size: 103903
>> Compacted row mean size: 11504
>>
>> Column Family: ColumnFamily11
>> SSTable count: 3
>> Space used (live): 6533310
>> Space used (total): 6533310
>> Memtable Columns Count: 8
>> Memtable Data Size: 334
>> Memtable Switch Count: 3
>> Read Count: 5431
>> Read Latency: NaN ms.
>> Write Count: 197
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 653
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 262
>> Compacted row maximum size: 17149
>> Compacted row mean size: 359
>>
>> Column Family: ColumnFamily12
>> SSTable count: 1
>> Space used (live): 5504796
>> Space used (total): 5504796
>> Memtable Columns Count: 722
>> Memtable Data Size: 5495928
>> Memtable Switch Count: 3
>> Read Count: 6413
>> Read Latency: NaN ms.
>> Write Count: 5784
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 167
>> Compacted row maximum size: 5498089
>> Compacted row mean size: 261974
>>
>> Column Family: ColumnFamily13
>> SSTable count: 3
>> Space used (live): 18208017
>> Space used (total): 18208017
>> Memtable Columns Count: 40
>> Memtable Data Size: 1730
>> Memtable Switch Count: 3
>> Read Count: 4617
>> Read Latency: NaN ms.
>> Write Count: 132
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 1604
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 512
>> Compacted row maximum size: 1776
>> Compacted row mean size: 683
>>
>> Column Family: ColumnFamily14
>> SSTable count: 9
>> Space used (live): 1631088415
>> Space used (total): 1631088415
>> Memtable Columns Count: 355843
>> Memtable Data Size: 15750980
>> Memtable Switch Count: 1142
>> Read Count: 37874988
>> Read Latency: 0.830 ms.
>> Write Count: 96285506
>> Write Latency: 0.032 ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 599
>> Compacted row maximum size: 1124
>> Compacted row mean size: 861
>>
>> Column Family: ColumnFamily15
>> SSTable count: 2
>> Space used (live): 69991756
>> Space used (total): 69991756
>> Memtable Columns Count: 315483
>> Memtable Data Size: 14827701
>> Memtable Switch Count: 65
>> Read Count: 5653028
>> Read Latency: 6.888 ms.
>> Write Count: 47666
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 160
>> Compacted row maximum size: 5683855
>> Compacted row mean size: 35031
>>
>> Column Family: ColumnFamily16
>> SSTable count: 1
>> Space used (live): 44605875
>> Space used (total): 44605875
>> Memtable Columns Count: 20273
>> Memtable Data Size: 645383
>> Memtable Switch Count: 3
>> Read Count: 373155
>> Read Latency: 0.058 ms.
>> Write Count: 199847
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 209
>> Compacted row maximum size: 82030
>> Compacted row mean size: 617
>>
>> Column Family: ColumnFamily17
>> SSTable count: 1
>> Space used (live): 5609
>> Space used (total): 5609
>> Memtable Columns Count: 166
>> Memtable Data Size: 2075
>> Memtable Switch Count: 3
>> Read Count: 17630
>> Read Latency: NaN ms.
>> Write Count: 542
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 522
>> Compacted row maximum size: 563
>> Compacted row mean size: 545
>>
>> Column Family: ColumnFamily18
>> SSTable count: 3
>> Space used (live): 20705514
>> Space used (total): 20705514
>> Memtable Columns Count: 164
>> Memtable Data Size: 5892
>> Memtable Switch Count: 3
>> Read Count: 5605
>> Read Latency: NaN ms.
>> Write Count: 318
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 2314
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 521
>> Compacted row maximum size: 1239
>> Compacted row mean size: 604
>>
>> Column Family: ColumnFamily19
>> SSTable count: 1
>> Space used (live): 29516821
>> Space used (total): 29516821
>> Memtable Columns Count: 8402
>> Memtable Data Size: 268574
>> Memtable Switch Count: 3
>> Read Count: 7587671
>> Read Latency: 0.193 ms.
>> Write Count: 72597
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 186
>> Compacted row maximum size: 18443
>> Compacted row mean size: 270
>>
>> Column Family: ColumnFamily20
>> SSTable count: 1
>> Space used (live): 711618
>> Space used (total): 711618
>> Memtable Columns Count: 16
>> Memtable Data Size: 496
>> Memtable Switch Count: 3
>> Read Count: 454
>> Read Latency: NaN ms.
>> Write Count: 97
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 385
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 238
>> Compacted row maximum size: 426
>> Compacted row mean size: 303
>>
>> Column Family: ColumnFamily21
>> SSTable count: 6
>> Space used (live): 499913576
>> Space used (total): 499913576
>> Memtable Columns Count: 79019
>> Memtable Data Size: 4820159
>> Memtable Switch Count: 11
>> Read Count: 65780
>> Read Latency: NaN ms.
>> Write Count: 6129833
>> Write Latency: 0.036 ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 278
>> Compacted row maximum size: 288
>> Compacted row mean size: 284
>>
>> Column Family: ColumnFamily22
>> SSTable count: 3
>> Space used (live): 378024
>> Space used (total): 378024
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 724
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily23
>> SSTable count: 3
>> Space used (live): 11425
>> Space used (total): 11425
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 1
>> Read Count: 578
>> Read Latency: NaN ms.
>> Write Count: 1
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> Column Family: ColumnFamily24
>> SSTable count: 2
>> Space used (live): 20577518
>> Space used (total): 20577518
>> Memtable Columns Count: 306
>> Memtable Data Size: 12731
>> Memtable Switch Count: 3
>> Read Count: 18
>> Read Latency: NaN ms.
>> Write Count: 3234
>> Write Latency: 0.013 ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 465334
>> Compacted row maximum size: 4552398
>> Compacted row mean size: 2815873
>>
>> Column Family: ColumnFamily25
>> SSTable count: 1
>> Space used (live): 2888
>> Space used (total): 2888
>> Memtable Columns Count: 2
>> Memtable Data Size: 68
>> Memtable Switch Count: 3
>> Read Count: 1385944
>> Read Latency: NaN ms.
>> Write Count: 19
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache: disabled
>> Row cache: disabled
>> Compacted row minimum size: 223
>> Compacted row maximum size: 234
>> Compacted row mean size: 230
>>
>> Column Family: ColumnFamily26
>> SSTable count: 0
>> Space used (live): 0
>> Space used (total): 0
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 0
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 200000
>> Key cache size: 0
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>> ----------------
>> Keyspace: system
>> Read Count: 14
>> Read Latency: 123.1865 ms.
>> Write Count: 6
>> Write Latency: 0.11666666666666667 ms.
>> Pending Tasks: 0
>> Column Family: LocationInfo
>> SSTable count: 1
>> Space used (live): 2871
>> Space used (total): 2871
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 2
>> Read Count: 2
>> Read Latency: NaN ms.
>> Write Count: 6
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 1
>> Key cache size: 1
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 203
>> Compacted row maximum size: 554
>> Compacted row mean size: 378
>>
>> Column Family: HintsColumnFamily
>> SSTable count: 2
>> Space used (live): 4876356
>> Space used (total): 4876356
>> Memtable Columns Count: 0
>> Memtable Data Size: 0
>> Memtable Switch Count: 0
>> Read Count: 12
>> Read Latency: NaN ms.
>> Write Count: 0
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Key cache capacity: 2
>> Key cache size: 2
>> Key cache hit rate: NaN
>> Row cache: disabled
>> Compacted row minimum size: 0
>> Compacted row maximum size: 0
>> Compacted row mean size: 0
>>
>>
>>
>> On 02/28/2012 06:51 PM, Dan Retzlaff wrote:
>>
>> Hi Stefan. Can you share the output of nodetool cfstats?
>>
>> On Tue, Feb 28, 2012 at 1:50 AM, Stefan Reek <st...@unitedgames.com>wrote:
>>
>>> Hi All,
>>>
>>> We are running a 3-node cluster with Cassandra 0.6.13.
>>> We are in the process of upgrading to 1.x, but can't do so for a while
>>> because we can't take the cluster offline.
>>> Until now 0.6.13 has run without problems, but lately we are getting
>>> some performance issues.
>>> We are getting timeouts on both reads and writes.
>>> The cluster contains a lot of "old" data which is only read very
>>> occasionally and not written to at all.
>>> Is it possible for this data to negatively impact the performance of the
>>> cluster?
>>> If so, would it help to move this data to another columnfamily/keyspace?
>>>
>>> Cheers,
>>>
>>> Stefan
>>>
>>
>>
>>
>
>