You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by André Cruz <an...@co.sapo.pt> on 2013/01/03 16:25:04 UTC
Re: Row cache and counters
Does anyone see anything wrong in these settings? Anything to account for a 8s timeout during a counter increment?
Thanks,
André
On 31/12/2012, at 14:35, André Cruz <an...@co.sapo.pt> wrote:
> On Dec 29, 2012, at 8:53 PM, Mohit Anchlia <mo...@gmail.com> wrote:
>
>> Can you post gc settings? Also check logs and see what it says
>
> These are the relevant jam settings:
>
> -home /usr/lib/jvm/j2re1.6-oracle/bin/../
> -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
> -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42
> -Xms8049M
> -Xmx8049M
> -Xmn800M
> -XX:+HeapDumpOnOutOfMemoryError
> -Xss196k
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly
> -Djava.net.preferIPv4Stack=true
>
> I have 3 servers (32GB RAM), with a RF of 3. I searched all of them for log messages related to a period when I had timeouts (19h20-19h30), only one of them showed messages for that timeframe, and none seem related with that CF:
>
> INFO [MemoryMeter:1] 2012-12-27 19:20:17,858 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='RevisionLog') liveRatio is 4.318314007200407 (just-counted was 4.318314007200407). calculation took 350ms for 8623 columns
> INFO [MemoryMeter:1] 2012-12-27 19:23:37,148 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='LinkPathsExist') liveRatio is 25.87012987012987 (just-counted was 25.87012987012987). calculation took 0ms for 10 columns
> INFO [MemoryMeter:1] 2012-12-27 19:28:32,736 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='BlockMetadata.BlockMetadata_used_idx') liveRatio is 1.7176206177506523 (just-counted was 1.7176206177506523). calculation took 62ms for 12941 columns
> INFO [MemoryMeter:1] 2012-12-27 19:30:12,752 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='Namespace') liveRatio is 20.097473571044617 (just-counted was 20.097473571044617). calculation took 10ms for 288 columns
> INFO [MemoryMeter:1] 2012-12-27 19:30:28,421 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='NamespaceDir') liveRatio is 4.801010311533358 (just-counted was 4.801010311533358). calculation took 96ms for 3138 columns
>
>
>> Also post how many writes and reads along with avg row size
>
> All rows have 3-6 counters. As for writes and reads:
>
> Column Family: UserQuotas
> SSTable count: 3
> Space used (live): 2609839
> Space used (total): 2609839
> Number of Keys (estimate): 22016
> Memtable Columns Count: 142705
> Memtable Data Size: 768117
> Memtable Switch Count: 26
> Read Count: 822203
> Read Latency: 0.305 ms.
> Write Count: 1024277
> Write Latency: 0.066 ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 3
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 42584
> Compacted row minimum size: 125
> Compacted row maximum size: 770
> Compacted row mean size: 298
>
>
> Is there anything wrong with my configuration?
>
> Best regards,
> André Cruz
>
>
Re: Row cache and counters
Posted by aaron morton <aa...@thelastpickle.com>.
The first thing I look for with timeouts like that is a flush storm causing blocking in the write path (due to the internal "switch lock").
Take a look in the logs, for a number of messages such as "enqueuing CF…" and "writing cf..". Look for a pattern of enqueuing cf messages that occur immediately after writing cf messages.
Hope that helps.
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 4/01/2013, at 4:25 AM, André Cruz <an...@co.sapo.pt> wrote:
> Does anyone see anything wrong in these settings? Anything to account for a 8s timeout during a counter increment?
>
> Thanks,
> André
>
> On 31/12/2012, at 14:35, André Cruz <an...@co.sapo.pt> wrote:
>
>> On Dec 29, 2012, at 8:53 PM, Mohit Anchlia <mo...@gmail.com> wrote:
>>
>>> Can you post gc settings? Also check logs and see what it says
>>
>> These are the relevant jam settings:
>>
>> -home /usr/lib/jvm/j2re1.6-oracle/bin/../
>> -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
>> -XX:+UseThreadPriorities
>> -XX:ThreadPriorityPolicy=42
>> -Xms8049M
>> -Xmx8049M
>> -Xmn800M
>> -XX:+HeapDumpOnOutOfMemoryError
>> -Xss196k
>> -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC
>> -XX:+CMSParallelRemarkEnabled
>> -XX:SurvivorRatio=8
>> -XX:MaxTenuringThreshold=1
>> -XX:CMSInitiatingOccupancyFraction=75
>> -XX:+UseCMSInitiatingOccupancyOnly
>> -Djava.net.preferIPv4Stack=true
>>
>> I have 3 servers (32GB RAM), with a RF of 3. I searched all of them for log messages related to a period when I had timeouts (19h20-19h30), only one of them showed messages for that timeframe, and none seem related with that CF:
>>
>> INFO [MemoryMeter:1] 2012-12-27 19:20:17,858 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='RevisionLog') liveRatio is 4.318314007200407 (just-counted was 4.318314007200407). calculation took 350ms for 8623 columns
>> INFO [MemoryMeter:1] 2012-12-27 19:23:37,148 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='LinkPathsExist') liveRatio is 25.87012987012987 (just-counted was 25.87012987012987). calculation took 0ms for 10 columns
>> INFO [MemoryMeter:1] 2012-12-27 19:28:32,736 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='BlockMetadata.BlockMetadata_used_idx') liveRatio is 1.7176206177506523 (just-counted was 1.7176206177506523). calculation took 62ms for 12941 columns
>> INFO [MemoryMeter:1] 2012-12-27 19:30:12,752 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='Namespace') liveRatio is 20.097473571044617 (just-counted was 20.097473571044617). calculation took 10ms for 288 columns
>> INFO [MemoryMeter:1] 2012-12-27 19:30:28,421 Memtable.java (line 213) CFS(Keyspace='Disco', ColumnFamily='NamespaceDir') liveRatio is 4.801010311533358 (just-counted was 4.801010311533358). calculation took 96ms for 3138 columns
>>
>>
>>> Also post how many writes and reads along with avg row size
>>
>> All rows have 3-6 counters. As for writes and reads:
>>
>> Column Family: UserQuotas
>> SSTable count: 3
>> Space used (live): 2609839
>> Space used (total): 2609839
>> Number of Keys (estimate): 22016
>> Memtable Columns Count: 142705
>> Memtable Data Size: 768117
>> Memtable Switch Count: 26
>> Read Count: 822203
>> Read Latency: 0.305 ms.
>> Write Count: 1024277
>> Write Latency: 0.066 ms.
>> Pending Tasks: 0
>> Bloom Filter False Postives: 3
>> Bloom Filter False Ratio: 0.00000
>> Bloom Filter Space Used: 42584
>> Compacted row minimum size: 125
>> Compacted row maximum size: 770
>> Compacted row mean size: 298
>>
>>
>> Is there anything wrong with my configuration?
>>
>> Best regards,
>> André Cruz
>>
>>