You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Yang <te...@gmail.com> on 2011/09/16 08:20:38 UTC
how to reduce disk read? (and bloom filter performance)
after I put my cassandra cluster on heavy load (1k/s write + 1k/s
read ) for 1 day,
I accumulated about 30GB of data in sstables. I think the caches have
warmed up to their
stable state.
when I started this, I manually cat all the sstables to /dev/null , so
that they are loaded into memory
(the system mem is 32GB, so a lot of extra space ), at that time, "sar
-B " shows about 100page in requests per second.
but after 1 day, I begin to see consistently 2000 page-in requests per
sec. and the end response latency seen by
application is also higher.
so I was curious about how often my Cassandra server resorted to
reading the sstables, I looked at the JMX attributes
CFS.BloomFilterFalseRatio, it's 1.0 , BloomFilterFalsePositives, it's
2810, ReadCount is about 1 million (these are numbers after a
restart, so smaller ), so I get about 0.2% of reads going to disk. I
am wondering what is the ball park number that you see with your
production
cluster? is 0.2% a good number?
besides bloomFilter, what other approaches do we have to avoid disk
reads? ----- as data grows, apparently we can't fit all that in
memory,
increase machine count so that the data volume per-box fits into memory?
Thanks
Yang
Re: how to reduce disk read? (and bloom filter performance)
Posted by Radim Kolar <hs...@sendmail.cz>.
> Look in jconcole -> org.apache.cassandra.db -> ColumnFamilies
bloom filter false ratio is on this server 0.0018 and 0,06% reads hits
more than 1 sstable.
From cassandra point of view, it looks good.
Re: how to reduce disk read? (and bloom filter performance)
Posted by Mohit Anchlia <mo...@gmail.com>.
On Sun, Oct 16, 2011 at 2:20 AM, Radim Kolar <hs...@sendmail.cz> wrote:
> Dne 10.10.2011 18:53, Mohit Anchlia napsal(a):
>>
>> Does it mean you are not updating a row or deleting them?
>
> yes. i have 350m rows and only about 100k of them are updated.
>>
>> Can you look at JMX values of
>>
>> BloomFilter* ?
>
> i could not find this in jconsole mbeans or in jmx over http in cassandra
> 1.0
Look in jconcole -> org.apache.cassandra.db -> ColumnFamilies
>
> I believe if you want to keep disk seeks to 1 ssTable you will need to
> compact more often.
>
> cassandra have max 2 disk seeks per 1 sstable read. one seek for index,
> second seek for data. If bloom filters are working fine, in my case there is
> no need to access more than 1 sstable.
>
unless your updates to a row are spread accross multiple uncompacted ssTables.
Re: how to reduce disk read? (and bloom filter performance)
Posted by Radim Kolar <hs...@sendmail.cz>.
Dne 10.10.2011 18:53, Mohit Anchlia napsal(a):
> Does it mean you are not updating a row or deleting them?
yes. i have 350m rows and only about 100k of them are updated.
> Can you look at JMX values of
>
> BloomFilter* ?
i could not find this in jconsole mbeans or in jmx over http in
cassandra 1.0
I believe if you want to keep disk seeks to 1 ssTable you will need to
compact more often.
cassandra have max 2 disk seeks per 1 sstable read. one seek for index,
second seek for data. If bloom filters are working fine, in my case
there is no need to access more than 1 sstable.
Re: how to reduce disk read? (and bloom filter performance)
Posted by Mohit Anchlia <mo...@gmail.com>.
Does it mean you are not updating a row or deleting them? Can you look
at JMX values of
BloomFilter* ?
I don't believe bloom filter false positive % value is configurable.
Someone else might be able to throw more light on this.
I believe if you want to keep disk seeks to 1 ssTable you will need to
compact more often.
On Sun, Oct 9, 2011 at 7:09 AM, Radim Kolar <hs...@sendmail.cz> wrote:
> Dne 7.10.2011 23:16, Mohit Anchlia napsal(a):
>>
>> You'll see output like:
>>
>> Offset SSTables
>> 1 8021
>> 2 783
>>
>> Which means 783 read operations accessed 2 SSTables
>
> thank you for explaining it to me. I see this:
>
> Offset SSTables
> 1 59323
> 2 857
> 3 56
>
> it means bloom filter failure ratio over 1%. Cassandra in unit tests expects
> bloom filter false positive less than 1.05%. HBase has configurable bloom
> filters. You can choose 1% or 0.5% - it can make difference for large cache.
>
> But result is that my poor read performance should not be caused by bloom
> filters.
>
Re: how to reduce disk read? (and bloom filter performance)
Posted by Radim Kolar <hs...@sendmail.cz>.
Dne 7.10.2011 23:16, Mohit Anchlia napsal(a):
> You'll see output like:
>
> Offset SSTables
> 1 8021
> 2 783
>
> Which means 783 read operations accessed 2 SSTables
thank you for explaining it to me. I see this:
Offset SSTables
1 59323
2 857
3 56
it means bloom filter failure ratio over 1%. Cassandra in unit tests
expects bloom filter false positive less than 1.05%. HBase has
configurable bloom filters. You can choose 1% or 0.5% - it can make
difference for large cache.
But result is that my poor read performance should not be caused by
bloom filters.
Re: how to reduce disk read? (and bloom filter performance)
Posted by Mohit Anchlia <mo...@gmail.com>.
You'll see output like:
Offset SSTables
1 8021
2 783
Which means 783 read operations accessed 2 SSTables
On Fri, Oct 7, 2011 at 2:03 PM, Radim Kolar <hs...@sendmail.cz> wrote:
> Dne 7.10.2011 15:55, Mohit Anchlia napsal(a):
>>
>> Check your disk utilization using iostat. Also, check if compactions
>> are causing reads to be slow. Check GC too.
>>
>> You can look at cfhistograms output or post it here.
>
> i dont know how to interpret cf historgrams. can you write it to wiki?
>
Re: how to reduce disk read? (and bloom filter performance)
Posted by Radim Kolar <hs...@sendmail.cz>.
Dne 7.10.2011 15:55, Mohit Anchlia napsal(a):
> Check your disk utilization using iostat. Also, check if compactions
> are causing reads to be slow. Check GC too.
>
> You can look at cfhistograms output or post it here.
i dont know how to interpret cf historgrams. can you write it to wiki?
Re: how to reduce disk read? (and bloom filter performance)
Posted by Mohit Anchlia <mo...@gmail.com>.
Check your disk utilization using iostat. Also, check if compactions
are causing reads to be slow. Check GC too.
You can look at cfhistograms output or post it here.
On Fri, Oct 7, 2011 at 1:44 AM, Radim Kolar <hs...@sendmail.cz> wrote:
> Dne 7.10.2011 10:04, aaron morton napsal(a):
>>
>> Of the top of my head I it's not exposed via nodetool.
>>
>
>> You can get it via HTTP if you install mx4j or if you could try
>> http://wiki.cyclopsgroup.org/jmxterm
>
> i have MX4J/Http but cant find that info in listing.
>
> i suspect that bloom filter performance is not so great on my 30GB CFs
> because one read is 44.846ms on machine without any workload and because CF
> has only single column and rarely (less that 0.01%) updated, data should not
> be in more than 1 ssfile.
>
> it should be 2 reads: one read to -Index and second to read to -Data: about
> 24 ms, but its double.
>
> I can also confirm other ppl experience that performance really drops after
> inserting more then 500k columns.
>
Re: how to reduce disk read? (and bloom filter performance)
Posted by Radim Kolar <hs...@sendmail.cz>.
Dne 7.10.2011 10:04, aaron morton napsal(a):
> Of the top of my head I it's not exposed via nodetool.
>
> You can get it via HTTP if you install mx4j or if you could try
> http://wiki.cyclopsgroup.org/jmxterm
i have MX4J/Http but cant find that info in listing.
i suspect that bloom filter performance is not so great on my 30GB CFs
because one read is 44.846ms on machine without any workload and because
CF has only single column and rarely (less that 0.01%) updated, data
should not be in more than 1 ssfile.
it should be 2 reads: one read to -Index and second to read to -Data:
about 24 ms, but its double.
I can also confirm other ppl experience that performance really drops
after inserting more then 500k columns.
Re: how to reduce disk read? (and bloom filter performance)
Posted by aaron morton <aa...@thelastpickle.com>.
Of the top of my head I it's not exposed via nodetool.
You can get it via HTTP if you install mx4j or if you could try http://wiki.cyclopsgroup.org/jmxterm
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 7/10/2011, at 8:09 PM, Radim Kolar wrote:
> Dne 16.9.2011 8:20, Yang napsal(a):
>> I looked at the JMX attributes
>> CFS.BloomFilterFalseRatio, it's 1.0 , BloomFilterFalsePositives, it's
>> 2810,
> its possible to query this bloom filter false ratio from command line?
Re: how to reduce disk read? (and bloom filter performance)
Posted by Radim Kolar <hs...@sendmail.cz>.
Dne 16.9.2011 8:20, Yang napsal(a):
> I looked at the JMX attributes
> CFS.BloomFilterFalseRatio, it's 1.0 , BloomFilterFalsePositives, it's
> 2810,
its possible to query this bloom filter false ratio from command line?