You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Bao Le <lv...@yahoo.com> on 2013/06/27 20:40:54 UTC
High-read latency for non-existing rows with LCS and 1.2.5
Hi,
We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size is 100M. On each node,
we have anywhere from 700+ to 800+ sstables (for all levels). The bloom_filter_fp_chance is set at 0.000744.
For read requests that ask for existing row records, the latency is great, mostly under 20 milliseconds with key cache and row cache set. For read requests that ask for non-existing row records (not because of deletes, but rather, have never been in the system to start with), the latency is running into hundreds of milliseconds and sometimes seconds.
Just wonder if anyone has come across this before and has some pointers on how to reduce the latency in this case.
Thanks
Bao
Re: High-read latency for non-existing rows with LCS and 1.2.5
Posted by aaron morton <aa...@thelastpickle.com>.
> We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size is 100M. On each node,
> we have anywhere from 700+ to 800+ sstables (for all levels). The bloom_filter_fp_chance is set at 0.000744.
The current default bloom_filter_fp_chance is 0.1 for levelled compaction. Reducing this (and running nodetool upgradesstables) will reduce the bloom filter size significantly.
> the latency is running into hundreds of milliseconds and sometimes seconds.
Check the number of SSTables per read using nodetool cfhistograms. With levelled compaction you should not see above 3 http://www.datastax.com/dev/blog/when-to-use-leveled-compaction
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 28/06/2013, at 7:44 AM, sankalp kohli <ko...@gmail.com> wrote:
> Try doing request tracing.
> http://www.datastax.com/dev/blog/tracing-in-cassandra-1-2
>
>
> On Thu, Jun 27, 2013 at 2:40 PM, Bao Le <lv...@yahoo.com> wrote:
> Hi,
>
> We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size is 100M. On each node,
> we have anywhere from 700+ to 800+ sstables (for all levels). The bloom_filter_fp_chance is set at 0.000744.
>
> For read requests that ask for existing row records, the latency is great, mostly under 20 milliseconds with key cache and row cache set. For read requests that ask for non-existing row records (not because of deletes, but rather, have never been in the system to start with), the latency is running into hundreds of milliseconds and sometimes seconds.
>
> Just wonder if anyone has come across this before and has some pointers on how to reduce the latency in this case.
>
> Thanks
> Bao
>
>
>
>
Re: High-read latency for non-existing rows with LCS and 1.2.5
Posted by sankalp kohli <ko...@gmail.com>.
Try doing request tracing.
http://www.datastax.com/dev/blog/tracing-in-cassandra-1-2
On Thu, Jun 27, 2013 at 2:40 PM, Bao Le <lv...@yahoo.com> wrote:
> Hi,
>
> We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size
> is 100M. On each node,
> we have anywhere from 700+ to 800+ sstables (for all levels). The
> bloom_filter_fp_chance is set at 0.000744.
>
> For read requests that ask for existing row records, the latency is
> great, mostly under 20 milliseconds with key cache and row cache set. For
> read requests that ask for non-existing row records (not because of
> deletes, but rather, have never been in the system to start with), the
> latency is running into hundreds of milliseconds and sometimes seconds.
>
> Just wonder if anyone has come across this before and has some pointers
> on how to reduce the latency in this case.
>
> Thanks
> Bao
>
>
>
>