You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jonathan Lacefield <jl...@datastax.com> on 2014/02/11 17:42:39 UTC
Re: Worse perf after Row Caching version 1.2.5:
Hello,
Please paste the output of cfhistograms for these tables. Also, what
does your environment look like, number of nodes, disk drive configs,
memory, C* version, etc.
Thanks,
Jonathan
Jonathan Lacefield
Solutions Architect, DataStax
(404) 822 3487
<http://www.linkedin.com/in/jlacefield>
<http://www.datastax.com/what-we-offer/products-services/training/virtual-training>
On Tue, Feb 11, 2014 at 10:26 AM, PARASHAR, BHASKARJYA JAY
<bp...@att.com>wrote:
> Hi,
>
>
>
> I have two tables and I enabled row caching for both of them using CQL.
> These two CF's are very small with one about 300 rows and other < 2000
> rows. The rows themselves are small.
>
> Cassandra heap: 8gb.
>
> a. alter table TABLE_X with caching = 'rows_only';
>
> b. alter table TABLE_Y with caching = 'rows_only';
>
> I also changed row_cache_size_in_mb: 1024 in the Cassandra.yaml file.
>
> After extensive testing, it seems the performance of Table_X degraded from
> 600ms to 750ms and Table_Y gained about 10 ms (from 188ms to 177 ms).
>
> More Info
>
> Table X is always queried with "Select * from Table_X"; Cfstats in
> Table_X shows Read Latency: NaN ms. I assumed that since we select all the
> rows, the entire table would be cached.
>
> Table_Y has a secondary index and is queried on that index.
>
>
>
>
>
> Would appreciate any input why the performance is worse and how to enable
> row caching for these two tables.
>
>
>
> Thanks
>
> Jay
>
>
>
>
>
RE: Worse perf after Row Caching version 1.2.5:
Posted by "PARASHAR, BHASKARJYA JAY" <bp...@att.com>.
Thanks a lot Marcus. That explains it.
From: Marcus Eriksson [mailto:krummas@gmail.com]
Sent: Thursday, February 13, 2014 12:33 AM
To: user@cassandra.apache.org
Subject: Re: Worse perf after Row Caching version 1.2.5:
"select * from <table>" will not populate row cache, but if the row is cached, it will be used. You need to use "select * from table where X=Y" to populate row cache.
when setting caching = "rows_only" you disable key cache which might hurt your performance.
On Wed, Feb 12, 2014 at 9:05 PM, PARASHAR, BHASKARJYA JAY <bp...@att.com>> wrote:
Thanks Jonathan,
I have the cfstats but our prod team has changed some configs after my post and I do not have the cfhistograms information now.
No Of nodes: 3
Ram: 472GB
Cassandra version: 1.2.5
I am pasting the cfstats below.
Regards
Jay
CREATE TABLE "EnablerCreditReasonInfo" (
key text PRIMARY KEY,
"creditReasonDescription" text
) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.010000 AND
caching='ROWS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'SnappyCompressor'};
CFStats
Column Family: EnablerCreditReasonInfo
SSTable count: 3
Space used (live): 108067
Space used (total): 108067
Number of Keys (estimate): 1920
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Bloom Filter False Positives: 0
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 2232
Compacted row minimum size: 61
Compacted row maximum size: 149
Compacted row mean size: 100
From: Jonathan Lacefield [mailto:jlacefield@datastax.com<ma...@datastax.com>]
Sent: Tuesday, February 11, 2014 10:43 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Worse perf after Row Caching version 1.2.5:
Hello,
Please paste the output of cfhistograms for these tables. Also, what does your environment look like, number of nodes, disk drive configs, memory, C* version, etc.
Thanks,
Jonathan
Jonathan Lacefield
Solutions Architect, DataStax
(404) 822 3487<tel:%28404%29%20822%203487>
[Image removed by sender.]<http://www.linkedin.com/in/jlacefield>
[Image removed by sender.]<http://www.datastax.com/what-we-offer/products-services/training/virtual-training>
On Tue, Feb 11, 2014 at 10:26 AM, PARASHAR, BHASKARJYA JAY <bp...@att.com>> wrote:
Hi,
I have two tables and I enabled row caching for both of them using CQL. These two CF's are very small with one about 300 rows and other < 2000 rows. The rows themselves are small.
Cassandra heap: 8gb.
a. alter table TABLE_X with caching = 'rows_only';
b. alter table TABLE_Y with caching = 'rows_only';
I also changed row_cache_size_in_mb: 1024 in the Cassandra.yaml file.
After extensive testing, it seems the performance of Table_X degraded from 600ms to 750ms and Table_Y gained about 10 ms (from 188ms to 177 ms).
More Info
Table X is always queried with "Select * from Table_X"; Cfstats in Table_X shows Read Latency: NaN ms. I assumed that since we select all the rows, the entire table would be cached.
Table_Y has a secondary index and is queried on that index.
Would appreciate any input why the performance is worse and how to enable row caching for these two tables.
Thanks
Jay
Re: Worse perf after Row Caching version 1.2.5:
Posted by Marcus Eriksson <kr...@gmail.com>.
"select * from <table>" will not populate row cache, but if the row is
cached, it will be used. You need to use "select * from table where X=Y" to
populate row cache.
when setting caching = "rows_only" you disable key cache which might hurt
your performance.
On Wed, Feb 12, 2014 at 9:05 PM, PARASHAR, BHASKARJYA JAY <bp...@att.com>wrote:
> Thanks Jonathan,
>
>
>
> I have the cfstats but our prod team has changed some configs after my
> post and I do not have the cfhistograms information now.
>
>
>
> No Of nodes: 3
>
> Ram: 472GB
>
> Cassandra version: 1.2.5
>
>
>
> I am pasting the cfstats below.
>
>
>
> Regards
>
> Jay
>
>
>
> CREATE TABLE "EnablerCreditReasonInfo" (
>
> key text PRIMARY KEY,
>
> "creditReasonDescription" text
>
> ) WITH COMPACT STORAGE AND
>
> bloom_filter_fp_chance=0.010000 AND
>
> caching='ROWS_ONLY' AND
>
> comment='' AND
>
> dclocal_read_repair_chance=0.000000 AND
>
> gc_grace_seconds=864000 AND
>
> read_repair_chance=0.100000 AND
>
> replicate_on_write='true' AND
>
> populate_io_cache_on_flush='false' AND
>
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
>
> compression={'sstable_compression': 'SnappyCompressor'};
>
>
>
>
>
> CFStats
>
> Column Family: EnablerCreditReasonInfo
>
> SSTable count: 3
>
> Space used (live): 108067
>
> Space used (total): 108067
>
> Number of Keys (estimate): 1920
>
> Memtable Columns Count: 0
>
> Memtable Data Size: 0
>
> Memtable Switch Count: 0
>
> Read Count: 0
>
> Read Latency: NaN ms.
>
> Write Count: 0
>
> Write Latency: NaN ms.
>
> Pending Tasks: 0
>
> Bloom Filter False Positives: 0
>
> Bloom Filter False Ratio: 0.00000
>
> Bloom Filter Space Used: 2232
>
> Compacted row minimum size: 61
>
> Compacted row maximum size: 149
>
> Compacted row mean size: 100
>
>
>
> *From:* Jonathan Lacefield [mailto:jlacefield@datastax.com]
> *Sent:* Tuesday, February 11, 2014 10:43 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Worse perf after Row Caching version 1.2.5:
>
>
>
> Hello,
>
>
>
> Please paste the output of cfhistograms for these tables. Also, what
> does your environment look like, number of nodes, disk drive configs,
> memory, C* version, etc.
>
>
>
> Thanks,
>
>
>
> Jonathan
>
>
> Jonathan Lacefield
>
> Solutions Architect, DataStax
>
> (404) 822 3487
>
> [image: Image removed by sender.] <http://www.linkedin.com/in/jlacefield>
>
>
>
>
>
> [image: Image removed by sender.]<http://www.datastax.com/what-we-offer/products-services/training/virtual-training>
>
>
>
> On Tue, Feb 11, 2014 at 10:26 AM, PARASHAR, BHASKARJYA JAY <bp...@att.com>
> wrote:
>
> Hi,
>
>
>
> I have two tables and I enabled row caching for both of them using CQL.
> These two CF's are very small with one about 300 rows and other < 2000
> rows. The rows themselves are small.
>
> Cassandra heap: 8gb.
>
> a. alter table TABLE_X with caching = 'rows_only';
>
> b. alter table TABLE_Y with caching = 'rows_only';
>
> I also changed row_cache_size_in_mb: 1024 in the Cassandra.yaml file.
>
> After extensive testing, it seems the performance of Table_X degraded from
> 600ms to 750ms and Table_Y gained about 10 ms (from 188ms to 177 ms).
>
> More Info
>
> Table X is always queried with "Select * from Table_X"; Cfstats in
> Table_X shows Read Latency: NaN ms. I assumed that since we select all the
> rows, the entire table would be cached.
>
> Table_Y has a secondary index and is queried on that index.
>
>
>
>
>
> Would appreciate any input why the performance is worse and how to enable
> row caching for these two tables.
>
>
>
> Thanks
>
> Jay
>
>
>
>
>
>
>
RE: Worse perf after Row Caching version 1.2.5:
Posted by "PARASHAR, BHASKARJYA JAY" <bp...@att.com>.
Thanks Jonathan,
I have the cfstats but our prod team has changed some configs after my post and I do not have the cfhistograms information now.
No Of nodes: 3
Ram: 472GB
Cassandra version: 1.2.5
I am pasting the cfstats below.
Regards
Jay
CREATE TABLE "EnablerCreditReasonInfo" (
key text PRIMARY KEY,
"creditReasonDescription" text
) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.010000 AND
caching='ROWS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'SnappyCompressor'};
CFStats
Column Family: EnablerCreditReasonInfo
SSTable count: 3
Space used (live): 108067
Space used (total): 108067
Number of Keys (estimate): 1920
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Bloom Filter False Positives: 0
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 2232
Compacted row minimum size: 61
Compacted row maximum size: 149
Compacted row mean size: 100
From: Jonathan Lacefield [mailto:jlacefield@datastax.com]
Sent: Tuesday, February 11, 2014 10:43 AM
To: user@cassandra.apache.org
Subject: Re: Worse perf after Row Caching version 1.2.5:
Hello,
Please paste the output of cfhistograms for these tables. Also, what does your environment look like, number of nodes, disk drive configs, memory, C* version, etc.
Thanks,
Jonathan
Jonathan Lacefield
Solutions Architect, DataStax
(404) 822 3487
[Image removed by sender.]<http://www.linkedin.com/in/jlacefield>
[Image removed by sender.]<http://www.datastax.com/what-we-offer/products-services/training/virtual-training>
On Tue, Feb 11, 2014 at 10:26 AM, PARASHAR, BHASKARJYA JAY <bp...@att.com>> wrote:
Hi,
I have two tables and I enabled row caching for both of them using CQL. These two CF's are very small with one about 300 rows and other < 2000 rows. The rows themselves are small.
Cassandra heap: 8gb.
a. alter table TABLE_X with caching = 'rows_only';
b. alter table TABLE_Y with caching = 'rows_only';
I also changed row_cache_size_in_mb: 1024 in the Cassandra.yaml file.
After extensive testing, it seems the performance of Table_X degraded from 600ms to 750ms and Table_Y gained about 10 ms (from 188ms to 177 ms).
More Info
Table X is always queried with "Select * from Table_X"; Cfstats in Table_X shows Read Latency: NaN ms. I assumed that since we select all the rows, the entire table would be cached.
Table_Y has a secondary index and is queried on that index.
Would appreciate any input why the performance is worse and how to enable row caching for these two tables.
Thanks
Jay