You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Arindam Barua <ab...@247-inc.com> on 2012/10/02 00:57:35 UTC
Read latency issue
We are trying to setup a Cassandra cluster and have low read latency requirements. Running some tests, we do not see the performance that we were hoping for. Wanted to check if anyone has thoughts on:
1. If these are expected latency times for the data/machine config, etc
2. If not, can do something to improve our read times?
We set up 4 boxes as a ring running Cassandra 1.1.5, and setup a keyspace with replication 3, and strategy_class SimpleStrategy. The column family being tested has 12 columns, 4 of which form a composite key.
We then wrote in 192,000 randomly generated test data rows into the column family. Most columns are either randomly generated UUIDs, or short strings. One of them however is a blob consisting of around 1K data (we later reduced the size of this blob data, but didn't seem to change our read times much)
Running a query to like "select * from <table_name> where atag=<foo>", where 'atag' is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network. The query returned around 800 results. Running the same query on a Cassandra host results in a read time of ~110-130 ms.
Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
Enabling row cache did not seem to change the performance much. Moreover, the row cache 'size' according to nodetool was very tiny. Here is a snapshot of the nodetool info after running few read tests:
Key Cache : size 2448 (bytes), capacity 104857584 (bytes), 231 hits, 266 requests, 1.000 recent hit rate, 14400 save period in seconds
Row Cache : size 96 (bytes), capacity 4194304000 (bytes), 9 hits, 13 requests, NaN recent hit rate, 0 save period in seconds
Hardware/OS specs:
Intel(R) Xeon(R) CPU L5640
OS: Solaris 5.10
RAM: 32 GB
Hard disk: 1 TB disk magnetic (not SSD)
Thanks,
Arindam
Re: Read latency issue
Posted by Sergey Tryuber <st...@gmail.com>.
Hi Arindam,
Just want to share our experience about latency. We use Cassandra to read
all columns for a specified key (5-10 columns, 10-15 each, no composites).
All our dataset fits memory (40GB on each node) and we use
LeveledCompactionStrategy. Our RF=2 and we read with CL=ONE. Also we use
Astyanax, which allows us to connect to that Cassandra node, which holds
data local (no additional network hops). The best performance we got is
with disabled row and key caches (no additional GC pressure) and reduced
heap size to 3.5GB. Our average read latency is 2ms, 99% fit in 10ms (1K
writes/updates/deletes and 1K reads per Cassandra node per second).
On 3 October 2012 11:32, aaron morton <aa...@thelastpickle.com> wrote:
> > Running a query to like “select * from <table_name> where atag=<foo>”,
> where ‘atag’ is the first column of the composite key, from either JDBC or
> Hector (equivalent code), results in read times of 200-300ms from a remote
> host on the same network.
>
> If you send a query to select columns from a row and do not fully specify
> the row key cassandra has to do a row scan.
>
> If you want fast performance specify the full row key.
>
> > Using read consistency of ONE reduces the read latency by ~20ms,
> compared to using QUORUM.
> It would only have read from the local node. (I think, may be confusing
> secondary index reads here).
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 3/10/2012, at 2:17 AM, Roshni Rajagopal <ro...@hotmail.com>
> wrote:
>
> > Arindam,
> >
> > Did you also try the cassandra stress tool & compare results?
> >
> > I havent done a performance test as yet, the only ones published on the
> internet are of YCSB on an older version of apache cassandra, and it
> doesn't seem to be actively supported or updated
> > http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf.
> >
> > The numbers you have sound very low, for a read of a row by key which
> should have been the fastest. I hope someone can help investigate or share
> numbers from their tests.
> >
> >
> >
> > Regards,
> > Roshni
> >
> >
> > > From: Dean.Hiller@nrel.gov
> > > To: user@cassandra.apache.org
> > > Date: Tue, 2 Oct 2012 06:41:09 -0600
> > > Subject: Re: Read latency issue
> > >
> > > Interesting results. With PlayOrm, we did a 6 node test of reading 100
> rows from 1,000,000 using PlayOrm Scalable SQL. It only took 60ms. Maybe we
> have better hardware though??? We are using 7200 RPM drives so nothing
> fancy on the disk side of things. More nodes puts at a higher throughput
> though as reading from more disks will be faster. Anyways, you may want to
> play with more nodes and re-run. If you run a test with PlayOrm, I would
> love to know the results there as well.
> > >
> > > Later,
> > > Dean
> > >
> > > From: Arindam Barua <ab...@247-inc.com>>
> > > Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>"
> <us...@cassandra.apache.org>>
> > > Date: Monday, October 1, 2012 4:57 PM
> > > To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org>>
> > > Subject: Read latency issue
> > >
> > > unning a query to like “select * from <table_name> where atag=<foo>”,
> where ‘atag’ is the first column of the composite key, from either JDBC or
> Hector (equivalent code), results in read times of 200-300ms from a remote
> host on the same network. The query returned around 800 results. Running
> the same query on a Cassandra host results in a read time of ~110-130 ms.
> > > Using read consistency of ONE reduces the read latency by ~20ms,
> compared to using QUORUM.
> > >
> > > Enabling row cache did not seem to change the performance much.
> Moreover, the row cache ‘size’ according to nodetool was very tiny. Here is
> a snapshot of the nodetool info after running few read tests:
> > > Key Cache : size 2448 (bytes), capacity 104857584 (bytes), 231 hits,
> 266 requests, 1.000 recent hit rate, 14400 save period in seconds
> > > Row Cache : size 96 (bytes), capacity 4194304000 (bytes), 9 hits, 13
> requests, NaN recent hit rate, 0 save period in seconds
> > >
>
>
RE: Read latency issue
Posted by Arindam Barua <ab...@247-inc.com>.
Hi Roshni,
In the JDBC request we do use CQL3 explicitly. We also attempted to read the data via Hector with similar read times.
Supplying all (or more) the components of the key reduces the response time and the number of columns in the result a lot. However, we do need back the entire row data in some scenarios, and so need to optimize that scenario.
Thanks,
Arindam
From: Roshni Rajagopal [mailto:roshni_rajagopal@hotmail.com]
Sent: Wednesday, October 03, 2012 4:53 PM
To: user@cassandra.apache.org
Subject: RE: Read latency issue
Hi Arindam,
There were some changes for CQL3 for composite keys storage , and you may be using CQL2 by default.
You could try for a non composite key or supply all the components of the key in the search...and see if you get different results...
Regards,
roshni
> From: abarua@247-inc.com<ma...@247-inc.com>
> To: user@cassandra.apache.org<ma...@cassandra.apache.org>
> Subject: RE: Read latency issue
> Date: Wed, 3 Oct 2012 17:53:46 +0000
>
>
> Thanks for your responses.
>
> Just to be clear our table declaration looks something like this:
> CREATE TABLE sessionevents (
> atag text,
> col2 uuid,
> col3 text,
> col4 uuid,
> col5 text,
> col6 text,
> col7 blob,
> col8 text,
> col9 timestamp,
> col10 uuid,
> col11 int,
> col12 uuid,
> PRIMARY KEY (atag, col2, col3, col4)
> )
>
> My understanding was that the (full) row key in this case would be the 'atag' values. The column names would then be composites like (<col2_value>:<col3_value>:<col4_value>:col5), (<col2_value>: <col3_value>: <col4_value>:col6), (<col2_value>:<col3_value>:<col4_value>:col7) ... (<col2_value>: <col3_value>: <col4_value>:col12). The columns would be sorted first by col2_values, then by col3 values, etc.
>
> Hence a query like "select * from sessionevents where atag=<foo>", we are specifying the entire row key, and Cassandra would return all the columns for that row.
>
> >> Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
> >It would only have read from the local node. (I think, may be confusing secondary index reads here).
> For read consistency ONE, reading only from one node is my expectation as well, and hence I'm seeing the reduced read latency compared to read consistency QUORUM. Does that not sound right?
> Btw, with read consistency ONE, we found the reading only happens from one node, but not necessarily the local node, even if the data is present in the local node. To check this, we turned on DEBUG logs on all the Cassandra hosts in the ring. We are using replication factor=3 on a 4 node ring, hence mostly the data is present locally. However, we noticed that the coordinator host on receiving the same request multiple times (i.e with the same row key) , would sometimes return the data locally, but sometimes would contact another host in the ring to fetch the data.
>
> Thanks,
> Arindam
>
> -----Original Message-----
> From: aaron morton [mailto:aaron@thelastpickle.com]<mailto:[mailto:aaron@thelastpickle.com]>
> Sent: Wednesday, October 03, 2012 12:32 AM
> To: user@cassandra.apache.org<ma...@cassandra.apache.org>
> Subject: Re: Read latency issue
>
> > Running a query to like "select * from <table_name> where atag=<foo>", where 'atag' is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network.
>
> If you send a query to select columns from a row and do not fully specify the row key cassandra has to do a row scan.
>
> If you want fast performance specify the full row key.
>
> > Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
> It would only have read from the local node. (I think, may be confusing secondary index reads here).
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 3/10/2012, at 2:17 AM, Roshni Rajagopal <ro...@hotmail.com>> wrote:
>
> > Arindam,
> >
> > Did you also try the cassandra stress tool & compare results?
> >
> > I havent done a performance test as yet, the only ones published on
> > the internet are of YCSB on an older version of apache cassandra, and it doesn't seem to be actively supported or updated http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf.
> >
> > The numbers you have sound very low, for a read of a row by key which should have been the fastest. I hope someone can help investigate or share numbers from their tests.
> >
> >
> >
> > Regards,
> > Roshni
> >
> >
> > > From: Dean.Hiller@nrel.gov<ma...@nrel.gov>
> > > To: user@cassandra.apache.org<ma...@cassandra.apache.org>
> > > Date: Tue, 2 Oct 2012 06:41:09 -0600
> > > Subject: Re: Read latency issue
> > >
> > > Interesting results. With PlayOrm, we did a 6 node test of reading 100 rows from 1,000,000 using PlayOrm Scalable SQL. It only took 60ms. Maybe we have better hardware though??? We are using 7200 RPM drives so nothing fancy on the disk side of things. More nodes puts at a higher throughput though as reading from more disks will be faster. Anyways, you may want to play with more nodes and re-run. If you run a test with PlayOrm, I would love to know the results there as well.
> > >
> > > Later,
> > > Dean
> > >
> > > From: Arindam Barua <ab...@247-inc.com>>>
> > > Reply-To:
> > > "user@cassandra.apache.org<ma...@cassandra.apache.org><mailto:user@cassandra.apache.org%3cmailto:user@cassandra.apache.org%3e>"
> > > <us...@cassandra.apache.org>>>
> > > Date: Monday, October 1, 2012 4:57 PM
> > > To: "user@cassandra.apache.org<ma...@cassandra.apache.org><mailto:user@cassandra.apache.org%3cmailto:user@cassandra.apache.org%3e>"
> > > <us...@cassandra.apache.org>>>
> > > Subject: Read latency issue
> > >
> > > unning a query to like "select * from <table_name> where atag=<foo>", where 'atag' is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network. The query returned around 800 results. Running the same query on a Cassandra host results in a read time of ~110-130 ms.
> > > Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
> > >
> > > Enabling row cache did not seem to change the performance much. Moreover, the row cache 'size' according to nodetool was very tiny. Here is a snapshot of the nodetool info after running few read tests:
> > > Key Cache : size 2448 (bytes), capacity 104857584 (bytes), 231 hits,
> > > 266 requests, 1.000 recent hit rate, 14400 save period in seconds
> > > Row Cache : size 96 (bytes), capacity 4194304000 (bytes), 9 hits, 13
> > > requests, NaN recent hit rate, 0 save period in seconds
> > >
>
>
>
RE: Read latency issue
Posted by Roshni Rajagopal <ro...@hotmail.com>.
Hi Arindam,
There were some changes for CQL3 for composite keys storage , and you may be using CQL2 by default.You could try for a non composite key or supply all the components of the key in the search...and see if you get different results...
Regards,roshni
> From: abarua@247-inc.com
> To: user@cassandra.apache.org
> Subject: RE: Read latency issue
> Date: Wed, 3 Oct 2012 17:53:46 +0000
>
>
> Thanks for your responses.
>
> Just to be clear our table declaration looks something like this:
> CREATE TABLE sessionevents (
> atag text,
> col2 uuid,
> col3 text,
> col4 uuid,
> col5 text,
> col6 text,
> col7 blob,
> col8 text,
> col9 timestamp,
> col10 uuid,
> col11 int,
> col12 uuid,
> PRIMARY KEY (atag, col2, col3, col4)
> )
>
> My understanding was that the (full) row key in this case would be the 'atag' values. The column names would then be composites like (<col2_value>:<col3_value>:<col4_value>:col5), (<col2_value>: <col3_value>: <col4_value>:col6), (<col2_value>:<col3_value>:<col4_value>:col7) ... (<col2_value>: <col3_value>: <col4_value>:col12). The columns would be sorted first by col2_values, then by col3 values, etc.
>
> Hence a query like "select * from sessionevents where atag=<foo>", we are specifying the entire row key, and Cassandra would return all the columns for that row.
>
> >> Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
> >It would only have read from the local node. (I think, may be confusing secondary index reads here).
> For read consistency ONE, reading only from one node is my expectation as well, and hence I'm seeing the reduced read latency compared to read consistency QUORUM. Does that not sound right?
> Btw, with read consistency ONE, we found the reading only happens from one node, but not necessarily the local node, even if the data is present in the local node. To check this, we turned on DEBUG logs on all the Cassandra hosts in the ring. We are using replication factor=3 on a 4 node ring, hence mostly the data is present locally. However, we noticed that the coordinator host on receiving the same request multiple times (i.e with the same row key) , would sometimes return the data locally, but sometimes would contact another host in the ring to fetch the data.
>
> Thanks,
> Arindam
>
> -----Original Message-----
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Wednesday, October 03, 2012 12:32 AM
> To: user@cassandra.apache.org
> Subject: Re: Read latency issue
>
> > Running a query to like "select * from <table_name> where atag=<foo>", where 'atag' is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network.
>
> If you send a query to select columns from a row and do not fully specify the row key cassandra has to do a row scan.
>
> If you want fast performance specify the full row key.
>
> > Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
> It would only have read from the local node. (I think, may be confusing secondary index reads here).
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 3/10/2012, at 2:17 AM, Roshni Rajagopal <ro...@hotmail.com> wrote:
>
> > Arindam,
> >
> > Did you also try the cassandra stress tool & compare results?
> >
> > I havent done a performance test as yet, the only ones published on
> > the internet are of YCSB on an older version of apache cassandra, and it doesn't seem to be actively supported or updated http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf.
> >
> > The numbers you have sound very low, for a read of a row by key which should have been the fastest. I hope someone can help investigate or share numbers from their tests.
> >
> >
> >
> > Regards,
> > Roshni
> >
> >
> > > From: Dean.Hiller@nrel.gov
> > > To: user@cassandra.apache.org
> > > Date: Tue, 2 Oct 2012 06:41:09 -0600
> > > Subject: Re: Read latency issue
> > >
> > > Interesting results. With PlayOrm, we did a 6 node test of reading 100 rows from 1,000,000 using PlayOrm Scalable SQL. It only took 60ms. Maybe we have better hardware though??? We are using 7200 RPM drives so nothing fancy on the disk side of things. More nodes puts at a higher throughput though as reading from more disks will be faster. Anyways, you may want to play with more nodes and re-run. If you run a test with PlayOrm, I would love to know the results there as well.
> > >
> > > Later,
> > > Dean
> > >
> > > From: Arindam Barua <ab...@247-inc.com>>
> > > Reply-To:
> > > "user@cassandra.apache.org<ma...@cassandra.apache.org>"
> > > <us...@cassandra.apache.org>>
> > > Date: Monday, October 1, 2012 4:57 PM
> > > To: "user@cassandra.apache.org<ma...@cassandra.apache.org>"
> > > <us...@cassandra.apache.org>>
> > > Subject: Read latency issue
> > >
> > > unning a query to like "select * from <table_name> where atag=<foo>", where 'atag' is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network. The query returned around 800 results. Running the same query on a Cassandra host results in a read time of ~110-130 ms.
> > > Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
> > >
> > > Enabling row cache did not seem to change the performance much. Moreover, the row cache 'size' according to nodetool was very tiny. Here is a snapshot of the nodetool info after running few read tests:
> > > Key Cache : size 2448 (bytes), capacity 104857584 (bytes), 231 hits,
> > > 266 requests, 1.000 recent hit rate, 14400 save period in seconds
> > > Row Cache : size 96 (bytes), capacity 4194304000 (bytes), 9 hits, 13
> > > requests, NaN recent hit rate, 0 save period in seconds
> > >
>
>
>
RE: Read latency issue
Posted by Arindam Barua <ab...@247-inc.com>.
We instrumented the Cassandra and Hector code adding more logs to check where the time was being spent.
We found the Cassandra read times to be very low, eg. CassandraServer.getSlice() is only 3ms.
However, on Hector's side, creating a ColumnFamilyTemplate<String, Composite>, and doing queryColumns() on it takes 90ms.
Looking at the breakup on the Hector side, it appears ExecutionResult.execute takes ~30ms, and ColumnFamilyResultWrapper takes ~47ms.
(we are reading around 800 composite columns of 1000 bytes each)
Any idea if this is the expected time to process stuff on Hector/other clients?
Btw, using Hector's SliceQuery() and reading into a List, or Astynax seem to result in similar times too.
Thanks,
Arindam
-----Original Message-----
From: Arindam Barua [mailto:abarua@247-inc.com]
Sent: Wednesday, October 03, 2012 10:54 AM
To: user@cassandra.apache.org
Subject: RE: Read latency issue
Thanks for your responses.
Just to be clear our table declaration looks something like this:
CREATE TABLE sessionevents (
atag text,
col2 uuid,
col3 text,
col4 uuid,
col5 text,
col6 text,
col7 blob,
col8 text,
col9 timestamp,
col10 uuid,
col11 int,
col12 uuid,
PRIMARY KEY (atag, col2, col3, col4)
)
My understanding was that the (full) row key in this case would be the 'atag' values. The column names would then be composites like (<col2_value>:<col3_value>:<col4_value>:col5), (<col2_value>: <col3_value>: <col4_value>:col6), (<col2_value>:<col3_value>:<col4_value>:col7) ... (<col2_value>: <col3_value>: <col4_value>:col12). The columns would be sorted first by col2_values, then by col3 values, etc.
Hence a query like "select * from sessionevents where atag=<foo>", we are specifying the entire row key, and Cassandra would return all the columns for that row.
>> Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
>It would only have read from the local node. (I think, may be confusing secondary index reads here).
For read consistency ONE, reading only from one node is my expectation as well, and hence I'm seeing the reduced read latency compared to read consistency QUORUM. Does that not sound right?
Btw, with read consistency ONE, we found the reading only happens from one node, but not necessarily the local node, even if the data is present in the local node. To check this, we turned on DEBUG logs on all the Cassandra hosts in the ring. We are using replication factor=3 on a 4 node ring, hence mostly the data is present locally. However, we noticed that the coordinator host on receiving the same request multiple times (i.e with the same row key) , would sometimes return the data locally, but sometimes would contact another host in the ring to fetch the data.
Thanks,
Arindam
-----Original Message-----
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, October 03, 2012 12:32 AM
To: user@cassandra.apache.org
Subject: Re: Read latency issue
> Running a query to like "select * from <table_name> where atag=<foo>", where 'atag' is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network.
If you send a query to select columns from a row and do not fully specify the row key cassandra has to do a row scan.
If you want fast performance specify the full row key.
> Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
It would only have read from the local node. (I think, may be confusing secondary index reads here).
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 3/10/2012, at 2:17 AM, Roshni Rajagopal <ro...@hotmail.com> wrote:
> Arindam,
>
> Did you also try the cassandra stress tool & compare results?
>
> I havent done a performance test as yet, the only ones published on
> the internet are of YCSB on an older version of apache cassandra, and it doesn't seem to be actively supported or updated http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf.
>
> The numbers you have sound very low, for a read of a row by key which should have been the fastest. I hope someone can help investigate or share numbers from their tests.
>
>
>
> Regards,
> Roshni
>
>
> > From: Dean.Hiller@nrel.gov
> > To: user@cassandra.apache.org
> > Date: Tue, 2 Oct 2012 06:41:09 -0600
> > Subject: Re: Read latency issue
> >
> > Interesting results. With PlayOrm, we did a 6 node test of reading 100 rows from 1,000,000 using PlayOrm Scalable SQL. It only took 60ms. Maybe we have better hardware though??? We are using 7200 RPM drives so nothing fancy on the disk side of things. More nodes puts at a higher throughput though as reading from more disks will be faster. Anyways, you may want to play with more nodes and re-run. If you run a test with PlayOrm, I would love to know the results there as well.
> >
> > Later,
> > Dean
> >
> > From: Arindam Barua <ab...@247-inc.com>>
> > Reply-To:
> > "user@cassandra.apache.org<ma...@cassandra.apache.org>"
> > <us...@cassandra.apache.org>>
> > Date: Monday, October 1, 2012 4:57 PM
> > To: "user@cassandra.apache.org<ma...@cassandra.apache.org>"
> > <us...@cassandra.apache.org>>
> > Subject: Read latency issue
> >
> > unning a query to like "select * from <table_name> where atag=<foo>", where 'atag' is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network. The query returned around 800 results. Running the same query on a Cassandra host results in a read time of ~110-130 ms.
> > Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
> >
> > Enabling row cache did not seem to change the performance much. Moreover, the row cache 'size' according to nodetool was very tiny. Here is a snapshot of the nodetool info after running few read tests:
> > Key Cache : size 2448 (bytes), capacity 104857584 (bytes), 231 hits,
> > 266 requests, 1.000 recent hit rate, 14400 save period in seconds
> > Row Cache : size 96 (bytes), capacity 4194304000 (bytes), 9 hits, 13
> > requests, NaN recent hit rate, 0 save period in seconds
> >
RE: Read latency issue
Posted by Arindam Barua <ab...@247-inc.com>.
Thanks for your responses.
Just to be clear our table declaration looks something like this:
CREATE TABLE sessionevents (
atag text,
col2 uuid,
col3 text,
col4 uuid,
col5 text,
col6 text,
col7 blob,
col8 text,
col9 timestamp,
col10 uuid,
col11 int,
col12 uuid,
PRIMARY KEY (atag, col2, col3, col4)
)
My understanding was that the (full) row key in this case would be the 'atag' values. The column names would then be composites like (<col2_value>:<col3_value>:<col4_value>:col5), (<col2_value>: <col3_value>: <col4_value>:col6), (<col2_value>:<col3_value>:<col4_value>:col7) ... (<col2_value>: <col3_value>: <col4_value>:col12). The columns would be sorted first by col2_values, then by col3 values, etc.
Hence a query like "select * from sessionevents where atag=<foo>", we are specifying the entire row key, and Cassandra would return all the columns for that row.
>> Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
>It would only have read from the local node. (I think, may be confusing secondary index reads here).
For read consistency ONE, reading only from one node is my expectation as well, and hence I'm seeing the reduced read latency compared to read consistency QUORUM. Does that not sound right?
Btw, with read consistency ONE, we found the reading only happens from one node, but not necessarily the local node, even if the data is present in the local node. To check this, we turned on DEBUG logs on all the Cassandra hosts in the ring. We are using replication factor=3 on a 4 node ring, hence mostly the data is present locally. However, we noticed that the coordinator host on receiving the same request multiple times (i.e with the same row key) , would sometimes return the data locally, but sometimes would contact another host in the ring to fetch the data.
Thanks,
Arindam
-----Original Message-----
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, October 03, 2012 12:32 AM
To: user@cassandra.apache.org
Subject: Re: Read latency issue
> Running a query to like "select * from <table_name> where atag=<foo>", where 'atag' is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network.
If you send a query to select columns from a row and do not fully specify the row key cassandra has to do a row scan.
If you want fast performance specify the full row key.
> Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
It would only have read from the local node. (I think, may be confusing secondary index reads here).
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 3/10/2012, at 2:17 AM, Roshni Rajagopal <ro...@hotmail.com> wrote:
> Arindam,
>
> Did you also try the cassandra stress tool & compare results?
>
> I havent done a performance test as yet, the only ones published on
> the internet are of YCSB on an older version of apache cassandra, and it doesn't seem to be actively supported or updated http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf.
>
> The numbers you have sound very low, for a read of a row by key which should have been the fastest. I hope someone can help investigate or share numbers from their tests.
>
>
>
> Regards,
> Roshni
>
>
> > From: Dean.Hiller@nrel.gov
> > To: user@cassandra.apache.org
> > Date: Tue, 2 Oct 2012 06:41:09 -0600
> > Subject: Re: Read latency issue
> >
> > Interesting results. With PlayOrm, we did a 6 node test of reading 100 rows from 1,000,000 using PlayOrm Scalable SQL. It only took 60ms. Maybe we have better hardware though??? We are using 7200 RPM drives so nothing fancy on the disk side of things. More nodes puts at a higher throughput though as reading from more disks will be faster. Anyways, you may want to play with more nodes and re-run. If you run a test with PlayOrm, I would love to know the results there as well.
> >
> > Later,
> > Dean
> >
> > From: Arindam Barua <ab...@247-inc.com>>
> > Reply-To:
> > "user@cassandra.apache.org<ma...@cassandra.apache.org>"
> > <us...@cassandra.apache.org>>
> > Date: Monday, October 1, 2012 4:57 PM
> > To: "user@cassandra.apache.org<ma...@cassandra.apache.org>"
> > <us...@cassandra.apache.org>>
> > Subject: Read latency issue
> >
> > unning a query to like "select * from <table_name> where atag=<foo>", where 'atag' is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network. The query returned around 800 results. Running the same query on a Cassandra host results in a read time of ~110-130 ms.
> > Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
> >
> > Enabling row cache did not seem to change the performance much. Moreover, the row cache 'size' according to nodetool was very tiny. Here is a snapshot of the nodetool info after running few read tests:
> > Key Cache : size 2448 (bytes), capacity 104857584 (bytes), 231 hits,
> > 266 requests, 1.000 recent hit rate, 14400 save period in seconds
> > Row Cache : size 96 (bytes), capacity 4194304000 (bytes), 9 hits, 13
> > requests, NaN recent hit rate, 0 save period in seconds
> >
Re: Read latency issue
Posted by aaron morton <aa...@thelastpickle.com>.
> Running a query to like “select * from <table_name> where atag=<foo>”, where ‘atag’ is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network.
If you send a query to select columns from a row and do not fully specify the row key cassandra has to do a row scan.
If you want fast performance specify the full row key.
> Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
It would only have read from the local node. (I think, may be confusing secondary index reads here).
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 3/10/2012, at 2:17 AM, Roshni Rajagopal <ro...@hotmail.com> wrote:
> Arindam,
>
> Did you also try the cassandra stress tool & compare results?
>
> I havent done a performance test as yet, the only ones published on the internet are of YCSB on an older version of apache cassandra, and it doesn't seem to be actively supported or updated
> http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf.
>
> The numbers you have sound very low, for a read of a row by key which should have been the fastest. I hope someone can help investigate or share numbers from their tests.
>
>
>
> Regards,
> Roshni
>
>
> > From: Dean.Hiller@nrel.gov
> > To: user@cassandra.apache.org
> > Date: Tue, 2 Oct 2012 06:41:09 -0600
> > Subject: Re: Read latency issue
> >
> > Interesting results. With PlayOrm, we did a 6 node test of reading 100 rows from 1,000,000 using PlayOrm Scalable SQL. It only took 60ms. Maybe we have better hardware though??? We are using 7200 RPM drives so nothing fancy on the disk side of things. More nodes puts at a higher throughput though as reading from more disks will be faster. Anyways, you may want to play with more nodes and re-run. If you run a test with PlayOrm, I would love to know the results there as well.
> >
> > Later,
> > Dean
> >
> > From: Arindam Barua <ab...@247-inc.com>>
> > Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
> > Date: Monday, October 1, 2012 4:57 PM
> > To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
> > Subject: Read latency issue
> >
> > unning a query to like “select * from <table_name> where atag=<foo>”, where ‘atag’ is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network. The query returned around 800 results. Running the same query on a Cassandra host results in a read time of ~110-130 ms.
> > Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
> >
> > Enabling row cache did not seem to change the performance much. Moreover, the row cache ‘size’ according to nodetool was very tiny. Here is a snapshot of the nodetool info after running few read tests:
> > Key Cache : size 2448 (bytes), capacity 104857584 (bytes), 231 hits, 266 requests, 1.000 recent hit rate, 14400 save period in seconds
> > Row Cache : size 96 (bytes), capacity 4194304000 (bytes), 9 hits, 13 requests, NaN recent hit rate, 0 save period in seconds
> >
RE: Read latency issue
Posted by Roshni Rajagopal <ro...@hotmail.com>.
Arindam,
Did you also try the cassandra stress tool & compare results?
I havent done a performance test as yet, the only ones published on the internet are of YCSB on an older version of apache cassandra, and it doesn't seem to be actively supported or updatedhttp://www.brianfrankcooper.net/pubs/ycsb-v4.pdf.
The numbers you have sound very low, for a read of a row by key which should have been the fastest. I hope someone can help investigate or share numbers from their tests.
Regards,Roshni
> From: Dean.Hiller@nrel.gov
> To: user@cassandra.apache.org
> Date: Tue, 2 Oct 2012 06:41:09 -0600
> Subject: Re: Read latency issue
>
> Interesting results. With PlayOrm, we did a 6 node test of reading 100 rows from 1,000,000 using PlayOrm Scalable SQL. It only took 60ms. Maybe we have better hardware though??? We are using 7200 RPM drives so nothing fancy on the disk side of things. More nodes puts at a higher throughput though as reading from more disks will be faster. Anyways, you may want to play with more nodes and re-run. If you run a test with PlayOrm, I would love to know the results there as well.
>
> Later,
> Dean
>
> From: Arindam Barua <ab...@247-inc.com>>
> Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
> Date: Monday, October 1, 2012 4:57 PM
> To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
> Subject: Read latency issue
>
> unning a query to like “select * from <table_name> where atag=<foo>”, where ‘atag’ is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network. The query returned around 800 results. Running the same query on a Cassandra host results in a read time of ~110-130 ms.
> Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
>
> Enabling row cache did not seem to change the performance much. Moreover, the row cache ‘size’ according to nodetool was very tiny. Here is a snapshot of the nodetool info after running few read tests:
> Key Cache : size 2448 (bytes), capacity 104857584 (bytes), 231 hits, 266 requests, 1.000 recent hit rate, 14400 save period in seconds
> Row Cache : size 96 (bytes), capacity 4194304000 (bytes), 9 hits, 13 requests, NaN recent hit rate, 0 save period in seconds
>
Re: Read latency issue
Posted by "Hiller, Dean" <De...@nrel.gov>.
Interesting results. With PlayOrm, we did a 6 node test of reading 100 rows from 1,000,000 using PlayOrm Scalable SQL. It only took 60ms. Maybe we have better hardware though??? We are using 7200 RPM drives so nothing fancy on the disk side of things. More nodes puts at a higher throughput though as reading from more disks will be faster. Anyways, you may want to play with more nodes and re-run. If you run a test with PlayOrm, I would love to know the results there as well.
Later,
Dean
From: Arindam Barua <ab...@247-inc.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Monday, October 1, 2012 4:57 PM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Read latency issue
unning a query to like “select * from <table_name> where atag=<foo>”, where ‘atag’ is the first column of the composite key, from either JDBC or Hector (equivalent code), results in read times of 200-300ms from a remote host on the same network. The query returned around 800 results. Running the same query on a Cassandra host results in a read time of ~110-130 ms.
Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
Enabling row cache did not seem to change the performance much. Moreover, the row cache ‘size’ according to nodetool was very tiny. Here is a snapshot of the nodetool info after running few read tests:
Key Cache : size 2448 (bytes), capacity 104857584 (bytes), 231 hits, 266 requests, 1.000 recent hit rate, 14400 save period in seconds
Row Cache : size 96 (bytes), capacity 4194304000 (bytes), 9 hits, 13 requests, NaN recent hit rate, 0 save period in seconds