You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Batranut Bogdan <ba...@yahoo.com> on 2014/03/14 13:50:39 UTC
Cassandra slow on some reads
Hello all,
Here is the environment:
I have a 6 node Cassandra cluster. On each node I have:
- 32 G RAM
- 24 G RAM for cassa
- ~150 - 200 MB/s disk speed
- tomcat 6 with axis2 webservice that uses the datastax java driver to make
asynch reads / writes
- replication factor for the keyspace is 3
(I know that there is a lot of heap but I also have write heavy tasks and I want them to get into mem fast) .
All nodes in the same data center
The clients that read / write are in the same datacenter so network is Gigabit.
The table structure is like this: PK(key String , timestam int, column1 string, col2 string) , list1 , list 2, list 3 .
There are about 300 milions individual keys.
There are about 100 timestamps for each key now, so the rows will get wider as time passes.
I am using datastax java driver to query the cluster.
I have ~450 queries that are like this: SELECT * FROM table where key = 'some string' and ts = some value; some value is close to present time.
The problem:
About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, the majority of those take around 10 seconds.
When investigating I saw that if I have a slow response and I redo the query it will finish in 8 - 10 MILIseconds like the rest of the queries that I have.
I could not see using JConsole any spikes in CPU / memory when executing the queries. The rise in resource consumtion is very small on all nodes on the cluster. I expect such delays to be generated by a BIG increase in resource consumption.
Any comments will be appreciated.
Thank you.
Re: Cassandra slow on some reads
Posted by Batranut Bogdan <ba...@yahoo.com>.
Hello,
I can't go this way... this cf will be used for time ranges.
On Friday, March 14, 2014 5:10 PM, "Laing, Michael" <mi...@nytimes.com> wrote:
If you do not need to do range queries on your 'timestam' (ts) column - and if you can change your schema (big if...), then you could move 'timestam' into the partition key like this (using your notation):
PK((key String , timestam int), column1 string, col2 string) , list1 , list 2, list 3 .
Now the select query you showed should execute more consistently.
But of course something else might break...!
ml
On Fri, Mar 14, 2014 at 8:50 AM, Batranut Bogdan <ba...@yahoo.com> wrote:
Hello all,
>
>
>Here is the environment:
>
>
>I have a 6 node Cassandra cluster. On each node I have:
>- 32 G RAM
>- 24 G RAM for cassa
>- ~150 - 200 MB/s disk speed
>- tomcat 6 with axis2 webservice that uses the datastax java driver to make
>asynch reads / writes
>- replication factor for the keyspace is 3
>
>(I know that there is a lot of heap but I
also have write heavy tasks and I want them to get into mem fast) .
>
>All nodes in the same data center
>The clients that read / write are in the same datacenter so network is Gigabit.
>
>
>
>The table structure is like this: PK(key String , timestam int, column1 string, col2 string) , list1 , list 2, list 3 .
>There are about 300 milions individual keys.
>There are about 100 timestamps for each key now, so the rows will get wider as time passes.
>
>
>I am using datastax java driver to query the cluster.
>
>
>I have ~450 queries that are like this: SELECT * FROM table where key = 'some string' and ts = some value; some value is close to present time.
>
>
>The problem:
>
>
>About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, the majority of those take around 10 seconds.
>When investigating I saw that if I have a slow response and I redo the query it will finish in 8 - 10 MILIseconds like the rest of the queries that I have.
>I could not see using JConsole any spikes in CPU / memory when executing the queries. The rise in resource consumtion is very small on all nodes on the cluster. I expect such delays to be generated by a BIG increase in resource consumption.
>
>
>Any comments will be appreciated.
>
>
>Thank you.
>
>
>
>
>
>
>
Re: Cassandra slow on some reads
Posted by "Laing, Michael" <mi...@nytimes.com>.
*If* you do not need to do range queries on your 'timestam' (ts) column -
*and* if you can change your schema (big if...), then you could move
'timestam' into the partition key like this (using your notation):
PK((key String , timestam int), column1 string, col2 string) , list1 , list
2, list 3 .
Now the select query you showed should execute more consistently.
But of course something else might break...!
ml
On Fri, Mar 14, 2014 at 8:50 AM, Batranut Bogdan <ba...@yahoo.com> wrote:
> Hello all,
>
> Here is the environment:
>
> I have a 6 node Cassandra cluster. On each node I have:
> - 32 G RAM
> - 24 G RAM for cassa
> - ~150 - 200 MB/s disk speed
> - tomcat 6 with axis2 webservice that uses the datastax java driver to make
> asynch reads / writes
> - replication factor for the keyspace is 3
>
> (I know that there is a lot of heap but I also have write heavy tasks and
> I want them to get into mem fast) .
>
> All nodes in the same data center
> The clients that read / write are in the same datacenter so network is
> Gigabit.
>
> The table structure is like this: PK(key String , timestam int, column1
> string, col2 string) , list1 , list 2, list 3 .
> There are about 300 milions individual keys.
> There are about 100 timestamps for each key now, so the rows will get
> wider as time passes.
>
> I am using datastax java driver to query the cluster.
>
> I have ~450 queries that are like this: SELECT * FROM table where key =
> 'some string' and ts = some value; some value is close to present time.
>
> The problem:
>
> About 10 - 20 % of these queries take more than 5 seconds to execute, in
> fact, the majority of those take around 10 seconds.
> When investigating I saw that if I have a slow response and I redo the
> query it will finish in 8 - 10 MILIseconds like the rest of the queries
> that I have.
> I could not see using JConsole any spikes in CPU / memory when executing
> the queries. The rise in resource consumtion is very small on all nodes on
> the cluster. I expect such delays to be generated by a BIG increase in
> resource consumption.
>
> Any comments will be appreciated.
>
> Thank you.
>
>
>
>
>
Re: Cassandra slow on some reads
Posted by Batranut Bogdan <ba...@yahoo.com>.
Well the problem still persists.
Giving cassandra 12G of heap and having a look at the table I saw that caching='KEYS_ONLY' . Did not find how to disable caching for rows (I'm not sure if setting to 0 will disable it)
On Friday, March 14, 2014 3:14 PM, Andras Szerdahelyi <an...@ignitionone.com> wrote:
Is row cache enabled on this CF? Try disabling it. Seems like you might have a very wide row there.
Can you grep for GCInspector in your Cassandra log? 24G might be a bit too much for the Cassandra JVM, bogging down GC, and not leaving much to page cache ( 32G -24G - Tomcat ). I don’t quite understand your reasoning here:
> (I know that there is a lot of heap but I also have write heavy tasks and I want them to get into mem fast) .
So, I would try with default cassandra-env.sh JVM params too
From: Batranut Bogdan <ba...@yahoo.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>, Batranut Bogdan <ba...@yahoo.com>
Date: Friday 14 March 2014 13:50
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: Cassandra slow on some reads
Hello all,
Here is the environment:
I have a 6 node Cassandra cluster. On each node I have:
- 32 G RAM
- 24 G RAM for cassa
- ~150 - 200 MB/s disk speed
- tomcat 6 with axis2 webservice that uses the datastax java driver to make
asynch reads / writes
- replication factor for the keyspace is 3
(I know that there is a lot of heap but I also have write heavy tasks and I want them to get into mem fast) .
All nodes in the same data center
The clients that read / write are in the same datacenter so network is Gigabit.
The table structure is like this: PK(key String , timestam int, column1 string, col2 string) , list1 , list 2, list 3 .
There are about 300 milions individual keys.
There are about 100 timestamps for each key now, so the rows will get wider as time passes.
I am using datastax java driver to query the cluster.
I have ~450 queries that are like this: SELECT * FROM table where key = 'some string' and ts = some value; some value is close to present time.
The problem:
About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, the majority of those take around 10 seconds.
When investigating I saw that if I have a slow response and I redo the query it will finish in 8 - 10 MILIseconds like the rest of the queries that I have.
I could not see using JConsole any spikes in CPU / memory when executing the queries. The rise in resource consumtion is very small on all nodes on the cluster. I expect such delays to be generated by a BIG increase in resource consumption.
Any comments will be appreciated.
Thank you.
Re: Cassandra slow on some reads
Posted by Batranut Bogdan <ba...@yahoo.com>.
Ok will try to reduce heap and see what happens. Thanks guys. I' get back with conclusions.<a href="https://overview.mail.yahoo.com?.src=iOS"><br/><br/>Sent from Yahoo Mail for iPhone</a>
Re: Cassandra slow on some reads
Posted by Benedict Elliott Smith <be...@datastax.com>.
To add to this, if the 24G is the JVM limit, Cassandra will actually be
using even more than this for bloom filters etc. that are managed off-heap.
So the amount of page cache left is almost certainly inadequate.
On 14 March 2014 13:01, Andras Szerdahelyi <
andras.szerdahelyi@ignitionone.com> wrote:
> Is row cache enabled on this CF? Try disabling it. Seems like you might
> have a very wide row there.
>
> Can you grep for GCInspector in your Cassandra log? 24G might be a bit
> too much for the Cassandra JVM, bogging down GC, and not leaving much to
> page cache ( 32G -24G - Tomcat ). I don't quite understand your reasoning
> here:
> > (I know that there is a lot of heap but I also have write heavy tasks
> and I want them to get into mem fast) .
>
> So, I would try with default cassandra-env.sh JVM params too
>
> From: Batranut Bogdan <ba...@yahoo.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>,
> Batranut Bogdan <ba...@yahoo.com>
> Date: Friday 14 March 2014 13:50
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Cassandra slow on some reads
>
> Hello all,
>
> Here is the environment:
>
> I have a 6 node Cassandra cluster. On each node I have:
> - 32 G RAM
> - 24 G RAM for cassa
> - ~150 - 200 MB/s disk speed
> - tomcat 6 with axis2 webservice that uses the datastax java driver to make
> asynch reads / writes
> - replication factor for the keyspace is 3
>
> (I know that there is a lot of heap but I also have write heavy tasks and
> I want them to get into mem fast) .
>
> All nodes in the same data center
> The clients that read / write are in the same datacenter so network is
> Gigabit.
>
> The table structure is like this: PK(key String , timestam int, column1
> string, col2 string) , list1 , list 2, list 3 .
> There are about 300 milions individual keys.
> There are about 100 timestamps for each key now, so the rows will get
> wider as time passes.
>
> I am using datastax java driver to query the cluster.
>
> I have ~450 queries that are like this: SELECT * FROM table where key =
> 'some string' and ts = some value; some value is close to present time.
>
> The problem:
>
> About 10 - 20 % of these queries take more than 5 seconds to execute, in
> fact, the majority of those take around 10 seconds.
> When investigating I saw that if I have a slow response and I redo the
> query it will finish in 8 - 10 MILIseconds like the rest of the queries
> that I have.
> I could not see using JConsole any spikes in CPU / memory when executing
> the queries. The rise in resource consumtion is very small on all nodes on
> the cluster. I expect such delays to be generated by a BIG increase in
> resource consumption.
>
> Any comments will be appreciated.
>
> Thank you.
>
>
>
>
>
Re: Cassandra slow on some reads
Posted by Andras Szerdahelyi <an...@ignitionone.com>.
Is row cache enabled on this CF? Try disabling it. Seems like you might have a very wide row there.
Can you grep for GCInspector in your Cassandra log? 24G might be a bit too much for the Cassandra JVM, bogging down GC, and not leaving much to page cache ( 32G -24G - Tomcat ). I don’t quite understand your reasoning here:
> (I know that there is a lot of heap but I also have write heavy tasks and I want them to get into mem fast) .
So, I would try with default cassandra-env.sh JVM params too
From: Batranut Bogdan <ba...@yahoo.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>, Batranut Bogdan <ba...@yahoo.com>>
Date: Friday 14 March 2014 13:50
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Cassandra slow on some reads
Hello all,
Here is the environment:
I have a 6 node Cassandra cluster. On each node I have:
- 32 G RAM
- 24 G RAM for cassa
- ~150 - 200 MB/s disk speed
- tomcat 6 with axis2 webservice that uses the datastax java driver to make
asynch reads / writes
- replication factor for the keyspace is 3
(I know that there is a lot of heap but I also have write heavy tasks and I want them to get into mem fast) .
All nodes in the same data center
The clients that read / write are in the same datacenter so network is Gigabit.
The table structure is like this: PK(key String , timestam int, column1 string, col2 string) , list1 , list 2, list 3 .
There are about 300 milions individual keys.
There are about 100 timestamps for each key now, so the rows will get wider as time passes.
I am using datastax java driver to query the cluster.
I have ~450 queries that are like this: SELECT * FROM table where key = 'some string' and ts = some value; some value is close to present time.
The problem:
About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, the majority of those take around 10 seconds.
When investigating I saw that if I have a slow response and I redo the query it will finish in 8 - 10 MILIseconds like the rest of the queries that I have.
I could not see using JConsole any spikes in CPU / memory when executing the queries. The rise in resource consumtion is very small on all nodes on the cluster. I expect such delays to be generated by a BIG increase in resource consumption.
Any comments will be appreciated.
Thank you.
Re: Cassandra slow on some reads
Posted by Batranut Bogdan <ba...@yahoo.com>.
Whooops
On one of the nodes when running my tests I found an exception
java FileNotFoundException : file .... xxxx-Data.db not found
at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52
also got another one coming from
java.io.RandomAccessFile.open(native method)
java.io.RandomAccessFile..<init> (RandomAccesFile.java:241)
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
On Saturday, March 15, 2014 10:52 PM, Benedict Elliott Smith <be...@datastax.com> wrote:
Post the server logs and traces of one of the lengthy queries?
On 15 Mar 2014 20:49, "Batranut Bogdan" <ba...@yahoo.com> wrote:
Hello,
>
>
>Yes for the first query the server can be slow but no matter what is should not take 10 seconds to get one key from a column family. I can see this happening if I have times like 20 ms for some queries and let's say 300 ms for uncached ones. But I have responses that take from 8 to 20 ms (420 out of 450 queries) and 30 response that take from 5 to 13 seconds. It is this HUGE difference that bothers me.
>
>
>
>On Friday, March 14, 2014 9:18 PM, Manoj Khangaonkar <kh...@gmail.com> wrote:
>
>
>>
>>
>>
>>I have ~450 queries that are like this: SELECT * FROM table where key = 'some string' and ts = some value; some value is close to present time.
>>
>>
>>The problem:
>>
>>
>>About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, the majority of those take around 10 seconds.
>>When investigating I saw that if I have a slow response and I redo the query it will finish in 8 - 10 MILIseconds like the rest of the queries that I have.
>>
>>
>>
>
>Most likely for the first slow query, the server has to go looking in SSTables on disk. 2nd fast response is returned from memory. I am not sure how much data each node is managing. But try adding more nodes , so that each node is managing fewer keys. More nodes with recommended heap size of 8G is better than fewer nodes with bigger heap size.
>
>
>regards
>
>
>--
>http://khangaonkar.blogspot.com/
>
>
Re: Cassandra slow on some reads
Posted by Benedict Elliott Smith <be...@datastax.com>.
Post the server logs and traces of one of the lengthy queries?
On 15 Mar 2014 20:49, "Batranut Bogdan" <ba...@yahoo.com> wrote:
> Hello,
>
> Yes for the first query the server can be slow but no matter what is
> should not take 10 seconds to get one key from a column family. I can see
> this happening if I have times like 20 ms for some queries and let's say
> 300 ms for uncached ones. But I have responses that take from 8 to 20 ms
> (420 out of 450 queries) and 30 response that take from 5 to 13 seconds. It
> is this HUGE difference that bothers me.
>
>
> On Friday, March 14, 2014 9:18 PM, Manoj Khangaonkar <
> khangaonkar@gmail.com> wrote:
>
>
>
> I have ~450 queries that are like this: SELECT * FROM table where key =
> 'some string' and ts = some value; some value is close to present time.
>
> The problem:
>
> About 10 - 20 % of these queries take more than 5 seconds to execute, in
> fact, the majority of those take around 10 seconds.
> When investigating I saw that if I have a slow response and I redo the
> query it will finish in 8 - 10 MILIseconds like the rest of the queries
> that I have.
>
>
>
> Most likely for the first slow query, the server has to go looking in
> SSTables on disk. 2nd fast response is returned from memory. I am not sure
> how much data each node is managing. But try adding more nodes , so that
> each node is managing fewer keys. More nodes with recommended heap size of
> 8G is better than fewer nodes with bigger heap size.
>
> regards
>
> --
> http://khangaonkar.blogspot.com/
>
>
>
Re: Cassandra slow on some reads
Posted by Batranut Bogdan <ba...@yahoo.com>.
Hello,
Yes for the first query the server can be slow but no matter what is should not take 10 seconds to get one key from a column family. I can see this happening if I have times like 20 ms for some queries and let's say 300 ms for uncached ones. But I have responses that take from 8 to 20 ms (420 out of 450 queries) and 30 response that take from 5 to 13 seconds. It is this HUGE difference that bothers me.
On Friday, March 14, 2014 9:18 PM, Manoj Khangaonkar <kh...@gmail.com> wrote:
>
>
>
>I have ~450 queries that are like this: SELECT * FROM table where key = 'some string' and ts = some value; some value is close to present time.
>
>
>The problem:
>
>
>About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, the majority of those take around 10 seconds.
>When investigating I saw that if I have a slow response and I redo the query it will finish in 8 - 10 MILIseconds like the rest of the queries that I have.
>
>
>
Most likely for the first slow query, the server has to go looking in SSTables on disk. 2nd fast response is returned from memory. I am not sure how much data each node is managing. But try adding more nodes , so that each node is managing fewer keys. More nodes with recommended heap size of 8G is better than fewer nodes with bigger heap size.
regards
--
http://khangaonkar.blogspot.com/
Re: Cassandra slow on some reads
Posted by Manoj Khangaonkar <kh...@gmail.com>.
>
>
>
> I have ~450 queries that are like this: SELECT * FROM table where key =
> 'some string' and ts = some value; some value is close to present time.
>
> The problem:
>
> About 10 - 20 % of these queries take more than 5 seconds to execute, in
> fact, the majority of those take around 10 seconds.
> When investigating I saw that if I have a slow response and I redo the
> query it will finish in 8 - 10 MILIseconds like the rest of the queries
> that I have.
>
>
>
Most likely for the first slow query, the server has to go looking in
SSTables on disk. 2nd fast response is returned from memory. I am not sure
how much data each node is managing. But try adding more nodes , so that
each node is managing fewer keys. More nodes with recommended heap size of
8G is better than fewer nodes with bigger heap size.
regards
--
http://khangaonkar.blogspot.com/