You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Subroto Barua <sb...@yahoo.com.INVALID> on 2019/02/01 16:45:05 UTC
Help with sudden spike in read requests
In our production cluster, we observed sudden spike (over 160 MB/s) in read requests on *all* Cassandra nodes for a very short period (less than a min); this event happens few times a day.
I am not able to get to the bottom of this issue, nothing interesting in system.log or from app level; repair was not running
Does anyone have any thoughts on what could have triggered this event? Under what condition C* (if it is tied to c*) will trigger this type of event?
Thanks!
Subroto
RE: Help with sudden spike in read requests
Posted by Kenneth Brotman <ke...@yahoo.com.INVALID>.
If it’s a legacy write table why does it write 10% of the time? Maybe it’s the design of the big legacy table you mentioned. It could be so many things.
Is it the same time of day?
Same days of the week or month?
Are there analytics run at that time?
What are you using for monitoring and how did you find out it was happening?
Is this a DSE cluster or OSS Cassandra cluster?
Kenneth Brotman
From: Subroto Barua [mailto:sbarua116@yahoo.com.INVALID]
Sent: Friday, February 01, 2019 10:48 AM
To: user@cassandra.apache.org
Subject: Re: Help with sudden spike in read requests
We migrated one of the application from on-Prem to aws; the queries are very light, more like registration info;
Queries from the new app is via pk of data type, “text”, no cc (this table has about 200 rows; however the legacy table (more like reference table) has several million rows, about 800 sstables per node, using lcs (9:1, read-write ratio)
Subroto
On Feb 1, 2019, at 10:33 AM, Kenneth Brotman <ke...@yahoo.com.invalid> wrote:
Do you have that many queries? You could just review them and your data model to see if there was an error of some kind. How long has it been happening? What changed since it started happening?
Kenneth Brotman
From: Subroto Barua [mailto:sbarua116@yahoo.com.INVALID]
Sent: Friday, February 01, 2019 10:13 AM
To: user@cassandra.apache.org
Subject: Re: Help with sudden spike in read requests
Vnode is 256
C*: 3.0.15 on m4.4xlarge gp2 vol
There are 2 more DCs on bare metal (raid 10 and older machines) attached to this cluster and we have not seen this behavior on on-prem servers
If this event is triggered by some bad query/queries, what is the best way to trap it?
Subroto
On Feb 1, 2019, at 8:55 AM, Kenneth Brotman <ke...@yahoo.com.invalid> wrote:
If you had a query that went across the partitions and especially if you had vNodes set high, that would do it.
Kenneth Brotman
From: Subroto Barua [mailto:sbarua116@yahoo.com.INVALID]
Sent: Friday, February 01, 2019 8:45 AM
To: User cassandra.apache.org <http://cassandraapache.org>
Subject: Help with sudden spike in read requests
In our production cluster, we observed sudden spike (over 160 MB/s) in read requests on *all* Cassandra nodes for a very short period (less than a min); this event happens few times a day.
I am not able to get to the bottom of this issue, nothing interesting in system.log or from app level; repair was not running
Does anyone have any thoughts on what could have triggered this event? Under what condition C* (if it is tied to c*) will trigger this type of event?
Thanks!
Subroto
Re: Help with sudden spike in read requests
Posted by Subroto Barua <sb...@yahoo.com.INVALID>.
We migrated one of the application from on-Prem to aws; the queries are very light, more like registration info;
Queries from the new app is via pk of data type, “text”, no cc (this table has about 200 rows; however the legacy table (more like reference table) has several million rows, about 800 sstables per node, using lcs (9:1, read-write ratio)
Subroto
> On Feb 1, 2019, at 10:33 AM, Kenneth Brotman <ke...@yahoo.com.invalid> wrote:
>
> Do you have that many queries? You could just review them and your data model to see if there was an error of some kind. How long has it been happening? What changed since it started happening?
>
> Kenneth Brotman
>
> From: Subroto Barua [mailto:sbarua116@yahoo.com.INVALID]
> Sent: Friday, February 01, 2019 10:13 AM
> To: user@cassandra.apache.org
> Subject: Re: Help with sudden spike in read requests
>
> Vnode is 256
> C*: 3.0.15 on m4.4xlarge gp2 vol
>
> There are 2 more DCs on bare metal (raid 10 and older machines) attached to this cluster and we have not seen this behavior on on-prem servers
>
> If this event is triggered by some bad query/queries, what is the best way to trap it?
>
> Subroto
>
> On Feb 1, 2019, at 8:55 AM, Kenneth Brotman <ke...@yahoo.com.invalid> wrote:
>
> If you had a query that went across the partitions and especially if you had vNodes set high, that would do it.
>
> Kenneth Brotman
>
> From: Subroto Barua [mailto:sbarua116@yahoo.com.INVALID]
> Sent: Friday, February 01, 2019 8:45 AM
> To: User cassandra.apache.org
> Subject: Help with sudden spike in read requests
>
> In our production cluster, we observed sudden spike (over 160 MB/s) in read requests on *all* Cassandra nodes for a very short period (less than a min); this event happens few times a day.
>
> I am not able to get to the bottom of this issue, nothing interesting in system.log or from app level; repair was not running
>
> Does anyone have any thoughts on what could have triggered this event? Under what condition C* (if it is tied to c*) will trigger this type of event?
>
> Thanks!
>
> Subroto
RE: Help with sudden spike in read requests
Posted by Kenneth Brotman <ke...@yahoo.com.INVALID>.
Do you have that many queries? You could just review them and your data model to see if there was an error of some kind. How long has it been happening? What changed since it started happening?
Kenneth Brotman
From: Subroto Barua [mailto:sbarua116@yahoo.com.INVALID]
Sent: Friday, February 01, 2019 10:13 AM
To: user@cassandra.apache.org
Subject: Re: Help with sudden spike in read requests
Vnode is 256
C*: 3.0.15 on m4.4xlarge gp2 vol
There are 2 more DCs on bare metal (raid 10 and older machines) attached to this cluster and we have not seen this behavior on on-prem servers
If this event is triggered by some bad query/queries, what is the best way to trap it?
Subroto
On Feb 1, 2019, at 8:55 AM, Kenneth Brotman <ke...@yahoo.com.invalid> wrote:
If you had a query that went across the partitions and especially if you had vNodes set high, that would do it.
Kenneth Brotman
From: Subroto Barua [mailto:sbarua116@yahoo.com.INVALID]
Sent: Friday, February 01, 2019 8:45 AM
To: User cassandra.apache.org <http://cassandraapache.org>
Subject: Help with sudden spike in read requests
In our production cluster, we observed sudden spike (over 160 MB/s) in read requests on *all* Cassandra nodes for a very short period (less than a min); this event happens few times a day.
I am not able to get to the bottom of this issue, nothing interesting in system.log or from app level; repair was not running
Does anyone have any thoughts on what could have triggered this event? Under what condition C* (if it is tied to c*) will trigger this type of event?
Thanks!
Subroto
Re: Help with sudden spike in read requests
Posted by Subroto Barua <sb...@yahoo.com.INVALID>.
Vnode is 256
C*: 3.0.15 on m4.4xlarge gp2 vol
There are 2 more DCs on bare metal (raid 10 and older machines) attached to this cluster and we have not seen this behavior on on-prem servers
If this event is triggered by some bad query/queries, what is the best way to trap it?
Subroto
> On Feb 1, 2019, at 8:55 AM, Kenneth Brotman <ke...@yahoo.com.invalid> wrote:
>
> If you had a query that went across the partitions and especially if you had vNodes set high, that would do it.
>
> Kenneth Brotman
>
> From: Subroto Barua [mailto:sbarua116@yahoo.com.INVALID]
> Sent: Friday, February 01, 2019 8:45 AM
> To: User cassandra.apache.org
> Subject: Help with sudden spike in read requests
>
> In our production cluster, we observed sudden spike (over 160 MB/s) in read requests on *all* Cassandra nodes for a very short period (less than a min); this event happens few times a day.
>
> I am not able to get to the bottom of this issue, nothing interesting in system.log or from app level; repair was not running
>
> Does anyone have any thoughts on what could have triggered this event? Under what condition C* (if it is tied to c*) will trigger this type of event?
>
> Thanks!
>
> Subroto
RE: Help with sudden spike in read requests
Posted by Kenneth Brotman <ke...@yahoo.com.INVALID>.
If you had a query that went across the partitions and especially if you had vNodes set high, that would do it.
Kenneth Brotman
From: Subroto Barua [mailto:sbarua116@yahoo.com.INVALID]
Sent: Friday, February 01, 2019 8:45 AM
To: User cassandra.apache.org
Subject: Help with sudden spike in read requests
In our production cluster, we observed sudden spike (over 160 MB/s) in read requests on *all* Cassandra nodes for a very short period (less than a min); this event happens few times a day.
I am not able to get to the bottom of this issue, nothing interesting in system.log or from app level; repair was not running
Does anyone have any thoughts on what could have triggered this event? Under what condition C* (if it is tied to c*) will trigger this type of event?
Thanks!
Subroto