CF performance suddenly degraded

[Paolo Crosato <pa...@targaubiest.com>]

Hi,

I declared this CF:

CREATE TABLE timesliceunitstate (
   day timestamp,
   unitbuckets text,
   PRIMARY KEY (day)
);

unitbuckets is a text column holding a fairly big amount of data, around 
30 MB of json text per row.

The table is holding 30 rows, I'm running cassandra 2.0.8 on a 3 nodes 
cluster with replication factor of 3, consistency of reads is quorum, so 
2 out of 3 nodes.

The table has a write hit about every 20 minutes, which updates only one 
row, the most recent.

I had no problem with read queries (I query the table one row at time) 
until this morning, when read latency jumped from around 300ms to 20 
seconds for each query.

I tried repairing the table on all the 3 nodes using range repair 
without success, the *Data.db file on the disk is aorund 30 MB, so a bit 
less than 1MB for each row.

I'm using latest version of datastax driver, 2.1. I changed nothing on 
the application level since days, so it's not something related to the 
applicacation or the driver.

Is there anyway I can troubleshoot the issue and discover what's making 
the table so slow?

Thanks for any advice,

Paolo

-- 
Paolo Crosato
Software engineer/Custom Solutions
e-mail: paolo.crosato@targaubiest.com

Re: CF performance suddenly degraded

[Ryan Svihla <rs...@datastax.com>]

There can be many root causes. Would need a lot more information such as
node hardware specs, cf histograms on the table, tpstats,GC settings (Max
heap, parnew, JVM version) and logs with specifically any ERROR, WARN, or
GCInspector messages

As a start a simple trace of the query in question is probably good enough.
On Dec 22, 2014 10:59 AM, "Paolo Crosato" <pa...@targaubiest.com>
wrote:

> Hi,
>
> I declared this CF:
>
> CREATE TABLE timesliceunitstate (
>   day timestamp,
>   unitbuckets text,
>   PRIMARY KEY (day)
> );
>
> unitbuckets is a text column holding a fairly big amount of data, around
> 30 MB of json text per row.
>
> The table is holding 30 rows, I'm running cassandra 2.0.8 on a 3 nodes
> cluster with replication factor of 3, consistency of reads is quorum, so 2
> out of 3 nodes.
>
> The table has a write hit about every 20 minutes, which updates only one
> row, the most recent.
>
> I had no problem with read queries (I query the table one row at time)
> until this morning, when read latency jumped from around 300ms to 20
> seconds for each query.
>
> I tried repairing the table on all the 3 nodes using range repair without
> success, the *Data.db file on the disk is aorund 30 MB, so a bit less than
> 1MB for each row.
>
> I'm using latest version of datastax driver, 2.1. I changed nothing on the
> application level since days, so it's not something related to the
> applicacation or the driver.
>
> Is there anyway I can troubleshoot the issue and discover what's making
> the table so slow?
>
> Thanks for any advice,
>
> Paolo
>
> --
> Paolo Crosato
> Software engineer/Custom Solutions
> e-mail: paolo.crosato@targaubiest.com
>
>