You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Joe Obernberger <jo...@gmail.com> on 2021/11/29 16:22:43 UTC
High read Latency
I have an 11 node cluster and am experiencing high read latency on one
table. This table has ~112 million rows:
nodetool tablehistograms doc.origdoc
doc/origdoc histograms
Percentile Read Latency Write Latency SSTables Partition
Size Cell Count
(micros) (micros) (bytes)
50% 36157.19 0.00 1.00
310 4
75% 74975.55 0.00 1.00
372 4
95% 155469.30 0.00 1.00
642 4
98% 223875.79 0.00 1.00
924 4
99% 268650.95 0.00 1.00
924 4
Min 152.32 0.00 1.00
180 4
Max 464228.84 0.00 1.00
9887 17
What should I look for to debug?
Thank you!
-Joe
Re: High read Latency
Posted by Joe Obernberger <jo...@gmail.com>.
To add onto this message:
Queries are all on the partition key (select
origvalue,ingestdate,mediatype from doc.origdoc where uuid=?). Queries
were very fast when the table was <10 million rows.
Table description:
describe doc.origdoc;
CREATE TABLE doc.origdoc (
uuid text,
ingestdate timestamp,
markings text,
mediatype text,
origvalue text,
source text,
PRIMARY KEY (uuid, ingestdate)
) WITH CLUSTERING ORDER BY (ingestdate ASC)
AND additional_write_policy = '99p'
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND cdc = false
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '16', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND default_time_to_live = 0
AND extensions = {}
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair = 'BLOCKING'
AND speculative_retry = '99p';
-Joe
On 11/29/2021 11:22 AM, Joe Obernberger wrote:
> I have an 11 node cluster and am experiencing high read latency on one
> table. This table has ~112 million rows:
>
> nodetool tablehistograms doc.origdoc
> doc/origdoc histograms
> Percentile Read Latency Write Latency SSTables Partition
> Size Cell Count
> (micros) (micros) (bytes)
> 50% 36157.19 0.00 1.00 310 4
> 75% 74975.55 0.00 1.00 372 4
> 95% 155469.30 0.00 1.00 642 4
> 98% 223875.79 0.00 1.00 924 4
> 99% 268650.95 0.00 1.00 924 4
> Min 152.32 0.00 1.00 180 4
> Max 464228.84 0.00 1.00
> 9887 17
>
> What should I look for to debug?
>
> Thank you!
>
> -Joe
>