You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ayub M <hi...@gmail.com> on 2021/04/28 22:43:52 UTC
tablehistogram shows high sstables
The table has 24 sstables with size tiered compaction, when I run nodetool
tablehistograms I see 99% percentile of the queries are showing up 24 as
the number of sstables. But the read latency is very low, my understanding
from the tableshistograms's sstable column is - it shows how many sstables
were read to complete the query. If so reading 24 sstables should take
sometime, at least maybe couple of seconds. Am I missing something here?
Does checking against index/bloom filters count towards sstable counter as
well?
Percentile SSTables Write Latency Read Latency Partition
Size Cell Count
(micros) (micros) (bytes)
50% 24.00 17.08 17436.92
310 6
75% 24.00 24.60 20924.30
446 6
95% 24.00 42.51 62479.63
770 10
98% 24.00 51.01 74975.55
1597 17
99% 24.00 61.21 74975.55
3311 24
Min 18.00 2.30 4866.32
87 0
Max 24.00 943.13 89970.66
545791 17084
Re: tablehistogram shows high sstables
Posted by MyWorld <ti...@gmail.com>.
High latency at table level could be a result of no of factors.
High tombstones
GC
Bloom filter fp
I/O bottleneck
Insufficient cache or threads
Since your partion size is not a problem here, try running major
compaction on this table.
Also you can turn on caching for this table.
As mentioned above check other factors and eliminate one by one.
Regards,
Ashish
On Sat 1 May, 2021, 06:41 vytenis silgalis, <vs...@gmail.com> wrote:
> 17ms read latency for the 50th percentile is actually a pretty high
> latency in my experience, I prefer to see the 75th percentile read latency
> to be around 1-2ms. Of course it depends on your use case and what your
> performance objectives are.
>
> On Thu, Apr 29, 2021 at 7:05 AM Kane Wilson <k...@raft.so> wrote:
>
>> It does imply the SSTables are being read - how big is your data size and
>> how much memory on the nodes? It's certainly possible to get low latencies
>> despite many SSTables, but I'd expect small read sizes paired with a lot of
>> memory.
>>
>>
>> raft.so - Cassandra consulting, support, managed services
>>
>> On Thu., 29 Apr. 2021, 08:44 Ayub M, <hi...@gmail.com> wrote:
>>
>>> The table has 24 sstables with size tiered compaction, when I run
>>> nodetool tablehistograms I see 99% percentile of the queries are showing up
>>> 24 as the number of sstables. But the read latency is very low, my
>>> understanding from the tableshistograms's sstable column is - it shows how
>>> many sstables were read to complete the query. If so reading 24 sstables
>>> should take sometime, at least maybe couple of seconds. Am I missing
>>> something here? Does checking against index/bloom filters count towards
>>> sstable counter as well?
>>>
>>> Percentile SSTables Write Latency Read Latency Partition Size Cell Count
>>> (micros) (micros) (bytes)
>>> 50% 24.00 17.08 17436.92 310 6
>>> 75% 24.00 24.60 20924.30 446 6
>>> 95% 24.00 42.51 62479.63 770 10
>>> 98% 24.00 51.01 74975.55 1597 17
>>> 99% 24.00 61.21 74975.55 3311 24
>>> Min 18.00 2.30 4866.32 87 0
>>> Max 24.00 943.13 89970.66 545791 17084
>>>
>>>
Re: tablehistogram shows high sstables
Posted by vytenis silgalis <vs...@gmail.com>.
17ms read latency for the 50th percentile is actually a pretty high latency
in my experience, I prefer to see the 75th percentile read latency to be
around 1-2ms. Of course it depends on your use case and what your
performance objectives are.
On Thu, Apr 29, 2021 at 7:05 AM Kane Wilson <k...@raft.so> wrote:
> It does imply the SSTables are being read - how big is your data size and
> how much memory on the nodes? It's certainly possible to get low latencies
> despite many SSTables, but I'd expect small read sizes paired with a lot of
> memory.
>
>
> raft.so - Cassandra consulting, support, managed services
>
> On Thu., 29 Apr. 2021, 08:44 Ayub M, <hi...@gmail.com> wrote:
>
>> The table has 24 sstables with size tiered compaction, when I run
>> nodetool tablehistograms I see 99% percentile of the queries are showing up
>> 24 as the number of sstables. But the read latency is very low, my
>> understanding from the tableshistograms's sstable column is - it shows how
>> many sstables were read to complete the query. If so reading 24 sstables
>> should take sometime, at least maybe couple of seconds. Am I missing
>> something here? Does checking against index/bloom filters count towards
>> sstable counter as well?
>>
>> Percentile SSTables Write Latency Read Latency Partition Size Cell Count
>> (micros) (micros) (bytes)
>> 50% 24.00 17.08 17436.92 310 6
>> 75% 24.00 24.60 20924.30 446 6
>> 95% 24.00 42.51 62479.63 770 10
>> 98% 24.00 51.01 74975.55 1597 17
>> 99% 24.00 61.21 74975.55 3311 24
>> Min 18.00 2.30 4866.32 87 0
>> Max 24.00 943.13 89970.66 545791 17084
>>
>>
Re: tablehistogram shows high sstables
Posted by Kane Wilson <k...@raft.so>.
It does imply the SSTables are being read - how big is your data size and
how much memory on the nodes? It's certainly possible to get low latencies
despite many SSTables, but I'd expect small read sizes paired with a lot of
memory.
raft.so - Cassandra consulting, support, managed services
On Thu., 29 Apr. 2021, 08:44 Ayub M, <hi...@gmail.com> wrote:
> The table has 24 sstables with size tiered compaction, when I run nodetool
> tablehistograms I see 99% percentile of the queries are showing up 24 as
> the number of sstables. But the read latency is very low, my understanding
> from the tableshistograms's sstable column is - it shows how many sstables
> were read to complete the query. If so reading 24 sstables should take
> sometime, at least maybe couple of seconds. Am I missing something here?
> Does checking against index/bloom filters count towards sstable counter as
> well?
>
> Percentile SSTables Write Latency Read Latency Partition Size Cell Count
> (micros) (micros) (bytes)
> 50% 24.00 17.08 17436.92 310 6
> 75% 24.00 24.60 20924.30 446 6
> 95% 24.00 42.51 62479.63 770 10
> 98% 24.00 51.01 74975.55 1597 17
> 99% 24.00 61.21 74975.55 3311 24
> Min 18.00 2.30 4866.32 87 0
> Max 24.00 943.13 89970.66 545791 17084
>
>