nodetool tablestats reporting local read count of 0, incorrectly

[Voytek Jarnot <vo...@gmail.com>]

Cassandra 3.9

Have a keyspace with 5 tables, one of which is exhibiting rather poor read
performance. In starting an attempt to get to the bottom of the issues, I
noticed that, when running nodetool tablestats against the keyspace, that
particular table reports "Local read count: 0" on all nodes - which is
incorrect.

It tallies "Local write count", presumably correctly, as at least it's not
0. Other tables in the keyspace do not exhibit this behavior, as they
provide non-zero numbers for both read and write values.

Is this perhaps indicative of a deeper issue with this particular table?

Thank you.

Re: nodetool tablestats reporting local read count of 0, incorrectly

[Jeff Jirsa <jj...@apache.org>]

On 2017-04-03 12:42 (-0700), Voytek Jarnot <vo...@gmail.com> wrote: 
> Continuing to grasp at straws...
> 
> Is it possible that indexing is modifying the read path such that the
> tablestats/tablehistograms output is no longer trustworthy?  I notice more
> realistic "local read count" numbers on tables which do not utilize SASI.
> 
> Would greatly appreciate any thoughts,
> Thanks.
> 

Not the most ridiculous thought, though that would not be expected. Would be great if you could open a JIRA.

Re: nodetool tablestats reporting local read count of 0, incorrectly

[Voytek Jarnot <vo...@gmail.com>]

Continuing to grasp at straws...

Is it possible that indexing is modifying the read path such that the
tablestats/tablehistograms output is no longer trustworthy?  I notice more
realistic "local read count" numbers on tables which do not utilize SASI.

Would greatly appreciate any thoughts,
Thanks.

On Mon, Apr 3, 2017 at 9:56 AM, Voytek Jarnot <vo...@gmail.com>
wrote:

> Further info - tablehistograms reports zeros for all percentiles for Read
> Latency; tablestats also reports really low numbers for Bloom filter usage
> (3-4 KiB, depending on node, whereas I'd expect orders of magnitude more
> given other - less accessed - tables in this keyspace).  This is the most
> written-to and read-from table in the keyspace, seems to keep up with
> tracking of writes, but not reads.
>
> Full repair on this table is the only thing I can think of; but that's a
> guess and doesn't get me any closer to understanding what has happened.
>
> On Fri, Mar 31, 2017 at 11:11 PM, Voytek Jarnot <vo...@gmail.com>
> wrote:
>
>> Cassandra 3.9
>>
>> Have a keyspace with 5 tables, one of which is exhibiting rather poor
>> read performance. In starting an attempt to get to the bottom of the
>> issues, I noticed that, when running nodetool tablestats against the
>> keyspace, that particular table reports "Local read count: 0" on all nodes
>> - which is incorrect.
>>
>> It tallies "Local write count", presumably correctly, as at least it's
>> not 0. Other tables in the keyspace do not exhibit this behavior, as they
>> provide non-zero numbers for both read and write values.
>>
>> Is this perhaps indicative of a deeper issue with this particular table?
>>
>> Thank you.
>>
>
>

Re: nodetool tablestats reporting local read count of 0, incorrectly

[Voytek Jarnot <vo...@gmail.com>]

Further info - tablehistograms reports zeros for all percentiles for Read
Latency; tablestats also reports really low numbers for Bloom filter usage
(3-4 KiB, depending on node, whereas I'd expect orders of magnitude more
given other - less accessed - tables in this keyspace).  This is the most
written-to and read-from table in the keyspace, seems to keep up with
tracking of writes, but not reads.

Full repair on this table is the only thing I can think of; but that's a
guess and doesn't get me any closer to understanding what has happened.

On Fri, Mar 31, 2017 at 11:11 PM, Voytek Jarnot <vo...@gmail.com>
wrote:

> Cassandra 3.9
>
> Have a keyspace with 5 tables, one of which is exhibiting rather poor read
> performance. In starting an attempt to get to the bottom of the issues, I
> noticed that, when running nodetool tablestats against the keyspace, that
> particular table reports "Local read count: 0" on all nodes - which is
> incorrect.
>
> It tallies "Local write count", presumably correctly, as at least it's not
> 0. Other tables in the keyspace do not exhibit this behavior, as they
> provide non-zero numbers for both read and write values.
>
> Is this perhaps indicative of a deeper issue with this particular table?
>
> Thank you.
>