Local read request going across DC

[manish khandelwal <ma...@gmail.com>]

Hi All

In one of our cluster, read request with consistency "LOCAL_QUORUM" is
going across DC. When we run query setting CONSISTECY to LOCAL_QUORUM in
cqlsh, with tracing on, we see READ and digest request sent across to nodes
on  other DC. I have checked gossipinfo, peers table, nodetool status. No
big gc pauses. Repairs are running fine.  In logs we see a lot of messages
informing us about slow queries running above 500 ms. Cluster is on
Cassandra version 3.11.2 and this particular cluster has reported this
issue. Any hint where I should see to further debug the issue would be of
great help.

Regards
Manish

Re: Local read request going across DC

[Raphael Mazelier <ra...@futomaki.net>]

I lost the beginning of the thread but I raised what seems the same 
issue recently in this list.

I have a worldwide c* cluster with 3dcs (eu, us, ap 8nodes each). For 
whatever reason I have read with consistency set to local_quorum which 
trigger reads on other dc(s). I would say it's 1% of the queries. 
read_repair is set to 0. I follow this thread with interest. C* version 
is 3.11.4.

--
Raphael Mazelier

On 21/09/2022 09:23, Erick Ramirez wrote:
> Would you be open to temporarily posting the full CQL command + full 
> trace output to gist.github.com <http://gist.github.com>? I'd like to 
> see what it shows. Cheers!
>

Re: Local read request going across DC

[manish khandelwal <ma...@gmail.com>]

Thanks a lot Erick for your help and support.

Regards
Manish

On Wed, Sep 21, 2022, 19:12 Erick Ramirez <er...@apache.org> wrote:

> Just to circle back here, I've reviewed the trace output and it shows
> multiple requests fired off for lookups on the `roles` table which
> indicated to me that the default `cassandra` superuser is being used. To be
> clear, the original read request is being executed at the configured
> consistency BUT authenticating with the default `cassandra` superuser
> requires `QUORUM` so it spans across DCs.
>
> For this reason, we recommend that the default `cassandra` superuser only
> be used for provisioning a new admin role then should not be used again.
> Cheers!
>
>>

Re: Local read request going across DC

[Erick Ramirez <er...@apache.org>]

Just to circle back here, I've reviewed the trace output and it shows
multiple requests fired off for lookups on the `roles` table which
indicated to me that the default `cassandra` superuser is being used. To be
clear, the original read request is being executed at the configured
consistency BUT authenticating with the default `cassandra` superuser
requires `QUORUM` so it spans across DCs.

For this reason, we recommend that the default `cassandra` superuser only
be used for provisioning a new admin role then should not be used again.
Cheers!

>

Re: Local read request going across DC

[Erick Ramirez <er...@apache.org>]

Would you be open to temporarily posting the full CQL command + full trace
output to gist.github.com? I'd like to see what it shows. Cheers!

>

Re: Local read request going across DC

[manish khandelwal <ma...@gmail.com>]

Thanks Erick for the response.

read_repair_chance is 0. Can speculative_retry cause this? We have that set
at 99 percentile.

Regards
Manish

On Wed, Sep 21, 2022 at 11:17 AM Erick Ramirez <er...@apache.org>
wrote:

> It sounds like read-repair chance is enabled on the table. Check the table
> schema for a non-zero read_repair_chance. Cheers!
>
>>

Re: Local read request going across DC

[Erick Ramirez <er...@apache.org>]

It sounds like read-repair chance is enabled on the table. Check the table
schema for a non-zero read_repair_chance. Cheers!

>