Multi dc replication failing

[Inquistive allen <in...@gmail.com>]

Hello Team,

I have a multi dc Cassandra setup.
We write and read using local quorum to a particular datacenter A

The replica definition in schema is correct.
Sometimes we don't see data getting correctly replicated to datacenter B.

I need to know what may be different ways to troubleshoot this.
What are the key metrics, details, log patterns I may look for

Thanks

Re: Multi dc replication failing

[Bowen Song <bo...@bso.ng>]

You've sent the email from a different email address. I will assume I'm 
writing to the same person for now.

Normally, you shouldn't persistently have a different schema version 
between DCs or between nodes in the same DC. However, this can 
occasionally happen, for example when multiple schema changing 
statements, such as "CREATE TABLE", are sent to two nodes concurrently 
and causing a schema collision. If this happens to you, you can follow 
this guide 
<https://docs.datastax.com/en/dse/5.1/cql/cql/cql_using/useCreateTableCollisionFix.html> 
to fix it.

On 21/02/2022 15:30, DB-men wrote:
> Sure, thanks Bowen for the reply.
>
> I found the DC where we are observing data not getting replicated is 
> currently having a different schema version as I see
>
> All other DC have a schema version in common.
>
> Does that signify anything. May that attribute towards data 
> inconsistency. I just want to understand what is the problem with 
> different schema versions. All i can quickly think of is tables 
> definitions may be different. What else are the repurcsions of 
> multiple schema versions.
>
>
> I just found this out that the schema versions are different.
> I think the only way to fix this is to restart the Cassandra dc.
>
> Kindly comment
>
> Thanks
>
> On Mon, 21 Feb, 2022, 7:56 pm Bowen Song, <bo...@bso.ng> wrote:
>
>     What do you mean by saying "correctly replicated"? Keep in mind by
>     using
>     LOCAL_QUORUM, you should *not* expect the data to be strongly
>     consistent
>     when the read and write are in different DCs. If that's desired, use
>     EACH_QUORUM on either read or write, and keep the other at
>     LOCAL_QUORUM.
>
>     On 21/02/2022 13:16, Inquistive allen wrote:
>     > Hello Team,
>     >
>     > I have a multi dc Cassandra setup.
>     > We write and read using local quorum to a particular datacenter A
>     >
>     > The replica definition in schema is correct.
>     > Sometimes we don't see data getting correctly replicated to
>     datacenter B.
>     >
>     > I need to know what may be different ways to troubleshoot this.
>     > What are the key metrics, details, log patterns I may look for
>     >
>     > Thanks
>     >
>

Re: Multi dc replication failing

[DB-men <ra...@gmail.com>]

Sure, thanks Bowen for the reply.

I found the DC where we are observing data not getting replicated is
currently having a different schema version as I see

All other DC have a schema version in common.

Does that signify anything. May that attribute towards data inconsistency.
I just want to understand what is the problem with different schema
versions. All i can quickly think of is tables definitions may be
different. What else are the repurcsions of multiple schema versions.


I just found this out that the schema versions are different.
I think the only way to fix this is to restart the Cassandra dc.

Kindly comment

Thanks

On Mon, 21 Feb, 2022, 7:56 pm Bowen Song, <bo...@bso.ng> wrote:

> What do you mean by saying "correctly replicated"? Keep in mind by using
> LOCAL_QUORUM, you should *not* expect the data to be strongly consistent
> when the read and write are in different DCs. If that's desired, use
> EACH_QUORUM on either read or write, and keep the other at LOCAL_QUORUM.
>
> On 21/02/2022 13:16, Inquistive allen wrote:
> > Hello Team,
> >
> > I have a multi dc Cassandra setup.
> > We write and read using local quorum to a particular datacenter A
> >
> > The replica definition in schema is correct.
> > Sometimes we don't see data getting correctly replicated to datacenter B.
> >
> > I need to know what may be different ways to troubleshoot this.
> > What are the key metrics, details, log patterns I may look for
> >
> > Thanks
> >
>

Re: Multi dc replication failing

[Bowen Song <bo...@bso.ng>]

What do you mean by saying "correctly replicated"? Keep in mind by using 
LOCAL_QUORUM, you should *not* expect the data to be strongly consistent 
when the read and write are in different DCs. If that's desired, use 
EACH_QUORUM on either read or write, and keep the other at LOCAL_QUORUM.

On 21/02/2022 13:16, Inquistive allen wrote:
> Hello Team,
>
> I have a multi dc Cassandra setup.
> We write and read using local quorum to a particular datacenter A
>
> The replica definition in schema is correct.
> Sometimes we don't see data getting correctly replicated to datacenter B.
>
> I need to know what may be different ways to troubleshoot this.
> What are the key metrics, details, log patterns I may look for
>
> Thanks
>