You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Jaydeep Chovatia <ch...@gmail.com> on 2022/09/02 20:56:08 UTC
Cassandra Token ownership split-brain (3.0.14)
Hi,
We are running a production Cassandra version (3.0.14) with 256 tokens
v-node configuration. Occasionally, we see that different nodes show
different ownership for the same key. Only a node restart corrects;
otherwise, it continues to behave in a split-brain.
Say, for example,
*NodeA*
nodetool getendpoints ks1 table1 10
- n1
- n2
- n3
*NodeB*
nodetool getendpoints ks1 table1 10
- n1
- n2
*- n5*
If I restart NodeB, then it shows the correct ownership {n1,n2,n3}. The
majority of the nodes in the ring show correct ownership {n1,n2,n3}, only a
few show this issue, and restarting them solves the problem.
To me, it seems I think Cassandra's Gossip cache and StorageService cache
(TokenMetadata) are having some sort of cache coherence.
Anyone has observed this behavior?
Any help would be highly appreciated.
Jaydeep
Re: Cassandra Token ownership split-brain (3.0.14)
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Thanks Scott. I will prioritize upgrading to 3.0.27 and will circle back if
this issue persists.
Jaydeep
On Tue, Sep 6, 2022 at 3:45 PM C. Scott Andreas <sc...@paradoxica.net>
wrote:
> Hi Jaydeep,
>
> Thanks for reaching out and for bumping this thread.
>
> This is probably not the answer you’re after, but mentioning as it may
> address the issue.
>
> C* 3.0.14 was released over five years ago, with many hundreds of
> important bug fixes landing since July 2017. These include fixes for issues
> that have affected gossip in the past which may be related to this issue.
> Note that 3.0.14 also is susceptible to several critical data loss bugs
> including C-14513 and C-14515.
>
> I’d strongly recommend upgrading to Cassandra 3.0.27 as a starting point.
> If this doesn’t resolve your issue, members of the community may be in a
> better position to help triage a bug report against a current release of
> the database.
>
> - Scott
>
> On Sep 6, 2022, at 5:13 PM, Jaydeep Chovatia <ch...@gmail.com>
> wrote:
>
>
> If anyone has seen this issue and knows a fix, it would be a great help!
> Thanks in advance.
>
> Jaydeep
>
> On Fri, Sep 2, 2022 at 1:56 PM Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> Hi,
>>
>> We are running a production Cassandra version (3.0.14) with 256 tokens
>> v-node configuration. Occasionally, we see that different nodes show
>> different ownership for the same key. Only a node restart corrects;
>> otherwise, it continues to behave in a split-brain.
>>
>> Say, for example,
>>
>> *NodeA*
>> nodetool getendpoints ks1 table1 10
>> - n1
>> - n2
>> - n3
>>
>> *NodeB*
>> nodetool getendpoints ks1 table1 10
>> - n1
>> - n2
>> *- n5*
>>
>> If I restart NodeB, then it shows the correct ownership {n1,n2,n3}. The
>> majority of the nodes in the ring show correct ownership {n1,n2,n3}, only a
>> few show this issue, and restarting them solves the problem.
>>
>> To me, it seems I think Cassandra's Gossip cache and StorageService cache
>> (TokenMetadata) are having some sort of cache coherence.
>>
>> Anyone has observed this behavior?
>> Any help would be highly appreciated.
>>
>> Jaydeep
>>
>
Re: Cassandra Token ownership split-brain (3.0.14)
Posted by "C. Scott Andreas" <sc...@paradoxica.net>.
Hi Jaydeep,
Thanks for reaching out and for bumping this thread.
This is probably not the answer you’re after, but mentioning as it may address
the issue.
C* 3.0.14 was released over five years ago, with many hundreds of important
bug fixes landing since July 2017. These include fixes for issues that have
affected gossip in the past which may be related to this issue. Note that
3.0.14 also is susceptible to several critical data loss bugs including
C-14513 and C-14515.
I’d strongly recommend upgrading to Cassandra 3.0.27 as a starting point. If
this doesn’t resolve your issue, members of the community may be in a better
position to help triage a bug report against a current release of the
database.
\- Scott
> On Sep 6, 2022, at 5:13 PM, Jaydeep Chovatia <ch...@gmail.com>
> wrote:
>
>
>
>
> If anyone has seen this issue and knows a fix, it would be a great help!
>
>
> Thanks in advance.
>
>
>
>
>
> Jaydeep
>
>
>
>
>
> On Fri, Sep 2, 2022 at 1:56 PM Jaydeep Chovatia
> <[chovatia.jaydeep@gmail.com](mailto:chovatia.jaydeep@gmail.com)> wrote:
>
>
>> Hi,
>>
>>
>
>>
>> We are running a production Cassandra version (3.0.14) with 256 tokens
v-node configuration. Occasionally, we see that different nodes show different
ownership for the same key. Only a node restart corrects; otherwise, it
continues to behave in a split-brain.
>>
>>
>
>>
>> Say, for example,
>
>>
>>
>
>>
>> _NodeA_
>
>>
>> nodetool getendpoints ks1 table1 10
>>
>> \- n1
>>
>> \- n2
>>
>> \- n3
>>
>>
>
>>
>> _NodeB_
>
>>
>> nodetool getendpoints ks1 table1 10
>>
>> \- n1
>>
>> \- n2
>>
>> **\- n5**
>>
>>
>
>>
>> If I restart NodeB, then it shows the correct ownership {n1,n2,n3}. The
majority of the nodes in the ring show correct ownership {n1,n2,n3}, only a
few show this issue, and restarting them solves the problem.
>
>>
>>
>
>>
>> To me, it seems I think Cassandra's Gossip cache and StorageService cache
(TokenMetadata) are having some sort of cache coherence.
>
>>
>>
>
>>
>> Anyone has observed this behavior?
>
>>
>> Any help would be highly appreciated.
>
>>
>>
>
>>
>> Jaydeep
>
Re: Cassandra Token ownership split-brain (3.0.14)
Posted by Jaydeep Chovatia <ch...@gmail.com>.
If anyone has seen this issue and knows a fix, it would be a great help!
Thanks in advance.
Jaydeep
On Fri, Sep 2, 2022 at 1:56 PM Jaydeep Chovatia <ch...@gmail.com>
wrote:
> Hi,
>
> We are running a production Cassandra version (3.0.14) with 256 tokens
> v-node configuration. Occasionally, we see that different nodes show
> different ownership for the same key. Only a node restart corrects;
> otherwise, it continues to behave in a split-brain.
>
> Say, for example,
>
> *NodeA*
> nodetool getendpoints ks1 table1 10
> - n1
> - n2
> - n3
>
> *NodeB*
> nodetool getendpoints ks1 table1 10
> - n1
> - n2
> *- n5*
>
> If I restart NodeB, then it shows the correct ownership {n1,n2,n3}. The
> majority of the nodes in the ring show correct ownership {n1,n2,n3}, only a
> few show this issue, and restarting them solves the problem.
>
> To me, it seems I think Cassandra's Gossip cache and StorageService cache
> (TokenMetadata) are having some sort of cache coherence.
>
> Anyone has observed this behavior?
> Any help would be highly appreciated.
>
> Jaydeep
>