Corrupt SSTable

[Pushpendra Rajpoot <pu...@gmail.com>]

Hi,

I have a cluster of 2 DC, each DC has 5 nodes in production. This cluster
is based on active-passive model i.e. application is writing data on one DC
(Active) & it's replicated to other DC (Passive).

My Passive DC has corrupt sstables (3 nodes out of 5 nodes) whereas there
are no corrupt sstables on any node in Active DC.

Now, I am planning to do the following activity on affected nodes in
Passive DC:
1. delete keyspace having corrupt sstables
2. run 'nodetool rebuild'

DO you see any problem in the above approach ?

Another approach which I am considering as 'Second Option' is as given
below:
1. Stop the affected node
2. Remove all corrupt sstables from the affected node
3. Start the affected node
4. Run repair on the affected node

In my case, data is not written to Passive DC and has corrupt sstables.

Does the 2nd approach lead to data resurrection ?

I found in one of the threads in 'Mailing List' that data can ressurect if
I run 'nodetool repair'. Because of this reason, I considered this as a
secondary approach.

Do you have any other approach to fix this problem OR I can go with one of
the above approaches?

Regards,
Pushpendra

Re: Corrupt SSTable

[Nitan Kainth <ni...@gmail.com>]

If you are not deleting or updating data then it should be safe to use 2nd approach. 

Regards,
Nitan
Cell: 510 449 9629

> On Aug 13, 2020, at 11:48 AM, Pushpendra Rajpoot <pu...@gmail.com> wrote:
> 
> 
> Hi,
> 
> I have a cluster of 2 DC, each DC has 5 nodes in production. This cluster is based on active-passive model i.e. application is writing data on one DC (Active) & it's replicated to other DC (Passive).
> 
> My Passive DC has corrupt sstables (3 nodes out of 5 nodes) whereas there are no corrupt sstables on any node in Active DC.
> 
> Now, I am planning to do the following activity on affected nodes in Passive DC:
> 1. delete keyspace having corrupt sstables
> 2. run 'nodetool rebuild'
> 
> DO you see any problem in the above approach ?
> 
> Another approach which I am considering as 'Second Option' is as given below:
> 1. Stop the affected node
> 2. Remove all corrupt sstables from the affected node
> 3. Start the affected node
> 4. Run repair on the affected node
> 
> In my case, data is not written to Passive DC and has corrupt sstables.
> 
> Does the 2nd approach lead to data resurrection ?
> 
> I found in one of the threads in 'Mailing List' that data can ressurect if I run 'nodetool repair'. Because of this reason, I considered this as a secondary approach.
> 
> Do you have any other approach to fix this problem OR I can go with one of the above approaches?
> 
> Regards,
> Pushpendra