Full repair results in uneven data distribution

[Inquistive allen <in...@gmail.com>]

Hello Team,

Sorry for this might be a simple question.

I was working on Cassandra 2.1.14

Node1 -- 4.5 mb data
Node2 -- 5.3 mb data
Node3 -- 4.9 mb data

Node3 was down since 90 days.
I brought it up and it joined the cluster.
To sync data I ran nodetool repair --full

Repair was successful...however just to be sure that the data is in sync..I
re-ran the repair process..expecting the process to exit and hence prove
that there is nothing to repair.

Each time I ran full repair.. repair did run completely and
successfully...it didn't exit immediately as I expected.


Running it 4 times I suddenly saw this

Node 1 -- 43 mb
Node2 -- 42 mb
Node3 -- 6 mb

I was clueless of this data growth on node1 and node2.

May anyone pls help me understand why this happened.

To bring back things to normal, I tried running nodetool repair -pr on all
the hosts one after another.. repair ran successfully...

Still there was difference in data size on 3 nodes.

Hence I decided to decommission each node  and re-add them one after
another..

I did that.  The data size is now

Just wanted to understand is there anyway my data is lost..why was there a
difference in data size after I ran full repair multiple times

Thanks

Re: Full repair results in uneven data distribution

[Bowen Song <bo...@bso.ng.INVALID>]

That sounds like the combined results from the anti-compaction and the 
size amplification from the default SizeTieredCompactionStrategy. If you 
keep repeating those steps, the disk usage will eventually stop growing. 
Of course, that's not an excuse to keep repeating it.

To fix this (if you really need to reclaim those disk space), you can 
shutdown a node, run "sstablerepairedset --really-set --is-unrepaired" 
on all SSTable files, then restart the node and run "nodetool compact 
-s". Repeat these steps on every node, including the seemingly 
unaffected node, one by one.

To avoid this issue in the future, I'd recommend you avoid causing 
Cassandra to do anti-compaction during repairs. You can achieve that by 
specifying a DC in the "nodetool repair" command, such as "nodetool 
repair -full -dc DC1". This will work even you only have one DC. You 
should also look into automation tools, such as Cassandra Reaper 
<http://cassandra-reaper.io/>, for running repairs.


On 16/03/2021 07:11, Inquistive allen wrote:
> Hello Team,
>
> Sorry for this might be a simple question.
>
> I was working on Cassandra 2.1.14
>
> Node1 -- 4.5 mb data
> Node2 -- 5.3 mb data
> Node3 -- 4.9 mb data
>
> Node3 was down since 90 days.
> I brought it up and it joined the cluster.
> To sync data I ran nodetool repair --full
>
> Repair was successful...however just to be sure that the data is in 
> sync..I re-ran the repair process..expecting the process to exit and 
> hence prove that there is nothing to repair.
>
> Each time I ran full repair.. repair did run completely and 
> successfully...it didn't exit immediately as I expected.
>
>
> Running it 4 times I suddenly saw this
>
> Node 1 -- 43 mb
> Node2 -- 42 mb
> Node3 -- 6 mb
>
> I was clueless of this data growth on node1 and node2.
>
> May anyone pls help me understand why this happened.
>
> To bring back things to normal, I tried running nodetool repair -pr on 
> all the hosts one after another.. repair ran successfully...
>
> Still there was difference in data size on 3 nodes.
>
> Hence I decided to decommission each node  and re-add them one after 
> another..
>
> I did that.  The data size is now
>
> Just wanted to understand is there anyway my data is lost..why was 
> there a difference in data size after I ran full repair multiple times
>
> Thanks
>
>