Data repairing on one node questionably affects data on others

["ssivikt@gmail.com" <ss...@gmail.com>]

Hi All,

I have cluster of 7 nodes completely balanced (each node owns ~500GB of 
data).
And I have one keyspace and one table and three replicas. Than, I just 
failed one node's disk, replace it with a new one and started repairing.
During that process I noticed that additional two nodes have started 
getting data, and at the end of the repairing three nodes have twice 
more data than at the beginning.
I'm curious, is it a normal behavior for Cassandra? Why not only one 
node, but three, have gotten data during repairing? May be it's because 
of clocks skew?

Thanks!

-- 
best regards,
Sergey

Re: Data repairing on one node questionably affects data on others

[Jean Carlo <je...@gmail.com>]

Well that may explain why you do have an unbalanced distribution of data.
It is not cassandra normal behaivor. If you lose your disk, and you don't
have a backup of your data (at least system because it is not replicated),
this is what I think should be the normal procedure to recover your node.

- stop cassandra on the node down
- nodetool removenode id
- update de conf of your cluster to remove the ip of the node removed
(cassandra.yaml, cassandra-topology.properties). You might want to make a
copy of your original conf
- restart your cluster
- check the node does not appear in nodetool status
- restore your previous conf of your cluster. including the node that was
down
- on the node down, if its ip address is in the list of seeds, remove it in
order to do the boobstrap
- restart cassandra in every node one by one.

It is what we do in our cluster.



Saludos

Jean Carlo

"The best way to predict the future is to invent it" Alan Kay

On Tue, Apr 26, 2016 at 2:25 PM, ssivikt@gmail.com <ss...@gmail.com>
wrote:

> No, I didn't. I just want to understand how C* handle such case and what
> is a predictable behavior.
>
>
> On 04/26/2016 02:51 PM, Jean Carlo wrote:
>
> Did you use a backup of the keyspace system?
>
> If not, you might do removenode of that node and re added to the cluster
> to re generate new tokens.
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Tue, Apr 26, 2016 at 12:06 AM, <ss...@gmail.com>ssivikt@gmail.com <
> ssivikt@gmail.com> wrote:
>
>> Hi All,
>>
>> I have cluster of 7 nodes completely balanced (each node owns ~500GB of
>> data).
>> And I have one keyspace and one table and three replicas. Than, I just
>> failed one node's disk, replace it with a new one and started repairing.
>> During that process I noticed that additional two nodes have started
>> getting data, and at the end of the repairing three nodes have twice more
>> data than at the beginning.
>> I'm curious, is it a normal behavior for Cassandra? Why not only one
>> node, but three, have gotten data during repairing? May be it's because of
>> clocks skew?
>>
>> Thanks!
>>
>> --
>> best regards,
>> Sergey
>>
>>
>
> --
> Thanks,
> Serj
>
>

Re: Data repairing on one node questionably affects data on others

["ssivikt@gmail.com" <ss...@gmail.com>]

No, I didn't. I just want to understand how C* handle such case and what 
is a predictable behavior.

On 04/26/2016 02:51 PM, Jean Carlo wrote:
> Did you use a backup of the keyspace system?
>
> If not, you might do removenode of that node and re added to the 
> cluster to re generate new tokens.
>
>
> Saludos
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
> On Tue, Apr 26, 2016 at 12:06 AM, ssivikt@gmail.com 
> <ma...@gmail.com> <ssivikt@gmail.com 
> <ma...@gmail.com>> wrote:
>
>     Hi All,
>
>     I have cluster of 7 nodes completely balanced (each node owns
>     ~500GB of data).
>     And I have one keyspace and one table and three replicas. Than, I
>     just failed one node's disk, replace it with a new one and started
>     repairing.
>     During that process I noticed that additional two nodes have
>     started getting data, and at the end of the repairing three nodes
>     have twice more data than at the beginning.
>     I'm curious, is it a normal behavior for Cassandra? Why not only
>     one node, but three, have gotten data during repairing? May be
>     it's because of clocks skew?
>
>     Thanks!
>
>     -- 
>     best regards,
>     Sergey
>
>

-- 
Thanks,
Serj

Re: Data repairing on one node questionably affects data on others

[Jean Carlo <je...@gmail.com>]

Did you use a backup of the keyspace system?

If not, you might do removenode of that node and re added to the cluster to
re generate new tokens.


Saludos

Jean Carlo

"The best way to predict the future is to invent it" Alan Kay

On Tue, Apr 26, 2016 at 12:06 AM, ssivikt@gmail.com <ss...@gmail.com>
wrote:

> Hi All,
>
> I have cluster of 7 nodes completely balanced (each node owns ~500GB of
> data).
> And I have one keyspace and one table and three replicas. Than, I just
> failed one node's disk, replace it with a new one and started repairing.
> During that process I noticed that additional two nodes have started
> getting data, and at the end of the repairing three nodes have twice more
> data than at the beginning.
> I'm curious, is it a normal behavior for Cassandra? Why not only one node,
> but three, have gotten data during repairing? May be it's because of clocks
> skew?
>
> Thanks!
>
> --
> best regards,
> Sergey
>
>