You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by shini gupta <gu...@gmail.com> on 2017/12/06 04:28:09 UTC
Running repair while Cassandra upgrade 2.0.X to 2.1.X
If we have upgraded Cassandra binaries from 2.0 to 2.1 on ALL the nodes but
upgradesstable is still pending, please provide the impact of following
scenarios:
1. Running nodetool repair on one of the nodes while upgradesstables is
still executing on one or more nodes in the cluster.
2. Running nodetool repair when upgradesstables failed abruptly on some of
the nodes such that some sstable files are in new format while other
sstable files are still in old format.
Even though it may not be recommended to run I/O intensive operations like
repair and upgradesstables simultaneously, can we assume that both the
above sceanrios are now supported and will not break anything, especially
after https://issues.apache.org/jira/browse/CASSANDRA-5772 has been fixed
in 2.0?
Regards
Shini
Re: Running repair while Cassandra upgrade 2.0.X to 2.1.X
Posted by kurt greaves <ku...@instaclustr.com>.
That ticket says that streaming SSTables that are older versions is
supported. Streaming is only one component of repairs, and this ticket
doesn't talk about repair at all, only bootstrap. For the most part it
should work but as Alain said, it's probably best avoided. Especially if
you can avoid it (your cluster isn't so huge it's going to take days to
upgrade). Just upgrade your nodes, run and re-run upgradesstables until it
stops kicking off upgrade compactions, then re-start your repairs.
Re: Running repair while Cassandra upgrade 2.0.X to 2.1.X
Posted by shini gupta <gu...@gmail.com>.
Thanks Alain !! During Cassandra upgrade, we halt repair scheduling for 24
hours as this is good time to finish upgradesstables..we dont want
continuous manual monitoring of upgradesstables and then rescheduling of
repair tasks when upgradesstables successfully finishes on all the
nodes..Here is the problem with the approach..sometimes we see that
upgradsstables has finished on a node but still some sstable files exist in
the old format. I want to understand how safe it is if our repair executes
after 24 hours when old and new sstables cooexist on some nodes. My
understanding of
https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-5772
says that this is supported and nothing could go wrong ..Can someone please
confirm my understanding of the JIRA and confirm that repairs cant break
anything in such scenarios?"
Thanks
Shini
On Fri, 8 Dec 2017 at 2:26 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> Hi Shini,
>
> First thing that comes to my mind honestly is "why?". Why would you do
> this? It's way safer to finish upgrade then repair I would say.
>
> That being said, I can make guesses for you here, but the best would
> probably be to test it.
>
> 1. Running nodetool repair on one of the nodes while upgradesstables is
>> still executing on one or more nodes in the cluster.
>
> 2. Running nodetool repair when upgradesstables failed abruptly on some of
>> the nodes such that some sstable files are in new format while other
>> sstable files are still in old format.
>
>
> These impacts are unknown for me since I never did this, I always heard
> around it was really bad. For me both case are similar. You are going to
> repair data with distinct SSTable formats. I don't remember the differences
> between Cassandra 2.0 and 2.1.
>
> In best case it will work.
>
> But my guess is it might lead to:
> - Schema disagreement (but I heard that on schema changes on multi-version
> clusters). I don't think you would hit this one.
> - Mixing all new and old over the cluster (even on nodes with upgraded
> SSTables)...
> - Maybe directly fail if networking changed between the 2 versions.
>
> I would stay away from repairs and upgrade first all the nodes.
>
> If that's really not doable, then I recommend you to test it (ccm / AWS /
> stage cluster, as you see fit).
>
> Sorry I cannot be more precise, impacts are not clear to me because I
> always stayed away from this kind of mixed operations. The need for
> repairing before upgrading sstables is not clear to me either. I would add
> that I found out in the past that working in a rush on Cassandra often
> creates more issues than solutions in general.
>
> C*heers,
> -----------------------
> Alain Rodriguez - @arodream - alain@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2017-12-08 5:26 GMT+00:00 shini gupta <gu...@gmail.com>:
>
>> Hi
>> Can someone please answer this query?
>>
>> Thanks
>>
>> On Wed, Dec 6, 2017 at 9:58 AM, shini gupta <gu...@gmail.com>
>> wrote:
>>
>>> If we have upgraded Cassandra binaries from 2.0 to 2.1 on ALL the nodes
>>> but upgradesstable is still pending, please provide the impact of following
>>> scenarios:
>>>
>>>
>>>
>>> 1. Running nodetool repair on one of the nodes while upgradesstables is
>>> still executing on one or more nodes in the cluster.
>>>
>>> 2. Running nodetool repair when upgradesstables failed abruptly on some
>>> of the nodes such that some sstable files are in new format while other
>>> sstable files are still in old format.
>>>
>>>
>>>
>>> Even though it may not be recommended to run I/O intensive operations
>>> like repair and upgradesstables simultaneously, can we assume that both the
>>> above sceanrios are now supported and will not break anything, especially
>>> after https://issues.apache.org/jira/browse/CASSANDRA-5772 has been
>>> fixed in 2.0?
>>>
>>>
>>> Regards
>>> Shini
>>>
>>>
>>
>>
>> --
>> -Shini Gupta
>>
>> ""Trusting in God won't make the mountain smaller,
>> But will make climbing easier.
>> Do not ask God for a lighter load
>> But ask Him for a stronger back... ""
>>
>
> --
-Shini Gupta
""Trusting in God won't make the mountain smaller,
But will make climbing easier.
Do not ask God for a lighter load
But ask Him for a stronger back... ""
Re: Running repair while Cassandra upgrade 2.0.X to 2.1.X
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi Shini,
First thing that comes to my mind honestly is "why?". Why would you do
this? It's way safer to finish upgrade then repair I would say.
That being said, I can make guesses for you here, but the best would
probably be to test it.
1. Running nodetool repair on one of the nodes while upgradesstables is
> still executing on one or more nodes in the cluster.
2. Running nodetool repair when upgradesstables failed abruptly on some of
> the nodes such that some sstable files are in new format while other
> sstable files are still in old format.
These impacts are unknown for me since I never did this, I always heard
around it was really bad. For me both case are similar. You are going to
repair data with distinct SSTable formats. I don't remember the differences
between Cassandra 2.0 and 2.1.
In best case it will work.
But my guess is it might lead to:
- Schema disagreement (but I heard that on schema changes on multi-version
clusters). I don't think you would hit this one.
- Mixing all new and old over the cluster (even on nodes with upgraded
SSTables)...
- Maybe directly fail if networking changed between the 2 versions.
I would stay away from repairs and upgrade first all the nodes.
If that's really not doable, then I recommend you to test it (ccm / AWS /
stage cluster, as you see fit).
Sorry I cannot be more precise, impacts are not clear to me because I
always stayed away from this kind of mixed operations. The need for
repairing before upgrading sstables is not clear to me either. I would add
that I found out in the past that working in a rush on Cassandra often
creates more issues than solutions in general.
C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France / Spain
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2017-12-08 5:26 GMT+00:00 shini gupta <gu...@gmail.com>:
> Hi
> Can someone please answer this query?
>
> Thanks
>
> On Wed, Dec 6, 2017 at 9:58 AM, shini gupta <gu...@gmail.com> wrote:
>
>> If we have upgraded Cassandra binaries from 2.0 to 2.1 on ALL the nodes
>> but upgradesstable is still pending, please provide the impact of following
>> scenarios:
>>
>>
>>
>> 1. Running nodetool repair on one of the nodes while upgradesstables is
>> still executing on one or more nodes in the cluster.
>>
>> 2. Running nodetool repair when upgradesstables failed abruptly on some
>> of the nodes such that some sstable files are in new format while other
>> sstable files are still in old format.
>>
>>
>>
>> Even though it may not be recommended to run I/O intensive operations
>> like repair and upgradesstables simultaneously, can we assume that both the
>> above sceanrios are now supported and will not break anything, especially
>> after https://issues.apache.org/jira/browse/CASSANDRA-5772 has been
>> fixed in 2.0?
>>
>>
>> Regards
>> Shini
>>
>>
>
>
> --
> -Shini Gupta
>
> ""Trusting in God won't make the mountain smaller,
> But will make climbing easier.
> Do not ask God for a lighter load
> But ask Him for a stronger back... ""
>
Re: Running repair while Cassandra upgrade 2.0.X to 2.1.X
Posted by shini gupta <gu...@gmail.com>.
Hi
Can someone please answer this query?
Thanks
On Wed, Dec 6, 2017 at 9:58 AM, shini gupta <gu...@gmail.com> wrote:
> If we have upgraded Cassandra binaries from 2.0 to 2.1 on ALL the nodes
> but upgradesstable is still pending, please provide the impact of following
> scenarios:
>
>
>
> 1. Running nodetool repair on one of the nodes while upgradesstables is
> still executing on one or more nodes in the cluster.
>
> 2. Running nodetool repair when upgradesstables failed abruptly on some of
> the nodes such that some sstable files are in new format while other
> sstable files are still in old format.
>
>
>
> Even though it may not be recommended to run I/O intensive operations like
> repair and upgradesstables simultaneously, can we assume that both the
> above sceanrios are now supported and will not break anything, especially
> after https://issues.apache.org/jira/browse/CASSANDRA-5772 has been fixed
> in 2.0?
>
>
> Regards
> Shini
>
>
--
-Shini Gupta
""Trusting in God won't make the mountain smaller,
But will make climbing easier.
Do not ask God for a lighter load
But ask Him for a stronger back... ""