You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mike <mt...@yahoo.com> on 2013/01/16 16:58:28 UTC
Cassandra 1.1.2 -> 1.1.8 upgrade
Hello,
We are looking to upgrade our Cassandra cluster from 1.1.2 -> 1.1.8 (or
possibly 1.1.9 depending on timing). It is my understanding that
rolling upgrades of Cassandra is supported, so as we upgrade our
cluster, we can do so one node at a time without experiencing downtime.
Has anyone had any gotchas recently that I should be aware of before
performing this upgrade?
In order to upgrade, is the only thing that needs to change are the JAR
files? Can everything remain as-is?
Thanks,
-Mike
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by Edward Capriolo <ed...@gmail.com>.
You always want to run upgrade stables, ASAP. Sooner or later as the tables
compact they will upgrade. I hit a bug once with large bloom filters where,
after an upgrade, reading old bloom filters caused 0 rows returned when
there was data.
Once bitten twice shy. Upgrade one node disable gossip, disable thrift,
bring up iptables if you are really paranoid. Run repair/scrub. Wait till
done deactivated.
On Tuesday, February 12, 2013, aaron morton <aa...@thelastpickle.com> wrote:
> Can anyone know the impact of not running upgrade sstables? Or possible
not running it for several days?
>
> nodetool repair will not work.
> Cheers
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
> @aaronmorton
> http://www.thelastpickle.com
> On 12/02/2013, at 11:54 AM, Mike <mt...@yahoo.com> wrote:
>
> So the upgrade sstables is recommended as part of the upgrade to 1.1.3 if
you are using counter columns
>
> Also, there was a general recommendation (in another response to my
question) to run upgrade sstables because of:
>
> "upgradesstables always needs to be done between majors. While 1.1.2 ->
1.1.8 is not a major, due to an unforeseen bug in the conversion to
microseconds you'll need to run upgradesstables."
>
> Is this referring to: https://issues.apache.org/jira/browse/CASSANDRA-4432
>
> Can anyone know the impact of not running upgrade sstables? Or possible
not running it for several days?
>
> Thanks,
> -Mike
>
> On 2/10/2013 3:27 PM, aaron morton wrote:
>
> I would do #1.
> You can play with nodetool setcompactionthroughput to speed things up,
but beware nothing comes for free.
> Cheers
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
> @aaronmorton
> http://www.thelastpickle.com
> On 10/02/2013, at 6:40 AM, Mike <mt...@yahoo.com> wrote:
>
> Thank you,
>
> Another question on this topic.
>
> Upgrading from 1.1.2->1.1.9 requires running upgradesstables, which will
take many hours on our dataset (about 12). For this upgrade, is it
recommended that I:
>
> 1) Upgrade all the DB nodes to 1.1.9 first, then go around the ring and
run a staggered upgrade of the sstables over a number of days.
> 2) Upgrade one node at a time, running the clustered in a mixed
1.1.2->1.1.9 configuration for a number of days.
>
> I would prefer #1, as with #2, streaming will not work until all the
nodes are upgraded.
>
> I appreciate your thoughts,
> -Mike
>
> On 1/16/2013 11:
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by aaron morton <aa...@thelastpickle.com>.
> Can anyone know the impact of not running upgrade sstables? Or possible not running it for several days?
nodetool repair will not work.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 12/02/2013, at 11:54 AM, Mike <mt...@yahoo.com> wrote:
> So the upgrade sstables is recommended as part of the upgrade to 1.1.3 if you are using counter columns
>
> Also, there was a general recommendation (in another response to my question) to run upgrade sstables because of:
>
> "upgradesstables always needs to be done between majors. While 1.1.2 -> 1.1.8 is not a major, due to an unforeseen bug in the conversion to microseconds you'll need to run upgradesstables."
>
> Is this referring to: https://issues.apache.org/jira/browse/CASSANDRA-4432
>
> Can anyone know the impact of not running upgrade sstables? Or possible not running it for several days?
>
> Thanks,
> -Mike
>
> On 2/10/2013 3:27 PM, aaron morton wrote:
>> I would do #1.
>>
>> You can play with nodetool setcompactionthroughput to speed things up, but beware nothing comes for free.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 10/02/2013, at 6:40 AM, Mike <mt...@yahoo.com> wrote:
>>
>>> Thank you,
>>>
>>> Another question on this topic.
>>>
>>> Upgrading from 1.1.2->1.1.9 requires running upgradesstables, which will take many hours on our dataset (about 12). For this upgrade, is it recommended that I:
>>>
>>> 1) Upgrade all the DB nodes to 1.1.9 first, then go around the ring and run a staggered upgrade of the sstables over a number of days.
>>> 2) Upgrade one node at a time, running the clustered in a mixed 1.1.2->1.1.9 configuration for a number of days.
>>>
>>> I would prefer #1, as with #2, streaming will not work until all the nodes are upgraded.
>>>
>>> I appreciate your thoughts,
>>> -Mike
>>>
>>> On 1/16/2013 11:08 AM, Jason Wee wrote:
>>>> always check NEWS.txt for instance for cassandra 1.1.3 you need to run nodetool upgradesstables if your cf has counter.
>>>>
>>>>
>>>> On Wed, Jan 16, 2013 at 11:58 PM, Mike <mt...@yahoo.com> wrote:
>>>> Hello,
>>>>
>>>> We are looking to upgrade our Cassandra cluster from 1.1.2 -> 1.1.8 (or possibly 1.1.9 depending on timing). It is my understanding that rolling upgrades of Cassandra is supported, so as we upgrade our cluster, we can do so one node at a time without experiencing downtime.
>>>>
>>>> Has anyone had any gotchas recently that I should be aware of before performing this upgrade?
>>>>
>>>> In order to upgrade, is the only thing that needs to change are the JAR files? Can everything remain as-is?
>>>>
>>>> Thanks,
>>>> -Mike
>>>>
>>>
>>
>
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by Mike <mt...@yahoo.com>.
So the upgrade sstables is recommended as part of the upgrade to 1.1.3
if you are using counter columns
Also, there was a general recommendation (in another response to my
question) to run upgrade sstables because of:
"upgradesstables always needs to be done between majors. While 1.1.2 ->
1.1.8 is not a major, due to an unforeseen bug in the conversion to
microseconds you'll need to run upgradesstables."
Is this referring to: https://issues.apache.org/jira/browse/CASSANDRA-4432
Can anyone know the impact of not running upgrade sstables? Or possible
not running it for several days?
Thanks,
-Mike
On 2/10/2013 3:27 PM, aaron morton wrote:
> I would do #1.
>
> You can play with nodetool setcompactionthroughput to speed things up,
> but beware nothing comes for free.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 10/02/2013, at 6:40 AM, Mike <mtheroux2@yahoo.com
> <ma...@yahoo.com>> wrote:
>
>> Thank you,
>>
>> Another question on this topic.
>>
>> Upgrading from 1.1.2->1.1.9 requires running upgradesstables, which
>> will take many hours on our dataset (about 12). For this upgrade, is
>> it recommended that I:
>>
>> 1) Upgrade all the DB nodes to 1.1.9 first, then go around the ring
>> and run a staggered upgrade of the sstables over a number of days.
>> 2) Upgrade one node at a time, running the clustered in a mixed
>> 1.1.2->1.1.9 configuration for a number of days.
>>
>> I would prefer #1, as with #2, streaming will not work until all the
>> nodes are upgraded.
>>
>> I appreciate your thoughts,
>> -Mike
>>
>> On 1/16/2013 11:08 AM, Jason Wee wrote:
>>> always check NEWS.txt for instance for cassandra 1.1.3 you need to
>>> run nodetool upgradesstables if your cf has counter.
>>>
>>>
>>> On Wed, Jan 16, 2013 at 11:58 PM, Mike <mtheroux2@yahoo.com
>>> <ma...@yahoo.com>> wrote:
>>>
>>> Hello,
>>>
>>> We are looking to upgrade our Cassandra cluster from 1.1.2 ->
>>> 1.1.8 (or possibly 1.1.9 depending on timing). It is my
>>> understanding that rolling upgrades of Cassandra is supported,
>>> so as we upgrade our cluster, we can do so one node at a time
>>> without experiencing downtime.
>>>
>>> Has anyone had any gotchas recently that I should be aware of
>>> before performing this upgrade?
>>>
>>> In order to upgrade, is the only thing that needs to change are
>>> the JAR files? Can everything remain as-is?
>>>
>>> Thanks,
>>> -Mike
>>>
>>>
>>
>
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by aaron morton <aa...@thelastpickle.com>.
I would do #1.
You can play with nodetool setcompactionthroughput to speed things up, but beware nothing comes for free.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 10/02/2013, at 6:40 AM, Mike <mt...@yahoo.com> wrote:
> Thank you,
>
> Another question on this topic.
>
> Upgrading from 1.1.2->1.1.9 requires running upgradesstables, which will take many hours on our dataset (about 12). For this upgrade, is it recommended that I:
>
> 1) Upgrade all the DB nodes to 1.1.9 first, then go around the ring and run a staggered upgrade of the sstables over a number of days.
> 2) Upgrade one node at a time, running the clustered in a mixed 1.1.2->1.1.9 configuration for a number of days.
>
> I would prefer #1, as with #2, streaming will not work until all the nodes are upgraded.
>
> I appreciate your thoughts,
> -Mike
>
> On 1/16/2013 11:08 AM, Jason Wee wrote:
>> always check NEWS.txt for instance for cassandra 1.1.3 you need to run nodetool upgradesstables if your cf has counter.
>>
>>
>> On Wed, Jan 16, 2013 at 11:58 PM, Mike <mt...@yahoo.com> wrote:
>> Hello,
>>
>> We are looking to upgrade our Cassandra cluster from 1.1.2 -> 1.1.8 (or possibly 1.1.9 depending on timing). It is my understanding that rolling upgrades of Cassandra is supported, so as we upgrade our cluster, we can do so one node at a time without experiencing downtime.
>>
>> Has anyone had any gotchas recently that I should be aware of before performing this upgrade?
>>
>> In order to upgrade, is the only thing that needs to change are the JAR files? Can everything remain as-is?
>>
>> Thanks,
>> -Mike
>>
>
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Not sure this will be useful for you but "nodetool drain" doesn't work
properly well for a while. If you are using counters I recommend you to
remove commit logs after you drained ans stopped the node, before
restarting the node to avoid replaying counts.
https://issues.apache.org/jira/browse/CASSANDRA-4446
Hope this will help,
Alain
2013/2/11 Michal Michalski <mi...@opera.com>
>
> 2) Upgrade one node at a time, running the clustered in a mixed
>> 1.1.2->1.1.9 configuration for a number of days.
>>
>
> I'm about to upgrade my 1.1.0 cluster and
> http://www.datastax.com/docs/**1.1/install/upgrading#info<http://www.datastax.com/docs/1.1/install/upgrading#info>says:
>
> "If you are upgrading to Cassandra 1.1.9 from a version earlier than
> 1.1.7, all nodes must be upgraded before any streaming can take place.
> Until you upgrade all nodes, you cannot add version 1.1.7 nodes or later to
> a 1.1.7 or earlier cluster."
>
> Which one is correct then? Can I run mixed 1.1.2 (in my case 1.1.0) &
> 1.1.9 cluster or not?
>
> M.
>
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by aaron morton <aa...@thelastpickle.com>.
You have linked to the 1.2 news file, which branched from 1.1 at some point.
Look at the news file in the distribution you are installing or here https://github.com/apache/cassandra/blob/cassandra-1.1/NEWS.txt
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 11/02/2013, at 11:14 PM, Michal Michalski <mi...@opera.com> wrote:
> OK, thanks Aaron. I ask because NEWS.txt is not a big help in case of > 1.1.5 versions because there's no info on them in it (especially on 1.1.7 which seems to be the most important one in this case, according to the DataStax' upgrade instructions) ;-)
>
> https://github.com/apache/cassandra/blob/trunk/NEWS.txt
>
> M.
>
>
> W dniu 11.02.2013 11:05, aaron morton pisze:
>> You can always run them. But in some situations repair cannot be used, and in this case new nodes cannot be added. The news.txt file is your friend there.
>>
>> As a general rule when upgrading a cluster I move one node to the new version and let it soak in for an hour or so. Just to catch any crazy. I then upgrade all the nodes and run through the upgrade table. You can stagger upgrade table to be every RF'th node in the cluster to reduce the impact.
>
>
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by Michal Michalski <mi...@opera.com>.
OK, thanks Aaron. I ask because NEWS.txt is not a big help in case of >
1.1.5 versions because there's no info on them in it (especially on
1.1.7 which seems to be the most important one in this case, according
to the DataStax' upgrade instructions) ;-)
https://github.com/apache/cassandra/blob/trunk/NEWS.txt
M.
W dniu 11.02.2013 11:05, aaron morton pisze:
> You can always run them. But in some situations repair cannot be used, and in this case new nodes cannot be added. The news.txt file is your friend there.
>
> As a general rule when upgrading a cluster I move one node to the new version and let it soak in for an hour or so. Just to catch any crazy. I then upgrade all the nodes and run through the upgrade table. You can stagger upgrade table to be every RF'th node in the cluster to reduce the impact.
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by aaron morton <aa...@thelastpickle.com>.
You can always run them. But in some situations repair cannot be used, and in this case new nodes cannot be added. The news.txt file is your friend there.
As a general rule when upgrading a cluster I move one node to the new version and let it soak in for an hour or so. Just to catch any crazy. I then upgrade all the nodes and run through the upgrade table. You can stagger upgrade table to be every RF'th node in the cluster to reduce the impact.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 11/02/2013, at 8:05 PM, Michal Michalski <mi...@opera.com> wrote:
>
>> 2) Upgrade one node at a time, running the clustered in a mixed
>> 1.1.2->1.1.9 configuration for a number of days.
>
> I'm about to upgrade my 1.1.0 cluster and
> http://www.datastax.com/docs/1.1/install/upgrading#info says:
>
> "If you are upgrading to Cassandra 1.1.9 from a version earlier than 1.1.7, all nodes must be upgraded before any streaming can take place. Until you upgrade all nodes, you cannot add version 1.1.7 nodes or later to a 1.1.7 or earlier cluster."
>
> Which one is correct then? Can I run mixed 1.1.2 (in my case 1.1.0) & 1.1.9 cluster or not?
>
> M.
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by Michal Michalski <mi...@opera.com>.
> 2) Upgrade one node at a time, running the clustered in a mixed
> 1.1.2->1.1.9 configuration for a number of days.
I'm about to upgrade my 1.1.0 cluster and
http://www.datastax.com/docs/1.1/install/upgrading#info says:
"If you are upgrading to Cassandra 1.1.9 from a version earlier than
1.1.7, all nodes must be upgraded before any streaming can take place.
Until you upgrade all nodes, you cannot add version 1.1.7 nodes or later
to a 1.1.7 or earlier cluster."
Which one is correct then? Can I run mixed 1.1.2 (in my case 1.1.0) &
1.1.9 cluster or not?
M.
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by Mike <mt...@yahoo.com>.
Thank you,
Another question on this topic.
Upgrading from 1.1.2->1.1.9 requires running upgradesstables, which will
take many hours on our dataset (about 12). For this upgrade, is it
recommended that I:
1) Upgrade all the DB nodes to 1.1.9 first, then go around the ring and
run a staggered upgrade of the sstables over a number of days.
2) Upgrade one node at a time, running the clustered in a mixed
1.1.2->1.1.9 configuration for a number of days.
I would prefer #1, as with #2, streaming will not work until all the
nodes are upgraded.
I appreciate your thoughts,
-Mike
On 1/16/2013 11:08 AM, Jason Wee wrote:
> always check NEWS.txt for instance for cassandra 1.1.3 you need to
> run nodetool upgradesstables if your cf has counter.
>
>
> On Wed, Jan 16, 2013 at 11:58 PM, Mike <mtheroux2@yahoo.com
> <ma...@yahoo.com>> wrote:
>
> Hello,
>
> We are looking to upgrade our Cassandra cluster from 1.1.2 ->
> 1.1.8 (or possibly 1.1.9 depending on timing). It is my
> understanding that rolling upgrades of Cassandra is supported, so
> as we upgrade our cluster, we can do so one node at a time without
> experiencing downtime.
>
> Has anyone had any gotchas recently that I should be aware of
> before performing this upgrade?
>
> In order to upgrade, is the only thing that needs to change are
> the JAR files? Can everything remain as-is?
>
> Thanks,
> -Mike
>
>
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by Michael Kjellman <mk...@barracuda.com>.
upgradesstables is safe, but it is essentially compaction (because sstables are immutable it rewrites the sstable in the new format) so you'll want to do it when traffic is low to avoid IO issues.
upgradesstables always needs to be done between majors. While 1.1.2 -> 1.1.8 is not a major, due to an unforeseen bug in the conversion to microseconds you'll need to run upgradesstables.
You can check if all of your sstables have been upgraded by looking at the file names of the sstables. Your files should be –hd–
1.1.8 will be –hf–
I don't remember there being changes to cassandra.yaml between 1.1.2 -> 1.1.7 but you might want to check out a clean copy and compare your yaml to make sure some of the recommended default haven't changed or required new config options.
Otherwise, nodetool drain, stop the service, upgrade to the new release, and start the service, upgradesstables
Hinted Handoff should take care of anything while the node is down but if you want to be extra safe you can do a repair –pr on every node but you should be doing that on a regular basis anyways!
Hope this helps.
Best,
michael
From: Mike <mt...@yahoo.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Wednesday, January 16, 2013 8:15 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Thanks for pointing that out.
Given upgradesstables can only be run on a live node, does anyone know if there is a danger of having this node in the cluster while this is being performed? Also, can anyone confirm this only needs to be done on counter counter column families, or all column families (the former makes sense, I'm just making sure).
-Mike
On 1/16/2013 11:08 AM, Jason Wee wrote:
always check NEWS.txt for instance for cassandra 1.1.3 you need to run nodetool upgradesstables if your cf has counter.
On Wed, Jan 16, 2013 at 11:58 PM, Mike <mt...@yahoo.com>> wrote:
Hello,
We are looking to upgrade our Cassandra cluster from 1.1.2 -> 1.1.8 (or possibly 1.1.9 depending on timing). It is my understanding that rolling upgrades of Cassandra is supported, so as we upgrade our cluster, we can do so one node at a time without experiencing downtime.
Has anyone had any gotchas recently that I should be aware of before performing this upgrade?
In order to upgrade, is the only thing that needs to change are the JAR files? Can everything remain as-is?
Thanks,
-Mike
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by Mike <mt...@yahoo.com>.
Thanks for pointing that out.
Given upgradesstables can only be run on a live node, does anyone know
if there is a danger of having this node in the cluster while this is
being performed? Also, can anyone confirm this only needs to be done on
counter counter column families, or all column families (the former
makes sense, I'm just making sure).
-Mike
On 1/16/2013 11:08 AM, Jason Wee wrote:
> always check NEWS.txt for instance for cassandra 1.1.3 you need to
> run nodetool upgradesstables if your cf has counter.
>
>
> On Wed, Jan 16, 2013 at 11:58 PM, Mike <mtheroux2@yahoo.com
> <ma...@yahoo.com>> wrote:
>
> Hello,
>
> We are looking to upgrade our Cassandra cluster from 1.1.2 ->
> 1.1.8 (or possibly 1.1.9 depending on timing). It is my
> understanding that rolling upgrades of Cassandra is supported, so
> as we upgrade our cluster, we can do so one node at a time without
> experiencing downtime.
>
> Has anyone had any gotchas recently that I should be aware of
> before performing this upgrade?
>
> In order to upgrade, is the only thing that needs to change are
> the JAR files? Can everything remain as-is?
>
> Thanks,
> -Mike
>
>
Re: Cassandra 1.1.2 -> 1.1.8 upgrade
Posted by Jason Wee <pe...@gmail.com>.
always check NEWS.txt for instance for cassandra 1.1.3 you need to
run nodetool upgradesstables if your cf has counter.
On Wed, Jan 16, 2013 at 11:58 PM, Mike <mt...@yahoo.com> wrote:
> Hello,
>
> We are looking to upgrade our Cassandra cluster from 1.1.2 -> 1.1.8 (or
> possibly 1.1.9 depending on timing). It is my understanding that rolling
> upgrades of Cassandra is supported, so as we upgrade our cluster, we can do
> so one node at a time without experiencing downtime.
>
> Has anyone had any gotchas recently that I should be aware of before
> performing this upgrade?
>
> In order to upgrade, is the only thing that needs to change are the JAR
> files? Can everything remain as-is?
>
> Thanks,
> -Mike
>