RE: [EXTERNAL] Re: rolling version upgrade, upgradesstables, and vulnerability window

["Durity, Sean R" <SE...@homedepot.com>]

Just to pile on:

I agree. On our upgrades, I always aim to get the binary part done on all nodes before worrying about upgradesstables. Upgrade is one node at a time (precautionary). Upgradesstables depends on cluster size, data size, compactionthroughput, etc. I usually start with running upgradesstables on 2 nodes per DC and watch how the application performs. On larger clusters (over 30 nodes), I usually work up to 4-5 nodes per DC running upgradesstables with staggered start times.

NOTE: I am rarely doing streaming operations outside of repairs. But I want to be able to handle a down node, etc., so I do not run in mixed version mode very long.


Sean Durity

From: Carl Mueller <ca...@smartthings.com.INVALID>
Sent: Tuesday, October 30, 2018 11:51 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: rolling version upgrade, upgradesstables, and vulnerability window

Thank you very much. I couldn't find any definitive answer on that on the list or stackoverflow.

It's clear that the safest for a prod cluster is rolling version upgrade of the binary, then the upgradesstables.

I will strongly consider cstar for the upgradesstables


On Tue, Oct 30, 2018 at 10:39 AM Alexander Dejanovski <al...@thelastpickle.com>> wrote:
Yes, as the new version can read both the old and the new sstables format.

Restrictions only apply when the cluster is in mixed versions.

On Tue, Oct 30, 2018 at 4:37 PM Carl Mueller <ca...@smartthings.com.invalid>> wrote:
But the topology change restrictions are only in place while there are heterogenous versions in the cluster? All the nodes at the upgraded version with "degraded" sstables does NOT preclude topology changes or node replacement/addition?


On Tue, Oct 30, 2018 at 10:33 AM Jeff Jirsa <jj...@gmail.com>> wrote:
Wait for 3.11.4 to be cut

I also vote for doing all the binary bounces and upgradesstables after the fact, largely because normal writes/compactions are going to naturally start upgrading sstables anyway, and there are some hard restrictions on mixed mode (e.g. schema changes won’t cross version) that can be far more impactful.



--
Jeff Jirsa


> On Oct 30, 2018, at 8:21 AM, Carl Mueller <ca...@smartthings.com>.INVALID> wrote:
>
> We are about to finally embark on some version upgrades for lots of clusters, 2.1.x and 2.2.x targetting eventually 3.11.x
>
> I have seen recipes that do the full binary upgrade + upgrade sstables for 1 node before moving forward, while I've seen a 2016 vote by Jon Haddad (a TLP guy) that backs doing the binary version upgrades through the cluster on a rolling basis, then doing the upgradesstables on a rolling basis.
>
> Under what cluster conditions are streaming/node replacement precluded, that is we are vulnerable to a cloud provided dumping one of our nodes under us or hardware failure? We ain't apple, but we do have 30+ node datacenters and 80-100 node clusters.
>
> Is the node replacement and streaming only disabled while there are heterogenous cassandra versions, or until all the sstables have been upgraded in the cluster?
>
> My instincts tell me the best thing to do is to get all the cassandra nodes to the same version without the upgradesstables step through the cluster, and then roll through the upgradesstables as needed, and that upgradesstables is a node-local concern that doesn't impact streaming or node replacement or other situations since cassandra can read old version sstables and new sstables would simply be the new format.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org<ma...@cassandra.apache.org>
For additional commands, e-mail: user-help@cassandra.apache.org<ma...@cassandra.apache.org>
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.thelastpickle.com_&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=G-Ap7nwUtxBiQjeKRqvd1erBWJv4wggybz60rOAzi2A&s=OcVCKQwOhlKxAFCc1WixWFEgw-NmKIQie1_65FtdQ_o&e=>

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.