You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Anshu Vajpayee <an...@gmail.com> on 2020/01/30 04:45:07 UTC
Cassandra OS Patching.
Hi Team,
What is the best way to patch OS of 1000 nodes Multi DC Cassandra cluster
where we cannot suspend application traffic( we can redirect traffic to one
DC).
Please suggest if anyone has any best practice around it.
--
*C*heers,*
*Anshu V*
Re: Cassandra OS Patching.
Posted by Carl Mueller <ca...@smartthings.com.INVALID>.
In AWS we are doing os upgrades by standing up an equivalent set of nodes,
and then in a rolling fashion we move the EBS mounts, selectively sync
necessary cassandra settings, and then start up the new node.
The downtime is fairly minimal since the ebs detach/attach is pretty quick,
you don't need to wait for a long exhaustive OS upgrade process (we are
hopping to ubu 18 LTS from 16 LTS, so that would be a loooong upgrade.
We have lots of code already to do checks, cassandra shutdown/startup,
waits, gossip stability checks and the like from our cassandra version
upgrade scripts.
We are getting bit by the us-east-1e region being "deprecated" and not
having m5 instances, so we have to rsync those, but we do a pre-rsync of
the node, then right before the shutdown we do another rsync, then
shutdown, do a final incremental rsync, which also reduces the downtime,
but the initial rsync can take a while depending on the node size.
Remember you don't have to rsync the snapshots.
On Tue, Feb 4, 2020 at 12:03 PM Jeff Jirsa <jj...@gmail.com> wrote:
> The original question was OS (and / or JDK) upgrades, and for those "do it
> in QA first", bounce non-replicas and let them come up before proceeding.
>
> If you're doing Cassandra itself, it's a REALLY REALLY REALLY good idea to
> try the upgrade on a backup of your production cluster first. While it's
> uncommon, there were certain things allowed in very old versions of
> Cassandra that are occasionally missed in upgrade tests, and can be found
> the hard way. Sometimes those incompatible schema changes should have
> disallowed but weren't - sometimes they leave data on disk that cassandra
> 3.0/3.11/etc know is illegal or invalid, and throws exceptions when trying
> to read or upgradesstables. If you don't have a backup of prod for whatever
> reason (high churn, ephemeral data, whatever), try at least a backup of one
> host.
>
> This is especially if you may have done a lot of schema changes in
> 1.0/1.2/2.0 era clusters - take a snapshot, fire it up in a new cluster,
> try upgrading offline first.
>
> On Tue, Feb 4, 2020 at 9:29 AM Reid Pinchback <rp...@tripadvisor.com>
> wrote:
>
>> Another thing I'll add, since I don't think any of the other responders
>> brought it up.
>>
>> This all assumes that you already believe that the update is safe. If
>> you have any kind of test cluster, I'd evaluate the change there first.
>>
>> While I haven't hit it with C* specifically, I have seen database
>> problems from the O/S related to shared library updates. With database
>> technologies mostly formed from HA pairs your blast radius on data
>> corruption can be limited, so by the time you realize the problem you
>> haven't risked more than you intended. C* is a bit more biased to propagate
>> a lot of data across a common infrastructure. Just keep that in mind when
>> reviewing what the changes are in the O/S updates. When the updates are
>> related to external utilities and services, but not to shared libraries,
>> usually your only degradation risks relate to performance and availability
>> so you are in a safer position to forge ahead and then it only comes down
>> to proper C* hygiene. Performance and connectivity risks can be mitigated
>> by having one DC you update first, and then letting stew awhile as you
>> evaluate the results. Plan in advance what you want to evaluate before
>> continuing on.
>>
>> R
>>
>> On 1/30/20, 11:09 AM, "Michael Shuler" <mi...@pbandjelly.org> wrote:
>>
>> Message from External Sender
>>
>> That is some good info. To add just a little more, knowing what the
>> pending security updates are for your nodes helps in knowing what to
>> do
>> after. Read the security update notes from your vendor.
>>
>> Java or Cassandra update? Of course the service needs restarted -
>> rolling upgrade and restart the `cassandra` service as usual.
>>
>> Linux kernel update? Node needs a full reboot, so follow a rolling
>> reboot plan.
>>
>> Other OS updates? Most can be done while not affecting Cassandra. For
>> instance, an OpenSSH security update to patch some vulnerability
>> should
>> most certainly be done as soon as possible, and the node updates can
>> be
>> even be in parallel without causing any problems with the JVM or
>> Cassandra service. Most intelligent package update systems will
>> install
>> the update and restart the affected service, in this hypothetical
>> case
>> `sshd`.
>>
>> Michael
>>
>> On 1/30/20 3:56 AM, Erick Ramirez wrote:
>> > There is no need to shutdown the application because you should be
>> able
>> > to carry out the operating system upgraded without an outage to the
>> > database particularly since you have a lot of nodes in your cluster.
>> >
>> > Provided your cluster has sufficient capacity, you might even have
>> the
>> > ability to upgrade multiple nodes in parallel to reduce the upgrade
>> > window. If you decide to do nodes in parallel and you fully
>> understand
>> > the token allocations and where the nodes are positioned in the
>> ring in
>> > each DC, make sure you only upgrade nodes which are at least 5
>> nodes
>> > "away" to the right so you know none of the nodes would have
>> overlapping
>> > token ranges and they're not replicas of each other.
>> >
>> > Other points to consider are:
>> >
>> > * If a node goes down (for whatever reason), I suggest you
>> upgrade the
>> > OS on the node before bringing back up. It's already down so you
>> > might as well take advantage of it since you have so many nodes
>> to
>> > upgrade.
>> > * Resist the urge to run nodetool decommission or nodetool
>> removenode
>> > if you encounter an issue while upgrading a node. This is a
>> common
>> > knee-jerk reaction which can prove costly because the cluster
>> will
>> > rebalance automatically, adding more time to your upgrade
>> window.
>> > Either fix the problem on the server or replace node using the
>> > "replace_address" flag.
>> > * Test, test, and test again. Familiarity with the process is your
>> > friend when the unexpected happens.
>> > * Plan ahead and rehearse your recovery method (i.e. replace the
>> node)
>> > should you run into unexpected issues.
>> > * Stick to the plan and be prepared to implement it -- don't
>> deviate.
>> > Don't spend 4 hours or more investigating why a server won't
>> start.
>> > * Be decisive. Activate your recovery/remediation plan
>> immediately.
>> >
>> > I'm sure others will chime in with their recommendations. Let us
>> know
>> > how you go as I'm sure others would be interested in hearing from
>> your
>> > experience. Not a lot of shops have a deployment as large as yours
>> so
>> > you are in an enviable position. Good luck!
>> >
>> > On Thu, Jan 30, 2020 at 3:45 PM Anshu Vajpayee <
>> anshu.vajpayee@gmail.com
>> > <ma...@gmail.com>> wrote:
>> >
>> > Hi Team,
>> > What is the best way to patch OS of 1000 nodes Multi DC
>> Cassandra
>> > cluster where we cannot suspend application traffic( we can
>> redirect
>> > traffic to one DC).
>> >
>> > Please suggest if anyone has any best practice around it.
>> >
>> > --
>> > *C*heers,*
>> > *Anshu V*
>> > *
>> > *
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
>>
>>
Re: Cassandra OS Patching.
Posted by Jeff Jirsa <jj...@gmail.com>.
The original question was OS (and / or JDK) upgrades, and for those "do it
in QA first", bounce non-replicas and let them come up before proceeding.
If you're doing Cassandra itself, it's a REALLY REALLY REALLY good idea to
try the upgrade on a backup of your production cluster first. While it's
uncommon, there were certain things allowed in very old versions of
Cassandra that are occasionally missed in upgrade tests, and can be found
the hard way. Sometimes those incompatible schema changes should have
disallowed but weren't - sometimes they leave data on disk that cassandra
3.0/3.11/etc know is illegal or invalid, and throws exceptions when trying
to read or upgradesstables. If you don't have a backup of prod for whatever
reason (high churn, ephemeral data, whatever), try at least a backup of one
host.
This is especially if you may have done a lot of schema changes in
1.0/1.2/2.0 era clusters - take a snapshot, fire it up in a new cluster,
try upgrading offline first.
On Tue, Feb 4, 2020 at 9:29 AM Reid Pinchback <rp...@tripadvisor.com>
wrote:
> Another thing I'll add, since I don't think any of the other responders
> brought it up.
>
> This all assumes that you already believe that the update is safe. If you
> have any kind of test cluster, I'd evaluate the change there first.
>
> While I haven't hit it with C* specifically, I have seen database problems
> from the O/S related to shared library updates. With database technologies
> mostly formed from HA pairs your blast radius on data corruption can be
> limited, so by the time you realize the problem you haven't risked more
> than you intended. C* is a bit more biased to propagate a lot of data
> across a common infrastructure. Just keep that in mind when reviewing what
> the changes are in the O/S updates. When the updates are related to
> external utilities and services, but not to shared libraries, usually your
> only degradation risks relate to performance and availability so you are in
> a safer position to forge ahead and then it only comes down to proper C*
> hygiene. Performance and connectivity risks can be mitigated by having one
> DC you update first, and then letting stew awhile as you evaluate the
> results. Plan in advance what you want to evaluate before continuing on.
>
> R
>
> On 1/30/20, 11:09 AM, "Michael Shuler" <mi...@pbandjelly.org> wrote:
>
> Message from External Sender
>
> That is some good info. To add just a little more, knowing what the
> pending security updates are for your nodes helps in knowing what to
> do
> after. Read the security update notes from your vendor.
>
> Java or Cassandra update? Of course the service needs restarted -
> rolling upgrade and restart the `cassandra` service as usual.
>
> Linux kernel update? Node needs a full reboot, so follow a rolling
> reboot plan.
>
> Other OS updates? Most can be done while not affecting Cassandra. For
> instance, an OpenSSH security update to patch some vulnerability
> should
> most certainly be done as soon as possible, and the node updates can
> be
> even be in parallel without causing any problems with the JVM or
> Cassandra service. Most intelligent package update systems will
> install
> the update and restart the affected service, in this hypothetical case
> `sshd`.
>
> Michael
>
> On 1/30/20 3:56 AM, Erick Ramirez wrote:
> > There is no need to shutdown the application because you should be
> able
> > to carry out the operating system upgraded without an outage to the
> > database particularly since you have a lot of nodes in your cluster.
> >
> > Provided your cluster has sufficient capacity, you might even have
> the
> > ability to upgrade multiple nodes in parallel to reduce the upgrade
> > window. If you decide to do nodes in parallel and you fully
> understand
> > the token allocations and where the nodes are positioned in the ring
> in
> > each DC, make sure you only upgrade nodes which are at least 5 nodes
> > "away" to the right so you know none of the nodes would have
> overlapping
> > token ranges and they're not replicas of each other.
> >
> > Other points to consider are:
> >
> > * If a node goes down (for whatever reason), I suggest you upgrade
> the
> > OS on the node before bringing back up. It's already down so you
> > might as well take advantage of it since you have so many nodes
> to
> > upgrade.
> > * Resist the urge to run nodetool decommission or nodetool
> removenode
> > if you encounter an issue while upgrading a node. This is a
> common
> > knee-jerk reaction which can prove costly because the cluster
> will
> > rebalance automatically, adding more time to your upgrade window.
> > Either fix the problem on the server or replace node using the
> > "replace_address" flag.
> > * Test, test, and test again. Familiarity with the process is your
> > friend when the unexpected happens.
> > * Plan ahead and rehearse your recovery method (i.e. replace the
> node)
> > should you run into unexpected issues.
> > * Stick to the plan and be prepared to implement it -- don't
> deviate.
> > Don't spend 4 hours or more investigating why a server won't
> start.
> > * Be decisive. Activate your recovery/remediation plan immediately.
> >
> > I'm sure others will chime in with their recommendations. Let us
> know
> > how you go as I'm sure others would be interested in hearing from
> your
> > experience. Not a lot of shops have a deployment as large as yours
> so
> > you are in an enviable position. Good luck!
> >
> > On Thu, Jan 30, 2020 at 3:45 PM Anshu Vajpayee <
> anshu.vajpayee@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Hi Team,
> > What is the best way to patch OS of 1000 nodes Multi DC Cassandra
> > cluster where we cannot suspend application traffic( we can
> redirect
> > traffic to one DC).
> >
> > Please suggest if anyone has any best practice around it.
> >
> > --
> > *C*heers,*
> > *Anshu V*
> > *
> > *
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
>
>
Re: Cassandra OS Patching.
Posted by Reid Pinchback <rp...@tripadvisor.com>.
Another thing I'll add, since I don't think any of the other responders brought it up.
This all assumes that you already believe that the update is safe. If you have any kind of test cluster, I'd evaluate the change there first.
While I haven't hit it with C* specifically, I have seen database problems from the O/S related to shared library updates. With database technologies mostly formed from HA pairs your blast radius on data corruption can be limited, so by the time you realize the problem you haven't risked more than you intended. C* is a bit more biased to propagate a lot of data across a common infrastructure. Just keep that in mind when reviewing what the changes are in the O/S updates. When the updates are related to external utilities and services, but not to shared libraries, usually your only degradation risks relate to performance and availability so you are in a safer position to forge ahead and then it only comes down to proper C* hygiene. Performance and connectivity risks can be mitigated by having one DC you update first, and then letting stew awhile as you evaluate the results. Plan in advance what you want to evaluate before continuing on.
R
On 1/30/20, 11:09 AM, "Michael Shuler" <mi...@pbandjelly.org> wrote:
Message from External Sender
That is some good info. To add just a little more, knowing what the
pending security updates are for your nodes helps in knowing what to do
after. Read the security update notes from your vendor.
Java or Cassandra update? Of course the service needs restarted -
rolling upgrade and restart the `cassandra` service as usual.
Linux kernel update? Node needs a full reboot, so follow a rolling
reboot plan.
Other OS updates? Most can be done while not affecting Cassandra. For
instance, an OpenSSH security update to patch some vulnerability should
most certainly be done as soon as possible, and the node updates can be
even be in parallel without causing any problems with the JVM or
Cassandra service. Most intelligent package update systems will install
the update and restart the affected service, in this hypothetical case
`sshd`.
Michael
On 1/30/20 3:56 AM, Erick Ramirez wrote:
> There is no need to shutdown the application because you should be able
> to carry out the operating system upgraded without an outage to the
> database particularly since you have a lot of nodes in your cluster.
>
> Provided your cluster has sufficient capacity, you might even have the
> ability to upgrade multiple nodes in parallel to reduce the upgrade
> window. If you decide to do nodes in parallel and you fully understand
> the token allocations and where the nodes are positioned in the ring in
> each DC, make sure you only upgrade nodes which are at least 5 nodes
> "away" to the right so you know none of the nodes would have overlapping
> token ranges and they're not replicas of each other.
>
> Other points to consider are:
>
> * If a node goes down (for whatever reason), I suggest you upgrade the
> OS on the node before bringing back up. It's already down so you
> might as well take advantage of it since you have so many nodes to
> upgrade.
> * Resist the urge to run nodetool decommission or nodetool removenode
> if you encounter an issue while upgrading a node. This is a common
> knee-jerk reaction which can prove costly because the cluster will
> rebalance automatically, adding more time to your upgrade window.
> Either fix the problem on the server or replace node using the
> "replace_address" flag.
> * Test, test, and test again. Familiarity with the process is your
> friend when the unexpected happens.
> * Plan ahead and rehearse your recovery method (i.e. replace the node)
> should you run into unexpected issues.
> * Stick to the plan and be prepared to implement it -- don't deviate.
> Don't spend 4 hours or more investigating why a server won't start.
> * Be decisive. Activate your recovery/remediation plan immediately.
>
> I'm sure others will chime in with their recommendations. Let us know
> how you go as I'm sure others would be interested in hearing from your
> experience. Not a lot of shops have a deployment as large as yours so
> you are in an enviable position. Good luck!
>
> On Thu, Jan 30, 2020 at 3:45 PM Anshu Vajpayee <anshu.vajpayee@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi Team,
> What is the best way to patch OS of 1000 nodes Multi DC Cassandra
> cluster where we cannot suspend application traffic( we can redirect
> traffic to one DC).
>
> Please suggest if anyone has any best practice around it.
>
> --
> *C*heers,*
> *Anshu V*
> *
> *
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Cassandra OS Patching.
Posted by Michael Shuler <mi...@pbandjelly.org>.
That is some good info. To add just a little more, knowing what the
pending security updates are for your nodes helps in knowing what to do
after. Read the security update notes from your vendor.
Java or Cassandra update? Of course the service needs restarted -
rolling upgrade and restart the `cassandra` service as usual.
Linux kernel update? Node needs a full reboot, so follow a rolling
reboot plan.
Other OS updates? Most can be done while not affecting Cassandra. For
instance, an OpenSSH security update to patch some vulnerability should
most certainly be done as soon as possible, and the node updates can be
even be in parallel without causing any problems with the JVM or
Cassandra service. Most intelligent package update systems will install
the update and restart the affected service, in this hypothetical case
`sshd`.
Michael
On 1/30/20 3:56 AM, Erick Ramirez wrote:
> There is no need to shutdown the application because you should be able
> to carry out the operating system upgraded without an outage to the
> database particularly since you have a lot of nodes in your cluster.
>
> Provided your cluster has sufficient capacity, you might even have the
> ability to upgrade multiple nodes in parallel to reduce the upgrade
> window. If you decide to do nodes in parallel and you fully understand
> the token allocations and where the nodes are positioned in the ring in
> each DC, make sure you only upgrade nodes which are at least 5 nodes
> "away" to the right so you know none of the nodes would have overlapping
> token ranges and they're not replicas of each other.
>
> Other points to consider are:
>
> * If a node goes down (for whatever reason), I suggest you upgrade the
> OS on the node before bringing back up. It's already down so you
> might as well take advantage of it since you have so many nodes to
> upgrade.
> * Resist the urge to run nodetool decommission or nodetool removenode
> if you encounter an issue while upgrading a node. This is a common
> knee-jerk reaction which can prove costly because the cluster will
> rebalance automatically, adding more time to your upgrade window.
> Either fix the problem on the server or replace node using the
> "replace_address" flag.
> * Test, test, and test again. Familiarity with the process is your
> friend when the unexpected happens.
> * Plan ahead and rehearse your recovery method (i.e. replace the node)
> should you run into unexpected issues.
> * Stick to the plan and be prepared to implement it -- don't deviate.
> Don't spend 4 hours or more investigating why a server won't start.
> * Be decisive. Activate your recovery/remediation plan immediately.
>
> I'm sure others will chime in with their recommendations. Let us know
> how you go as I'm sure others would be interested in hearing from your
> experience. Not a lot of shops have a deployment as large as yours so
> you are in an enviable position. Good luck!
>
> On Thu, Jan 30, 2020 at 3:45 PM Anshu Vajpayee <anshu.vajpayee@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi Team,
> What is the best way to patch OS of 1000 nodes Multi DC Cassandra
> cluster where we cannot suspend application traffic( we can redirect
> traffic to one DC).
>
> Please suggest if anyone has any best practice around it.
>
> --
> *C*heers,*
> *Anshu V*
> *
> *
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Cassandra OS Patching.
Posted by Erick Ramirez <fl...@gmail.com>.
There is no need to shutdown the application because you should be able to
carry out the operating system upgraded without an outage to the database
particularly since you have a lot of nodes in your cluster.
Provided your cluster has sufficient capacity, you might even have the
ability to upgrade multiple nodes in parallel to reduce the upgrade window.
If you decide to do nodes in parallel and you fully understand the token
allocations and where the nodes are positioned in the ring in each DC, make
sure you only upgrade nodes which are at least 5 nodes "away" to the right
so you know none of the nodes would have overlapping token ranges and
they're not replicas of each other.
Other points to consider are:
- If a node goes down (for whatever reason), I suggest you upgrade the
OS on the node before bringing back up. It's already down so you might as
well take advantage of it since you have so many nodes to upgrade.
- Resist the urge to run nodetool decommission or nodetool removenode if
you encounter an issue while upgrading a node. This is a common knee-jerk
reaction which can prove costly because the cluster will rebalance
automatically, adding more time to your upgrade window. Either fix the
problem on the server or replace node using the "replace_address" flag.
- Test, test, and test again. Familiarity with the process is your
friend when the unexpected happens.
- Plan ahead and rehearse your recovery method (i.e. replace the node)
should you run into unexpected issues.
- Stick to the plan and be prepared to implement it -- don't deviate.
Don't spend 4 hours or more investigating why a server won't start.
- Be decisive. Activate your recovery/remediation plan immediately.
I'm sure others will chime in with their recommendations. Let us know how
you go as I'm sure others would be interested in hearing from your
experience. Not a lot of shops have a deployment as large as yours so you
are in an enviable position. Good luck!
On Thu, Jan 30, 2020 at 3:45 PM Anshu Vajpayee <an...@gmail.com>
wrote:
> Hi Team,
> What is the best way to patch OS of 1000 nodes Multi DC Cassandra cluster
> where we cannot suspend application traffic( we can redirect traffic to one
> DC).
>
> Please suggest if anyone has any best practice around it.
>
> --
> *C*heers,*
> *Anshu V*
>
>
>