You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Lance Albertson <la...@osuosl.org> on 2021/08/19 20:25:43 UTC
[Hosting] mysql-vip2 (mysql3/mysql4) 5.6 -> 5.7 Upgrade, August 23,
2021 2-4PM PDT
All,
We will be performing an upgrade on one of our MySQL clusters (mysql-vip2
(mysql3/mysql4)) next Monday, August 23, 2021 between 2-4PM PDT.
`date -d '1400 PDT` <-- Display this time in your local time zone.
Services Impacted:
- OpenStack control plane and API endpoints (all clusters)
- OSL RT
- SQLGrey on OSL mail relays
- Various projects which use this cluster
Summary:
We are upgrading from MySQL (Percona Server) from 5.6 to 5.7. This will
bring us to a version which includes additional features that have been
required with some projects.
Outage summary:
We have tested the upgrade in a variety of ways (thanks to our student
Robert Detjens for doing this hard work!) and believe our migration should
have minimal impacts to services. However, there will likely be a small
outage when we run mysql_upgrade since it requires a write lock.
Migration steps:
Here's a summary of the steps we're planning on taking for the migration:
1. Stop configuration management cron job
2. Stop replication on both nodes
3. Perform upgrade on mysql4 (does not have the VIP)
4. Start daemon on mysql4 (will join as secondary), wait for catch up
5. Switch VIP to mysql4
6. Stop replication again on both nodes
7. Perform upgrade on mysql3
8. Start daemon on mysql3 (will join as secondary), wait for catch up
9. Switch VIP to mysql3
10. Run mysql_upgrade on mysql3 (needs write lock)
11. Run mysql_upgrade on mysql4 (needs write lock)
12. Run and re-enable configuration management
If you have any questions or concerns, please let us know. We'll be in
#osuosl on Libera Chat in case any one needs to run us directly.
We plan on doing a similar upgrade to our other MySQL cluster, but that
will happen later and will require additional steps.
Thanks!
--
Lance Albertson
Director
Oregon State University | Open Source Lab
Re: [Hosting] mysql-vip2 (mysql3/mysql4) 5.6 -> 5.7 Upgrade,
August 23, 2021 2-4PM PDT
Posted by Robert Detjens <de...@osuosl.org>.
Updated certificates have been deployed and clients should now be able to
use SSL connections correctly. If there are any further issues, please open
a support ticket or reach us in #osuosl on Libera Chat.
On Mon, Aug 23, 2021 at 4:24 PM Lance Albertson <la...@osuosl.org> wrote:
> The upgrade has been completed, but there is an issue with SSL
> connections. The new version is using self-signed certificates which is
> causing problems for some clients. We are preparing a fix and will have
> that rolled out shortly.
>
> On Thu, Aug 19, 2021 at 1:25 PM Lance Albertson <la...@osuosl.org> wrote:
>
>> All,
>>
>> We will be performing an upgrade on one of our MySQL clusters (mysql-vip2
>> (mysql3/mysql4)) next Monday, August 23, 2021 between 2-4PM PDT.
>>
>> `date -d '1400 PDT` <-- Display this time in your local time zone.
>>
>> Services Impacted:
>>
>> - OpenStack control plane and API endpoints (all clusters)
>> - OSL RT
>> - SQLGrey on OSL mail relays
>> - Various projects which use this cluster
>>
>> Summary:
>>
>> We are upgrading from MySQL (Percona Server) from 5.6 to 5.7. This will
>> bring us to a version which includes additional features that have been
>> required with some projects.
>>
>> Outage summary:
>>
>> We have tested the upgrade in a variety of ways (thanks to our student
>> Robert Detjens for doing this hard work!) and believe our migration should
>> have minimal impacts to services. However, there will likely be a small
>> outage when we run mysql_upgrade since it requires a write lock.
>>
>> Migration steps:
>>
>> Here's a summary of the steps we're planning on taking for the migration:
>>
>> 1. Stop configuration management cron job
>> 2. Stop replication on both nodes
>> 3. Perform upgrade on mysql4 (does not have the VIP)
>> 4. Start daemon on mysql4 (will join as secondary), wait for catch up
>> 5. Switch VIP to mysql4
>> 6. Stop replication again on both nodes
>> 7. Perform upgrade on mysql3
>> 8. Start daemon on mysql3 (will join as secondary), wait for catch up
>> 9. Switch VIP to mysql3
>> 10. Run mysql_upgrade on mysql3 (needs write lock)
>> 11. Run mysql_upgrade on mysql4 (needs write lock)
>> 12. Run and re-enable configuration management
>>
>> If you have any questions or concerns, please let us know. We'll be in
>> #osuosl on Libera Chat in case any one needs to run us directly.
>>
>> We plan on doing a similar upgrade to our other MySQL cluster, but that
>> will happen later and will require additional steps.
>>
>> Thanks!
>>
>> --
>> Lance Albertson
>> Director
>> Oregon State University | Open Source Lab
>>
>
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
> _______________________________________________
> Hosting mailing list
> Hosting@osuosl.org
> https://lists.osuosl.org/mailman/listinfo/hosting
>
Re: [Hosting] mysql-vip2 (mysql3/mysql4) 5.6 -> 5.7 Upgrade,
August 23, 2021 2-4PM PDT
Posted by Lance Albertson <la...@osuosl.org>.
The upgrade has been completed, but there is an issue with SSL connections.
The new version is using self-signed certificates which is causing problems
for some clients. We are preparing a fix and will have that rolled out
shortly.
On Thu, Aug 19, 2021 at 1:25 PM Lance Albertson <la...@osuosl.org> wrote:
> All,
>
> We will be performing an upgrade on one of our MySQL clusters (mysql-vip2
> (mysql3/mysql4)) next Monday, August 23, 2021 between 2-4PM PDT.
>
> `date -d '1400 PDT` <-- Display this time in your local time zone.
>
> Services Impacted:
>
> - OpenStack control plane and API endpoints (all clusters)
> - OSL RT
> - SQLGrey on OSL mail relays
> - Various projects which use this cluster
>
> Summary:
>
> We are upgrading from MySQL (Percona Server) from 5.6 to 5.7. This will
> bring us to a version which includes additional features that have been
> required with some projects.
>
> Outage summary:
>
> We have tested the upgrade in a variety of ways (thanks to our student
> Robert Detjens for doing this hard work!) and believe our migration should
> have minimal impacts to services. However, there will likely be a small
> outage when we run mysql_upgrade since it requires a write lock.
>
> Migration steps:
>
> Here's a summary of the steps we're planning on taking for the migration:
>
> 1. Stop configuration management cron job
> 2. Stop replication on both nodes
> 3. Perform upgrade on mysql4 (does not have the VIP)
> 4. Start daemon on mysql4 (will join as secondary), wait for catch up
> 5. Switch VIP to mysql4
> 6. Stop replication again on both nodes
> 7. Perform upgrade on mysql3
> 8. Start daemon on mysql3 (will join as secondary), wait for catch up
> 9. Switch VIP to mysql3
> 10. Run mysql_upgrade on mysql3 (needs write lock)
> 11. Run mysql_upgrade on mysql4 (needs write lock)
> 12. Run and re-enable configuration management
>
> If you have any questions or concerns, please let us know. We'll be in
> #osuosl on Libera Chat in case any one needs to run us directly.
>
> We plan on doing a similar upgrade to our other MySQL cluster, but that
> will happen later and will require additional steps.
>
> Thanks!
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>
--
Lance Albertson
Director
Oregon State University | Open Source Lab