You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Lance Albertson <la...@osuosl.org> on 2022/09/06 22:01:08 UTC
Re: [Hosting] MAINTENANCE: Network Uplink migration to LinkOregon - Tuesday, Sep 6 2200 UTC
We're going to be doing this outage here in the next 5-10 minutes. Wish us luck!
On Tue, Aug 30, 2022 at 3:12 PM Lance Albertson <la...@osuosl.org> wrote:
>
> All,
>
> Next Tuesday, September 6th between 3-5PM PDT (2200-0000UTC) we will be migrating our network uplink from NERO to LinkOregon.
>
> Over the past several years, NERO has been merged into a new organization called LinkOregon [1]. They have built a completely new network and this migration will finally move us over to that new network. This will eventually give us access to even more bandwidth and will also move us off of their "legacy" network. For now we're just moving from one 10G uplink to another 10G. But we're planning on upgrading this to 40G or 100G eventually.
>
> I expect there to be a 5-10min system-wide outage (hopefully much less) while we make this migration. We will be shutting down the current uplink on one router, and bringing up the new uplink on a different router. I will be on-site in case there are any major issues where I need to get direct access to the switches.
>
> If you have any questions or concerns, please let me know.
>
> [1] https://www.linkoregon.org/
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
--
Lance Albertson
Director
Oregon State University | Open Source Lab
_______________________________________________
Hosting mailing list
Hosting@osuosl.org
https://lists.osuosl.org/mailman/listinfo/hosting
Re: [Hosting] On going IPv6 issues with some subnets
Posted by Lance Albertson <la...@osuosl.org>.
I have confirmed this has fixed the problem for IPv6 on the affected
subnets.
On Mon, Sep 12, 2022 at 3:15 PM Lance Albertson <la...@osuosl.org> wrote:
> All,
>
> LinkOregon will be pushing a fix for this tonight at midnight PDT.
>
> On Wed, Sep 7, 2022 at 2:22 PM Lance Albertson <la...@osuosl.org> wrote:
>
>> All,
>>
>> I wanted to send you an update on where we're at with the migration. We
>> appear to have hit some kind of bug on the LinkOregon side where any of our
>> IPv6 subnets above a certain range just don't work outside of the router.
>> They have a case open with the vendor and they are looking into it. I've
>> reached out to the affected parties and they're all OK with waiting until
>> it gets resolved (even if it's a few days away).
>>
>> Something else I noticed until this morning was that our email relays had
>> grabbed a SLAAC IPv6 address and were using it instead of the statically
>> assigned one. This resulted in some email not getting delivered to gmail
>> addresses due to their rules with IPv6 (i.e. no reverse DNS or included in
>> SPF). I resolved this at around 9:30AM PDT so any mail onward should have
>> gone through.
>>
>> I should also note that we have enabled IPv6 on _one_ OpenStack subnet
>> (public6 on OpenPOWER) that is _also_ affected by this outage. We are in
>> the process of testing IPv6 on OpenStack and hope to have this available
>> system wide in a few weeks (assuming this current issue is resolved soon).
>> We _will_ be utilizing SLAAC for the OpenStack instances once we have that
>> ready so make sure your VMs are ready for that.
>>
>> I will also be updating the reverse DNS entries for all of our gateways
>> soon to reflect the new physical paths. So you may notice that if you're
>> doing any kind of traceroute.
>>
>> Thanks for your patience and please let us know if you have any
>> additional questions or issues since our change yesterday.
>>
>> Thanks!
>>
>> On Tue, Sep 6, 2022 at 6:04 PM Lance Albertson <la...@osuosl.org> wrote:
>>
>>> All,
>>>
>>> I have an update on some issues we did discover after the fact. For
>>> now, everything with IPv4 seems to be 100% OK. Please let me know if
>>> that isn't the case.
>>>
>>> We did however have SLAAC enabled for a while on the other subnets
>>> where we normally didn't have it enabled. So if you see an IPv6
>>> address that is automatically assigned but it's not routable, that
>>> would be the case.
>>>
>>> HOWEVER, we are having some very strange issues with IPv6 on _some_ of
>>> the subnets. We weren't able to track down the root cause today and
>>> will try tomorrow.
>>>
>>> As of right now, it seems that this is only happening on the following
>>> IPv6 subnets:
>>>
>>> 2605:bc80:3010:200::/64 (Gentoo)
>>> 2605:bc80:3010:400::/64 (Drupal)
>>> 2605:bc80:3010:401::/64 (Drupal Virt)
>>> 2605:bc80:3010:600::/64 (Fedora)
>>> 2605:bc80:3010:700::/64 (OSM)
>>> 2605:bc80:3010:a00::/64 (Buildbot/RTEMS)
>>> 2605:bc80:3010:b00::/64 (Debian)
>>>
>>> If I am missing any IPv6 subnet that isn't working, please let me
>>> know. Since this impacted a smaller set of users, I decided to move
>>> forward keeping us on the new connection while we work out these IPv6
>>> issues and not roll back.
>>>
>>> Thanks again for all your support and patience!
>>>
>>
>>
>> --
>> Lance Albertson
>> Director
>> Oregon State University | Open Source Lab
>>
>
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>
--
Lance Albertson
Director
Oregon State University | Open Source Lab
Re: [Hosting] On going IPv6 issues with some subnets
Posted by Lance Albertson <la...@osuosl.org>.
All,
LinkOregon will be pushing a fix for this tonight at midnight PDT.
On Wed, Sep 7, 2022 at 2:22 PM Lance Albertson <la...@osuosl.org> wrote:
> All,
>
> I wanted to send you an update on where we're at with the migration. We
> appear to have hit some kind of bug on the LinkOregon side where any of our
> IPv6 subnets above a certain range just don't work outside of the router.
> They have a case open with the vendor and they are looking into it. I've
> reached out to the affected parties and they're all OK with waiting until
> it gets resolved (even if it's a few days away).
>
> Something else I noticed until this morning was that our email relays had
> grabbed a SLAAC IPv6 address and were using it instead of the statically
> assigned one. This resulted in some email not getting delivered to gmail
> addresses due to their rules with IPv6 (i.e. no reverse DNS or included in
> SPF). I resolved this at around 9:30AM PDT so any mail onward should have
> gone through.
>
> I should also note that we have enabled IPv6 on _one_ OpenStack subnet
> (public6 on OpenPOWER) that is _also_ affected by this outage. We are in
> the process of testing IPv6 on OpenStack and hope to have this available
> system wide in a few weeks (assuming this current issue is resolved soon).
> We _will_ be utilizing SLAAC for the OpenStack instances once we have that
> ready so make sure your VMs are ready for that.
>
> I will also be updating the reverse DNS entries for all of our gateways
> soon to reflect the new physical paths. So you may notice that if you're
> doing any kind of traceroute.
>
> Thanks for your patience and please let us know if you have any additional
> questions or issues since our change yesterday.
>
> Thanks!
>
> On Tue, Sep 6, 2022 at 6:04 PM Lance Albertson <la...@osuosl.org> wrote:
>
>> All,
>>
>> I have an update on some issues we did discover after the fact. For
>> now, everything with IPv4 seems to be 100% OK. Please let me know if
>> that isn't the case.
>>
>> We did however have SLAAC enabled for a while on the other subnets
>> where we normally didn't have it enabled. So if you see an IPv6
>> address that is automatically assigned but it's not routable, that
>> would be the case.
>>
>> HOWEVER, we are having some very strange issues with IPv6 on _some_ of
>> the subnets. We weren't able to track down the root cause today and
>> will try tomorrow.
>>
>> As of right now, it seems that this is only happening on the following
>> IPv6 subnets:
>>
>> 2605:bc80:3010:200::/64 (Gentoo)
>> 2605:bc80:3010:400::/64 (Drupal)
>> 2605:bc80:3010:401::/64 (Drupal Virt)
>> 2605:bc80:3010:600::/64 (Fedora)
>> 2605:bc80:3010:700::/64 (OSM)
>> 2605:bc80:3010:a00::/64 (Buildbot/RTEMS)
>> 2605:bc80:3010:b00::/64 (Debian)
>>
>> If I am missing any IPv6 subnet that isn't working, please let me
>> know. Since this impacted a smaller set of users, I decided to move
>> forward keeping us on the new connection while we work out these IPv6
>> issues and not roll back.
>>
>> Thanks again for all your support and patience!
>>
>
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>
--
Lance Albertson
Director
Oregon State University | Open Source Lab
[Hosting] On going IPv6 issues with some subnets
Posted by Lance Albertson <la...@osuosl.org>.
All,
I wanted to send you an update on where we're at with the migration. We
appear to have hit some kind of bug on the LinkOregon side where any of our
IPv6 subnets above a certain range just don't work outside of the router.
They have a case open with the vendor and they are looking into it. I've
reached out to the affected parties and they're all OK with waiting until
it gets resolved (even if it's a few days away).
Something else I noticed until this morning was that our email relays had
grabbed a SLAAC IPv6 address and were using it instead of the statically
assigned one. This resulted in some email not getting delivered to gmail
addresses due to their rules with IPv6 (i.e. no reverse DNS or included in
SPF). I resolved this at around 9:30AM PDT so any mail onward should have
gone through.
I should also note that we have enabled IPv6 on _one_ OpenStack subnet
(public6 on OpenPOWER) that is _also_ affected by this outage. We are in
the process of testing IPv6 on OpenStack and hope to have this available
system wide in a few weeks (assuming this current issue is resolved soon).
We _will_ be utilizing SLAAC for the OpenStack instances once we have that
ready so make sure your VMs are ready for that.
I will also be updating the reverse DNS entries for all of our gateways
soon to reflect the new physical paths. So you may notice that if you're
doing any kind of traceroute.
Thanks for your patience and please let us know if you have any additional
questions or issues since our change yesterday.
Thanks!
On Tue, Sep 6, 2022 at 6:04 PM Lance Albertson <la...@osuosl.org> wrote:
> All,
>
> I have an update on some issues we did discover after the fact. For
> now, everything with IPv4 seems to be 100% OK. Please let me know if
> that isn't the case.
>
> We did however have SLAAC enabled for a while on the other subnets
> where we normally didn't have it enabled. So if you see an IPv6
> address that is automatically assigned but it's not routable, that
> would be the case.
>
> HOWEVER, we are having some very strange issues with IPv6 on _some_ of
> the subnets. We weren't able to track down the root cause today and
> will try tomorrow.
>
> As of right now, it seems that this is only happening on the following
> IPv6 subnets:
>
> 2605:bc80:3010:200::/64 (Gentoo)
> 2605:bc80:3010:400::/64 (Drupal)
> 2605:bc80:3010:401::/64 (Drupal Virt)
> 2605:bc80:3010:600::/64 (Fedora)
> 2605:bc80:3010:700::/64 (OSM)
> 2605:bc80:3010:a00::/64 (Buildbot/RTEMS)
> 2605:bc80:3010:b00::/64 (Debian)
>
> If I am missing any IPv6 subnet that isn't working, please let me
> know. Since this impacted a smaller set of users, I decided to move
> forward keeping us on the new connection while we work out these IPv6
> issues and not roll back.
>
> Thanks again for all your support and patience!
>
--
Lance Albertson
Director
Oregon State University | Open Source Lab
Re: [Hosting] MAINTENANCE: Network Uplink migration to LinkOregon - Tuesday, Sep 6 2200 UTC
Posted by Lance Albertson <la...@osuosl.org>.
All,
I have an update on some issues we did discover after the fact. For
now, everything with IPv4 seems to be 100% OK. Please let me know if
that isn't the case.
We did however have SLAAC enabled for a while on the other subnets
where we normally didn't have it enabled. So if you see an IPv6
address that is automatically assigned but it's not routable, that
would be the case.
HOWEVER, we are having some very strange issues with IPv6 on _some_ of
the subnets. We weren't able to track down the root cause today and
will try tomorrow.
As of right now, it seems that this is only happening on the following
IPv6 subnets:
2605:bc80:3010:200::/64 (Gentoo)
2605:bc80:3010:400::/64 (Drupal)
2605:bc80:3010:401::/64 (Drupal Virt)
2605:bc80:3010:600::/64 (Fedora)
2605:bc80:3010:700::/64 (OSM)
2605:bc80:3010:a00::/64 (Buildbot/RTEMS)
2605:bc80:3010:b00::/64 (Debian)
If I am missing any IPv6 subnet that isn't working, please let me
know. Since this impacted a smaller set of users, I decided to move
forward keeping us on the new connection while we work out these IPv6
issues and not roll back.
Thanks again for all your support and patience!
On Tue, Sep 6, 2022 at 3:21 PM Lance Albertson <la...@osuosl.org> wrote:
>
> We should be back online now. Please let us know if there are any issues.
>
> On Tue, Sep 6, 2022 at 3:01 PM Lance Albertson <la...@osuosl.org> wrote:
> >
> > We're going to be doing this outage here in the next 5-10 minutes. Wish us luck!
> >
> > On Tue, Aug 30, 2022 at 3:12 PM Lance Albertson <la...@osuosl.org> wrote:
> > >
> > > All,
> > >
> > > Next Tuesday, September 6th between 3-5PM PDT (2200-0000UTC) we will be migrating our network uplink from NERO to LinkOregon.
> > >
> > > Over the past several years, NERO has been merged into a new organization called LinkOregon [1]. They have built a completely new network and this migration will finally move us over to that new network. This will eventually give us access to even more bandwidth and will also move us off of their "legacy" network. For now we're just moving from one 10G uplink to another 10G. But we're planning on upgrading this to 40G or 100G eventually.
> > >
> > > I expect there to be a 5-10min system-wide outage (hopefully much less) while we make this migration. We will be shutting down the current uplink on one router, and bringing up the new uplink on a different router. I will be on-site in case there are any major issues where I need to get direct access to the switches.
> > >
> > > If you have any questions or concerns, please let me know.
> > >
> > > [1] https://www.linkoregon.org/
> > >
> > > --
> > > Lance Albertson
> > > Director
> > > Oregon State University | Open Source Lab
> >
> >
> >
> > --
> > Lance Albertson
> > Director
> > Oregon State University | Open Source Lab
>
>
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
--
Lance Albertson
Director
Oregon State University | Open Source Lab
_______________________________________________
Hosting mailing list
Hosting@osuosl.org
https://lists.osuosl.org/mailman/listinfo/hosting
Re: [Hosting] MAINTENANCE: Network Uplink migration to LinkOregon - Tuesday, Sep 6 2200 UTC
Posted by Lance Albertson <la...@osuosl.org>.
We should be back online now. Please let us know if there are any issues.
On Tue, Sep 6, 2022 at 3:01 PM Lance Albertson <la...@osuosl.org> wrote:
>
> We're going to be doing this outage here in the next 5-10 minutes. Wish us luck!
>
> On Tue, Aug 30, 2022 at 3:12 PM Lance Albertson <la...@osuosl.org> wrote:
> >
> > All,
> >
> > Next Tuesday, September 6th between 3-5PM PDT (2200-0000UTC) we will be migrating our network uplink from NERO to LinkOregon.
> >
> > Over the past several years, NERO has been merged into a new organization called LinkOregon [1]. They have built a completely new network and this migration will finally move us over to that new network. This will eventually give us access to even more bandwidth and will also move us off of their "legacy" network. For now we're just moving from one 10G uplink to another 10G. But we're planning on upgrading this to 40G or 100G eventually.
> >
> > I expect there to be a 5-10min system-wide outage (hopefully much less) while we make this migration. We will be shutting down the current uplink on one router, and bringing up the new uplink on a different router. I will be on-site in case there are any major issues where I need to get direct access to the switches.
> >
> > If you have any questions or concerns, please let me know.
> >
> > [1] https://www.linkoregon.org/
> >
> > --
> > Lance Albertson
> > Director
> > Oregon State University | Open Source Lab
>
>
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
--
Lance Albertson
Director
Oregon State University | Open Source Lab
_______________________________________________
Hosting mailing list
Hosting@osuosl.org
https://lists.osuosl.org/mailman/listinfo/hosting