[Hosting] On going IPv6 issues with some subnets

[Lance Albertson <la...@osuosl.org>]

All,

I wanted to send you an update on where we're at with the migration. We
appear to have hit some kind of bug on the LinkOregon side where any of our
IPv6 subnets above a certain range just don't work outside of the router.
They have a case open with the vendor and they are looking into it. I've
reached out to the affected parties and they're all OK with waiting until
it gets resolved (even if it's a few days away).

Something else I noticed until this morning was that our email relays had
grabbed a SLAAC IPv6 address and were using it instead of the statically
assigned one. This resulted in some email not getting delivered to gmail
addresses due to their rules with IPv6 (i.e. no reverse DNS or included in
SPF). I resolved this at around 9:30AM PDT so any mail onward should have
gone through.

I should also note that we have enabled IPv6 on _one_ OpenStack subnet
(public6 on OpenPOWER) that is _also_ affected by this outage. We are in
the process of testing IPv6 on OpenStack and hope to have this available
system wide in a few weeks (assuming this current issue is resolved soon).
We _will_ be utilizing SLAAC for the OpenStack instances once we have that
ready so make sure your VMs are ready for that.

I will also be updating the reverse DNS entries for all of our gateways
soon to reflect the new physical paths. So you may notice that if you're
doing any kind of traceroute.

Thanks for your patience and please let us know if you have any additional
questions or issues since our change yesterday.

Thanks!

On Tue, Sep 6, 2022 at 6:04 PM Lance Albertson <la...@osuosl.org> wrote:

> All,
>
> I have an update on some issues we did discover after the fact. For
> now, everything with IPv4 seems to be 100% OK. Please let me know if
> that isn't the case.
>
> We did however have SLAAC enabled for a while on the other subnets
> where we normally didn't have it enabled. So if you see an IPv6
> address that is automatically assigned but it's not routable, that
> would be the case.
>
> HOWEVER, we are having some very strange issues with IPv6 on _some_ of
> the subnets. We weren't able to track down the root cause today and
> will try tomorrow.
>
> As of right now, it seems that this is only happening on the following
> IPv6 subnets:
>
> 2605:bc80:3010:200::/64 (Gentoo)
> 2605:bc80:3010:400::/64 (Drupal)
> 2605:bc80:3010:401::/64 (Drupal Virt)
> 2605:bc80:3010:600::/64 (Fedora)
> 2605:bc80:3010:700::/64 (OSM)
> 2605:bc80:3010:a00::/64 (Buildbot/RTEMS)
> 2605:bc80:3010:b00::/64 (Debian)
>
> If I am missing any IPv6 subnet that isn't working, please let me
> know. Since this impacted a smaller set of users, I decided to move
> forward keeping us on the new connection while we work out these IPv6
> issues and not roll back.
>
> Thanks again for all your support and patience!
>


-- 
Lance Albertson
Director
Oregon State University | Open Source Lab

Re: [Hosting] On going IPv6 issues with some subnets

[Lance Albertson <la...@osuosl.org>]

I have confirmed this has fixed the problem for IPv6 on the affected
subnets.

On Mon, Sep 12, 2022 at 3:15 PM Lance Albertson <la...@osuosl.org> wrote:

> All,
>
> LinkOregon will be pushing a fix for this tonight at midnight PDT.
>
> On Wed, Sep 7, 2022 at 2:22 PM Lance Albertson <la...@osuosl.org> wrote:
>
>> All,
>>
>> I wanted to send you an update on where we're at with the migration. We
>> appear to have hit some kind of bug on the LinkOregon side where any of our
>> IPv6 subnets above a certain range just don't work outside of the router.
>> They have a case open with the vendor and they are looking into it. I've
>> reached out to the affected parties and they're all OK with waiting until
>> it gets resolved (even if it's a few days away).
>>
>> Something else I noticed until this morning was that our email relays had
>> grabbed a SLAAC IPv6 address and were using it instead of the statically
>> assigned one. This resulted in some email not getting delivered to gmail
>> addresses due to their rules with IPv6 (i.e. no reverse DNS or included in
>> SPF). I resolved this at around 9:30AM PDT so any mail onward should have
>> gone through.
>>
>> I should also note that we have enabled IPv6 on _one_ OpenStack subnet
>> (public6 on OpenPOWER) that is _also_ affected by this outage. We are in
>> the process of testing IPv6 on OpenStack and hope to have this available
>> system wide in a few weeks (assuming this current issue is resolved soon).
>> We _will_ be utilizing SLAAC for the OpenStack instances once we have that
>> ready so make sure your VMs are ready for that.
>>
>> I will also be updating the reverse DNS entries for all of our gateways
>> soon to reflect the new physical paths. So you may notice that if you're
>> doing any kind of traceroute.
>>
>> Thanks for your patience and please let us know if you have any
>> additional questions or issues since our change yesterday.
>>
>> Thanks!
>>
>> On Tue, Sep 6, 2022 at 6:04 PM Lance Albertson <la...@osuosl.org> wrote:
>>
>>> All,
>>>
>>> I have an update on some issues we did discover after the fact. For
>>> now, everything with IPv4 seems to be 100% OK. Please let me know if
>>> that isn't the case.
>>>
>>> We did however have SLAAC enabled for a while on the other subnets
>>> where we normally didn't have it enabled. So if you see an IPv6
>>> address that is automatically assigned but it's not routable, that
>>> would be the case.
>>>
>>> HOWEVER, we are having some very strange issues with IPv6 on _some_ of
>>> the subnets. We weren't able to track down the root cause today and
>>> will try tomorrow.
>>>
>>> As of right now, it seems that this is only happening on the following
>>> IPv6 subnets:
>>>
>>> 2605:bc80:3010:200::/64 (Gentoo)
>>> 2605:bc80:3010:400::/64 (Drupal)
>>> 2605:bc80:3010:401::/64 (Drupal Virt)
>>> 2605:bc80:3010:600::/64 (Fedora)
>>> 2605:bc80:3010:700::/64 (OSM)
>>> 2605:bc80:3010:a00::/64 (Buildbot/RTEMS)
>>> 2605:bc80:3010:b00::/64 (Debian)
>>>
>>> If I am missing any IPv6 subnet that isn't working, please let me
>>> know. Since this impacted a smaller set of users, I decided to move
>>> forward keeping us on the new connection while we work out these IPv6
>>> issues and not roll back.
>>>
>>> Thanks again for all your support and patience!
>>>
>>
>>
>> --
>> Lance Albertson
>> Director
>> Oregon State University | Open Source Lab
>>
>
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>


-- 
Lance Albertson
Director
Oregon State University | Open Source Lab

Re: [Hosting] On going IPv6 issues with some subnets

[Lance Albertson <la...@osuosl.org>]

All,

LinkOregon will be pushing a fix for this tonight at midnight PDT.

On Wed, Sep 7, 2022 at 2:22 PM Lance Albertson <la...@osuosl.org> wrote:

> All,
>
> I wanted to send you an update on where we're at with the migration. We
> appear to have hit some kind of bug on the LinkOregon side where any of our
> IPv6 subnets above a certain range just don't work outside of the router.
> They have a case open with the vendor and they are looking into it. I've
> reached out to the affected parties and they're all OK with waiting until
> it gets resolved (even if it's a few days away).
>
> Something else I noticed until this morning was that our email relays had
> grabbed a SLAAC IPv6 address and were using it instead of the statically
> assigned one. This resulted in some email not getting delivered to gmail
> addresses due to their rules with IPv6 (i.e. no reverse DNS or included in
> SPF). I resolved this at around 9:30AM PDT so any mail onward should have
> gone through.
>
> I should also note that we have enabled IPv6 on _one_ OpenStack subnet
> (public6 on OpenPOWER) that is _also_ affected by this outage. We are in
> the process of testing IPv6 on OpenStack and hope to have this available
> system wide in a few weeks (assuming this current issue is resolved soon).
> We _will_ be utilizing SLAAC for the OpenStack instances once we have that
> ready so make sure your VMs are ready for that.
>
> I will also be updating the reverse DNS entries for all of our gateways
> soon to reflect the new physical paths. So you may notice that if you're
> doing any kind of traceroute.
>
> Thanks for your patience and please let us know if you have any additional
> questions or issues since our change yesterday.
>
> Thanks!
>
> On Tue, Sep 6, 2022 at 6:04 PM Lance Albertson <la...@osuosl.org> wrote:
>
>> All,
>>
>> I have an update on some issues we did discover after the fact. For
>> now, everything with IPv4 seems to be 100% OK. Please let me know if
>> that isn't the case.
>>
>> We did however have SLAAC enabled for a while on the other subnets
>> where we normally didn't have it enabled. So if you see an IPv6
>> address that is automatically assigned but it's not routable, that
>> would be the case.
>>
>> HOWEVER, we are having some very strange issues with IPv6 on _some_ of
>> the subnets. We weren't able to track down the root cause today and
>> will try tomorrow.
>>
>> As of right now, it seems that this is only happening on the following
>> IPv6 subnets:
>>
>> 2605:bc80:3010:200::/64 (Gentoo)
>> 2605:bc80:3010:400::/64 (Drupal)
>> 2605:bc80:3010:401::/64 (Drupal Virt)
>> 2605:bc80:3010:600::/64 (Fedora)
>> 2605:bc80:3010:700::/64 (OSM)
>> 2605:bc80:3010:a00::/64 (Buildbot/RTEMS)
>> 2605:bc80:3010:b00::/64 (Debian)
>>
>> If I am missing any IPv6 subnet that isn't working, please let me
>> know. Since this impacted a smaller set of users, I decided to move
>> forward keeping us on the new connection while we work out these IPv6
>> issues and not roll back.
>>
>> Thanks again for all your support and patience!
>>
>
>
> --
> Lance Albertson
> Director
> Oregon State University | Open Source Lab
>


-- 
Lance Albertson
Director
Oregon State University | Open Source Lab