You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "S.Fuller" <st...@gmail.com> on 2024/01/24 03:04:05 UTC
Hot Added NICs - PCI slot changes on instance reboot
I am running Cloudstack 4.11 and have encountered an issue with hot added
NICs on Windows guests. I add the NIC to a running guest, configure it, and
it works as expected. After the guest reboots, the NIC is assigned to a
different PCI slot, and then needs to be reconfigured. This has caused
issues when those hot-added network interfaces are used for ISCSI traffic.
I can see the slot reassignment in both Windows and in the XML when I do a
virsh dumpxml on the guest. I've also seen this behavior when hot adding
volumes. That sometimes results in the volumes being in an offline state
when the guest restarts. We can bring them online manually, and all of the
data is there.
I assume someone else has encountered this. If so, have you worked around
it somehow? For now, we're going to add interfaces and volumes only when
the guest is powered off, but that does make things a bit more work.
Thanks.
--
Steve Fuller
stevefuller@gmail.com
Re: Hot Added NICs - PCI slot changes on instance reboot
Posted by "S.Fuller" <st...@gmail.com>.
Wei,
Issue 7457 is very imilar to what I am experiencing. My test servers have
been Windows 2022 with the latest 1.240 build of the virtio drivers, and
Windows 2019 with an older build of the drivers installed. In my case, I am
seeing the slot ID / rediscovery occur when the added interface
is associated with either a L2 or an isolated network. My testing to this
point has only had the slot ID change on the first reboot. The NICs and
their configurations appear to be stable afterwards.
On Wed, Jan 24, 2024 at 3:30 AM Wei ZHOU <us...@gmail.com> wrote:
> Hi,
>
> Are the issues similar as
> https://github.com/apache/cloudstack/issues/7457
> https://github.com/apache/cloudstack/issues/7490
> ?
>
>
> -Wei
>
> On Wed, 24 Jan 2024 at 04:06, S.Fuller <st...@gmail.com> wrote:
>
> > I am running Cloudstack 4.11 and have encountered an issue with hot added
> > NICs on Windows guests. I add the NIC to a running guest, configure it,
> and
> > it works as expected. After the guest reboots, the NIC is assigned to a
> > different PCI slot, and then needs to be reconfigured. This has caused
> > issues when those hot-added network interfaces are used for ISCSI
> traffic.
> >
> > I can see the slot reassignment in both Windows and in the XML when I do
> a
> > virsh dumpxml on the guest. I've also seen this behavior when hot adding
> > volumes. That sometimes results in the volumes being in an offline state
> > when the guest restarts. We can bring them online manually, and all of
> the
> > data is there.
> >
> > I assume someone else has encountered this. If so, have you worked around
> > it somehow? For now, we're going to add interfaces and volumes only when
> > the guest is powered off, but that does make things a bit more work.
> >
> > Thanks.
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com
> >
>
--
Steve Fuller
stevefuller@gmail.com
Re: Hot Added NICs - PCI slot changes on instance reboot
Posted by Wei ZHOU <us...@gmail.com>.
Hi,
Are the issues similar as
https://github.com/apache/cloudstack/issues/7457
https://github.com/apache/cloudstack/issues/7490
?
-Wei
On Wed, 24 Jan 2024 at 04:06, S.Fuller <st...@gmail.com> wrote:
> I am running Cloudstack 4.11 and have encountered an issue with hot added
> NICs on Windows guests. I add the NIC to a running guest, configure it, and
> it works as expected. After the guest reboots, the NIC is assigned to a
> different PCI slot, and then needs to be reconfigured. This has caused
> issues when those hot-added network interfaces are used for ISCSI traffic.
>
> I can see the slot reassignment in both Windows and in the XML when I do a
> virsh dumpxml on the guest. I've also seen this behavior when hot adding
> volumes. That sometimes results in the volumes being in an offline state
> when the guest restarts. We can bring them online manually, and all of the
> data is there.
>
> I assume someone else has encountered this. If so, have you worked around
> it somehow? For now, we're going to add interfaces and volumes only when
> the guest is powered off, but that does make things a bit more work.
>
> Thanks.
>
> --
> Steve Fuller
> stevefuller@gmail.com
>
Re: Hot Added NICs - PCI slot changes on instance reboot
Posted by "S.Fuller" <st...@gmail.com>.
Gary - Yeah. It was set to "Online Shared" on my 2022 template. I ran into
that same issue with the Windows recovery partition and had to play the
same partition games as well.
On Thu, Jan 25, 2024 at 2:11 AM Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> No worries
>
> I think by default it is set to OfflineAll
>
> I see in your reply to Wei that you mention also you use Server 2022 -
> bear in mind Microsoft changed the OS disk partitioning scheme in Windows
> 10/11 and Server 2022 and now puts the recovery partition at the end of the
> disk. On our Windows server 2022 template we have made it so the recovery
> partition is at the beginning of the disk so that you still have the
> ability to expand the C drive in the future if you need to.
>
> Ref:
> https://superuser.com/questions/1453790/how-to-move-the-recovery-partition-on-windows-10
>
> BR
>
> Gary
>
>
> Gary Dixon
> Quadris Cloud Manager
> 0161 537 4980 <0161%20537%204980>
> +44 7989717661 <+44%207989717661>
> Gary.Dixon@quadris.co.uk
> www.quadris.com
> Innovation House, 12‑13 Bredbury Business Park
> Bredbury Park Way, Bredbury, Stockport, SK6 2SN
> -----Original Message-----
> From: S.Fuller <st...@gmail.com>
> Sent: Wednesday, January 24, 2024 6:58 PM
> To: users@cloudstack.apache.org
> Subject: Re: Hot Added NICs - PCI slot changes on instance reboot
>
> Gary,
>
> Thanks for chiming in. I'll check the SAN policy on and see what it is set
> to.
>
> On Wed, Jan 24, 2024 at 2:55 AM Gary Dixon
> <Ga...@quadris.co.uk.invalid>
> wrote:
>
> > Hi Steve
> >
> > I don't know about the hot-add nic issue - but with the volumes on
> > windows guests you need to set the SAN policy to OnlineAll in diskpart
> > - then additional volumes will always be available (we bake this into
> > our windows
> > templates)
> > Ref:
> > https://lear/
> > n.microsoft.com%2Fen-us%2Fwindows-server%2Fadministration%2Fwindows-co
> > mmands%2Fsan&data=05%7C02%7CGary.Dixon%40quadris.co.uk%7Cf3bdc0b8e4464
> > a9662e608dc1d0e73cf%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63841
> > 7195111998648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2lu
> > MzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=095UW0PnIp6Xl
> > vhqdlK8zVV3DsRQLwUqj2DI1im3rH4%3D&reserved=0
> >
> > BR
> >
> > Gary
> >
> >
> > Gary Dixon
> > Quadris Cloud Manager
> > 0161 537 4980 <0161%20537%204980>
> > +44 7989717661 <+44%207989717661>
> > Gary.Dixon@quadris.co.uk
> > http://www.q/
> > uadris.com%2F&data=05%7C02%7CGary.Dixon%40quadris.co.uk%7Cf3bdc0b8e446
> > 4a9662e608dc1d0e73cf%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C6384
> > 17195112010357%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2l
> > uMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aGZJHEFXbBID
> > 7Yx68x3dVN6F9StkB%2F0CP3hRNUbKTs4%3D&reserved=0
> > Innovation House, 12‑13 Bredbury Business Park Bredbury Park Way,
> > Bredbury, Stockport, SK6 2SN -----Original Message-----
> > From: S.Fuller <st...@gmail.com>
> > Sent: Wednesday, January 24, 2024 3:04 AM
> > To: users@cloudstack.apache.org
> > Subject: Hot Added NICs - PCI slot changes on instance reboot
> >
> > I am running Cloudstack 4.11 and have encountered an issue with hot
> > added NICs on Windows guests. I add the NIC to a running guest,
> > configure it, and it works as expected. After the guest reboots, the
> > NIC is assigned to a different PCI slot, and then needs to be
> > reconfigured. This has caused issues when those hot-added network
> interfaces are used for ISCSI traffic.
> >
> > I can see the slot reassignment in both Windows and in the XML when I
> > do a virsh dumpxml on the guest. I've also seen this behavior when hot
> > adding volumes. That sometimes results in the volumes being in an
> > offline state when the guest restarts. We can bring them online
> > manually, and all of the data is there.
> >
> > I assume someone else has encountered this. If so, have you worked
> > around it somehow? For now, we're going to add interfaces and volumes
> > only when the guest is powered off, but that does make things a bit more
> work.
> >
> > Thanks.
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com
> >
>
>
> --
> Steve Fuller
> stevefuller@gmail.com
>
--
Steve Fuller
stevefuller@gmail.com
RE: Hot Added NICs - PCI slot changes on instance reboot
Posted by Gary Dixon <Ga...@quadris.co.uk.INVALID>.
No worries
I think by default it is set to OfflineAll
I see in your reply to Wei that you mention also you use Server 2022 - bear in mind Microsoft changed the OS disk partitioning scheme in Windows 10/11 and Server 2022 and now puts the recovery partition at the end of the disk. On our Windows server 2022 template we have made it so the recovery partition is at the beginning of the disk so that you still have the ability to expand the C drive in the future if you need to.
Ref: https://superuser.com/questions/1453790/how-to-move-the-recovery-partition-on-windows-10
BR
Gary
Gary Dixon
Quadris Cloud Manager
0161 537 4980 +44 7989717661
Gary.Dixon@quadris.co.uk
www.quadris.com
Innovation House, 12-13 Bredbury Business Park
Bredbury Park Way, Bredbury, Stockport, SK6 2SN
-----Original Message-----
From: S.Fuller <st...@gmail.com>
Sent: Wednesday, January 24, 2024 6:58 PM
To: users@cloudstack.apache.org
Subject: Re: Hot Added NICs - PCI slot changes on instance reboot
Gary,
Thanks for chiming in. I'll check the SAN policy on and see what it is set to.
On Wed, Jan 24, 2024 at 2:55 AM Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi Steve
>
> I don't know about the hot-add nic issue - but with the volumes on
> windows guests you need to set the SAN policy to OnlineAll in diskpart
> - then additional volumes will always be available (we bake this into
> our windows
> templates)
> Ref:
> https://lear/
> n.microsoft.com%2Fen-us%2Fwindows-server%2Fadministration%2Fwindows-co
> mmands%2Fsan&data=05%7C02%7CGary.Dixon%40quadris.co.uk%7Cf3bdc0b8e4464
> a9662e608dc1d0e73cf%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C63841
> 7195111998648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2lu
> MzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=095UW0PnIp6Xl
> vhqdlK8zVV3DsRQLwUqj2DI1im3rH4%3D&reserved=0
>
> BR
>
> Gary
>
>
> Gary Dixon
> Quadris Cloud Manager
> 0161 537 4980 <0161%20537%204980>
> +44 7989717661 <+44%207989717661>
> Gary.Dixon@quadris.co.uk
> http://www.q/
> uadris.com%2F&data=05%7C02%7CGary.Dixon%40quadris.co.uk%7Cf3bdc0b8e446
> 4a9662e608dc1d0e73cf%7Cf1d6abf3d3b44894ae16db0fb93a96a2%7C0%7C0%7C6384
> 17195112010357%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2l
> uMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aGZJHEFXbBID
> 7Yx68x3dVN6F9StkB%2F0CP3hRNUbKTs4%3D&reserved=0
> Innovation House, 12‑13 Bredbury Business Park Bredbury Park Way,
> Bredbury, Stockport, SK6 2SN -----Original Message-----
> From: S.Fuller <st...@gmail.com>
> Sent: Wednesday, January 24, 2024 3:04 AM
> To: users@cloudstack.apache.org
> Subject: Hot Added NICs - PCI slot changes on instance reboot
>
> I am running Cloudstack 4.11 and have encountered an issue with hot
> added NICs on Windows guests. I add the NIC to a running guest,
> configure it, and it works as expected. After the guest reboots, the
> NIC is assigned to a different PCI slot, and then needs to be
> reconfigured. This has caused issues when those hot-added network interfaces are used for ISCSI traffic.
>
> I can see the slot reassignment in both Windows and in the XML when I
> do a virsh dumpxml on the guest. I've also seen this behavior when hot
> adding volumes. That sometimes results in the volumes being in an
> offline state when the guest restarts. We can bring them online
> manually, and all of the data is there.
>
> I assume someone else has encountered this. If so, have you worked
> around it somehow? For now, we're going to add interfaces and volumes
> only when the guest is powered off, but that does make things a bit more work.
>
> Thanks.
>
> --
> Steve Fuller
> stevefuller@gmail.com
>
--
Steve Fuller
stevefuller@gmail.com
Re: Hot Added NICs - PCI slot changes on instance reboot
Posted by "S.Fuller" <st...@gmail.com>.
Gary,
Thanks for chiming in. I'll check the SAN policy on and see what it is set
to.
On Wed, Jan 24, 2024 at 2:55 AM Gary Dixon <Ga...@quadris.co.uk.invalid>
wrote:
> Hi Steve
>
> I don't know about the hot-add nic issue - but with the volumes on windows
> guests you need to set the SAN policy to OnlineAll in diskpart - then
> additional volumes will always be available (we bake this into our windows
> templates)
> Ref:
> https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/san
>
> BR
>
> Gary
>
>
> Gary Dixon
> Quadris Cloud Manager
> 0161 537 4980 <0161%20537%204980>
> +44 7989717661 <+44%207989717661>
> Gary.Dixon@quadris.co.uk
> www.quadris.com
> Innovation House, 12‑13 Bredbury Business Park
> Bredbury Park Way, Bredbury, Stockport, SK6 2SN
> -----Original Message-----
> From: S.Fuller <st...@gmail.com>
> Sent: Wednesday, January 24, 2024 3:04 AM
> To: users@cloudstack.apache.org
> Subject: Hot Added NICs - PCI slot changes on instance reboot
>
> I am running Cloudstack 4.11 and have encountered an issue with hot added
> NICs on Windows guests. I add the NIC to a running guest, configure it, and
> it works as expected. After the guest reboots, the NIC is assigned to a
> different PCI slot, and then needs to be reconfigured. This has caused
> issues when those hot-added network interfaces are used for ISCSI traffic.
>
> I can see the slot reassignment in both Windows and in the XML when I do a
> virsh dumpxml on the guest. I've also seen this behavior when hot adding
> volumes. That sometimes results in the volumes being in an offline state
> when the guest restarts. We can bring them online manually, and all of the
> data is there.
>
> I assume someone else has encountered this. If so, have you worked around
> it somehow? For now, we're going to add interfaces and volumes only when
> the guest is powered off, but that does make things a bit more work.
>
> Thanks.
>
> --
> Steve Fuller
> stevefuller@gmail.com
>
--
Steve Fuller
stevefuller@gmail.com
RE: Hot Added NICs - PCI slot changes on instance reboot
Posted by Gary Dixon <Ga...@quadris.co.uk.INVALID>.
Hi Steve
I don't know about the hot-add nic issue - but with the volumes on windows guests you need to set the SAN policy to OnlineAll in diskpart - then additional volumes will always be available (we bake this into our windows templates)
Ref: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/san
BR
Gary
Gary Dixon
Quadris Cloud Manager
0161 537 4980 +44 7989717661
Gary.Dixon@quadris.co.uk
www.quadris.com
Innovation House, 12-13 Bredbury Business Park
Bredbury Park Way, Bredbury, Stockport, SK6 2SN
-----Original Message-----
From: S.Fuller <st...@gmail.com>
Sent: Wednesday, January 24, 2024 3:04 AM
To: users@cloudstack.apache.org
Subject: Hot Added NICs - PCI slot changes on instance reboot
I am running Cloudstack 4.11 and have encountered an issue with hot added NICs on Windows guests. I add the NIC to a running guest, configure it, and it works as expected. After the guest reboots, the NIC is assigned to a different PCI slot, and then needs to be reconfigured. This has caused issues when those hot-added network interfaces are used for ISCSI traffic.
I can see the slot reassignment in both Windows and in the XML when I do a virsh dumpxml on the guest. I've also seen this behavior when hot adding volumes. That sometimes results in the volumes being in an offline state when the guest restarts. We can bring them online manually, and all of the data is there.
I assume someone else has encountered this. If so, have you worked around it somehow? For now, we're going to add interfaces and volumes only when the guest is powered off, but that does make things a bit more work.
Thanks.
--
Steve Fuller
stevefuller@gmail.com