You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/11/24 11:40:30 UTC
[GitHub] [cloudstack] rhtyd opened a new issue #5716: Isolated Network VR with Vmxnet3 fails deployment when router.extra.public.nics is not 0
rhtyd opened a new issue #5716:
URL: https://github.com/apache/cloudstack/issues/5716
With 4.16.0.0 release, if a VMware environment uses Vmxnet3 as the router nic controller then the isolated network VR deployment fails when router.extra.public.nics is not 0. However, VPC VR works in either case.
The workaround with 4.16.0.0 is to either use (a) E1000 nic controller, or (b) use Vmxnet3 controller with the router.extra.public.nics set to 0 if the IaaS env has all public IP ranges in the same VLAN. The additional router nics are used when users acquire public IPs from IP ranges with different VLANs, otherwise these additional router nics aren't used in isolated network VR.
As described in an old email by Kelvin below, the global setting and feature were introduced for legacy templates where hot-plugging wasn't supported:
http://mail-archives.apache.org/mod_mbox/cloudstack-users/201304.mbox/%3CCD81EFED.3B701%25kelven.yang@citrix.com%3E
All modern systemvmtemplates had open-vm-tools installed and support hot-plugging which is what is used in the case of VPC VR, but nic-hotpluging isn't implemented for isolated network VR.
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
VR, Isolated Networks
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main branch.
-->
~~~
4.16.0.0
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
VMware 6.7u3, with Vmxnet3 used a nic controller for routers/systemvms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #5716: Isolated Network VR with Vmxnet3 fails deployment when router.extra.public.nics is not 0
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #5716:
URL: https://github.com/apache/cloudstack/issues/5716#issuecomment-1028727536
> @nvazquez @weizhouapache yes, this was identified and has been fixed with #5863. With that PR we no longer enforce the systemVM template to be registered as deploy-as-is (by setting the flag to true in the upgrade path).
thanks @nvazquez @Pearl1594 for investigating and fixing the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on issue #5716: Isolated Network VR with Vmxnet3 fails deployment when router.extra.public.nics is not 0
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on issue #5716:
URL: https://github.com/apache/cloudstack/issues/5716#issuecomment-1025596224
@nvazquez @weizhouapache yes, this was identified and has been fixed with #5863. With that PR we no longer enforce the systemVM template to be registered as deploy-as-is (by setting the flag to true in the upgrade path).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on issue #5716: Isolated Network VR with Vmxnet3 fails deployment when router.extra.public.nics is not 0
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on issue #5716:
URL: https://github.com/apache/cloudstack/issues/5716#issuecomment-1029912335
Fixed by #5912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland closed issue #5716: Isolated Network VR with Vmxnet3 fails deployment when router.extra.public.nics is not 0
Posted by GitBox <gi...@apache.org>.
DaanHoogland closed issue #5716:
URL: https://github.com/apache/cloudstack/issues/5716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #5716: Isolated Network VR with Vmxnet3 fails deployment when router.extra.public.nics is not 0
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #5716:
URL: https://github.com/apache/cloudstack/issues/5716#issuecomment-1024057976
I had some discussion and investigation with @harikrishna-patnala
when router.extra.public.nics is not 0 and vmware.systemvm.nic.device.type is set to "Vmxnet3" , the network VR will have more than 3 nics: 3 (guest, linklocal, source nat) + router.extra.public.nics (for extra public ip ranges). Therefore it will hit the problem which is not fixed by vmware for many years: the nic order inside VR is not same as defined in vmx file. please refer to https://communities.vmware.com/t5/ESXi-Discussions/Problem-with-the-order-of-interfaces-using-vmxnet3/td-p/2088500
I have tested cloudstack 4.14, which seem to be working. in cloudstack VR, there is a udev rule file to rename the network device name
```
root@r-6-VM:~# cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="02:00:6e:89:00:02", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="02:00:52:a4:00:04", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="1e:00:37:00:00:04", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="02:00:3b:02:00:05", NAME="eth3"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="02:00:25:30:00:06", NAME="eth4"
```
and it works fine (please notice the device name)
```
root@r-6-VM:~# ip a |egrep 'BROADCAST|ether'
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 02:00:6e:89:00:02 brd ff:ff:ff:ff:ff:ff
3: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 02:00:25:30:00:06 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 02:00:52:a4:00:04 brd ff:ff:ff:ff:ff:ff
5: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 1e:00:37:00:00:04 brd ff:ff:ff:ff:ff:ff
6: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 02:00:3b:02:00:05 brd ff:ff:ff:ff:ff:ff
```
Unfortunately, it does not work in 4.16 (VR cannot be started). the /etc/udev/rules.d/70-persistent-net.rules still exists, but the device name are not changed, for example eth0, eth1, eth2, eth3, eth4, not the order above.
this might be caused by some code changes (e.g. /opt/cloud/bin/setup/common.sh, /opt/cloud/bin/setup/router.sh), or systemvm template changes (e.g. systemd-udevd is used in latest Debian releases).
@harikrishna-patnala is currently working on removing router.extra.public.nics, which I think is a good idea and should fix the issue above (there will be only 3 nics on network VR: guest, link local and public).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #5716: Isolated Network VR with Vmxnet3 fails deployment when router.extra.public.nics is not 0
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #5716:
URL: https://github.com/apache/cloudstack/issues/5716#issuecomment-1025593404
@weizhouapache I have checked the codebase and it seems the auto-upgrade system VM mechanism sets it to true for VMware (https://github.com/apache/cloudstack/blob/main/engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java#L484, https://github.com/apache/cloudstack/blob/main/engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java#L816 and https://github.com/apache/cloudstack/blob/main/engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java#L838)
@davidjumani @Pearl1594 would it cause any potential regression if the above lines are removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #5716: Isolated Network VR with Vmxnet3 fails deployment when router.extra.public.nics is not 0
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #5716:
URL: https://github.com/apache/cloudstack/issues/5716#issuecomment-1024060474
Another issue we found in our testing is, cloudstack VR always use E1000 nic controller if systemvm template has deploy_as_is =1. see code below
https://github.com/apache/cloudstack/blob/44c08b5acc598972b4f0af576ffdea4e2447cb41/plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java#L2357-L2364
I checked all platforms, the template SystemVM Template (vSphere) (id=8) has deploy_as_is = 1.
but on Hari's testing platform, it is set to 0.
@nvazquez do you have idea why it is set to 1 ? can we change it to 0 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org