You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by David Jumani <Da...@shapeblue.com> on 2020/06/01 05:08:57 UTC
Re: ACS 4.13.1 failed to create PVLAN network
Hi Jerry, could you trace the DHCP packet coming out of the vm and the router ?
ovs-appctl ofproto/trace vlan_tci=0x0000,dl_src=<vm_mac>
ovs-appctl ofproto/trace vlan_tci=0x0000,dl_src=<vm_mac>,dl_dst=ff:ff:ff:ff:ff:ff
ovs-appctl ofproto/trace dl_vlan=<primary_vlan>,dl_src=<vm_mac>,dl_dst=ff:ff:ff:ff:ff:ff
________________________________
From: li jerry <di...@hotmail.com>
Sent: Saturday, May 30, 2020 1:13 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: ACS 4.13.1 failed to create PVLAN network
hi David, I cleared all the experimental environment and redeployed the test.
pvlan can work normally, and a VM with a PVLAN network + an isolated network can also work normally.
However, I found that when the VR where the PVLAN is located and the VM are not in the same HOST, the VM cannot obtain the IP of the PVLAN.
I checked ovs flows and the flow table was issued.
(VM has Pvlan Network and Isolated Network, Isolated Network can DHCP to IP, Pvlan Network can not get.
(Migrate VM to HOST where PVLAN VR is located, then VM can get all IPs)
Does PVLAN need other switch support?
My switch all port trunk all
-----邮件原件-----
发件人: David Jumani <Da...@shapeblue.com>
发送时间: 2020年5月29日 12:36
收件人: users@cloudstack.apache.org
主题: Re: ACS 4.13.1 failed to create PVLAN network
That's great!
There will be no communication between the devices on a PVLAN if they're isolated except DHCP (since isolated should not be able to communicate with each other).
About multiple NICs, in my setup, I've had a VM attached to a PVLAN as well as an Isolated and it worked. Haven't tried multiple PVLANs though ________________________________
From: li jerry <di...@hotmail.com>
Sent: Thursday, May 28, 2020 2:05 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: ACS 4.13.1 failed to create PVLAN network
Thanks to David, Boris and all friends.
I am on the master branch of clone github.com, and then merge PR [https://github.com/apache/cloudstack/pull/4040].
I have successfully implemented a PVLAN network in my test environment.
Realize no communication between VM and VM; Communication between VM and DHCP (dhcp request)
However, during the test, the following two problems were encountered.
1. VM does not support multiple NICs
When creating vm> 1 NIC, the cloudstack-agent not create ovs flows.
2. L3 PVLAN, Secondary Isolated VLAN Type = Isolated. VM cannot get userdata information
I tested in the VM and found that only DHCP requests can be sent to the VR, and other requests HTTP 80 AND 8080 can not communicate.
Is this a bug? Or am I doing something wrong?
-----邮件原件-----
发件人: David Jumani <Da...@shapeblue.com>
发送时间: 2020年5月27日 19:38
收件人: users@cloudstack.apache.org
主题: Re: ACS 4.13.1 failed to create PVLAN network
It's off master, I've built it using the PR that I shared, and yes, two bridges via ovs in my setup too!
________________________________
From: li jerry <di...@hotmail.com>
Sent: Wednesday, May 27, 2020 3:34 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: 回复: ACS 4.13.1 failed to create PVLAN network
Thanks David, is your environment 4.13.1 or 4.15?
I can test it according to your version.
In 4.13.1 I used openvswitch and created two bridges through ovs-vsctl
-Jerry
发件人: David Jumani<ma...@shapeblue.com>
发送时间: 2020年5月27日 16:28
收件人: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
主题: Re: ACS 4.13.1 failed to create PVLAN network
Hi Jerry,
The way PVLAN works on KVM is that it uses OpenFlow rules to emulate PVLAN, so OVS must be used for Linux networking, not the default Linux bridge. Apart from that, there were certain issues with it which have been addressed in the PR provided by Boris. It's been tested and working on OVS 2.9.2 When creating an L3 PVLAN, the VR is automatically created only when an instance is brought up on that network.
The steps I followed are :
1. Create the PVLAN network
2. Create an instance and attach it to the network
3. Wait until the router comes up
The OpenFlow scripts run in the background on the agent Once the instance is up, it should work as expected
I haven't tried adding a live host to a PVLAN network, but it worked when I tried it following the above steps.
You can test the PVLAN connectivity by pinging another host using the interface attached to the PVLAN network ________________________________
From: Boris Stoyanov <bo...@shapeblue.com>
Sent: Wednesday, May 27, 2020 1:45 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: ACS 4.13.1 failed to create PVLAN network
I can't advise if simply merging this code in 4.13 will work fine, it may cause some side issues since it's a big gap in code bases, therefore I'll advise you to install/upgrade you test env to this PR ( 4.15 )
Bobby.
On 27.05.20, 11:07, "li jerry" <di...@hotmail.com> wrote:
Thank you Boris for the information.
I now go to merge this PR into 4.13.1 for testing. Provide test results later.
-Jerry
发件人: Boris Stoyanov<ma...@shapeblue.com>
发送时间: 2020年5月27日 16:01
收件人: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
主题: Re: ACS 4.13.1 failed to create PVLAN network
Hi Li,
Can you try your tests with this PR: https://github.com/apache/cloudstack/pull/4040
This one ^^ is enabling it on L2 networks, to be honest I'm not sure about L3 as I'm currently testing it.
But to be able to test I'll need to facilitate an L3 network (arping requires IP), and then specify the interface to the other L2 network attached to the same VM.
Please note that this is 4.15+ and it's currently under testing, so do it in a testing lab.
Thanks,
On 26.05.20, 18:14, "li jerry" <di...@hotmail.com> wrote:
Dear All
Who has successfully deployed a PVLAN network on ACS 4.13.1?
I created the PVLAN network through the following process, but it failed
Server:
CentOS7
CloudStack 4.13.1
Zone 1：
l Physical Network1 ：Public and Guest , Isolation method=VLAN, KVM traffic label=br1
l Physical Network2 : Management , Isolation method=VLAN, KVM traffic label=br2
SystemVM Template (KVM): Version 4.11.3
Hypervisor:
CentOS7.7 3.10.0-1062
Openvswitch-2.12.0
libvirt 4.5.0
QEMU 1.5.3
1. Create L3 Guest Network, name= Pvlan-Net-01, vlan=700, isolatedpvlan=1700
( this is api command)
command=createNetwork&zoneId=409b04ea-d128-48ac-8e33-4df700da89cc&networkOfferingId=9778a4ab-0de8-4440-9879-a488416e0572&physicalnetworkid=ca0768c8-f068-4d88-b7bd-2766414a6415&name=Pvlan-Net-01&displayText=Pvlan-Net-01&vlan=700&bypassVlanOverlapCheck=false&isolatedpvlan=1700&acltype=domain&gateway=172.17.0.1&netmask=255.255.0.0&startip=172.17.1.1&endip=172.17.1.254&networkdomain=hyperx.com&response=json&_=1590497900407
2. attache network to VM vm1 (this VM is running and has an isolated network 10.0.0.x / 24)
3. attache failed, management throws the following error:
this is management log
2020-05-26 21:59:40,268 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) Seq 4-6374282322589515787: Received: { Ans: , MgmtId: 113349236140, via: 4(2222), Ver: v1, Flags: 10, { StartAnswer, CheckSshAnswer, GetDomRVersionAnswer, NetworkUsageAnswer, Answer, Answer, Answer, Answer, Answer } }
2020-05-26 21:59:40,280 DEBUG [o.a.c.n.t.AdvancedNetworkTopology] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) SETUP DHCP PVLAN RULES
2020-05-26 21:59:40,292 DEBUG [c.c.n.r.NetworkHelperImpl] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) Router requires upgrade. Unable to send command to router:5, router template version : null, minimal required version : 4.10.0
2020-05-26 21:59:40,294 WARN [o.a.c.n.t.AdvancedNetworkVisitor] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) Timed Out
com.cloud.exception.ResourceUnavailableException: Resource [VirtualRouter:5] is unreachable: Unable to send command. Router requires upgrade
at com.cloud.network.router.NetworkHelperImpl.sendCommandsToRouter(NetworkHelperImpl.java:175)
at org.apache.cloudstack.network.topology.AdvancedNetworkVisitor.visit(AdvancedNetworkVisitor.java:185)
at com.cloud.network.rules.DhcpPvlanRules.accept(DhcpPvlanRules.java:61)
at org.apache.cloudstack.network.topology.AdvancedNetworkTopology.setupDhcpForPvlan(AdvancedNetworkTopology.java:131)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeStart(VirtualNetworkApplianceManagerImpl.java:2080)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1170)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:904)
at com.cloud.network.router.NetworkHelperImpl.start(NetworkHelperImpl.java:277)
at com.cloud.network.router.NetworkHelperImpl.startVirtualRouter(NetworkHelperImpl.java:356)
at com.cloud.network.router.NetworkHelperImpl.startRouters(NetworkHelperImpl.java:341)
at org.cloud.network.router.deployment.RouterDeploymentDefinition.deployVirtualRouter(RouterDeploymentDefinition.java:205)
at com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement.java:278)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1380)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1715)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.createNicForVm(NetworkOrchestrator.java:3767)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateAddVmToNetwork(VirtualMachineManagerImpl.java:3504)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateAddVmToNetwork(VirtualMachineManagerImpl.java:5264)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5326)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:603)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:551)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-05-26 21:59:40,297 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) The guru did not like the answers so stopping VM[DomainRouter|r-5-VM]
2020-05-26 21:59:40,302 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-905dd7d7 job-44/job-45 ctx-4c296832) (logid:2009fc05) Seq 4-6374282322589515789: Sending { Cmd , MgmtId: 113349236140, via: 4(2222), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.188.7","forceStop":false,"volumesToDisconnect":[],"vmName":"r-5-VM","executeInSequence":false,"wait":0}}] }
Agent error
20-05-26 22:53:44,801 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-2:null) (logid:) Execution is successful.
2020-05-26 22:53:44,802 DEBUG [kvm.resource.LibvirtConnection] (UgentTask-2:null) (logid:) Looking for libvirtd connection at: qemu:///system
2020-05-26 22:53:44,807 DEBUG [cloud.agent.Agent] (UgentTask-2:null) (logid:) Sending ping: Seq 4-4: { Cmd , MgmtId: -1, via: 4, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{"r-11-VM":{"state":"PowerOn","host":"2222"},"r-4-VM":{"state":"PowerOn","host":"2222"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":4,"wait":0}}] }
2020-05-26 22:53:44,903 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null) (logid:) Received response: Seq 4-4: { Ans: , MgmtId: 113349236140, via: 4, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":4,"wait":0},"result":true,"wait":0}}] }
2020-05-26 22:53:48,475 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Trying to connect to 169.254.208.61
2020-05-26 22:53:48,477 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.check.CheckSshCommand
2020-05-26 22:53:48,477 DEBUG [resource.wrapper.LibvirtOvsVpcRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-3:null) (logid:03678ec5) Ping command port, 169.254.208.61:3922
2020-05-26 22:53:48,477 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Trying to connect to 169.254.208.61
2020-05-26 22:53:48,477 DEBUG [resource.wrapper.LibvirtOvsVpcRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-3:null) (logid:03678ec5) Ping command port succeeded for vm r-11-VM
2020-05-26 22:53:48,477 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.GetDomRVersionCmd
2020-05-26 22:53:48,480 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh get_template_version.sh 169.254.208.61
2020-05-26 22:53:48,482 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing while with timeout : 1800000
2020-05-26 22:53:48,986 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Execution is successful.
2020-05-26 22:53:48,987 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing script in VR: get_template_version.sh
2020-05-26 22:53:48,988 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.NetworkUsageCommand
2020-05-26 22:53:48,988 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh netusage.sh 169.254.208.61 -c
2020-05-26 22:53:48,989 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing while with timeout : 3600000
2020-05-26 22:53:49,571 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Execution is successful.
2020-05-26 22:53:49,572 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.routing.AggregationControlCommand
2020-05-26 22:53:49,572 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.routing.SetMonitorServiceCommand
2020-05-26 22:53:49,573 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:03678ec5) Processing command: com.cloud.agent.api.routing.AggregationControlCommand
2020-05-26 22:53:49,573 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Transforming com.cloud.agent.api.routing.SetMonitorServiceCommand to ConfigItems
2020-05-26 22:53:49,602 DEBUG [virtualnetwork.facade.AbstractConfigItemFacade] (agentRequest-Handler-3:null) (logid:03678ec5) Transformed filename: monitor_service.json to: monitor_service.json.e2aeaa96-5a74-4753-8edf-6a040717a8aa
2020-05-26 22:53:49,604 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Aggregate action timeout in seconds is 600
2020-05-26 22:53:49,605 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Creating file in VR, with ip: 169.254.208.61, file: VR-1e88cb1c-f82d-4994-9a80-b8c5fc22fd9e.cfg
2020-05-26 22:53:50,236 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vr_cfg.sh 169.254.208.61 -c /var/cache/cloud/VR-1e88cb1c-f82d-4994-9a80-b8c5fc22fd9e.cfg
2020-05-26 22:53:50,239 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing while with timeout : 600600
2020-05-26 22:53:52,488 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Execution is successful.
2020-05-26 22:53:52,488 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:03678ec5) Executing script in VR: vr_cfg.sh
Note：
1. I used the same template to create the Isolated network and attache to the VM, everything works fine.
2. When I created the PVLAN VR, I used the virsh console to enter the VM and executed the get_template_version.sh script, which output cloudstack 4.11.3
I do n’t know what causes the PVLAN network to be created!
Any help would be greatly appreciated!
If you need more detailed information, please let me know
Thank you
-Jerry
boris.stoyanov@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
boris.stoyanov@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue
David.Jumani@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue
David.Jumani@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue
David.Jumani@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue
David.Jumani@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue