You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Shashi Dahal <s....@leaseweb.com> on 2013/04/15 17:10:08 UTC
RE: upgrade from 2.2.14 to 4.0.0: inbound public network not
working -- worked
Hi All,
I got the system to work using the following workaround:
## rules for system VM
iptables -N v-2-VM
iptables -A v-2-VM -j ACCEPT
iptables -A BF-cloudVirBr110-IN -m physdev --physdev-in vnet9 --physdev-is-bridged -j v-2-VM
iptables -A BF-cloudVirBr110-OUT -m physdev --physdev-out vnet9 --physdev-is-bridged -j v-2-VM
## rules for console PROXY
iptables -N s-1-VM
iptables -A s-1-VM -j ACCEPT
iptables -A BF-cloudVirBr110-IN -m physdev --physdev-in vnet2 --physdev-is-bridged -j s-1-VM
iptables -A BF-cloudVirBr110-OUT -m physdev --physdev-out vnet2 --physdev-is-bridged -j s-1-VM
# Change to the VMS that were not working after reboot ( due to default being DROP , despite all tcp/udp/icmp being allowed before upgrade )
#drop the default rule of DROP that was in the VM.
iptables -D i-2-8-VM 1
# set it to accept.
iptables -A i-28-VM -j ACCEPT
Right now, all my basic network connectivity and VMs are working. I have still not rebooted the hypervisor box, in case someone wants to send a debug log or something.
I will perform rest of the tests and update.
Cheers,
Shashi Dahal
-----Original Message-----
From: Shashi Dahal [mailto:s.dahal@leaseweb.com]
Sent: maandag, 15 april, 2013 16:22
To: users@cloudstack.apache.org
Subject: RE: upgrade from 2.2.14 to 4.0.0: inbound public network not working.
Hi All,
While trying to pinpoint why my incoming does not work, I ruled out ebtables rule and routing, routing cache in the hypervisor.
I found that the entry for the console-VM is missing from the iptables rule.
Comparing it with a working console-proxy, here is the iptables -L -nvx output.
V-14-VM is the console-proxy VM
-------------------------
Chain BF-cloudVirBr110 (2 references)
pkts bytes target prot opt in out source destination
117116 20293461 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out eth0.110 --physdev-is-bridged
112742 15003917 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
95079 6614436 BF-cloudVirBr110-OUT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-out --physdev-is-bridged
0 0 BF-cloudVirBr110-IN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-in --physdev-is-bridged
Chain BF-cloudVirBr110-IN (1 references)
pkts bytes target prot opt in out source destination
0 0 i-4-11-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet0 --physdev-is-bridged
0 0 i-5-13-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet1 --physdev-is-bridged
0 0 i-4-10-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet2 --physdev-is-bridged
0 0 v-14-VM all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet3 --physdev-is-bridged
0 0 v-14-VM all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet4 --physdev-is-bridged
0 0 v-14-VM all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet5 --physdev-is-bridged
Chain BF-cloudVirBr110-OUT (1 references)
pkts bytes target prot opt in out source destination
24446 1680960 i-4-11-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet0 --physdev-is-bridged
24386 1665594 i-5-13-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet1 --physdev-is-bridged
24263 1663826 i-4-10-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet2 --physdev-is-bridged
0 0 v-14-VM all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet3 --physdev-is-bridged
0 0 v-14-VM all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet4 --physdev-is-bridged
21886 1592406 v-14-VM all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet5 --physdev-is-bridged
-------------------------
After upgrade, there is no entry for the console-proxy VM in the iptables rule.
No entry for v-2-VM
Chain BF-cloudVirBr110 (2 references)
num pkts bytes target prot opt in out source destination
1 11651 1060945 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out em1.110 --physdev-is-bridged
2 9604 4429936 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
3 42221 3637781 BF-cloudVirBr110-OUT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-out --physdev-is-bridged
4 21372 1926303 BF-cloudVirBr110-IN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-in --physdev-is-bridged
Chain BF-cloudVirBr110-IN (1 references)
num pkts bytes target prot opt in out source destination
1 18 1239 i-2-3-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet6 --physdev-is-bridged
2 15 1011 i-2-6-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet7 --physdev-is-bridged
3 0 0 i-2-7-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet10 --physdev-is-bridged
4 12 783 i-2-8-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet11 --physdev-is-bridged
5 70 5184 i-3-11-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet12 --physdev-is-bridged
6 15 964 i-4-12-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vnet13 --physdev-is-bridged
Chain BF-cloudVirBr110-OUT (1 references)
num pkts bytes target prot opt in out source destination
1 3477 285403 i-2-3-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet6 --physdev-is-bridged
2 3476 285359 i-2-6-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet7 --physdev-is-bridged
3 3478 285495 i-2-7-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet10 --physdev-is-bridged
4 3478 285495 i-2-8-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet11 --physdev-is-bridged
5 3458 283991 i-3-11-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet12 --physdev-is-bridged
6 3482 285735 i-4-12-def all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vnet13 --physdev-is-bridged
To compare my findings, I tried to re-create our 2.2.14 setup in 4.0.1 setup, but I have the following constraints.
We used Advanced Zone ( with VLAN ) and Security Groups in 2.2.14 setup.
In 4.0.1,
1. If I select Basic Zone, which provides Security Groups, I cannot select/use VLAN's 2. If I select Advanced Zone, it does not have Security Groups, which I need to compare with.
Cheers,
Shashi Dahal
-----Original Message-----
From: Shashi Dahal [mailto:s.dahal@leaseweb.com]
Sent: Monday, April 15, 2013 3:09 PM
To: users@cloudstack.apache.org
Subject: upgrade from 2.2.14 to 4.0.0: inbound public network not working.
Hi.
I upgraded from 2.2.14 to 4.0.0 and now, the public network is not working.
2.2.14 setup
Single Management Server (centos 6)
2 Hypervisors (KVM) (centos 6)
Network Type: Advanced
Security Groups Enabled: Yes
Primary and Secondary Storage: NFS
Public Network: VLAN 110
Public and Guest on: cloudbr0
Private: cloudbr1
Upgrade Instructions followed from this URL:
http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.0-incubating/html-single/release-notes/index.html#upgrade-from-2.2.x-to-4.0
On step 20:
"nohup cloud-sysvmadm -d 127.0.0.1 -u cloud -p password -c -r > sysvm.log 2>&1 &"
I had to change this command to : "nohup cloud-sysvmadm -d 127.0.0.1 -u cloud -p password -s -r > sysvm.log 2>&1 &"
In the cloud-systemadm command, there isno -c but an -s doing the same.
" -s - stop then start all running SSVMs and Console Proxies "
My sysvm.log output:
-------------------------
nohup: ignoring input
Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)
Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s).
Stopping and starting 1 running routing vm(s)...
Done restarting router(s).
-------------------------
Different from the Document: My SystemVMs have this version on /etc/cloudstack-release: Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012 While the upgrade document says it should show: Cloudstack Release 4.0.0-incubating Mon Oct 9 15:10:04 PST 2012
Of the 2 hypervisors, all VMS are in a single hypervisor node01, and after the upgrade, this hypervisor has not been rebooted.
The other hypervisor, which does not have any VM, has been rebooted.
After the upgrade:
CloudStack management interface is upgraded to the new ones.
All VMS were running and reachable via their public IP
Network Offering: Guest Network: Name, Account, VLAN and CIDR is blank, Type has shared.
Network Offering: DefaultSharedNetworkOfferingWithSGService
Security Groups: All 2.2.14 settings of various Security Groups are present.
Zone: Network Type: Advanced
Local Storage Enabled: NO
Status of Virtual Router and System VMS - Running I can ssh to all the system VM's using the local link ip address Ping to private IPs of the system VM's work. ( from management server, hypervisors and from each other ) Ping to the PUBLIC IPs of the system VM's do not work from anywhere.
I rebooted 2 VMs.. Their status shows running, but they are not reachable anymore( via public IP) I migrated these 2 VMS that were rebooted to the 2nd hypervisor.
They migrated OK, but they are still unaccessible.
>From the management interface, their status is Running.
Right now , I have the following:
1. A successfully upgraded (no errors reported in any of the upgrade step given in the document) 2.2.14 to 4.0.0 platform
2. 4 running VMs that are reachable via their public IP.
3. 2 unreachable ( but running) VMS that were rebooted after the upgrade process.
4. Running System VMS
5. I can reach the outside world from the system VMs via public network
6. I can reach the system VMS via their private IP address
7. I cannot reach from the internet to the public IP to any of the system VMS.
Infrastructure >> Zone >> Zone01 >> Physical Network I see zone-name as: ZONE01-pNtwk200 , State: Enabled
Public:
Details Page: Does not Load, spinning/spawning circle IP ranges: All blank
Guest:
KVM traffic label: cloudbr0
Network: Type Shared, Scope: Domain(ROOT) , Name|VlanID|CIDR: blank
Management:
KVM traffic label: cloudbr1
IP Ranges: POD01 - has the IPs.
Storage:
All default
IP ranges: All Blank
----
When I ping the VMS, via tcpdump I see the hypervisor gets the ping request, but I do not see it being forwarded, or any arp requests being made.
I did a iptables -Z to clear out all counters and did a ping of 5000 bytes, to check if any rules are there are any iptables rule, but I was not able to find anything.
When I ping for example the public IP of the console proxy, I get the following in tcpdump log in the hypervisor:
10.0.110.230 - public IP of the console proxy
10.0.28.16 - system doing the ping request.
tcpdump -i any host 10.11.110.230
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
11:35:31.918978 P 00:1d:e6:a5:9b:e4 (oui Unknown) ethertype Unknown (0x006e), length 80:
0x0000: 0000 0800 4500 003c 35af 0000 7d01 6911 ....E..<5...}.i.
0x0010: 0a00 1c10 0a0b 6ee6 0800 4d48 0001 0013 ......n...MH....
0x0020: 6162 6364 6566 6768 696a 6b6c 6d6e 6f70 abcdefghijklmnop
0x0030: 7172 7374 7576 7761 6263 6465 6667 6869 qrstuvwabcdefghi
11:35:31.918978 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 19, length 40
11:35:31.919022 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 19, length 40
11:35:31.919029 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 19, length 40
11:35:31.919038 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 19, length 40
11:35:31.919043 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 19, length 40
11:35:36.904310 P 00:1d:e6:a5:9b:e4 (oui Unknown) ethertype Unknown (0x006e), length 80:
0x0000: 0000 0800 4500 003c 35bf 0000 7d01 6901 ....E..<5...}.i.
0x0010: 0a00 1c10 0a0b 6ee6 0800 4d47 0001 0014 ......n...MG....
0x0020: 6162 6364 6566 6768 696a 6b6c 6d6e 6f70 abcdefghijklmnop
0x0030: 7172 7374 7576 7761 6263 6465 6667 6869 qrstuvwabcdefghi
11:35:36.904310 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 20, length 40
11:35:36.904362 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 20, length 40
11:35:36.904371 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 20, length 40
11:35:36.904382 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 20, length 40
11:35:36.904388 IP 10.0.28.16 > 10.11.110.230: ICMP echo request, id 1, seq 20, length 40
Tcpdump from inside the console proxy at the same time does not output anything. The ICMP requests do not reach the console proxy at all.
Same case for all the system VMS where its not working.
Cheers,
Shashi Dahal