You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by citrix-dev <ci...@clogeny.com> on 2012/08/15 20:00:32 UTC

Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Hello,
We have a KVM-based setup with the following configurations:
Private Network : 10.88.88.0/28
Public Network: 108.x.x.x/24

Each KVM host has a statically assigned IP Address to it from the
10.88.88.x pool. 10.88.88.1 is the default gateway which is used to route
traffic to the external world. The VMs that we spawn via Cloudstack come-up
fine on the 10.1.1.x Cloudstack network. But we would like to have the
10.88.88.x network-based IP addresses assigned the VMs as well. i.e bridged
mode.

Is it possible to achieve the same? or will the VMs always come-up on the
10.1.1.x network *only* ?

--
Regards, Kalpak

RE: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by "Kelcey Damage (BBITS)" <ke...@bbits.ca>.
I did have to stop using VRs as gateways because the ARP table was
constantly screwing up.

Kelcey Jamison-Damage
Infrastructure Systems Architect

Backbone Technology | Backbone Datavault | Backbone IT Services
55 East 7th Ave Vancouver - BC Canada V5T 1M4
kelcey@bbits.ca | 604-331-1152 ext. 114


-----Original Message-----
From: Matthew Patton [mailto:mpatton@inforelay.com] 
Sent: Monday, August 20, 2012 5:38 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Given what appears to be a serious (though probably simple) bug in
CentOS/RHEL 6.3 and libvirt 0.9.10-21.el6 I don't know how ANYONE has VLANs
working (bonded or otherwise) in a KVM environment.

I've opened https://bugzilla.redhat.com/show_bug.cgi?id=849766

Anyone here with an actual, working example?

--
Cloud Services Architect, Senior System Administrator InfoRelay Online
Systems (www.inforelay.com)


RE: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by "Kelcey Damage (BBITS)" <ke...@bbits.ca>.
On CentOS 6.3 hosts with 6.3 guests, latest libvirt, I have wiresharked the
lines and tagging is working.

I am not using bonding.

Thanks

Kelcey Jamison-Damage
Infrastructure Systems Architect

Backbone Technology | Backbone Datavault | Backbone IT Services
55 East 7th Ave Vancouver - BC Canada V5T 1M4
kelcey@bbits.ca | 604-331-1152 ext. 114


-----Original Message-----
From: Matthew Patton [mailto:mpatton@inforelay.com] 
Sent: Monday, August 20, 2012 5:38 PM
To: cloudstack-dev@incubator.apache.org
Subject: Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Given what appears to be a serious (though probably simple) bug in
CentOS/RHEL 6.3 and libvirt 0.9.10-21.el6 I don't know how ANYONE has VLANs
working (bonded or otherwise) in a KVM environment.

I've opened https://bugzilla.redhat.com/show_bug.cgi?id=849766

Anyone here with an actual, working example?

--
Cloud Services Architect, Senior System Administrator InfoRelay Online
Systems (www.inforelay.com)


Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by Matthew Patton <mp...@inforelay.com>.
Given what appears to be a serious (though probably simple) bug in  
CentOS/RHEL 6.3 and libvirt 0.9.10-21.el6 I don't know how ANYONE has  
VLANs working (bonded or otherwise) in a KVM environment.

I've opened https://bugzilla.redhat.com/show_bug.cgi?id=849766

Anyone here with an actual, working example?

-- 
Cloud Services Architect, Senior System Administrator
InfoRelay Online Systems (www.inforelay.com)

Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by "Kelceydamage@bbits" <ke...@bbits.ca>.
Just to add, a tagged interface needs no configuration to work.

Here is a centos example.

Create bridge:

DEVICE=Br0
TYPE=Bridge

Ifup Br0

now tag eth0:

vconfig add eth0 1

add tag to bridge:

brctl addif Br0 eth.1

At this point in theory traffic over Br0 should be tagged VLAN 1 and route out eth0. however on centos the next couple steps are needed:

Create tagged eth0.1:

DEVICE=eth0.1
BRIDGE=Br0

ifup eth0.1

Now traffic is routing. I'm not sure how CS works without that last dummy interface config that is needed when manually creating VLAN bridges.

Sent from my iPhone

On Aug 17, 2012, at 5:54 AM, "Matthew Patton" <mp...@inforelay.com> wrote:

> 'cloudVirBr' is hard-coded in /usr/lib[64]/cloud/agent/scripts/vm/network/vnet/modifyvlan.sh in function addVlan().
> 
> What I don't understand is how the script is even working. It does
> 
> vconfig add ethN V
> brctl addbr brV
> brctl addif brV ethN.V
> 
> Furthermore, in scripts/storage/qcow2/modifyvlan.sh (anyone looked at why there are 3 different copies of this shell script?!?) it does the same thing but using bonds.
> 
> But when I run thru the sequence by hand (eth or bond) it doesn't pass traffic. The only way I get it to work is:
> 
> brctl addbr br
> brctl addif br ethN
> vconfig add ethN V
> ifconfig ethN.V <ipaddr> <netmask>
> 
> And I can define as many ethN.[1..9] as I like with all different VLAN IDs and attach them to the single bridge. My way also works when adding a bond to the bridge and adding as many VLAN tagged bonds as I like.
> 
> But the guest isn't receiving response packets to things like ARP. When I sniff bond0 on the hypervisor I see the VLAN tag in the frame AS LONG AS in the guest I configure ethN.V; as would be logical. The ARP reply also gets as far as bond0 on the way back. Using tcpdump on interface 'vnet0' doesn't show any packets going in either direction though they are transiting bond0 just fine. I can see the ARP replies on bond0.N which is again as expected.
> 
> If I can sniff bond0, how come I can't sniff vnet0?
> There is something odd in the 'brctl showmacs <bridge>' output which probably explains why vnet0 sees no packets.
> 
> The guest's MAC is 52:54:00:c9:8d:d1. It's on port #2, marked local=no and eventually ages out. But there is another MAC on port 2 that is the same except it starts with 'fe'. This one is marked local=yes and doesn't age out. It seems this is the MAC given to the host-side of the tap. So I can understand it being semi-permanent. So then why is the the guest's (?private?) MAC address showing up in the bridge? If the ARP response packets have the wrong (guest's private MAC) in them (and they do) that would explain why it can't return to the guest since it "missed" the on-ramp that is the hypervisor's end of the TUN interface.
> 
> 
> Looks like a mistake in the tap code that's forgetting to rewrite the MAC value? Maybe because there is a VLAN tag in the frame?
> 
> 
> At first there were no "QEMU networks" defined since I had undefined the default one since I wasn't interested in NAT.
> [root@kvm1b ~]# virsh net-list
> Name                 State      Autostart
> -----------------------------------------
> 
> But seeing as that might be a problem I defined one for bridging.
> <network>
>  <name>bridge-shared</name>
>  <uuid>0824e08b-bb0e-452a-e2c9-dcca52af2341</uuid>
>  <forward mode='bridge'/>
>  <bridge name='shared' />
> </network>
> 
> 
> Now I have a network
> [root@kvm1b networks]# virsh net-list
> Name                 State      Autostart
> -----------------------------------------
> bridge-shared        active     yes
> 
> 
> This is the interface section from my guest's XML.
>    <interface type='bridge'>
>      <mac address='52:54:00:c9:8d:d1'/>
>      <source bridge='shared'/>
>      <model type='virtio'/>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>    </interface>
> 
> [root@kvm1b ~]# virsh domiflist test
> Interface  Type       Source     Model       MAC
> -------------------------------------------------------
> vnet0      bridge     shared     virtio      52:54:00:c9:8d:d1
> 
> [root@kvm1b networks]# virsh iface-list
> Name                 State      MAC Address
> --------------------------------------------
> bond0.39             active     00:25:90:4c:ba:92
> bond0.398            active     00:25:90:4c:ba:92
> eth2                 active     90:e2:ba:04:f6:28
> eth3                 active     90:e2:ba:04:f6:29
> lo                   active     00:00:00:00:00:00
> shared               active     00:25:90:4c:ba:92
> 
> and for good measure
> 
> [root@kvm1b networks]# virsh iface-dumpxml shared
> <interface type='bridge' name='shared'>
>  <protocol family='ipv6'>
>    <ip address='fe80::225:90ff:fe4c:ba92' prefix='64'/>
>  </protocol>
>  <bridge>
>    <interface type='bond' name='bond0'>
>      <bond>
>        <interface type='ethernet' name='eth0'>
>          <mac address='00:25:90:4c:ba:92'/>
>        </interface>
>        <interface type='ethernet' name='eth1'>
>          <mac address='00:25:90:4c:ba:92'/>
>        </interface>
>      </bond>
>    </interface>
>    <interface type='ethernet' name='vnet0'>
>      <mac address='fe:54:00:c9:8d:d1'/>
>    </interface>
>  </bridge>
> </interface>
> 
> [root@kvm1b networks]# virsh domifstat test vnet0
> vnet0 rx_bytes 44592
> vnet0 rx_packets 847
> vnet0 rx_errs 0
> vnet0 rx_drop 0
> vnet0 tx_bytes 2150
> vnet0 tx_packets 35
> vnet0 tx_errs 0
> vnet0 tx_drop 0
> 
> So you'd think something was getting back to the guest. I just don't know what. tcpdump running inside the guest on eth0 shows nothing at all, just like sniffing vnet0.
> 
> The MAC hitting the wire on the hypervisor's bond0 is still the guest's private MAC and it seems the response packets don't know where to go once they come back to the bridge.
> 
> 
> On a different note I ran across this: http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbptap.htm
> So QEMU has it's own bastard redefinition of the term "VLAN"? WTF!

Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by "Kelceydamage@bbits" <ke...@bbits.ca>.
Read that IBM article you referenced, it made a lot of things clear to me.

Sent from my iPhone

On Aug 17, 2012, at 5:54 AM, "Matthew Patton" <mp...@inforelay.com> wrote:

> 'cloudVirBr' is hard-coded in /usr/lib[64]/cloud/agent/scripts/vm/network/vnet/modifyvlan.sh in function addVlan().
> 
> What I don't understand is how the script is even working. It does
> 
> vconfig add ethN V
> brctl addbr brV
> brctl addif brV ethN.V
> 
> Furthermore, in scripts/storage/qcow2/modifyvlan.sh (anyone looked at why there are 3 different copies of this shell script?!?) it does the same thing but using bonds.
> 
> But when I run thru the sequence by hand (eth or bond) it doesn't pass traffic. The only way I get it to work is:
> 
> brctl addbr br
> brctl addif br ethN
> vconfig add ethN V
> ifconfig ethN.V <ipaddr> <netmask>
> 
> And I can define as many ethN.[1..9] as I like with all different VLAN IDs and attach them to the single bridge. My way also works when adding a bond to the bridge and adding as many VLAN tagged bonds as I like.
> 
> But the guest isn't receiving response packets to things like ARP. When I sniff bond0 on the hypervisor I see the VLAN tag in the frame AS LONG AS in the guest I configure ethN.V; as would be logical. The ARP reply also gets as far as bond0 on the way back. Using tcpdump on interface 'vnet0' doesn't show any packets going in either direction though they are transiting bond0 just fine. I can see the ARP replies on bond0.N which is again as expected.
> 
> If I can sniff bond0, how come I can't sniff vnet0?
> There is something odd in the 'brctl showmacs <bridge>' output which probably explains why vnet0 sees no packets.
> 
> The guest's MAC is 52:54:00:c9:8d:d1. It's on port #2, marked local=no and eventually ages out. But there is another MAC on port 2 that is the same except it starts with 'fe'. This one is marked local=yes and doesn't age out. It seems this is the MAC given to the host-side of the tap. So I can understand it being semi-permanent. So then why is the the guest's (?private?) MAC address showing up in the bridge? If the ARP response packets have the wrong (guest's private MAC) in them (and they do) that would explain why it can't return to the guest since it "missed" the on-ramp that is the hypervisor's end of the TUN interface.
> 
> 
> Looks like a mistake in the tap code that's forgetting to rewrite the MAC value? Maybe because there is a VLAN tag in the frame?
> 
> 
> At first there were no "QEMU networks" defined since I had undefined the default one since I wasn't interested in NAT.
> [root@kvm1b ~]# virsh net-list
> Name                 State      Autostart
> -----------------------------------------
> 
> But seeing as that might be a problem I defined one for bridging.
> <network>
>  <name>bridge-shared</name>
>  <uuid>0824e08b-bb0e-452a-e2c9-dcca52af2341</uuid>
>  <forward mode='bridge'/>
>  <bridge name='shared' />
> </network>
> 
> 
> Now I have a network
> [root@kvm1b networks]# virsh net-list
> Name                 State      Autostart
> -----------------------------------------
> bridge-shared        active     yes
> 
> 
> This is the interface section from my guest's XML.
>    <interface type='bridge'>
>      <mac address='52:54:00:c9:8d:d1'/>
>      <source bridge='shared'/>
>      <model type='virtio'/>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>    </interface>
> 
> [root@kvm1b ~]# virsh domiflist test
> Interface  Type       Source     Model       MAC
> -------------------------------------------------------
> vnet0      bridge     shared     virtio      52:54:00:c9:8d:d1
> 
> [root@kvm1b networks]# virsh iface-list
> Name                 State      MAC Address
> --------------------------------------------
> bond0.39             active     00:25:90:4c:ba:92
> bond0.398            active     00:25:90:4c:ba:92
> eth2                 active     90:e2:ba:04:f6:28
> eth3                 active     90:e2:ba:04:f6:29
> lo                   active     00:00:00:00:00:00
> shared               active     00:25:90:4c:ba:92
> 
> and for good measure
> 
> [root@kvm1b networks]# virsh iface-dumpxml shared
> <interface type='bridge' name='shared'>
>  <protocol family='ipv6'>
>    <ip address='fe80::225:90ff:fe4c:ba92' prefix='64'/>
>  </protocol>
>  <bridge>
>    <interface type='bond' name='bond0'>
>      <bond>
>        <interface type='ethernet' name='eth0'>
>          <mac address='00:25:90:4c:ba:92'/>
>        </interface>
>        <interface type='ethernet' name='eth1'>
>          <mac address='00:25:90:4c:ba:92'/>
>        </interface>
>      </bond>
>    </interface>
>    <interface type='ethernet' name='vnet0'>
>      <mac address='fe:54:00:c9:8d:d1'/>
>    </interface>
>  </bridge>
> </interface>
> 
> [root@kvm1b networks]# virsh domifstat test vnet0
> vnet0 rx_bytes 44592
> vnet0 rx_packets 847
> vnet0 rx_errs 0
> vnet0 rx_drop 0
> vnet0 tx_bytes 2150
> vnet0 tx_packets 35
> vnet0 tx_errs 0
> vnet0 tx_drop 0
> 
> So you'd think something was getting back to the guest. I just don't know what. tcpdump running inside the guest on eth0 shows nothing at all, just like sniffing vnet0.
> 
> The MAC hitting the wire on the hypervisor's bond0 is still the guest's private MAC and it seems the response packets don't know where to go once they come back to the bridge.
> 
> 
> On a different note I ran across this: http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbptap.htm
> So QEMU has it's own bastard redefinition of the term "VLAN"? WTF!

Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by Matthew Patton <mp...@inforelay.com>.
'cloudVirBr' is hard-coded in  
/usr/lib[64]/cloud/agent/scripts/vm/network/vnet/modifyvlan.sh in function  
addVlan().

What I don't understand is how the script is even working. It does

vconfig add ethN V
brctl addbr brV
brctl addif brV ethN.V

Furthermore, in scripts/storage/qcow2/modifyvlan.sh (anyone looked at why  
there are 3 different copies of this shell script?!?) it does the same  
thing but using bonds.

But when I run thru the sequence by hand (eth or bond) it doesn't pass  
traffic. The only way I get it to work is:

brctl addbr br
brctl addif br ethN
vconfig add ethN V
ifconfig ethN.V <ipaddr> <netmask>

And I can define as many ethN.[1..9] as I like with all different VLAN IDs  
and attach them to the single bridge. My way also works when adding a bond  
to the bridge and adding as many VLAN tagged bonds as I like.

But the guest isn't receiving response packets to things like ARP. When I  
sniff bond0 on the hypervisor I see the VLAN tag in the frame AS LONG AS  
in the guest I configure ethN.V; as would be logical. The ARP reply also  
gets as far as bond0 on the way back. Using tcpdump on interface 'vnet0'  
doesn't show any packets going in either direction though they are  
transiting bond0 just fine. I can see the ARP replies on bond0.N which is  
again as expected.

If I can sniff bond0, how come I can't sniff vnet0?
There is something odd in the 'brctl showmacs <bridge>' output which  
probably explains why vnet0 sees no packets.

The guest's MAC is 52:54:00:c9:8d:d1. It's on port #2, marked local=no and  
eventually ages out. But there is another MAC on port 2 that is the same  
except it starts with 'fe'. This one is marked local=yes and doesn't age  
out. It seems this is the MAC given to the host-side of the tap. So I can  
understand it being semi-permanent. So then why is the the guest's  
(?private?) MAC address showing up in the bridge? If the ARP response  
packets have the wrong (guest's private MAC) in them (and they do) that  
would explain why it can't return to the guest since it "missed" the  
on-ramp that is the hypervisor's end of the TUN interface.


Looks like a mistake in the tap code that's forgetting to rewrite the MAC  
value? Maybe because there is a VLAN tag in the frame?


At first there were no "QEMU networks" defined since I had undefined the  
default one since I wasn't interested in NAT.
[root@kvm1b ~]# virsh net-list
Name                 State      Autostart
-----------------------------------------

But seeing as that might be a problem I defined one for bridging.
<network>
   <name>bridge-shared</name>
   <uuid>0824e08b-bb0e-452a-e2c9-dcca52af2341</uuid>
   <forward mode='bridge'/>
   <bridge name='shared' />
</network>


Now I have a network
[root@kvm1b networks]# virsh net-list
Name                 State      Autostart
-----------------------------------------
bridge-shared        active     yes


This is the interface section from my guest's XML.
     <interface type='bridge'>
       <mac address='52:54:00:c9:8d:d1'/>
       <source bridge='shared'/>
       <model type='virtio'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'  
function='0x0'/>
     </interface>

[root@kvm1b ~]# virsh domiflist test
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      bridge     shared     virtio      52:54:00:c9:8d:d1

[root@kvm1b networks]# virsh iface-list
Name                 State      MAC Address
--------------------------------------------
bond0.39             active     00:25:90:4c:ba:92
bond0.398            active     00:25:90:4c:ba:92
eth2                 active     90:e2:ba:04:f6:28
eth3                 active     90:e2:ba:04:f6:29
lo                   active     00:00:00:00:00:00
shared               active     00:25:90:4c:ba:92

and for good measure

[root@kvm1b networks]# virsh iface-dumpxml shared
<interface type='bridge' name='shared'>
   <protocol family='ipv6'>
     <ip address='fe80::225:90ff:fe4c:ba92' prefix='64'/>
   </protocol>
   <bridge>
     <interface type='bond' name='bond0'>
       <bond>
         <interface type='ethernet' name='eth0'>
           <mac address='00:25:90:4c:ba:92'/>
         </interface>
         <interface type='ethernet' name='eth1'>
           <mac address='00:25:90:4c:ba:92'/>
         </interface>
       </bond>
     </interface>
     <interface type='ethernet' name='vnet0'>
       <mac address='fe:54:00:c9:8d:d1'/>
     </interface>
   </bridge>
</interface>

[root@kvm1b networks]# virsh domifstat test vnet0
vnet0 rx_bytes 44592
vnet0 rx_packets 847
vnet0 rx_errs 0
vnet0 rx_drop 0
vnet0 tx_bytes 2150
vnet0 tx_packets 35
vnet0 tx_errs 0
vnet0 tx_drop 0

So you'd think something was getting back to the guest. I just don't know  
what. tcpdump running inside the guest on eth0 shows nothing at all, just  
like sniffing vnet0.

The MAC hitting the wire on the hypervisor's bond0 is still the guest's  
private MAC and it seems the response packets don't know where to go once  
they come back to the bridge.


On a different note I ran across this:  
http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbptap.htm
So QEMU has it's own bastard redefinition of the term "VLAN"? WTF!

Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by "Kelceydamage@bbits" <ke...@bbits.ca>.
Also run "brctl show" from the hypervisor and note if there is an interface called eth{x}.0 attached to the bridge. If so, then check "cat /proc/net/vlan/config" and "cat /proc/net/vlan/eth{x}.0, where x=interface number.

I'm curious how the system is interpreting the bridge.

Sent from my iPhone

On Aug 16, 2012, at 9:25 PM, citrix-dev <ci...@clogeny.com> wrote:

> Hello Kelcey,
> I think i have identified the issue. It seems that Cloudstack is associating the VM's interface with a non-existent bridge cloudVirBr0 instead of the cloudbr0 which it creates. I figured it from the virsh dump xml :
> 
>     <interface type='bridge'>
>       <mac address='06:66:e4:00:00:20'/>
>       <source bridge='cloudVirBr0'/>
>       <target dev='vnet7'/>
>       <model type='virtio'/>
>       <alias name='net0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>     </interface>
> 
> 
> I grep'ped to the cloudstack config as well as database, i cannot seem to find any configuration pointing to cloudVirBr0. 
> 
> It would be great if you can share your thoughts.
> 
> --
> Regards, Kalpak
> 
> On Thu, Aug 16, 2012 at 12:23 AM, citrix-dev <ci...@clogeny.com> wrote:
> Hello Kelcey,
> Thanks for the quick response.
> 
> I create the additional guest network and the another network interface gets added to the VM, although the VM isnt accessible from either of the interfaces. On accessing the VM from the console, i cannot seem ping anyone outside.
> 
> The default gateway in the route table is to the router-instance (r-2-xxx). I seem to be missing something.
> 
> Please let me know.
> 
> --
> Regards, Kalpak
> 
> 
> On Wed, Aug 15, 2012 at 11:42 PM, Kelcey Damage (BBITS) <ke...@bbits.ca> wrote:
> Create a shared network with that IP range, then when you create(instance)
> the VMs add both networks to the instance and it will come up with 2
> interfaces, one on each network.
> 
> Kelcey Jamison-Damage
> Infrastructure Systems Architect
> 
> Backbone Technology | Backbone Datavault | Backbone IT Services
> 55 East 7th Ave Vancouver - BC Canada V5T 1M4
> kelcey@bbits.ca | 604-331-1152 ext. 114
> 
> -----Original Message-----
> From: chirag@clogeny.com [mailto:chirag@clogeny.com] On Behalf Of citrix-dev
> Sent: Wednesday, August 15, 2012 11:01 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)
> 
> Hello,
> We have a KVM-based setup with the following configurations:
> Private Network : 10.88.88.0/28
> Public Network: 108.x.x.x/24
> 
> Each KVM host has a statically assigned IP Address to it from the 10.88.88.x
> pool. 10.88.88.1 is the default gateway which is used to route traffic to
> the external world. The VMs that we spawn via Cloudstack come-up fine on the
> 10.1.1.x Cloudstack network. But we would like to have the 10.88.88.x
> network-based IP addresses assigned the VMs as well. i.e bridged mode.
> 
> Is it possible to achieve the same? or will the VMs always come-up on the
> 10.1.1.x network *only* ?
> 
> --
> Regards, Kalpak
> 
> 
> 

Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by "Kelceydamage@bbits" <ke...@bbits.ca>.
I've seen cloudVirBr0 before in build tests. I know all network bridges are created from one of the shell scripts that is called by one of the networking gurus.

When you specify a VLAN it's creates a bridge called "cloudVirBr" + "vlanID". Since yours is tagged VLAN 0, my guess is it tried to assign a VLAN in a zone using untagged networking, and defaulted to 0.

Could also be some kind of system bridge used in basic networking.

I would need to know more to formulate a stronger hypothesis.

Sent from my iPhone

On Aug 16, 2012, at 9:25 PM, citrix-dev <ci...@clogeny.com> wrote:

> Hello Kelcey,
> I think i have identified the issue. It seems that Cloudstack is associating the VM's interface with a non-existent bridge cloudVirBr0 instead of the cloudbr0 which it creates. I figured it from the virsh dump xml :
> 
>     <interface type='bridge'>
>       <mac address='06:66:e4:00:00:20'/>
>       <source bridge='cloudVirBr0'/>
>       <target dev='vnet7'/>
>       <model type='virtio'/>
>       <alias name='net0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>     </interface>
> 
> 
> I grep'ped to the cloudstack config as well as database, i cannot seem to find any configuration pointing to cloudVirBr0. 
> 
> It would be great if you can share your thoughts.
> 
> --
> Regards, Kalpak
> 
> On Thu, Aug 16, 2012 at 12:23 AM, citrix-dev <ci...@clogeny.com> wrote:
> Hello Kelcey,
> Thanks for the quick response.
> 
> I create the additional guest network and the another network interface gets added to the VM, although the VM isnt accessible from either of the interfaces. On accessing the VM from the console, i cannot seem ping anyone outside.
> 
> The default gateway in the route table is to the router-instance (r-2-xxx). I seem to be missing something.
> 
> Please let me know.
> 
> --
> Regards, Kalpak
> 
> 
> On Wed, Aug 15, 2012 at 11:42 PM, Kelcey Damage (BBITS) <ke...@bbits.ca> wrote:
> Create a shared network with that IP range, then when you create(instance)
> the VMs add both networks to the instance and it will come up with 2
> interfaces, one on each network.
> 
> Kelcey Jamison-Damage
> Infrastructure Systems Architect
> 
> Backbone Technology | Backbone Datavault | Backbone IT Services
> 55 East 7th Ave Vancouver - BC Canada V5T 1M4
> kelcey@bbits.ca | 604-331-1152 ext. 114
> 
> -----Original Message-----
> From: chirag@clogeny.com [mailto:chirag@clogeny.com] On Behalf Of citrix-dev
> Sent: Wednesday, August 15, 2012 11:01 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)
> 
> Hello,
> We have a KVM-based setup with the following configurations:
> Private Network : 10.88.88.0/28
> Public Network: 108.x.x.x/24
> 
> Each KVM host has a statically assigned IP Address to it from the 10.88.88.x
> pool. 10.88.88.1 is the default gateway which is used to route traffic to
> the external world. The VMs that we spawn via Cloudstack come-up fine on the
> 10.1.1.x Cloudstack network. But we would like to have the 10.88.88.x
> network-based IP addresses assigned the VMs as well. i.e bridged mode.
> 
> Is it possible to achieve the same? or will the VMs always come-up on the
> 10.1.1.x network *only* ?
> 
> --
> Regards, Kalpak
> 
> 
> 

Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by citrix-dev <ci...@clogeny.com>.
After following the documentation here:
http://wiki.cloudstack.org/display/~kdamage/CloudStack+3+and+advanced+networking
http://confluence.cloudstack.org/display/~kdamage/Anatomy+of+Basic+Virtual+Networking

It seems that i would need to create the cloudVirBr<VLAN-no> bridge and
associate that with a physical interface manually for the shared network.

I will give it a shot and share my findings.


On Fri, Aug 17, 2012 at 9:55 AM, citrix-dev <ci...@clogeny.com> wrote:

> Hello Kelcey,
> I think i have identified the issue. It seems that Cloudstack is
> associating the VM's interface with a non-existent bridge cloudVirBr0
> instead of the cloudbr0 which it creates. I figured it from the virsh dump
> xml :
>
>     <interface type='bridge'>
>       <mac address='06:66:e4:00:00:20'/>
>       <source bridge='*cloudVirBr0*'/>
>       <target dev='vnet7'/>
>       <model type='virtio'/>
>       <alias name='net0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
>
>
> I grep'ped to the cloudstack config as well as database, i cannot seem to
> find any configuration pointing to *cloudVirBr0*.
>
> It would be great if you can share your thoughts.
>
> --
> Regards, Kalpak
>
> On Thu, Aug 16, 2012 at 12:23 AM, citrix-dev <ci...@clogeny.com>wrote:
>
>> Hello Kelcey,
>> Thanks for the quick response.
>>
>> I create the additional guest network and the another network interface
>> gets added to the VM, although the VM isnt accessible from either of the
>> interfaces. On accessing the VM from the console, i cannot seem ping anyone
>> outside.
>>
>> The default gateway in the route table is to the router-instance
>> (r-2-xxx). I seem to be missing something.
>>
>> Please let me know.
>>
>> --
>> Regards, Kalpak
>>
>>
>> On Wed, Aug 15, 2012 at 11:42 PM, Kelcey Damage (BBITS) <ke...@bbits.ca>wrote:
>>
>>> Create a shared network with that IP range, then when you
>>> create(instance)
>>> the VMs add both networks to the instance and it will come up with 2
>>> interfaces, one on each network.
>>>
>>> Kelcey Jamison-Damage
>>> Infrastructure Systems Architect
>>>
>>> Backbone Technology | Backbone Datavault | Backbone IT Services
>>> 55 East 7th Ave Vancouver - BC Canada V5T 1M4
>>> kelcey@bbits.ca | 604-331-1152 ext. 114
>>>
>>> -----Original Message-----
>>> From: chirag@clogeny.com [mailto:chirag@clogeny.com] On Behalf Of
>>> citrix-dev
>>> Sent: Wednesday, August 15, 2012 11:01 AM
>>> To: cloudstack-dev@incubator.apache.org
>>> Subject: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)
>>>
>>> Hello,
>>> We have a KVM-based setup with the following configurations:
>>> Private Network : 10.88.88.0/28
>>> Public Network: 108.x.x.x/24
>>>
>>> Each KVM host has a statically assigned IP Address to it from the
>>> 10.88.88.x
>>> pool. 10.88.88.1 is the default gateway which is used to route traffic to
>>> the external world. The VMs that we spawn via Cloudstack come-up fine on
>>> the
>>> 10.1.1.x Cloudstack network. But we would like to have the 10.88.88.x
>>> network-based IP addresses assigned the VMs as well. i.e bridged mode.
>>>
>>> Is it possible to achieve the same? or will the VMs always come-up on the
>>> 10.1.1.x network *only* ?
>>>
>>> --
>>> Regards, Kalpak
>>>
>>>
>>
>

Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by citrix-dev <ci...@clogeny.com>.
Hello Kelcey,
I think i have identified the issue. It seems that Cloudstack is
associating the VM's interface with a non-existent bridge cloudVirBr0
instead of the cloudbr0 which it creates. I figured it from the virsh dump
xml :

    <interface type='bridge'>
      <mac address='06:66:e4:00:00:20'/>
      <source bridge='*cloudVirBr0*'/>
      <target dev='vnet7'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
    </interface>


I grep'ped to the cloudstack config as well as database, i cannot seem to
find any configuration pointing to *cloudVirBr0*.

It would be great if you can share your thoughts.

--
Regards, Kalpak

On Thu, Aug 16, 2012 at 12:23 AM, citrix-dev <ci...@clogeny.com> wrote:

> Hello Kelcey,
> Thanks for the quick response.
>
> I create the additional guest network and the another network interface
> gets added to the VM, although the VM isnt accessible from either of the
> interfaces. On accessing the VM from the console, i cannot seem ping anyone
> outside.
>
> The default gateway in the route table is to the router-instance
> (r-2-xxx). I seem to be missing something.
>
> Please let me know.
>
> --
> Regards, Kalpak
>
>
> On Wed, Aug 15, 2012 at 11:42 PM, Kelcey Damage (BBITS) <ke...@bbits.ca>wrote:
>
>> Create a shared network with that IP range, then when you create(instance)
>> the VMs add both networks to the instance and it will come up with 2
>> interfaces, one on each network.
>>
>> Kelcey Jamison-Damage
>> Infrastructure Systems Architect
>>
>> Backbone Technology | Backbone Datavault | Backbone IT Services
>> 55 East 7th Ave Vancouver - BC Canada V5T 1M4
>> kelcey@bbits.ca | 604-331-1152 ext. 114
>>
>> -----Original Message-----
>> From: chirag@clogeny.com [mailto:chirag@clogeny.com] On Behalf Of
>> citrix-dev
>> Sent: Wednesday, August 15, 2012 11:01 AM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)
>>
>> Hello,
>> We have a KVM-based setup with the following configurations:
>> Private Network : 10.88.88.0/28
>> Public Network: 108.x.x.x/24
>>
>> Each KVM host has a statically assigned IP Address to it from the
>> 10.88.88.x
>> pool. 10.88.88.1 is the default gateway which is used to route traffic to
>> the external world. The VMs that we spawn via Cloudstack come-up fine on
>> the
>> 10.1.1.x Cloudstack network. But we would like to have the 10.88.88.x
>> network-based IP addresses assigned the VMs as well. i.e bridged mode.
>>
>> Is it possible to achieve the same? or will the VMs always come-up on the
>> 10.1.1.x network *only* ?
>>
>> --
>> Regards, Kalpak
>>
>>
>

Re: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by citrix-dev <ci...@clogeny.com>.
Hello Kelcey,
Thanks for the quick response.

I create the additional guest network and the another network interface
gets added to the VM, although the VM isnt accessible from either of the
interfaces. On accessing the VM from the console, i cannot seem ping anyone
outside.

The default gateway in the route table is to the router-instance (r-2-xxx).
I seem to be missing something.

Please let me know.

--
Regards, Kalpak


On Wed, Aug 15, 2012 at 11:42 PM, Kelcey Damage (BBITS) <ke...@bbits.ca>wrote:

> Create a shared network with that IP range, then when you create(instance)
> the VMs add both networks to the instance and it will come up with 2
> interfaces, one on each network.
>
> Kelcey Jamison-Damage
> Infrastructure Systems Architect
>
> Backbone Technology | Backbone Datavault | Backbone IT Services
> 55 East 7th Ave Vancouver - BC Canada V5T 1M4
> kelcey@bbits.ca | 604-331-1152 ext. 114
>
> -----Original Message-----
> From: chirag@clogeny.com [mailto:chirag@clogeny.com] On Behalf Of
> citrix-dev
> Sent: Wednesday, August 15, 2012 11:01 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)
>
> Hello,
> We have a KVM-based setup with the following configurations:
> Private Network : 10.88.88.0/28
> Public Network: 108.x.x.x/24
>
> Each KVM host has a statically assigned IP Address to it from the
> 10.88.88.x
> pool. 10.88.88.1 is the default gateway which is used to route traffic to
> the external world. The VMs that we spawn via Cloudstack come-up fine on
> the
> 10.1.1.x Cloudstack network. But we would like to have the 10.88.88.x
> network-based IP addresses assigned the VMs as well. i.e bridged mode.
>
> Is it possible to achieve the same? or will the VMs always come-up on the
> 10.1.1.x network *only* ?
>
> --
> Regards, Kalpak
>
>

RE: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Posted by "Kelcey Damage (BBITS)" <ke...@bbits.ca>.
Create a shared network with that IP range, then when you create(instance)
the VMs add both networks to the instance and it will come up with 2
interfaces, one on each network.

Kelcey Jamison-Damage
Infrastructure Systems Architect

Backbone Technology | Backbone Datavault | Backbone IT Services
55 East 7th Ave Vancouver - BC Canada V5T 1M4
kelcey@bbits.ca | 604-331-1152 ext. 114

-----Original Message-----
From: chirag@clogeny.com [mailto:chirag@clogeny.com] On Behalf Of citrix-dev
Sent: Wednesday, August 15, 2012 11:01 AM
To: cloudstack-dev@incubator.apache.org
Subject: Spawning VMs in bridged mode on KVM ( Cloudstack 3.0.2)

Hello,
We have a KVM-based setup with the following configurations:
Private Network : 10.88.88.0/28
Public Network: 108.x.x.x/24

Each KVM host has a statically assigned IP Address to it from the 10.88.88.x
pool. 10.88.88.1 is the default gateway which is used to route traffic to
the external world. The VMs that we spawn via Cloudstack come-up fine on the
10.1.1.x Cloudstack network. But we would like to have the 10.88.88.x
network-based IP addresses assigned the VMs as well. i.e bridged mode.

Is it possible to achieve the same? or will the VMs always come-up on the
10.1.1.x network *only* ?

--
Regards, Kalpak


Re: [Events] Free Passes to LInuxCon/CloudOpen

Posted by Duncan Johnston Watt <du...@cloudsoftcorp.com>.
Done. I have two more if anyone is interested.

On 21 August 2012 01:06, Outback Dingo <ou...@gmail.com> wrote:

> i could use two... ill be in san diego for the dates
>
> On Mon, Aug 20, 2012 at 7:05 PM, Duncan Johnston Watt
> <du...@cloudsoftcorp.com> wrote:
> > Mark
> >
> > Cloudsoft also has a few free passes (first come first served) to
> CloudOpen.
> >
> > Best
> >
> > Duncan
> >
> > On 15 August 2012 21:26, Hinkle Mark R. <mr...@cloudstack.org> wrote:
> >
> >> I think I'll have some free passes to the Linux Foundation's LinuxCon
> >> North America and the Cloud Open Conference this month on August 29th
> >> through August 31st.
> >>
> >> CloudOpen - http://events.linuxfoundation.org/events/cloudopen/
> >>
> >> LinuxCon North America -
> http://events.linuxfoundation.org/events/linuxcon
> >>
> >> If you are interested in them please send me a note with subject
> "LinuxCon
> >> passes" your preferred email address, organization and title for their
> >> registration and I'll get back to you by the end of the week. If you are
> >> planning on attending the event and want to help out with the booth let
> me
> >> know as well. We'll have an Apache CloudStack booth on the 29th and
> 30th.
> >> If you want to help out and tell people about Apache CloudStack at the
> >> expo portion of the event let me know and we'll put you to work.
> >>
> >> Regards,
> >>
> >> Mark R. Hinkle
> >> Senior Director, Cloud Computing Community
> >> Citrix Systems
> >> p: 919.228.8049
> >> m: 919.522.3520
> >> e: mark.hinkle@citrix.com
> >> twitter: @mrhinkle
> >>
> >>
> >> >
> >>
> >>
> >>
> >
> >
> > --
> > Duncan Johnston-Watt
> > CEO | Cloudsoft Corporation
> >
> > Twitter | @duncanjw
> > Mobile | +44 777 190 2653
> > Skype | duncan_johnstonwatt
> > Linkedin | www.linkedin.com/in/duncanjohnstonwatt
> >
> > Cloudsoft Corporation Limited, Registered in Scotland No: SC349230.
> >  Registered Office: 13 Dryden Place, Edinburgh, EH9 1RP
> >
> > This e-mail message is confidential and for use by the addressee only. If
> > the message is received by anyone other than the addressee, please return
> > the message to the sender by replying to it and then delete the message
> > from your computer. Internet e-mails are not necessarily secure.
> Cloudsoft
> > Corporation Limited does not accept responsibility for changes made to
> this
> > message after it was sent.
> >
> > Whilst all reasonable care has been taken to avoid the transmission of
> > viruses, it is the responsibility of the recipient to ensure that the
> > onward transmission, opening or use of this message and any attachments
> > will not adversely affect its systems or data. No responsibility is
> > accepted by Cloudsoft Corporation Limited in this regard and the
> recipient
> > should carry out such virus and other checks as it considers appropriate.
>



-- 
Duncan Johnston-Watt
CEO | Cloudsoft Corporation

Twitter | @duncanjw
Mobile | +44 777 190 2653
Skype | duncan_johnstonwatt
Linkedin | www.linkedin.com/in/duncanjohnstonwatt

Cloudsoft Corporation Limited, Registered in Scotland No: SC349230.
 Registered Office: 13 Dryden Place, Edinburgh, EH9 1RP

This e-mail message is confidential and for use by the addressee only. If
the message is received by anyone other than the addressee, please return
the message to the sender by replying to it and then delete the message
from your computer. Internet e-mails are not necessarily secure. Cloudsoft
Corporation Limited does not accept responsibility for changes made to this
message after it was sent.

Whilst all reasonable care has been taken to avoid the transmission of
viruses, it is the responsibility of the recipient to ensure that the
onward transmission, opening or use of this message and any attachments
will not adversely affect its systems or data. No responsibility is
accepted by Cloudsoft Corporation Limited in this regard and the recipient
should carry out such virus and other checks as it considers appropriate.

Re: [Events] Free Passes to LInuxCon/CloudOpen

Posted by Outback Dingo <ou...@gmail.com>.
i could use two... ill be in san diego for the dates

On Mon, Aug 20, 2012 at 7:05 PM, Duncan Johnston Watt
<du...@cloudsoftcorp.com> wrote:
> Mark
>
> Cloudsoft also has a few free passes (first come first served) to CloudOpen.
>
> Best
>
> Duncan
>
> On 15 August 2012 21:26, Hinkle Mark R. <mr...@cloudstack.org> wrote:
>
>> I think I'll have some free passes to the Linux Foundation's LinuxCon
>> North America and the Cloud Open Conference this month on August 29th
>> through August 31st.
>>
>> CloudOpen - http://events.linuxfoundation.org/events/cloudopen/
>>
>> LinuxCon North America - http://events.linuxfoundation.org/events/linuxcon
>>
>> If you are interested in them please send me a note with subject "LinuxCon
>> passes" your preferred email address, organization and title for their
>> registration and I'll get back to you by the end of the week. If you are
>> planning on attending the event and want to help out with the booth let me
>> know as well. We'll have an Apache CloudStack booth on the 29th and 30th.
>> If you want to help out and tell people about Apache CloudStack at the
>> expo portion of the event let me know and we'll put you to work.
>>
>> Regards,
>>
>> Mark R. Hinkle
>> Senior Director, Cloud Computing Community
>> Citrix Systems
>> p: 919.228.8049
>> m: 919.522.3520
>> e: mark.hinkle@citrix.com
>> twitter: @mrhinkle
>>
>>
>> >
>>
>>
>>
>
>
> --
> Duncan Johnston-Watt
> CEO | Cloudsoft Corporation
>
> Twitter | @duncanjw
> Mobile | +44 777 190 2653
> Skype | duncan_johnstonwatt
> Linkedin | www.linkedin.com/in/duncanjohnstonwatt
>
> Cloudsoft Corporation Limited, Registered in Scotland No: SC349230.
>  Registered Office: 13 Dryden Place, Edinburgh, EH9 1RP
>
> This e-mail message is confidential and for use by the addressee only. If
> the message is received by anyone other than the addressee, please return
> the message to the sender by replying to it and then delete the message
> from your computer. Internet e-mails are not necessarily secure. Cloudsoft
> Corporation Limited does not accept responsibility for changes made to this
> message after it was sent.
>
> Whilst all reasonable care has been taken to avoid the transmission of
> viruses, it is the responsibility of the recipient to ensure that the
> onward transmission, opening or use of this message and any attachments
> will not adversely affect its systems or data. No responsibility is
> accepted by Cloudsoft Corporation Limited in this regard and the recipient
> should carry out such virus and other checks as it considers appropriate.

Re: [Events] Free Passes to LInuxCon/CloudOpen

Posted by Duncan Johnston Watt <du...@cloudsoftcorp.com>.
Mark

Cloudsoft also has a few free passes (first come first served) to CloudOpen.

Best

Duncan

On 15 August 2012 21:26, Hinkle Mark R. <mr...@cloudstack.org> wrote:

> I think I'll have some free passes to the Linux Foundation's LinuxCon
> North America and the Cloud Open Conference this month on August 29th
> through August 31st.
>
> CloudOpen - http://events.linuxfoundation.org/events/cloudopen/
>
> LinuxCon North America - http://events.linuxfoundation.org/events/linuxcon
>
> If you are interested in them please send me a note with subject "LinuxCon
> passes" your preferred email address, organization and title for their
> registration and I'll get back to you by the end of the week. If you are
> planning on attending the event and want to help out with the booth let me
> know as well. We'll have an Apache CloudStack booth on the 29th and 30th.
> If you want to help out and tell people about Apache CloudStack at the
> expo portion of the event let me know and we'll put you to work.
>
> Regards,
>
> Mark R. Hinkle
> Senior Director, Cloud Computing Community
> Citrix Systems
> p: 919.228.8049
> m: 919.522.3520
> e: mark.hinkle@citrix.com
> twitter: @mrhinkle
>
>
> >
>
>
>


-- 
Duncan Johnston-Watt
CEO | Cloudsoft Corporation

Twitter | @duncanjw
Mobile | +44 777 190 2653
Skype | duncan_johnstonwatt
Linkedin | www.linkedin.com/in/duncanjohnstonwatt

Cloudsoft Corporation Limited, Registered in Scotland No: SC349230.
 Registered Office: 13 Dryden Place, Edinburgh, EH9 1RP

This e-mail message is confidential and for use by the addressee only. If
the message is received by anyone other than the addressee, please return
the message to the sender by replying to it and then delete the message
from your computer. Internet e-mails are not necessarily secure. Cloudsoft
Corporation Limited does not accept responsibility for changes made to this
message after it was sent.

Whilst all reasonable care has been taken to avoid the transmission of
viruses, it is the responsibility of the recipient to ensure that the
onward transmission, opening or use of this message and any attachments
will not adversely affect its systems or data. No responsibility is
accepted by Cloudsoft Corporation Limited in this regard and the recipient
should carry out such virus and other checks as it considers appropriate.

[Events] Free Passes to LInuxCon/CloudOpen

Posted by "Hinkle Mark R." <mr...@cloudstack.org>.
I think I'll have some free passes to the Linux Foundation's LinuxCon
North America and the Cloud Open Conference this month on August 29th
through August 31st.

CloudOpen - http://events.linuxfoundation.org/events/cloudopen/

LinuxCon North America - http://events.linuxfoundation.org/events/linuxcon

If you are interested in them please send me a note with subject "LinuxCon
passes" your preferred email address, organization and title for their
registration and I'll get back to you by the end of the week. If you are
planning on attending the event and want to help out with the booth let me
know as well. We'll have an Apache CloudStack booth on the 29th and 30th.
If you want to help out and tell people about Apache CloudStack at the
expo portion of the event let me know and we'll put you to work.

Regards, 

Mark R. Hinkle
Senior Director, Cloud Computing Community
Citrix Systems
p: 919.228.8049
m: 919.522.3520
e: mark.hinkle@citrix.com
twitter: @mrhinkle


>