You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by "Vogel, Sven" <Sv...@kupper-computer.com> on 2017/10/31 14:23:36 UTC
4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip on
eth2?
Hi Devs,
i have the following problem.
When I look this jira ticket I see no solution. https://issues.apache.org/jira/browse/CLOUDSTACK-6801 but I think the problem is not solved correctly.
1. KVM
2. Bridges
bond with two interfaces and trunk (0,129,180,100-1500) to cloudbr0
Cloudbr0 (0 - guest network)
Fakebridge pub129 (public network)
Fakebridge sto180 (secondary storage network)
Fakebridge mgmt0 (management)
If I have a vpc all things work until I add a secondary ip and add a static nat.
The following will happen, first address will be on the the correct interface 146.0.122.134/26 but static nat will be on the false network. Its on the eth2…
root@r-29-VM:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 0e:00:a9:fe:03:81 brd ff:ff:ff:ff:ff:ff
inet 169.254.3.129/16 brd 169.254.255.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 1e:00:2c:00:00:68 brd ff:ff:ff:ff:ff:ff
inet 146.0.122.134/26 brd 146.0.122.191 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:00:57:07:00:0c brd ff:ff:ff:ff:ff:ff
inet 192.168.1.254/24 brd 192.168.1.255 scope global eth2
inet 146.0.122.135/26 brd 146.0.122.191 scope global eth2
Normally I think the secondary ip should be on signed to eth1 not eth2! It sets my ip on the guest network vlan range on my cloudbr0 but it should be pub129. vnet6 has 1353 guest tag and not the public tag.
[root@kvm01 ~]# ovs-vsctl list-br
cloud0
cloudbr0
mgmt0
pub129
sto180
[root@kvm01 ~]# virsh domiflist r-29-VM
Interface Type Source Model MAC
-------------------------------------------------------
vnet4 bridge cloud0 virtio 0e:00:a9:fe:03:81
vnet5 bridge pub129 virtio 1e:00:2c:00:00:68
vnet6 bridge cloudbr0 virtio 02:00:57:07:00:0c
Bridge "cloud0"
Port "vnet4"
Interface "vnet4"
Port "vnet5"
tag: 129
Interface "vnet5"
Port "vnet6"
tag: 1353
Interface "vnet6"
Whats wrong or what can I do to fix this? I hope anybody has an idea.
greetings
Sven Vogel
Head of Cloud Solutions / Consultants
Kupper Computer GmbH
Prager Str. 15
04103 Leipzig
Re: 4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip
on eth2?
Posted by "Vogel, Sven" <Sv...@kupper-computer.com>.
Hi Frank,
root@r-29-VM:~# cat /etc/cloudstack/ips.json
{
"eth0": [
{
"add": true,
"broadcast": "169.254.255.255",
"cidr": "169.254.3.129/16",
"device": "eth0",
"gateway": "None",
"netmask": "255.255.0.0",
"network": "169.254.0.0/16",
"nic_dev_id": "0",
"nw_type": "control",
"one_to_one_nat": false,
"public_ip": "169.254.3.129",
"size": "16",
"source_nat": false
}
],
"eth1": [
{
"add": true,
"broadcast": "146.0.122.191",
"cidr": "146.0.122.134/26",
"device": "eth1",
"first_i_p": true,
"gateway": "146.0.122.130",
"netmask": "255.255.255.192",
"network": "146.0.122.128/26",
"new_nic": false,
"nic_dev_id": 1,
"nw_type": "public",
"one_to_one_nat": false,
"public_ip": "146.0.122.134",
"size": "26",
"source_nat": true,
"vif_mac_address": "1e:00:2c:00:00:68"
}
],
"eth2": [
{
"add": true,
"broadcast": "146.0.122.191",
"cidr": "146.0.122.135/26",
"device": "eth2",
"first_i_p": true,
"gateway": "146.0.122.130",
"netmask": "255.255.255.192",
"network": "146.0.122.128/26",
"new_nic": false,
"nic_dev_id": 2,
"nw_type": "public",
"one_to_one_nat": true,
"public_ip": "146.0.122.135",
"size": "26",
"source_nat": true,
"vif_mac_address": "1e:00:2c:00:00:68"
},
{
"add": false,
"broadcast": "146.0.122.191",
"cidr": "146.0.122.136/26",
"device": "eth2",
"first_i_p": true,
"gateway": "146.0.122.130",
"netmask": "255.255.255.192",
"network": "146.0.122.128/26",
"new_nic": false,
"nic_dev_id": 2,
"nw_type": "public",
"one_to_one_nat": true,
"public_ip": "146.0.122.136",
"size": "26",
"source_nat": true,
"vif_mac_address": "1e:00:2c:00:00:68"
},
{
"add": true,
"broadcast": "192.168.1.255",
"cidr": "192.168.1.254/24",
"device": "eth2",
"gateway": "192.168.1.254",
"netmask": "255.255.255.0",
"network": "192.168.1.0/24",
"nic_dev_id": "2",
"nw_type": "guest",
"one_to_one_nat": false,
"public_ip": "192.168.1.254",
"size": "24",
"source_nat": false
}
],
"id": "ips"
}
do you have an idea?
Thanks
Sven
Am 31.10.17, 20:35 schrieb "Frank Maximus" <fr...@nuagenetworks.net>:
I think that the bug you mentioned might not be related.
Could you please send the content of the file /etc/cloudstack/ips.json on
the VR.
That might provide useful information.
Kind regards,
Frank
On Tue, 31 Oct 2017 15:23 Vogel, Sven, <Sv...@kupper-computer.com>
wrote:
> Hi Devs,
>
> i have the following problem.
>
> When I look this jira ticket I see no solution.
> https://issues.apache.org/jira/browse/CLOUDSTACK-6801 but I think the
> problem is not solved correctly.
>
> 1. KVM
> 2. Bridges
> bond with two interfaces and trunk (0,129,180,100-1500) to cloudbr0
> Cloudbr0 (0 - guest network)
> Fakebridge pub129 (public network)
> Fakebridge sto180 (secondary storage network)
> Fakebridge mgmt0 (management)
>
> If I have a vpc all things work until I add a secondary ip and add a
> static nat.
>
> The following will happen, first address will be on the the correct
> interface 146.0.122.134/26 but static nat will be on the false network.
> Its on the eth2…
>
> root@r-29-VM:~# ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UP qlen 1000
> link/ether 0e:00:a9:fe:03:81 brd ff:ff:ff:ff:ff:ff
> inet 169.254.3.129/16 brd 169.254.255.255 scope global eth0
> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UP qlen 1000
> link/ether 1e:00:2c:00:00:68 brd ff:ff:ff:ff:ff:ff
> inet 146.0.122.134/26 brd 146.0.122.191 scope global eth1
> 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UP qlen 1000
> link/ether 02:00:57:07:00:0c brd ff:ff:ff:ff:ff:ff
> inet 192.168.1.254/24 brd 192.168.1.255 scope global eth2
> inet 146.0.122.135/26 brd 146.0.122.191 scope global eth2
>
> Normally I think the secondary ip should be on signed to eth1 not eth2! It
> sets my ip on the guest network vlan range on my cloudbr0 but it should be
> pub129. vnet6 has 1353 guest tag and not the public tag.
>
> [root@kvm01 ~]# ovs-vsctl list-br
> cloud0
> cloudbr0
> mgmt0
> pub129
> sto180
>
> [root@kvm01 ~]# virsh domiflist r-29-VM
> Interface Type Source Model MAC
> -------------------------------------------------------
> vnet4 bridge cloud0 virtio 0e:00:a9:fe:03:81
> vnet5 bridge pub129 virtio 1e:00:2c:00:00:68
> vnet6 bridge cloudbr0 virtio 02:00:57:07:00:0c
>
> Bridge "cloud0"
> Port "vnet4"
> Interface "vnet4"
>
> Port "vnet5"
> tag: 129
> Interface "vnet5"
> Port "vnet6"
> tag: 1353
> Interface "vnet6"
>
> Whats wrong or what can I do to fix this? I hope anybody has an idea.
>
>
> greetings
> Sven Vogel
> Head of Cloud Solutions / Consultants
> Kupper Computer GmbH
> Prager Str. 15
> 04103 Leipzig
>
>
>
Re: 4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip
on eth2?
Posted by "Vogel, Sven" <Sv...@kupper-computer.com>.
Hi Frank, Hi Remi,
great thank to both of you and really to Frank for do the work. Good Work!
I can test it on real setup. We will compile and test it. …
Really cool community work. Thanks
Sven
Am 01.11.17, 18:10 schrieb "Remi Bergsma" <RB...@schubergphilis.com>:
Hi Sven, Frank,
We hit this issue too last year in our CloudStack fork, and this is how we resolved it:
https://github.com/MissionCriticalCloudOldRepos/cosmic-plugin-hypervisor-kvm/pull/14
Shouldn’t be too hard to port to current CloudStack, but I have no time to do it and test it. We have been running like this ever since (18 months) without issues.
Regards,
Remi
On 01/11/2017, 11:06, "Vogel, Sven" <Sv...@kupper-computer.com> wrote:
Hi Frank,
i filed a bug report. I hope anybody can fix that fastly.
https://issues.apache.org/jira/browse/CLOUDSTACK-10127
Greetings
Sven
-----Ursprüngliche Nachricht-----
Von: Vogel, Sven [mailto:Sven.Vogel@kupper-computer.com]
Gesendet: Mittwoch, 1. November 2017 00:14
An: dev@cloudstack.apache.org
Betreff: Re: 4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip on eth2?
Hi Frank,
Great analysis, but what can we do now? Is there a workaround or who can fix this?
Greetings
Sven
Am 31.10.2017 um 22:41 schrieb Frank Maximus <fr...@nuagenetworks.net>>:
That seems to be a bug in the lookup of the device number, in case of openvswitch.
The config clearly sets device to eth2, while it should be eth1.
More specifically:
in LibvirtComputingResource.prepareNetworkElementCommand()
The broadcastUriToNicNum map is filled depending on the VR nics.
In openvswitch the guest bridge is used as is, so it overwrites the mapping of public.
This was not an issue until 4.6 as then VR was using the macaddress to do lookup, while now it is using the device number.
Kind Regards,
Frank
On Tue, Oct 31, 2017 at 8:35 PM Frank Maximus < frank.maximus@nuagenetworks.net<ma...@nuagenetworks.net>> wrote:
I think that the bug you mentioned might not be related.
Could you please send the content of the file /etc/cloudstack/ips.json on the VR.
That might provide useful information.
Kind regards,
Frank
On Tue, 31 Oct 2017 15:23 Vogel, Sven, <Sv...@kupper-computer.com>>
wrote:
Hi Devs,
i have the following problem.
When I look this jira ticket I see no solution.
https://issues.apache.org/jira/browse/CLOUDSTACK-6801 but I think the problem is not solved correctly.
1. KVM
2. Bridges
bond with two interfaces and trunk (0,129,180,100-1500) to cloudbr0
Cloudbr0 (0 - guest network)
Fakebridge pub129 (public network)
Fakebridge sto180 (secondary storage network)
Fakebridge mgmt0 (management)
If I have a vpc all things work until I add a secondary ip and add a static nat.
The following will happen, first address will be on the the correct interface 146.0.122.134/26 but static nat will be on the false network.
Its on the eth2.
root@r-29-VM:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 0e:00:a9:fe:03:81 brd ff:ff:ff:ff:ff:ff
inet 169.254.3.129/16 brd 169.254.255.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 1e:00:2c:00:00:68 brd ff:ff:ff:ff:ff:ff
inet 146.0.122.134/26 brd 146.0.122.191 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:00:57:07:00:0c brd ff:ff:ff:ff:ff:ff
inet 192.168.1.254/24 brd 192.168.1.255 scope global eth2
inet 146.0.122.135/26 brd 146.0.122.191 scope global eth2
Normally I think the secondary ip should be on signed to eth1 not eth2!
It sets my ip on the guest network vlan range on my cloudbr0 but it should be pub129. vnet6 has 1353 guest tag and not the public tag.
[root@kvm01 ~]# ovs-vsctl list-br
cloud0
cloudbr0
mgmt0
pub129
sto180
[root@kvm01 ~]# virsh domiflist r-29-VM
Interface Type Source Model MAC
-------------------------------------------------------
vnet4 bridge cloud0 virtio 0e:00:a9:fe:03:81
vnet5 bridge pub129 virtio 1e:00:2c:00:00:68
vnet6 bridge cloudbr0 virtio 02:00:57:07:00:0c
Bridge "cloud0"
Port "vnet4"
Interface "vnet4"
Port "vnet5"
tag: 129
Interface "vnet5"
Port "vnet6"
tag: 1353
Interface "vnet6"
Whats wrong or what can I do to fix this? I hope anybody has an idea.
greetings
Sven Vogel
Head of Cloud Solutions / Consultants
Kupper Computer GmbH
Prager Str. 15
04103 Leipzig
Re: 4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip on eth2?
Posted by Frank Maximus <fr...@nuagenetworks.net>.
Thank you, Remi.
I made some simplifications, and have a PR
<https://github.com/apache/cloudstack/pull/2304>ready.
Tests on a real setup still needed,
I'm also thinking, perhaps it would be better to check the mac address,
as is the case in XenServer, as this would cover the case
of adding a secondary ip to a guest ip on VR.
Which would fail now on KVM + LinuxBridge.
Regards,
Frank
On Wed, Nov 1, 2017 at 6:10 PM Remi Bergsma <RB...@schubergphilis.com>
wrote:
> Hi Sven, Frank,
>
> We hit this issue too last year in our CloudStack fork, and this is how we
> resolved it:
>
> https://github.com/MissionCriticalCloudOldRepos/cosmic-plugin-hypervisor-kvm/pull/14
>
> Shouldn’t be too hard to port to current CloudStack, but I have no time to
> do it and test it. We have been running like this ever since (18 months)
> without issues.
>
> Regards,
> Remi
>
>
>
> On 01/11/2017, 11:06, "Vogel, Sven" <Sv...@kupper-computer.com>
> wrote:
>
> Hi Frank,
>
> i filed a bug report. I hope anybody can fix that fastly.
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-10127
>
> Greetings
>
> Sven
>
> -----Ursprüngliche Nachricht-----
> Von: Vogel, Sven [mailto:Sven.Vogel@kupper-computer.com]
> Gesendet: Mittwoch, 1. November 2017 00:14
> An: dev@cloudstack.apache.org
> Betreff: Re: 4.9 / 4.10 KVM + openvswitch + vpc + static nat /
> secondary ip on eth2?
>
> Hi Frank,
>
> Great analysis, but what can we do now? Is there a workaround or who
> can fix this?
>
>
> Greetings
>
> Sven
>
>
>
> Am 31.10.2017 um 22:41 schrieb Frank Maximus <
> frank.maximus@nuagenetworks.net<ma...@nuagenetworks.net>>:
>
> That seems to be a bug in the lookup of the device number, in case of
> openvswitch.
> The config clearly sets device to eth2, while it should be eth1.
>
> More specifically:
> in LibvirtComputingResource.prepareNetworkElementCommand()
> The broadcastUriToNicNum map is filled depending on the VR nics.
> In openvswitch the guest bridge is used as is, so it overwrites the
> mapping of public.
> This was not an issue until 4.6 as then VR was using the macaddress to
> do lookup, while now it is using the device number.
>
> Kind Regards,
> Frank
>
>
> On Tue, Oct 31, 2017 at 8:35 PM Frank Maximus <
> frank.maximus@nuagenetworks.net<ma...@nuagenetworks.net>>
> wrote:
>
> I think that the bug you mentioned might not be related.
> Could you please send the content of the file /etc/cloudstack/ips.json
> on the VR.
>
> That might provide useful information.
>
> Kind regards,
> Frank
>
> On Tue, 31 Oct 2017 15:23 Vogel, Sven, <Sven.Vogel@kupper-computer.com
> <ma...@kupper-computer.com>>
> wrote:
>
> Hi Devs,
>
> i have the following problem.
>
> When I look this jira ticket I see no solution.
> https://issues.apache.org/jira/browse/CLOUDSTACK-6801 but I think
> the problem is not solved correctly.
>
> 1. KVM
> 2. Bridges
> bond with two interfaces and trunk (0,129,180,100-1500) to cloudbr0
> Cloudbr0 (0 - guest network)
> Fakebridge pub129 (public network)
> Fakebridge sto180 (secondary storage network)
> Fakebridge mgmt0 (management)
>
> If I have a vpc all things work until I add a secondary ip and add
> a static nat.
>
> The following will happen, first address will be on the the correct
> interface 146.0.122.134/26 but static nat will be on the false network.
> Its on the eth2.
>
> root@r-29-VM:~# ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> pfifo_fast state UP qlen 1000
> link/ether 0e:00:a9:fe:03:81 brd ff:ff:ff:ff:ff:ff
> inet 169.254.3.129/16 brd 169.254.255.255 scope global eth0
> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> pfifo_fast state UP qlen 1000
> link/ether 1e:00:2c:00:00:68 brd ff:ff:ff:ff:ff:ff
> inet 146.0.122.134/26 brd 146.0.122.191 scope global eth1
> 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> pfifo_fast state UP qlen 1000
> link/ether 02:00:57:07:00:0c brd ff:ff:ff:ff:ff:ff
> inet 192.168.1.254/24 brd 192.168.1.255 scope global eth2
> inet 146.0.122.135/26 brd 146.0.122.191 scope global eth2
>
> Normally I think the secondary ip should be on signed to eth1 not eth2!
> It sets my ip on the guest network vlan range on my cloudbr0 but it
> should be pub129. vnet6 has 1353 guest tag and not the public tag.
>
> [root@kvm01 ~]# ovs-vsctl list-br
> cloud0
> cloudbr0
> mgmt0
> pub129
> sto180
>
> [root@kvm01 ~]# virsh domiflist r-29-VM
> Interface Type Source Model MAC
> -------------------------------------------------------
> vnet4 bridge cloud0 virtio 0e:00:a9:fe:03:81
> vnet5 bridge pub129 virtio 1e:00:2c:00:00:68
> vnet6 bridge cloudbr0 virtio 02:00:57:07:00:0c
>
> Bridge "cloud0"
> Port "vnet4"
> Interface "vnet4"
>
> Port "vnet5"
> tag: 129
> Interface "vnet5"
> Port "vnet6"
> tag: 1353
> Interface "vnet6"
>
> Whats wrong or what can I do to fix this? I hope anybody has an idea.
>
>
> greetings
> Sven Vogel
> Head of Cloud Solutions / Consultants
> Kupper Computer GmbH
> Prager Str. 15
> 04103 Leipzig
>
>
>
>
>
>
Re: 4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip on
eth2?
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Hi Sven, Frank,
We hit this issue too last year in our CloudStack fork, and this is how we resolved it:
https://github.com/MissionCriticalCloudOldRepos/cosmic-plugin-hypervisor-kvm/pull/14
Shouldn’t be too hard to port to current CloudStack, but I have no time to do it and test it. We have been running like this ever since (18 months) without issues.
Regards,
Remi
On 01/11/2017, 11:06, "Vogel, Sven" <Sv...@kupper-computer.com> wrote:
Hi Frank,
i filed a bug report. I hope anybody can fix that fastly.
https://issues.apache.org/jira/browse/CLOUDSTACK-10127
Greetings
Sven
-----Ursprüngliche Nachricht-----
Von: Vogel, Sven [mailto:Sven.Vogel@kupper-computer.com]
Gesendet: Mittwoch, 1. November 2017 00:14
An: dev@cloudstack.apache.org
Betreff: Re: 4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip on eth2?
Hi Frank,
Great analysis, but what can we do now? Is there a workaround or who can fix this?
Greetings
Sven
Am 31.10.2017 um 22:41 schrieb Frank Maximus <fr...@nuagenetworks.net>>:
That seems to be a bug in the lookup of the device number, in case of openvswitch.
The config clearly sets device to eth2, while it should be eth1.
More specifically:
in LibvirtComputingResource.prepareNetworkElementCommand()
The broadcastUriToNicNum map is filled depending on the VR nics.
In openvswitch the guest bridge is used as is, so it overwrites the mapping of public.
This was not an issue until 4.6 as then VR was using the macaddress to do lookup, while now it is using the device number.
Kind Regards,
Frank
On Tue, Oct 31, 2017 at 8:35 PM Frank Maximus < frank.maximus@nuagenetworks.net<ma...@nuagenetworks.net>> wrote:
I think that the bug you mentioned might not be related.
Could you please send the content of the file /etc/cloudstack/ips.json on the VR.
That might provide useful information.
Kind regards,
Frank
On Tue, 31 Oct 2017 15:23 Vogel, Sven, <Sv...@kupper-computer.com>>
wrote:
Hi Devs,
i have the following problem.
When I look this jira ticket I see no solution.
https://issues.apache.org/jira/browse/CLOUDSTACK-6801 but I think the problem is not solved correctly.
1. KVM
2. Bridges
bond with two interfaces and trunk (0,129,180,100-1500) to cloudbr0
Cloudbr0 (0 - guest network)
Fakebridge pub129 (public network)
Fakebridge sto180 (secondary storage network)
Fakebridge mgmt0 (management)
If I have a vpc all things work until I add a secondary ip and add a static nat.
The following will happen, first address will be on the the correct interface 146.0.122.134/26 but static nat will be on the false network.
Its on the eth2.
root@r-29-VM:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 0e:00:a9:fe:03:81 brd ff:ff:ff:ff:ff:ff
inet 169.254.3.129/16 brd 169.254.255.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 1e:00:2c:00:00:68 brd ff:ff:ff:ff:ff:ff
inet 146.0.122.134/26 brd 146.0.122.191 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:00:57:07:00:0c brd ff:ff:ff:ff:ff:ff
inet 192.168.1.254/24 brd 192.168.1.255 scope global eth2
inet 146.0.122.135/26 brd 146.0.122.191 scope global eth2
Normally I think the secondary ip should be on signed to eth1 not eth2!
It sets my ip on the guest network vlan range on my cloudbr0 but it should be pub129. vnet6 has 1353 guest tag and not the public tag.
[root@kvm01 ~]# ovs-vsctl list-br
cloud0
cloudbr0
mgmt0
pub129
sto180
[root@kvm01 ~]# virsh domiflist r-29-VM
Interface Type Source Model MAC
-------------------------------------------------------
vnet4 bridge cloud0 virtio 0e:00:a9:fe:03:81
vnet5 bridge pub129 virtio 1e:00:2c:00:00:68
vnet6 bridge cloudbr0 virtio 02:00:57:07:00:0c
Bridge "cloud0"
Port "vnet4"
Interface "vnet4"
Port "vnet5"
tag: 129
Interface "vnet5"
Port "vnet6"
tag: 1353
Interface "vnet6"
Whats wrong or what can I do to fix this? I hope anybody has an idea.
greetings
Sven Vogel
Head of Cloud Solutions / Consultants
Kupper Computer GmbH
Prager Str. 15
04103 Leipzig
AW: 4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip
on eth2?
Posted by "Vogel, Sven" <Sv...@kupper-computer.com>.
Hi Frank,
i filed a bug report. I hope anybody can fix that fastly.
https://issues.apache.org/jira/browse/CLOUDSTACK-10127
Greetings
Sven
-----Ursprüngliche Nachricht-----
Von: Vogel, Sven [mailto:Sven.Vogel@kupper-computer.com]
Gesendet: Mittwoch, 1. November 2017 00:14
An: dev@cloudstack.apache.org
Betreff: Re: 4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip on eth2?
Hi Frank,
Great analysis, but what can we do now? Is there a workaround or who can fix this?
Greetings
Sven
Am 31.10.2017 um 22:41 schrieb Frank Maximus <fr...@nuagenetworks.net>>:
That seems to be a bug in the lookup of the device number, in case of openvswitch.
The config clearly sets device to eth2, while it should be eth1.
More specifically:
in LibvirtComputingResource.prepareNetworkElementCommand()
The broadcastUriToNicNum map is filled depending on the VR nics.
In openvswitch the guest bridge is used as is, so it overwrites the mapping of public.
This was not an issue until 4.6 as then VR was using the macaddress to do lookup, while now it is using the device number.
Kind Regards,
Frank
On Tue, Oct 31, 2017 at 8:35 PM Frank Maximus < frank.maximus@nuagenetworks.net<ma...@nuagenetworks.net>> wrote:
I think that the bug you mentioned might not be related.
Could you please send the content of the file /etc/cloudstack/ips.json on the VR.
That might provide useful information.
Kind regards,
Frank
On Tue, 31 Oct 2017 15:23 Vogel, Sven, <Sv...@kupper-computer.com>>
wrote:
Hi Devs,
i have the following problem.
When I look this jira ticket I see no solution.
https://issues.apache.org/jira/browse/CLOUDSTACK-6801 but I think the problem is not solved correctly.
1. KVM
2. Bridges
bond with two interfaces and trunk (0,129,180,100-1500) to cloudbr0
Cloudbr0 (0 - guest network)
Fakebridge pub129 (public network)
Fakebridge sto180 (secondary storage network)
Fakebridge mgmt0 (management)
If I have a vpc all things work until I add a secondary ip and add a static nat.
The following will happen, first address will be on the the correct interface 146.0.122.134/26 but static nat will be on the false network.
Its on the eth2.
root@r-29-VM:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 0e:00:a9:fe:03:81 brd ff:ff:ff:ff:ff:ff
inet 169.254.3.129/16 brd 169.254.255.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 1e:00:2c:00:00:68 brd ff:ff:ff:ff:ff:ff
inet 146.0.122.134/26 brd 146.0.122.191 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:00:57:07:00:0c brd ff:ff:ff:ff:ff:ff
inet 192.168.1.254/24 brd 192.168.1.255 scope global eth2
inet 146.0.122.135/26 brd 146.0.122.191 scope global eth2
Normally I think the secondary ip should be on signed to eth1 not eth2!
It sets my ip on the guest network vlan range on my cloudbr0 but it should be pub129. vnet6 has 1353 guest tag and not the public tag.
[root@kvm01 ~]# ovs-vsctl list-br
cloud0
cloudbr0
mgmt0
pub129
sto180
[root@kvm01 ~]# virsh domiflist r-29-VM
Interface Type Source Model MAC
-------------------------------------------------------
vnet4 bridge cloud0 virtio 0e:00:a9:fe:03:81
vnet5 bridge pub129 virtio 1e:00:2c:00:00:68
vnet6 bridge cloudbr0 virtio 02:00:57:07:00:0c
Bridge "cloud0"
Port "vnet4"
Interface "vnet4"
Port "vnet5"
tag: 129
Interface "vnet5"
Port "vnet6"
tag: 1353
Interface "vnet6"
Whats wrong or what can I do to fix this? I hope anybody has an idea.
greetings
Sven Vogel
Head of Cloud Solutions / Consultants
Kupper Computer GmbH
Prager Str. 15
04103 Leipzig
Re: 4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip
on eth2?
Posted by "Vogel, Sven" <Sv...@kupper-computer.com>.
Hi Frank,
Great analysis, but what can we do now? Is there a workaround or who can fix this?
Greetings
Sven
Am 31.10.2017 um 22:41 schrieb Frank Maximus <fr...@nuagenetworks.net>>:
That seems to be a bug in the lookup of the device number, in case of
openvswitch.
The config clearly sets device to eth2, while it should be eth1.
More specifically:
in LibvirtComputingResource.prepareNetworkElementCommand()
The broadcastUriToNicNum map is filled depending on the VR nics.
In openvswitch the guest bridge is used as is, so it overwrites the mapping
of public.
This was not an issue until 4.6 as then VR was using the macaddress to do
lookup,
while now it is using the device number.
Kind Regards,
Frank
On Tue, Oct 31, 2017 at 8:35 PM Frank Maximus <
frank.maximus@nuagenetworks.net<ma...@nuagenetworks.net>> wrote:
I think that the bug you mentioned might not be related.
Could you please send the content of the file /etc/cloudstack/ips.json on
the VR.
That might provide useful information.
Kind regards,
Frank
On Tue, 31 Oct 2017 15:23 Vogel, Sven, <Sv...@kupper-computer.com>>
wrote:
Hi Devs,
i have the following problem.
When I look this jira ticket I see no solution.
https://issues.apache.org/jira/browse/CLOUDSTACK-6801 but I think the
problem is not solved correctly.
1. KVM
2. Bridges
bond with two interfaces and trunk (0,129,180,100-1500) to cloudbr0
Cloudbr0 (0 - guest network)
Fakebridge pub129 (public network)
Fakebridge sto180 (secondary storage network)
Fakebridge mgmt0 (management)
If I have a vpc all things work until I add a secondary ip and add a
static nat.
The following will happen, first address will be on the the correct
interface 146.0.122.134/26 but static nat will be on the false network.
Its on the eth2…
root@r-29-VM:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
link/ether 0e:00:a9:fe:03:81 brd ff:ff:ff:ff:ff:ff
inet 169.254.3.129/16 brd 169.254.255.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
link/ether 1e:00:2c:00:00:68 brd ff:ff:ff:ff:ff:ff
inet 146.0.122.134/26 brd 146.0.122.191 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
link/ether 02:00:57:07:00:0c brd ff:ff:ff:ff:ff:ff
inet 192.168.1.254/24 brd 192.168.1.255 scope global eth2
inet 146.0.122.135/26 brd 146.0.122.191 scope global eth2
Normally I think the secondary ip should be on signed to eth1 not eth2!
It sets my ip on the guest network vlan range on my cloudbr0 but it should
be pub129. vnet6 has 1353 guest tag and not the public tag.
[root@kvm01 ~]# ovs-vsctl list-br
cloud0
cloudbr0
mgmt0
pub129
sto180
[root@kvm01 ~]# virsh domiflist r-29-VM
Interface Type Source Model MAC
-------------------------------------------------------
vnet4 bridge cloud0 virtio 0e:00:a9:fe:03:81
vnet5 bridge pub129 virtio 1e:00:2c:00:00:68
vnet6 bridge cloudbr0 virtio 02:00:57:07:00:0c
Bridge "cloud0"
Port "vnet4"
Interface "vnet4"
Port "vnet5"
tag: 129
Interface "vnet5"
Port "vnet6"
tag: 1353
Interface "vnet6"
Whats wrong or what can I do to fix this? I hope anybody has an idea.
greetings
Sven Vogel
Head of Cloud Solutions / Consultants
Kupper Computer GmbH
Prager Str. 15
04103 Leipzig
Re: 4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip on eth2?
Posted by Frank Maximus <fr...@nuagenetworks.net>.
That seems to be a bug in the lookup of the device number, in case of
openvswitch.
The config clearly sets device to eth2, while it should be eth1.
More specifically:
in LibvirtComputingResource.prepareNetworkElementCommand()
The broadcastUriToNicNum map is filled depending on the VR nics.
In openvswitch the guest bridge is used as is, so it overwrites the mapping
of public.
This was not an issue until 4.6 as then VR was using the macaddress to do
lookup,
while now it is using the device number.
Kind Regards,
Frank
On Tue, Oct 31, 2017 at 8:35 PM Frank Maximus <
frank.maximus@nuagenetworks.net> wrote:
> I think that the bug you mentioned might not be related.
> Could you please send the content of the file /etc/cloudstack/ips.json on
> the VR.
>
> That might provide useful information.
>
> Kind regards,
> Frank
>
> On Tue, 31 Oct 2017 15:23 Vogel, Sven, <Sv...@kupper-computer.com>
> wrote:
>
>> Hi Devs,
>>
>> i have the following problem.
>>
>> When I look this jira ticket I see no solution.
>> https://issues.apache.org/jira/browse/CLOUDSTACK-6801 but I think the
>> problem is not solved correctly.
>>
>> 1. KVM
>> 2. Bridges
>> bond with two interfaces and trunk (0,129,180,100-1500) to cloudbr0
>> Cloudbr0 (0 - guest network)
>> Fakebridge pub129 (public network)
>> Fakebridge sto180 (secondary storage network)
>> Fakebridge mgmt0 (management)
>>
>> If I have a vpc all things work until I add a secondary ip and add a
>> static nat.
>>
>> The following will happen, first address will be on the the correct
>> interface 146.0.122.134/26 but static nat will be on the false network.
>> Its on the eth2…
>>
>> root@r-29-VM:~# ip a
>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>> inet 127.0.0.1/8 scope host lo
>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>> state UP qlen 1000
>> link/ether 0e:00:a9:fe:03:81 brd ff:ff:ff:ff:ff:ff
>> inet 169.254.3.129/16 brd 169.254.255.255 scope global eth0
>> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>> state UP qlen 1000
>> link/ether 1e:00:2c:00:00:68 brd ff:ff:ff:ff:ff:ff
>> inet 146.0.122.134/26 brd 146.0.122.191 scope global eth1
>> 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>> state UP qlen 1000
>> link/ether 02:00:57:07:00:0c brd ff:ff:ff:ff:ff:ff
>> inet 192.168.1.254/24 brd 192.168.1.255 scope global eth2
>> inet 146.0.122.135/26 brd 146.0.122.191 scope global eth2
>>
>> Normally I think the secondary ip should be on signed to eth1 not eth2!
>> It sets my ip on the guest network vlan range on my cloudbr0 but it should
>> be pub129. vnet6 has 1353 guest tag and not the public tag.
>>
>> [root@kvm01 ~]# ovs-vsctl list-br
>> cloud0
>> cloudbr0
>> mgmt0
>> pub129
>> sto180
>>
>> [root@kvm01 ~]# virsh domiflist r-29-VM
>> Interface Type Source Model MAC
>> -------------------------------------------------------
>> vnet4 bridge cloud0 virtio 0e:00:a9:fe:03:81
>> vnet5 bridge pub129 virtio 1e:00:2c:00:00:68
>> vnet6 bridge cloudbr0 virtio 02:00:57:07:00:0c
>>
>> Bridge "cloud0"
>> Port "vnet4"
>> Interface "vnet4"
>>
>> Port "vnet5"
>> tag: 129
>> Interface "vnet5"
>> Port "vnet6"
>> tag: 1353
>> Interface "vnet6"
>>
>> Whats wrong or what can I do to fix this? I hope anybody has an idea.
>>
>>
>> greetings
>> Sven Vogel
>> Head of Cloud Solutions / Consultants
>> Kupper Computer GmbH
>> Prager Str. 15
>> 04103 Leipzig
>>
>>
>>
Re: 4.9 / 4.10 KVM + openvswitch + vpc + static nat / secondary ip on eth2?
Posted by Frank Maximus <fr...@nuagenetworks.net>.
I think that the bug you mentioned might not be related.
Could you please send the content of the file /etc/cloudstack/ips.json on
the VR.
That might provide useful information.
Kind regards,
Frank
On Tue, 31 Oct 2017 15:23 Vogel, Sven, <Sv...@kupper-computer.com>
wrote:
> Hi Devs,
>
> i have the following problem.
>
> When I look this jira ticket I see no solution.
> https://issues.apache.org/jira/browse/CLOUDSTACK-6801 but I think the
> problem is not solved correctly.
>
> 1. KVM
> 2. Bridges
> bond with two interfaces and trunk (0,129,180,100-1500) to cloudbr0
> Cloudbr0 (0 - guest network)
> Fakebridge pub129 (public network)
> Fakebridge sto180 (secondary storage network)
> Fakebridge mgmt0 (management)
>
> If I have a vpc all things work until I add a secondary ip and add a
> static nat.
>
> The following will happen, first address will be on the the correct
> interface 146.0.122.134/26 but static nat will be on the false network.
> Its on the eth2…
>
> root@r-29-VM:~# ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UP qlen 1000
> link/ether 0e:00:a9:fe:03:81 brd ff:ff:ff:ff:ff:ff
> inet 169.254.3.129/16 brd 169.254.255.255 scope global eth0
> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UP qlen 1000
> link/ether 1e:00:2c:00:00:68 brd ff:ff:ff:ff:ff:ff
> inet 146.0.122.134/26 brd 146.0.122.191 scope global eth1
> 4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
> state UP qlen 1000
> link/ether 02:00:57:07:00:0c brd ff:ff:ff:ff:ff:ff
> inet 192.168.1.254/24 brd 192.168.1.255 scope global eth2
> inet 146.0.122.135/26 brd 146.0.122.191 scope global eth2
>
> Normally I think the secondary ip should be on signed to eth1 not eth2! It
> sets my ip on the guest network vlan range on my cloudbr0 but it should be
> pub129. vnet6 has 1353 guest tag and not the public tag.
>
> [root@kvm01 ~]# ovs-vsctl list-br
> cloud0
> cloudbr0
> mgmt0
> pub129
> sto180
>
> [root@kvm01 ~]# virsh domiflist r-29-VM
> Interface Type Source Model MAC
> -------------------------------------------------------
> vnet4 bridge cloud0 virtio 0e:00:a9:fe:03:81
> vnet5 bridge pub129 virtio 1e:00:2c:00:00:68
> vnet6 bridge cloudbr0 virtio 02:00:57:07:00:0c
>
> Bridge "cloud0"
> Port "vnet4"
> Interface "vnet4"
>
> Port "vnet5"
> tag: 129
> Interface "vnet5"
> Port "vnet6"
> tag: 1353
> Interface "vnet6"
>
> Whats wrong or what can I do to fix this? I hope anybody has an idea.
>
>
> greetings
> Sven Vogel
> Head of Cloud Solutions / Consultants
> Kupper Computer GmbH
> Prager Str. 15
> 04103 Leipzig
>
>
>