You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Chi vediamo <ta...@gmail.com> on 2025/04/03 16:47:42 UTC
CloudStack 4.20 VXLAN-EVPN - Management VXLAN generates error - Public VXLAN not advertising - Initial VMs running but Grey State for Agent State
Using [Network 1] Only for all three [Management], [guest],[public] Tagged as VXLAN and with cloudbr1
- **Should I use a separate Network for each ? ? **
What should I check, in addition to the official manual: I reread again the instructions and saw videos
https://www.youtube.com/watch?v=9gXEmWbgX2o <<Actually this video shown both VMs are down'
https://www.youtube.com/watch?v=DlJg3LYvIIs << this shows both on green.
https://www.youtube.com/watch?v=vZfHfoYEMdY << this one shows the [Agent State] green while the Vms are in starting mode with a bue dot.
I have Read the https://rohityadav.cloud/blog/cloudstack-kvm/
- Management servers are two physical separated servers. all Ubuntu 22.04
- DB is mysql running on a separate server. Mysql 8
- Hypervisor is running on a separate server. Ubuntu22.0 KVM
- Cloudstack 4.20.0.0
- Primary Storage CEPH RDB
- Secondary storage EMC NFS
**Issue 1) I can't add a second Management server, No idea why !!**
Thank you Wei, as soon i shut the second Management server the initial VMs proxy and secondary storage come up
**Issue 2) Management Network is Tagged as VXLAN, the UI shows vlan://untagged - "no typo vlan" every-time i use the UI to change to vxlan://untagged**
I did try vxlan://untagged parameter Got the message: Vlan parameter : vxlan://untagged is not in valid format
I did try vxlan:// Got the message: Unable to convert to broadcast URI: vxlan://
I did try vxlan:// Got the message: Vlan parameter : vxlan://100 is not in valid format'
**Issue 3) s-VM and Proxy come up but Public IP addresses are not being routed Advertised.**
v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x> [State-Running] [Agent State-Gray]
s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y> [State-Running] [Agent State-Gray]
If both [State] and [Agent State] need to be green, What other debug can I enable ??
**NOTE** The VXLAN parameter for the Public network was taken vxlan://1000, But are not routed.
I have to use a Single management server as everything messes UP when I run a second Management server with the Same Database.
I reiterate The initial VMs are running:
v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x>
s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y>
I do not see the Public IP anywhere, not even advertised on vni1000 or any other, the public VNI is there was created by the system, but nothing is there - No IP addresses advertised
I used VLAN/VNI vxlan://1000
vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 master brvx-1000 state forwarding priority
There are 2 VNETS assigned
virsh domiflist s-285-VM
Interface Type Source Model MAC
--------------------------------------------------------------
vnet3 bridge cloud0 virtio 0e:00:a9:fe:a1:14
vnet4 bridge cloudbr1 virtio 1e:00:3f:00:03:33
vnet5 bridge brvx-1000 virtio 1e:00:b8:00:0c:fe
virsh domifaddr s-285-VM
Name MAC address Protocol Address
-------------------------------------------------------------------------------
----- Nothing here -----
the command arp -n, does not show Public IP addresses neither the management IP addresses, just the private IP addresses and the <control-ip-169.x.x.x>
sh bgp evpn route vni 1000
BGP table version is 1, local router ID is 10.1.1.14
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
*> [3]:[0]:[32]:[10.1.1.14]
10.1.1.14(hv1)
32768 i
ET:8 RT:5641:1000
Displayed 1 prefixes (1 paths)
At the Agent side only WARNINGS I am not using uefi.
2025-04-03 00:01:44,744 ERROR [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:[]) (logid:) Process [15539] for command [/bin/bash -c virt-v2v --version ] encountered the error: [127].
2025-04-03 00:01:47,921 ERROR [cloud.agent.Agent] (agentRequest-Handler-5:[]) (logid:09929bd6) Unexpected arch null, expected x86_64
2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:[]) (logid:) Execution of process [15539] for command [/bin/bash -c virt-v2v --version ] failed.
2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:[]) (logid:) Process [15539] for command [/bin/bash -c virt-v2v --version ] encountered the error: [127].
AT the management the rror is:
2025-04-03 00:00:28,115 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Executing ClusterServicePdu with service URL: https://10.1.1.1:9090/clusterservice
2025-04-03 00:21:28,122 ERROR [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Exception from : https://10.1.1.1:9090/clusterservice, method : null, exception : [javax.net](http://javax.net/).ssl.SSLPeerUnverifiedException: Certificate for <10.1.1.1> doesn't match any of the subject alternative names: [fde0:f:2897:1:1:0:0:1, 172.1.1.1, fe80:0:0:0:e643:4bff:fe81:9660, cs1.myinternaldomain.int, cloudstack.internal]
Tata Y.
Re: More bugs seems related to #9920 Re: CloudStack 4.20 VXLAN-EVPN - Management VXLAN generates error - Public VXLAN not advertising - Initial VMs running but Grey State for Agent State
Posted by Chi vediamo <ta...@gmail.com>.
**### Regarding the VXLAN**
i found the Cloudstack does not link the Interface Loopback to the VXLAN interface,
I did manually and now it propagates the routes
**### On the AGENT side:**
Not sure if this is just related to Ubuntu systems:
there is an error on the management server every time i reload it:
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management stop
DEBUG:root:execute:sudo update-rc.d -f cloudstack-management remove
DEBUG:root:execute:sudo update-rc.d -f cloudstack-management defaults
DEBUG:root:Failed to execute:update-rc.d: **### error: unable to read /etc/init.d/cloudstack-management**
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management status
DEBUG:root:Failed to execute:× cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2025-04-02 13:13:48 UTC; 1 day 8h ago
Main PID: 537731 (code=exited, status=143)
CPU: 27min 54.476s
Obviously there is no such file: Should I create One?
ls -al /etc/init.d/
cryptdisks grub-common iwpmd nfs-common plymouth rng-tools-debian ufw
apparmor cryptdisks-early hwclock.sh keyboard-setup.sh ntp plymouth-log rpcbind unattended-upgrades
apport dbus ipmievd kmod open-iscsi procps rsync uuidd
console-setup.sh dpdk irqbalance lvm2 openipmi quota ssh x11-common
cron frr iscsid lvm2-lvmpolld openvswitch-switch
**### On the management side**, only when i restart the agent:
Apr 03 18:41:23 kvmvcompatl2001 systemd[1]: Starting Uncomplicated firewall...
Apr 03 18:41:23 kvmvcompatl2001 systemd[1]: Finished Uncomplicated firewall.
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: Stopping Uncomplicated firewall...
Apr 03 23:30:14 kvmvcompatl2001 ufw-init[56782]: Skip stopping firewall: ufw (not enabled)
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: ufw.service: Deactivated successfully.
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: Stopped Uncomplicated firewall.
DEBUG:root:execute:sudo /usr/sbin/service ufw start
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent stop
DEBUG:root:execute:sleep 30
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent remove
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent defaults
DEBUG:root:**### Failed to execute:update-rc.d: error: unable to read /etc/init.d/cloudstack-agent**
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:Failed to execute:○ cloudstack-agent.service - CloudStack Agent
Loaded: loaded (/lib/systemd/system/cloudstack-agent.service; enabled; vendor preset: enabled)
Active: inactive (dead) (Result: exit-code) since Thu 2025-04-03 23:30:17 UTC; 31s ago
Docs: http://www.cloudstack.org/
Main PID: 56558 (code=exited, status=1/FAILURE)
CPU: 14.716s
Apr 03 23:30:17 kvmvcompatl2001 systemd[1]: Stopped CloudStack Agent.
Apr 03 23:30:17 kvmvcompatl2001 systemd[1]: cloudstack-agent.service: Consumed 14.716s CPU time.
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start
**### Regarding the VXLAN** i found the Cloudstack does not link the Interface Loopback to the VXLAN interface,
I did manually and now it propagates the routes
> On Apr 4, 2025, at 7:07 PM, Chi vediamo <ta...@gmail.com> wrote:
>
> I created this one with the pictures,
>
> https://github.com/apache/cloudstack/issues/10659
>
>
> Pearl opened somethis similar: https://github.com/apache/cloudstack/issues/9920
>
> Which seems related to one Pearl was working, but if this is for 4.21 is unrealistic from the time line perspective for me !!! and Wido have this working Something is really messed up !
>
>
> Additionally: Issue 4) The VNI's I Choose for Guest are showing as VLANs, Wido what did you put on guest Vlan Range and Which Format Or nothing at all.
>
> Wido can you share which version of cloudstack are you running?
>
>
> Tata Y.
>
>
>
>
>> On Apr 4, 2025, at 6:27 AM, Chi vediamo <ta...@gmail.com> wrote:
>>
>>
>> The cloud0 was added by the system, i did not created that.
>> I do not have a cloudbr0
>>
>> what does it show for your in your agent.properties: public.network.device and guest.network.properties
>>
>>
>> My agent.properties
>> private.network.device=cloudbr1
>> guest.network.device=cloudbr1
>> hypervisor.type=kvm
>> public.network.device=cloudbr1
>> host=10.1.1.1@static
>> guid=579fafc9-5706-31f1-ae98-9f041ac90c80
>> workers=5
>>
>> the vxlan1000 I configured in the UI for the PUblic network and the brvx-1000 that was added by Cloudstack, I did not do it manually
>>>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 qdisc noqueue master brvx-1000 state UNKNOWN group default qlen 1000
>>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>>>> brvx-1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP group default qlen 1000
>>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>>
>>
>> I created a vxlan100 for the management and the management IPs are being propagates
>>
>> my VNi
>> ~# vtysh -c 'sh evpn vni'
>> VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs Tenant VRF
>> 1000 L2 vxlan1000 2 0 0 default <<< Added by Cloudstack Public but no Public IPs advertised
>> 100 L2 vxlan100 7 11 4 default <<< created manually for the initial clourbr1 and management sec storage
>> 222 L2 vxlan222 10 20 9 default <<< Separate vxlan for primary storage, its working
>>
>> My concerns are: Is normal the UI to show agent state gray for the 2 initial SYSTEM-VMs the proxy and the secndarystorage
>>>>> ***Issue 3) s-VM and Proxy come up but Public IP addresses are not being routed Advertised.***
>>>>> v-154-VM <public-IP-3> <Private-IP-2> <control-ip-169.x.x.x> [State-Running] [Agent State-Gray]
>>>>> s-285-VM <public-IP-2> <Private-IP-3> <control-ip-169.x.x.y> [State-Running] [Agent State-Gray]
>>
>> And where are my public IPs that should be added to the vxlan1000 ?
>>
>>
>> Also the UI does not allow me to add Public IPv6 address not even using square brackets []
>>
>> I can send snapshots but apache.org removes them.
>>
>> I know it should be a little detail but I do not know what I am missing, or what is the normal status for the UI buttons.
>>
>>
>> Can any of you send me a snapshot of the UI configuration for the ZONe
>>
>> and for your zone for example Attached a PDF with the snapshots of mine.
>> the network storage I did not add any, cloudstack added itself to the cloudbr1
>>
>> Also How do I Assign Public IPv6 Addresses ?
>>
>> thank you
>>
>> Tata Y.
>>
>> see PDF attached
>>
>> <TataY-Cloudstack-VXLAN-UI-2025-V1.pdf>
>>
>>
>>
>>> On Apr 4, 2025, at 5:00 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>>>
>>> Let me post a couple of outputs of a hypervisor we run:
>>>
>>>
>>> 9: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr1 state UNKNOWN group default qlen 1000
>>> link/ether 8e:7d:67:a7:c9:72 brd ff:ff:ff:ff:ff:ff
>>> inet6 fe80::8c7d:67ff:fea7:c972/64 scope link
>>> valid_lft forever preferred_lft forever
>>> 10: cloudbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
>>> link/ether 1e:11:99:1e:6d:46 brd ff:ff:ff:ff:ff:ff
>>> inet 10.100.2.2/20 scope global cloudbr1
>>> valid_lft forever preferred_lft forever
>>>
>>>
>>> As you can see there is a /20 IPv4 allocated there. The agent.properties shows:
>>>
>>> host=10.100.1.6@static
>>> control.cidr=169.254.240.0/20
>>> private.network.device=cloudbr1
>>> guid=44e94fad-4aec-414e-8cac-cea3264dd431
>>>
>>> Those are the relevant ones in this case.
>>>
>>>
>>> 6: enxb8599f206ba6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP group default qlen 1000
>>> link/ether b8:59:9f:20:6b:a6 brd ff:ff:ff:ff:ff:ff
>>> inet6 2a05:xxxx:601:117:ba59:9fff:fe20:6ba6/64 scope global deprecated dynamic mngtmpaddr
>>> valid_lft 3592sec preferred_lft 0sec
>>> inet6 fe80::ba59:9fff:fe20:6ba6/64 scope link
>>> valid_lft forever preferred_lft forever
>>> 7: enxb8599f206ba7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP group default qlen 1000
>>> link/ether b8:59:9f:20:6b:a7 brd ff:ff:ff:ff:ff:ff
>>> inet6 2a05:xxxx:601:197:ba59:9fff:fe20:6ba7/64 scope global deprecated dynamic mngtmpaddr
>>> valid_lft 3594sec preferred_lft 0sec
>>> inet6 fe80::ba59:9fff:fe20:6ba7/64 scope link
>>> valid_lft forever preferred_lft forever
>>>
>>>
>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
>>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>> inet 127.0.0.1/8 scope host lo
>>> valid_lft forever preferred_lft forever
>>> inet 10.255.255.10/32 brd 10.255.255.10 scope global lo
>>> valid_lft forever preferred_lft forever
>>> inet6 2a05:xxxx:601:2::10/128 scope global
>>> valid_lft forever preferred_lft forever
>>> inet6 ::1/128 scope host
>>> valid_lft forever preferred_lft forever
>>>
>>>
>>> With two BGP sessions over these two uplinks:
>>>
>>> L2VPN EVPN Summary:
>>> BGP router identifier 10.255.255.10, local AS number 4200100124 VRF default vrf-id 0
>>> BGP table version 0
>>> RIB entries 4341, using 543 KiB of memory
>>> Peers 2, using 48 KiB of memory
>>> Peer groups 1, using 64 bytes of memory
>>>
>>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
>>> enxb8599f206ba6 4 4200100002 6027297 5839388 39828 0 0 28w1d10h 6635 6706 N/A
>>> enxb8599f206ba7 4 4200100002 6035334 5835598 39828 0 0 28w1d10h 6635 6706 N/A
>>>
>>>
>>>
>>> That's all! There is no cloudbr0 on the hypervisor at all.
>>>
>>> hv-138-a05-zz.xxxx.cldin.net# sh evpn vni
>>> VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs Tenant VRF
>>> 539 L2 vxlan539 19 8 15 default
>>> 100 L2 vxlan100 107 106 103 default
>>> 528 L2 vxlan528 92 8 36 default
>>> 507 L2 vxlan507 43 8 22 default
>>> ....
>>> ....
>>> hv-138-a05-zz.xxxx.cldin.net#
>>>
>>>
>>> Wido
>>>
>>> On 4/4/25 04:07, Chi vediamo wrote:
>>>> I decided to reinstall everything clean again, drop the tables completely and start all over again.
>>>> The management IPs I see them routed. even the ones for the s-VM and v- VM. I see them on the transport switches.
>>>> The Public VXLAN is created, the Public IP address I am not able to see them anywhere !
>>>> FROM THE UI: If I run a diagnostic Ping to the public IP for the secondarystoragevm or for the consoleproxy - success !!!
>>>> Why or how is not being routed ? How can I troubleshoot this?
>>>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 qdisc noqueue master brvx-1000 state UNKNOWN group default qlen 1000
>>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>>>> brvx-1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP group default qlen 1000
>>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>>>> Any tip will be appreciated.
>>>> Thank you Teams.
>>>> Tata Y.
>>>>> On Apr 3, 2025, at 12:47 PM, Chi vediamo <ta...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Using [Network 1] Only for all three [Management], [guest],[public] Tagged as VXLAN and with cloudbr1
>>>>>
>>>>> *- **Should I use a separate Network for each ? ? ***
>>>>>
>>>>> What should I check, in addition to the official manual: I reread again the instructions and saw videos
>>>>> https://www.youtube.com/watch?v=9gXEmWbgX2o <<Actually this video shown both VMs are down'
>>>>> https://www.youtube.com/watch?v=DlJg3LYvIIs << this shows both on green.
>>>>> https://www.youtube.com/watch?v=vZfHfoYEMdY << this one shows the [Agent State] green while the Vms are in starting mode with a bue dot.
>>>>> I have Read the https://rohityadav.cloud/blog/cloudstack-kvm/
>>>>>
>>>>> - Management servers are two physical separated servers. all Ubuntu 22.04
>>>>> - DB is mysql running on a separate server. Mysql 8
>>>>> - Hypervisor is running on a separate server. Ubuntu22.0 KVM
>>>>> - Cloudstack 4.20.0.0
>>>>> - Primary Storage CEPH RDB
>>>>> - Secondary storage EMC NFS
>>>>>
>>>>>
>>>>> ***Issue 1) I can't add a second Management server, No idea why !!** *
>>>>> Thank you Wei, as soon i shut the second Management server the initial VMs proxy and secondary storage come up
>>>>>
>>>>> ***Issue 2) Management Network is Tagged as VXLAN, the UI shows vlan://untagged - "no typo vlan" every-time i use the UI to change to vxlan://untagged** *
>>>>> I did try vxlan://untagged parameter Got the message: Vlan parameter : vxlan://untagged is not in valid format
>>>>> I did try vxlan:// Got the message: Unable to convert to broadcast URI: vxlan://
>>>>> I did try vxlan:// Got the message: Vlan parameter : vxlan://100 is not in valid format'
>>>>>
>>>>> ***Issue 3) s-VM and Proxy come up but Public IP addresses are not being routed Advertised.***
>>>>> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x> [State-Running] [Agent State-Gray]
>>>>> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y> [State-Running] [Agent State-Gray]
>>>>> If both [State] and [Agent State] need to be green, What other debug can I enable ??
>>>>>
>>>>> **NOTE** The VXLAN parameter for the Public network was taken vxlan://1000, But are not routed.
>>>>>
>>>>> I have to use a Single management server as everything messes UP when I run a second Management server with the Same Database.
>>>>>
>>>>> I reiterate The initial VMs are running:
>>>>> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x>
>>>>> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y>
>>>>>
>>>>>
>>>>>
>>>>> I do not see the Public IP anywhere, not even advertised on vni1000 or any other, the public VNI is there was created by the system, but nothing is there - No IP addresses advertised
>>>>>
>>>>> I used VLAN/VNI vxlan://1000
>>>>>
>>>>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 master brvx-1000 state forwarding priority
>>>>> There are 2 VNETS assigned
>>>>>
>>>>> virsh domiflist s-285-VM
>>>>> Interface Type Source Model MAC
>>>>> --------------------------------------------------------------
>>>>> vnet3 bridge cloud0 virtio 0e:00:a9:fe:a1:14
>>>>> vnet4 bridge cloudbr1 virtio 1e:00:3f:00:03:33
>>>>> vnet5 bridge brvx-1000 virtio 1e:00:b8:00:0c:fe
>>>>>
>>>>> virsh domifaddr s-285-VM
>>>>> Name MAC address Protocol Address
>>>>> -------------------------------------------------------------------------------
>>>>> ----- Nothing here -----
>>>>>
>>>>>
>>>>> the command arp -n, does not show Public IP addresses neither the management IP addresses, just the private IP addresses and the <control-ip-169.x.x.x>
>>>>>
>>>>> sh bgp evpn route vni 1000
>>>>> BGP table version is 1, local router ID is 10.1.1.14
>>>>> Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
>>>>> Origin codes: i - IGP, e - EGP, ? - incomplete
>>>>> EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
>>>>> EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
>>>>> EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
>>>>> EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
>>>>> EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
>>>>>
>>>>> Network Next Hop Metric LocPrf Weight Path
>>>>> *> [3]:[0]:[32]:[10.1.1.14]
>>>>> 10.1.1.14(hv1)
>>>>> 32768 i
>>>>> ET:8 RT:5641:1000
>>>>>
>>>>> Displayed 1 prefixes (1 paths)
>>>>>
>>>>> At the Agent side only WARNINGS I am not using uefi.
>>>>>
>>>>> 2025-04-03 00:01:44,744 ERROR [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
>>>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: []) (logid:) Process [15539] for command [/bin/bash -c virt-v2v -- version ] encountered the error: [127].
>>>>> 2025-04-03 00:01:47,921 ERROR [cloud.agent.Agent] (agentRequest- Handler-5:[]) (logid:09929bd6) Unexpected arch null, expected x86_64
>>>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: []) (logid:) Execution of process [15539] for command [/bin/bash -c virt-v2v --version ] failed.
>>>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: []) (logid:) Process [15539] for command [/bin/bash -c virt-v2v -- version ] encountered the error: [127].
>>>>>
>>>>>
>>>>> AT the management the rror is:
>>>>> 2025-04-03 00:00:28,115 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Executing ClusterServicePdu with service URL: https://10.1.1.1:9090/clusterservice
>>>>> 2025-04-03 00:21:28,122 ERROR [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Exception from : https://10.1.1.1:9090/clusterservice, method : null, exception : [javax.net](http://javax.net/).ssl.SSLPeerUnverifiedException: Certificate for <10.1.1.1> doesn't match any of the subject alternative names: [fde0:f:2897:1:1:0:0:1, 172.1.1.1, fe80:0:0:0:e643:4bff:fe81:9660, cs1.myinternaldomain.int, cloudstack.internal]
>>>>>
>>>>>
>>>>> Tata Y.
>>>
>>
>
More bugs seems related to #9920 Re: CloudStack 4.20 VXLAN-EVPN - Management VXLAN generates error - Public VXLAN not advertising - Initial VMs running but Grey State for Agent State
Posted by Chi vediamo <ta...@gmail.com>.
I created this one with the pictures,
https://github.com/apache/cloudstack/issues/10659
Pearl opened somethis similar: https://github.com/apache/cloudstack/issues/9920
Which seems related to one Pearl was working, but if this is for 4.21 is unrealistic from the time line perspective for me !!! and Wido have this working Something is really messed up !
Additionally: Issue 4) The VNI's I Choose for Guest are showing as VLANs, Wido what did you put on guest Vlan Range and Which Format Or nothing at all.
Wido can you share which version of cloudstack are you running?
Tata Y.
> On Apr 4, 2025, at 6:27 AM, Chi vediamo <ta...@gmail.com> wrote:
>
>
> The cloud0 was added by the system, i did not created that.
> I do not have a cloudbr0
>
> what does it show for your in your agent.properties: public.network.device and guest.network.properties
>
>
> My agent.properties
> private.network.device=cloudbr1
> guest.network.device=cloudbr1
> hypervisor.type=kvm
> public.network.device=cloudbr1
> host=10.1.1.1@static
> guid=579fafc9-5706-31f1-ae98-9f041ac90c80
> workers=5
>
> the vxlan1000 I configured in the UI for the PUblic network and the brvx-1000 that was added by Cloudstack, I did not do it manually
>>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 qdisc noqueue master brvx-1000 state UNKNOWN group default qlen 1000
>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>>> brvx-1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP group default qlen 1000
>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>
>
> I created a vxlan100 for the management and the management IPs are being propagates
>
> my VNi
> ~# vtysh -c 'sh evpn vni'
> VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs Tenant VRF
> 1000 L2 vxlan1000 2 0 0 default <<< Added by Cloudstack Public but no Public IPs advertised
> 100 L2 vxlan100 7 11 4 default <<< created manually for the initial clourbr1 and management sec storage
> 222 L2 vxlan222 10 20 9 default <<< Separate vxlan for primary storage, its working
>
> My concerns are: Is normal the UI to show agent state gray for the 2 initial SYSTEM-VMs the proxy and the secndarystorage
>>>> ***Issue 3) s-VM and Proxy come up but Public IP addresses are not being routed Advertised.***
>>>> v-154-VM <public-IP-3> <Private-IP-2> <control-ip-169.x.x.x> [State-Running] [Agent State-Gray]
>>>> s-285-VM <public-IP-2> <Private-IP-3> <control-ip-169.x.x.y> [State-Running] [Agent State-Gray]
>
> And where are my public IPs that should be added to the vxlan1000 ?
>
>
> Also the UI does not allow me to add Public IPv6 address not even using square brackets []
>
> I can send snapshots but apache.org removes them.
>
> I know it should be a little detail but I do not know what I am missing, or what is the normal status for the UI buttons.
>
>
> Can any of you send me a snapshot of the UI configuration for the ZONe
>
> and for your zone for example Attached a PDF with the snapshots of mine.
> the network storage I did not add any, cloudstack added itself to the cloudbr1
>
> Also How do I Assign Public IPv6 Addresses ?
>
> thank you
>
> Tata Y.
>
> see PDF attached
>
> <TataY-Cloudstack-VXLAN-UI-2025-V1.pdf>
>
>
>
>> On Apr 4, 2025, at 5:00 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>>
>> Let me post a couple of outputs of a hypervisor we run:
>>
>>
>> 9: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr1 state UNKNOWN group default qlen 1000
>> link/ether 8e:7d:67:a7:c9:72 brd ff:ff:ff:ff:ff:ff
>> inet6 fe80::8c7d:67ff:fea7:c972/64 scope link
>> valid_lft forever preferred_lft forever
>> 10: cloudbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
>> link/ether 1e:11:99:1e:6d:46 brd ff:ff:ff:ff:ff:ff
>> inet 10.100.2.2/20 scope global cloudbr1
>> valid_lft forever preferred_lft forever
>>
>>
>> As you can see there is a /20 IPv4 allocated there. The agent.properties shows:
>>
>> host=10.100.1.6@static
>> control.cidr=169.254.240.0/20
>> private.network.device=cloudbr1
>> guid=44e94fad-4aec-414e-8cac-cea3264dd431
>>
>> Those are the relevant ones in this case.
>>
>>
>> 6: enxb8599f206ba6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP group default qlen 1000
>> link/ether b8:59:9f:20:6b:a6 brd ff:ff:ff:ff:ff:ff
>> inet6 2a05:xxxx:601:117:ba59:9fff:fe20:6ba6/64 scope global deprecated dynamic mngtmpaddr
>> valid_lft 3592sec preferred_lft 0sec
>> inet6 fe80::ba59:9fff:fe20:6ba6/64 scope link
>> valid_lft forever preferred_lft forever
>> 7: enxb8599f206ba7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP group default qlen 1000
>> link/ether b8:59:9f:20:6b:a7 brd ff:ff:ff:ff:ff:ff
>> inet6 2a05:xxxx:601:197:ba59:9fff:fe20:6ba7/64 scope global deprecated dynamic mngtmpaddr
>> valid_lft 3594sec preferred_lft 0sec
>> inet6 fe80::ba59:9fff:fe20:6ba7/64 scope link
>> valid_lft forever preferred_lft forever
>>
>>
>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>> inet 127.0.0.1/8 scope host lo
>> valid_lft forever preferred_lft forever
>> inet 10.255.255.10/32 brd 10.255.255.10 scope global lo
>> valid_lft forever preferred_lft forever
>> inet6 2a05:xxxx:601:2::10/128 scope global
>> valid_lft forever preferred_lft forever
>> inet6 ::1/128 scope host
>> valid_lft forever preferred_lft forever
>>
>>
>> With two BGP sessions over these two uplinks:
>>
>> L2VPN EVPN Summary:
>> BGP router identifier 10.255.255.10, local AS number 4200100124 VRF default vrf-id 0
>> BGP table version 0
>> RIB entries 4341, using 543 KiB of memory
>> Peers 2, using 48 KiB of memory
>> Peer groups 1, using 64 bytes of memory
>>
>> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
>> enxb8599f206ba6 4 4200100002 6027297 5839388 39828 0 0 28w1d10h 6635 6706 N/A
>> enxb8599f206ba7 4 4200100002 6035334 5835598 39828 0 0 28w1d10h 6635 6706 N/A
>>
>>
>>
>> That's all! There is no cloudbr0 on the hypervisor at all.
>>
>> hv-138-a05-zz.xxxx.cldin.net# sh evpn vni
>> VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs Tenant VRF
>> 539 L2 vxlan539 19 8 15 default
>> 100 L2 vxlan100 107 106 103 default
>> 528 L2 vxlan528 92 8 36 default
>> 507 L2 vxlan507 43 8 22 default
>> ....
>> ....
>> hv-138-a05-zz.xxxx.cldin.net#
>>
>>
>> Wido
>>
>> On 4/4/25 04:07, Chi vediamo wrote:
>>> I decided to reinstall everything clean again, drop the tables completely and start all over again.
>>> The management IPs I see them routed. even the ones for the s-VM and v- VM. I see them on the transport switches.
>>> The Public VXLAN is created, the Public IP address I am not able to see them anywhere !
>>> FROM THE UI: If I run a diagnostic Ping to the public IP for the secondarystoragevm or for the consoleproxy - success !!!
>>> Why or how is not being routed ? How can I troubleshoot this?
>>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 qdisc noqueue master brvx-1000 state UNKNOWN group default qlen 1000
>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>>> brvx-1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP group default qlen 1000
>>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>>> Any tip will be appreciated.
>>> Thank you Teams.
>>> Tata Y.
>>>> On Apr 3, 2025, at 12:47 PM, Chi vediamo <ta...@gmail.com> wrote:
>>>>
>>>>
>>>> Using [Network 1] Only for all three [Management], [guest],[public] Tagged as VXLAN and with cloudbr1
>>>>
>>>> *- **Should I use a separate Network for each ? ? ***
>>>>
>>>> What should I check, in addition to the official manual: I reread again the instructions and saw videos
>>>> https://www.youtube.com/watch?v=9gXEmWbgX2o <<Actually this video shown both VMs are down'
>>>> https://www.youtube.com/watch?v=DlJg3LYvIIs << this shows both on green.
>>>> https://www.youtube.com/watch?v=vZfHfoYEMdY << this one shows the [Agent State] green while the Vms are in starting mode with a bue dot.
>>>> I have Read the https://rohityadav.cloud/blog/cloudstack-kvm/
>>>>
>>>> - Management servers are two physical separated servers. all Ubuntu 22.04
>>>> - DB is mysql running on a separate server. Mysql 8
>>>> - Hypervisor is running on a separate server. Ubuntu22.0 KVM
>>>> - Cloudstack 4.20.0.0
>>>> - Primary Storage CEPH RDB
>>>> - Secondary storage EMC NFS
>>>>
>>>>
>>>> ***Issue 1) I can't add a second Management server, No idea why !!** *
>>>> Thank you Wei, as soon i shut the second Management server the initial VMs proxy and secondary storage come up
>>>>
>>>> ***Issue 2) Management Network is Tagged as VXLAN, the UI shows vlan://untagged - "no typo vlan" every-time i use the UI to change to vxlan://untagged** *
>>>> I did try vxlan://untagged parameter Got the message: Vlan parameter : vxlan://untagged is not in valid format
>>>> I did try vxlan:// Got the message: Unable to convert to broadcast URI: vxlan://
>>>> I did try vxlan:// Got the message: Vlan parameter : vxlan://100 is not in valid format'
>>>>
>>>> ***Issue 3) s-VM and Proxy come up but Public IP addresses are not being routed Advertised.***
>>>> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x> [State-Running] [Agent State-Gray]
>>>> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y> [State-Running] [Agent State-Gray]
>>>> If both [State] and [Agent State] need to be green, What other debug can I enable ??
>>>>
>>>> **NOTE** The VXLAN parameter for the Public network was taken vxlan://1000, But are not routed.
>>>>
>>>> I have to use a Single management server as everything messes UP when I run a second Management server with the Same Database.
>>>>
>>>> I reiterate The initial VMs are running:
>>>> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x>
>>>> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y>
>>>>
>>>>
>>>>
>>>> I do not see the Public IP anywhere, not even advertised on vni1000 or any other, the public VNI is there was created by the system, but nothing is there - No IP addresses advertised
>>>>
>>>> I used VLAN/VNI vxlan://1000
>>>>
>>>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 master brvx-1000 state forwarding priority
>>>> There are 2 VNETS assigned
>>>>
>>>> virsh domiflist s-285-VM
>>>> Interface Type Source Model MAC
>>>> --------------------------------------------------------------
>>>> vnet3 bridge cloud0 virtio 0e:00:a9:fe:a1:14
>>>> vnet4 bridge cloudbr1 virtio 1e:00:3f:00:03:33
>>>> vnet5 bridge brvx-1000 virtio 1e:00:b8:00:0c:fe
>>>>
>>>> virsh domifaddr s-285-VM
>>>> Name MAC address Protocol Address
>>>> -------------------------------------------------------------------------------
>>>> ----- Nothing here -----
>>>>
>>>>
>>>> the command arp -n, does not show Public IP addresses neither the management IP addresses, just the private IP addresses and the <control-ip-169.x.x.x>
>>>>
>>>> sh bgp evpn route vni 1000
>>>> BGP table version is 1, local router ID is 10.1.1.14
>>>> Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
>>>> Origin codes: i - IGP, e - EGP, ? - incomplete
>>>> EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
>>>> EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
>>>> EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
>>>> EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
>>>> EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
>>>>
>>>> Network Next Hop Metric LocPrf Weight Path
>>>> *> [3]:[0]:[32]:[10.1.1.14]
>>>> 10.1.1.14(hv1)
>>>> 32768 i
>>>> ET:8 RT:5641:1000
>>>>
>>>> Displayed 1 prefixes (1 paths)
>>>>
>>>> At the Agent side only WARNINGS I am not using uefi.
>>>>
>>>> 2025-04-03 00:01:44,744 ERROR [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
>>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: []) (logid:) Process [15539] for command [/bin/bash -c virt-v2v -- version ] encountered the error: [127].
>>>> 2025-04-03 00:01:47,921 ERROR [cloud.agent.Agent] (agentRequest- Handler-5:[]) (logid:09929bd6) Unexpected arch null, expected x86_64
>>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: []) (logid:) Execution of process [15539] for command [/bin/bash -c virt-v2v --version ] failed.
>>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: []) (logid:) Process [15539] for command [/bin/bash -c virt-v2v -- version ] encountered the error: [127].
>>>>
>>>>
>>>> AT the management the rror is:
>>>> 2025-04-03 00:00:28,115 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Executing ClusterServicePdu with service URL: https://10.1.1.1:9090/clusterservice
>>>> 2025-04-03 00:21:28,122 ERROR [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Exception from : https://10.1.1.1:9090/clusterservice, method : null, exception : [javax.net](http://javax.net/).ssl.SSLPeerUnverifiedException: Certificate for <10.1.1.1> doesn't match any of the subject alternative names: [fde0:f:2897:1:1:0:0:1, 172.1.1.1, fe80:0:0:0:e643:4bff:fe81:9660, cs1.myinternaldomain.int, cloudstack.internal]
>>>>
>>>>
>>>> Tata Y.
>>
>
Re: CloudStack 4.20 VXLAN-EVPN - Management VXLAN generates error - Public VXLAN not advertising - Initial VMs running but Grey State for Agent State
Posted by Chi vediamo <ta...@gmail.com>.
The cloud0 was added by the system, i did not created that.
I do not have a cloudbr0
what does it show for your in your agent.properties: public.network.device and guest.network.properties
My agent.properties
private.network.device=cloudbr1
guest.network.device=cloudbr1
hypervisor.type=kvm
public.network.device=cloudbr1
host=10.1.1.1@static
guid=579fafc9-5706-31f1-ae98-9f041ac90c80
workers=5
the vxlan1000 I configured in the UI for the PUblic network and the brvx-1000 that was added by Cloudstack, I did not do it manually
>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 qdisc noqueue master brvx-1000 state UNKNOWN group default qlen 1000
>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>> brvx-1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP group default qlen 1000
>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
I created a vxlan100 for the management and the management IPs are being propagates
my VNi
~# vtysh -c 'sh evpn vni'
VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs Tenant VRF
1000 L2 vxlan1000 2 0 0 default <<< Added by Cloudstack Public but no Public IPs advertised
100 L2 vxlan100 7 11 4 default <<< created manually for the initial clourbr1 and management sec storage
222 L2 vxlan222 10 20 9 default <<< Separate vxlan for primary storage, its working
My concerns are: Is normal the UI to show agent state gray for the 2 initial SYSTEM-VMs the proxy and the secndarystorage
>>> ***Issue 3) s-VM and Proxy come up but Public IP addresses are not being routed Advertised.***
>>> v-154-VM <public-IP-3> <Private-IP-2> <control-ip-169.x.x.x> [State-Running] [Agent State-Gray]
>>> s-285-VM <public-IP-2> <Private-IP-3> <control-ip-169.x.x.y> [State-Running] [Agent State-Gray]
And where are my public IPs that should be added to the vxlan1000 ?
Also the UI does not allow me to add Public IPv6 address not even using square brackets []
I can send snapshots but apache.org removes them.
I know it should be a little detail but I do not know what I am missing, or what is the normal status for the UI buttons.
Can any of you send me a snapshot of the UI configuration for the ZONe
and for your zone for example Attached a PDF with the snapshots of mine.
the network storage I did not add any, cloudstack added itself to the cloudbr1
Also How do I Assign Public IPv6 Addresses ?
thank you
Tata Y.
see PDF attached

> On Apr 4, 2025, at 5:00 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>
> Let me post a couple of outputs of a hypervisor we run:
>
>
> 9: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloudbr1 state UNKNOWN group default qlen 1000
> link/ether 8e:7d:67:a7:c9:72 brd ff:ff:ff:ff:ff:ff
> inet6 fe80::8c7d:67ff:fea7:c972/64 scope link
> valid_lft forever preferred_lft forever
> 10: cloudbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
> link/ether 1e:11:99:1e:6d:46 brd ff:ff:ff:ff:ff:ff
> inet 10.100.2.2/20 scope global cloudbr1
> valid_lft forever preferred_lft forever
>
>
> As you can see there is a /20 IPv4 allocated there. The agent.properties shows:
>
> host=10.100.1.6@static
> control.cidr=169.254.240.0/20
> private.network.device=cloudbr1
> guid=44e94fad-4aec-414e-8cac-cea3264dd431
>
> Those are the relevant ones in this case.
>
>
> 6: enxb8599f206ba6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP group default qlen 1000
> link/ether b8:59:9f:20:6b:a6 brd ff:ff:ff:ff:ff:ff
> inet6 2a05:xxxx:601:117:ba59:9fff:fe20:6ba6/64 scope global deprecated dynamic mngtmpaddr
> valid_lft 3592sec preferred_lft 0sec
> inet6 fe80::ba59:9fff:fe20:6ba6/64 scope link
> valid_lft forever preferred_lft forever
> 7: enxb8599f206ba7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP group default qlen 1000
> link/ether b8:59:9f:20:6b:a7 brd ff:ff:ff:ff:ff:ff
> inet6 2a05:xxxx:601:197:ba59:9fff:fe20:6ba7/64 scope global deprecated dynamic mngtmpaddr
> valid_lft 3594sec preferred_lft 0sec
> inet6 fe80::ba59:9fff:fe20:6ba7/64 scope link
> valid_lft forever preferred_lft forever
>
>
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> inet 10.255.255.10/32 brd 10.255.255.10 scope global lo
> valid_lft forever preferred_lft forever
> inet6 2a05:xxxx:601:2::10/128 scope global
> valid_lft forever preferred_lft forever
> inet6 ::1/128 scope host
> valid_lft forever preferred_lft forever
>
>
> With two BGP sessions over these two uplinks:
>
> L2VPN EVPN Summary:
> BGP router identifier 10.255.255.10, local AS number 4200100124 VRF default vrf-id 0
> BGP table version 0
> RIB entries 4341, using 543 KiB of memory
> Peers 2, using 48 KiB of memory
> Peer groups 1, using 64 bytes of memory
>
> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
> enxb8599f206ba6 4 4200100002 6027297 5839388 39828 0 0 28w1d10h 6635 6706 N/A
> enxb8599f206ba7 4 4200100002 6035334 5835598 39828 0 0 28w1d10h 6635 6706 N/A
>
>
>
> That's all! There is no cloudbr0 on the hypervisor at all.
>
> hv-138-a05-zz.xxxx.cldin.net# sh evpn vni
> VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs Tenant VRF
> 539 L2 vxlan539 19 8 15 default
> 100 L2 vxlan100 107 106 103 default
> 528 L2 vxlan528 92 8 36 default
> 507 L2 vxlan507 43 8 22 default
> ....
> ....
> hv-138-a05-zz.xxxx.cldin.net#
>
>
> Wido
>
> On 4/4/25 04:07, Chi vediamo wrote:
>> I decided to reinstall everything clean again, drop the tables completely and start all over again.
>> The management IPs I see them routed. even the ones for the s-VM and v- VM. I see them on the transport switches.
>> The Public VXLAN is created, the Public IP address I am not able to see them anywhere !
>> FROM THE UI: If I run a diagnostic Ping to the public IP for the secondarystoragevm or for the consoleproxy - success !!!
>> Why or how is not being routed ? How can I troubleshoot this?
>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 qdisc noqueue master brvx-1000 state UNKNOWN group default qlen 1000
>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>> brvx-1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP group default qlen 1000
>> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>> Any tip will be appreciated.
>> Thank you Teams.
>> Tata Y.
>>> On Apr 3, 2025, at 12:47 PM, Chi vediamo <ta...@gmail.com> wrote:
>>>
>>>
>>> Using [Network 1] Only for all three [Management], [guest],[public] Tagged as VXLAN and with cloudbr1
>>>
>>> *- **Should I use a separate Network for each ? ? ***
>>>
>>> What should I check, in addition to the official manual: I reread again the instructions and saw videos
>>> https://www.youtube.com/watch?v=9gXEmWbgX2o <<Actually this video shown both VMs are down'
>>> https://www.youtube.com/watch?v=DlJg3LYvIIs << this shows both on green.
>>> https://www.youtube.com/watch?v=vZfHfoYEMdY << this one shows the [Agent State] green while the Vms are in starting mode with a bue dot.
>>> I have Read the https://rohityadav.cloud/blog/cloudstack-kvm/
>>>
>>> - Management servers are two physical separated servers. all Ubuntu 22.04
>>> - DB is mysql running on a separate server. Mysql 8
>>> - Hypervisor is running on a separate server. Ubuntu22.0 KVM
>>> - Cloudstack 4.20.0.0
>>> - Primary Storage CEPH RDB
>>> - Secondary storage EMC NFS
>>>
>>>
>>> ***Issue 1) I can't add a second Management server, No idea why !!** *
>>> Thank you Wei, as soon i shut the second Management server the initial VMs proxy and secondary storage come up
>>>
>>> ***Issue 2) Management Network is Tagged as VXLAN, the UI shows vlan://untagged - "no typo vlan" every-time i use the UI to change to vxlan://untagged** *
>>> I did try vxlan://untagged parameter Got the message: Vlan parameter : vxlan://untagged is not in valid format
>>> I did try vxlan:// Got the message: Unable to convert to broadcast URI: vxlan://
>>> I did try vxlan:// Got the message: Vlan parameter : vxlan://100 is not in valid format'
>>>
>>> ***Issue 3) s-VM and Proxy come up but Public IP addresses are not being routed Advertised.***
>>> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x> [State-Running] [Agent State-Gray]
>>> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y> [State-Running] [Agent State-Gray]
>>> If both [State] and [Agent State] need to be green, What other debug can I enable ??
>>>
>>> **NOTE** The VXLAN parameter for the Public network was taken vxlan://1000, But are not routed.
>>>
>>> I have to use a Single management server as everything messes UP when I run a second Management server with the Same Database.
>>>
>>> I reiterate The initial VMs are running:
>>> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x>
>>> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y>
>>>
>>>
>>>
>>> I do not see the Public IP anywhere, not even advertised on vni1000 or any other, the public VNI is there was created by the system, but nothing is there - No IP addresses advertised
>>>
>>> I used VLAN/VNI vxlan://1000
>>>
>>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 master brvx-1000 state forwarding priority
>>> There are 2 VNETS assigned
>>>
>>> virsh domiflist s-285-VM
>>> Interface Type Source Model MAC
>>> --------------------------------------------------------------
>>> vnet3 bridge cloud0 virtio 0e:00:a9:fe:a1:14
>>> vnet4 bridge cloudbr1 virtio 1e:00:3f:00:03:33
>>> vnet5 bridge brvx-1000 virtio 1e:00:b8:00:0c:fe
>>>
>>> virsh domifaddr s-285-VM
>>> Name MAC address Protocol Address
>>> -------------------------------------------------------------------------------
>>> ----- Nothing here -----
>>>
>>>
>>> the command arp -n, does not show Public IP addresses neither the management IP addresses, just the private IP addresses and the <control-ip-169.x.x.x>
>>>
>>> sh bgp evpn route vni 1000
>>> BGP table version is 1, local router ID is 10.1.1.14
>>> Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
>>> Origin codes: i - IGP, e - EGP, ? - incomplete
>>> EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
>>> EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
>>> EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
>>> EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
>>> EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
>>>
>>> Network Next Hop Metric LocPrf Weight Path
>>> *> [3]:[0]:[32]:[10.1.1.14]
>>> 10.1.1.14(hv1)
>>> 32768 i
>>> ET:8 RT:5641:1000
>>>
>>> Displayed 1 prefixes (1 paths)
>>>
>>> At the Agent side only WARNINGS I am not using uefi.
>>>
>>> 2025-04-03 00:01:44,744 ERROR [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: []) (logid:) Process [15539] for command [/bin/bash -c virt-v2v -- version ] encountered the error: [127].
>>> 2025-04-03 00:01:47,921 ERROR [cloud.agent.Agent] (agentRequest- Handler-5:[]) (logid:09929bd6) Unexpected arch null, expected x86_64
>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: []) (logid:) Execution of process [15539] for command [/bin/bash -c virt-v2v --version ] failed.
>>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1: []) (logid:) Process [15539] for command [/bin/bash -c virt-v2v -- version ] encountered the error: [127].
>>>
>>>
>>> AT the management the rror is:
>>> 2025-04-03 00:00:28,115 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Executing ClusterServicePdu with service URL: https://10.1.1.1:9090/clusterservice
>>> 2025-04-03 00:21:28,122 ERROR [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Exception from : https://10.1.1.1:9090/clusterservice, method : null, exception : [javax.net](http://javax.net/).ssl.SSLPeerUnverifiedException: Certificate for <10.1.1.1> doesn't match any of the subject alternative names: [fde0:f:2897:1:1:0:0:1, 172.1.1.1, fe80:0:0:0:e643:4bff:fe81:9660, cs1.myinternaldomain.int, cloudstack.internal]
>>>
>>>
>>> Tata Y.
>
Re: CloudStack 4.20 VXLAN-EVPN - Management VXLAN generates error - Public VXLAN not advertising - Initial VMs running but Grey State for Agent State
Posted by Wido den Hollander <wi...@widodh.nl.INVALID>.
Let me post a couple of outputs of a hypervisor we run:
9: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
master cloudbr1 state UNKNOWN group default qlen 1000
link/ether 8e:7d:67:a7:c9:72 brd ff:ff:ff:ff:ff:ff
inet6 fe80::8c7d:67ff:fea7:c972/64 scope link
valid_lft forever preferred_lft forever
10: cloudbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP group default qlen 1000
link/ether 1e:11:99:1e:6d:46 brd ff:ff:ff:ff:ff:ff
inet 10.100.2.2/20 scope global cloudbr1
valid_lft forever preferred_lft forever
As you can see there is a /20 IPv4 allocated there. The agent.properties
shows:
host=10.100.1.6@static
control.cidr=169.254.240.0/20
private.network.device=cloudbr1
guid=44e94fad-4aec-414e-8cac-cea3264dd431
Those are the relevant ones in this case.
6: enxb8599f206ba6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq
state UP group default qlen 1000
link/ether b8:59:9f:20:6b:a6 brd ff:ff:ff:ff:ff:ff
inet6 2a05:xxxx:601:117:ba59:9fff:fe20:6ba6/64 scope global
deprecated dynamic mngtmpaddr
valid_lft 3592sec preferred_lft 0sec
inet6 fe80::ba59:9fff:fe20:6ba6/64 scope link
valid_lft forever preferred_lft forever
7: enxb8599f206ba7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq
state UP group default qlen 1000
link/ether b8:59:9f:20:6b:a7 brd ff:ff:ff:ff:ff:ff
inet6 2a05:xxxx:601:197:ba59:9fff:fe20:6ba7/64 scope global
deprecated dynamic mngtmpaddr
valid_lft 3594sec preferred_lft 0sec
inet6 fe80::ba59:9fff:fe20:6ba7/64 scope link
valid_lft forever preferred_lft forever
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.255.255.10/32 brd 10.255.255.10 scope global lo
valid_lft forever preferred_lft forever
inet6 2a05:xxxx:601:2::10/128 scope global
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
With two BGP sessions over these two uplinks:
L2VPN EVPN Summary:
BGP router identifier 10.255.255.10, local AS number 4200100124 VRF
default vrf-id 0
BGP table version 0
RIB entries 4341, using 543 KiB of memory
Peers 2, using 48 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
Up/Down State/PfxRcd PfxSnt Desc
enxb8599f206ba6 4 4200100002 6027297 5839388 39828 0 0
28w1d10h 6635 6706 N/A
enxb8599f206ba7 4 4200100002 6035334 5835598 39828 0 0
28w1d10h 6635 6706 N/A
That's all! There is no cloudbr0 on the hypervisor at all.
hv-138-a05-zz.xxxx.cldin.net# sh evpn vni
VNI Type VxLAN IF # MACs # ARPs # Remote VTEPs
Tenant VRF
539 L2 vxlan539 19 8 15
default
100 L2 vxlan100 107 106 103
default
528 L2 vxlan528 92 8 36
default
507 L2 vxlan507 43 8 22
default
....
....
hv-138-a05-zz.xxxx.cldin.net#
Wido
On 4/4/25 04:07, Chi vediamo wrote:
> I decided to reinstall everything clean again, drop the tables
> completely and start all over again.
>
>
> The management IPs I see them routed. even the ones for the s-VM and v-
> VM. I see them on the transport switches.
>
>
> The Public VXLAN is created, the Public IP address I am not able to see
> them anywhere !
>
> FROM THE UI: If I run a diagnostic Ping to the public IP for the
> secondarystoragevm or for the consoleproxy - success !!!
>
> Why or how is not being routed ? How can I troubleshoot this?
>
>
> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 qdisc noqueue
> master brvx-1000 state UNKNOWN group default qlen 1000
> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
> brvx-1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue
> state UP group default qlen 1000
> link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
>
>
> Any tip will be appreciated.
>
>
> Thank you Teams.
>
> Tata Y.
>
>
>
>
>> On Apr 3, 2025, at 12:47 PM, Chi vediamo <ta...@gmail.com> wrote:
>>
>>
>> Using [Network 1] Only for all three [Management], [guest],[public]
>> Tagged as VXLAN and with cloudbr1
>>
>> *- **Should I use a separate Network for each ? ? ***
>>
>> What should I check, in addition to the official manual: I reread
>> again the instructions and saw videos
>> https://www.youtube.com/watch?v=9gXEmWbgX2o <<Actually this video
>> shown both VMs are down'
>> https://www.youtube.com/watch?v=DlJg3LYvIIs << this shows both on green.
>> https://www.youtube.com/watch?v=vZfHfoYEMdY << this one shows the
>> [Agent State] green while the Vms are in starting mode with a bue dot.
>> I have Read the https://rohityadav.cloud/blog/cloudstack-kvm/
>>
>> - Management servers are two physical separated servers. all Ubuntu 22.04
>> - DB is mysql running on a separate server. Mysql 8
>> - Hypervisor is running on a separate server. Ubuntu22.0 KVM
>> - Cloudstack 4.20.0.0
>> - Primary Storage CEPH RDB
>> - Secondary storage EMC NFS
>>
>>
>> ***Issue 1) I can't add a second Management server, No idea why !!** *
>> Thank you Wei, as soon i shut the second Management server
>> the initial VMs proxy and secondary storage come up
>>
>> ***Issue 2) Management Network is Tagged as VXLAN, the UI shows
>> vlan://untagged - "no typo vlan" every-time i use the UI to change to
>> vxlan://untagged** *
>> I did try vxlan://untagged parameter Got the message: Vlan
>> parameter : vxlan://untagged is not in valid format
>> I did try vxlan:// Got the
>> message: Unable to convert to broadcast URI: vxlan://
>> I did try vxlan:// Got the
>> message: Vlan parameter : vxlan://100 is not in valid format'
>>
>> ***Issue 3) s-VM and Proxy come up but Public IP addresses are not
>> being routed Advertised.***
>> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x>
>> [State-Running] [Agent State-Gray]
>> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y>
>> [State-Running] [Agent State-Gray]
>> If both [State] and [Agent State] need to be green, What other
>> debug can I enable ??
>>
>> **NOTE** The VXLAN parameter for the Public network was taken
>> vxlan://1000, But are not routed.
>>
>> I have to use a Single management server as everything messes UP when
>> I run a second Management server with the Same Database.
>>
>> I reiterate The initial VMs are running:
>> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x>
>> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y>
>>
>>
>>
>> I do not see the Public IP anywhere, not even advertised on vni1000 or
>> any other, the public VNI is there was created by the system, but
>> nothing is there - No IP addresses advertised
>>
>> I used VLAN/VNI vxlan://1000
>>
>> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 master brvx-1000
>> state forwarding priority
>> There are 2 VNETS assigned
>>
>> virsh domiflist s-285-VM
>> Interface Type Source Model MAC
>> --------------------------------------------------------------
>> vnet3 bridge cloud0 virtio 0e:00:a9:fe:a1:14
>> vnet4 bridge cloudbr1 virtio 1e:00:3f:00:03:33
>> vnet5 bridge brvx-1000 virtio 1e:00:b8:00:0c:fe
>>
>> virsh domifaddr s-285-VM
>> Name MAC address Protocol Address
>> -------------------------------------------------------------------------------
>> ----- Nothing here -----
>>
>>
>> the command arp -n, does not show Public IP addresses neither the
>> management IP addresses, just the private IP addresses and the
>> <control-ip-169.x.x.x>
>>
>> sh bgp evpn route vni 1000
>> BGP table version is 1, local router ID is 10.1.1.14
>> Status codes: s suppressed, d damped, h history, * valid, > best, i -
>> internal
>> Origin codes: i - IGP, e - EGP, ? - incomplete
>> EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
>> EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
>> EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
>> EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
>> EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
>>
>> Network Next Hop Metric LocPrf Weight Path
>> *> [3]:[0]:[32]:[10.1.1.14]
>> 10.1.1.14(hv1)
>> 32768 i
>> ET:8 RT:5641:1000
>>
>> Displayed 1 prefixes (1 paths)
>>
>> At the Agent side only WARNINGS I am not using uefi.
>>
>> 2025-04-03 00:01:44,744 ERROR [kvm.resource.LibvirtComputingResource]
>> (main:[]) (logid:) uefi properties file not found due to: Unable to
>> find file uefi.properties.
>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:
>> []) (logid:) Process [15539] for command [/bin/bash -c virt-v2v --
>> version ] encountered the error: [127].
>> 2025-04-03 00:01:47,921 ERROR [cloud.agent.Agent] (agentRequest-
>> Handler-5:[]) (logid:09929bd6) Unexpected arch null, expected x86_64
>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:
>> []) (logid:) Execution of process [15539] for command [/bin/bash -c
>> virt-v2v --version ] failed.
>> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:
>> []) (logid:) Process [15539] for command [/bin/bash -c virt-v2v --
>> version ] encountered the error: [127].
>>
>>
>> AT the management the rror is:
>> 2025-04-03 00:00:28,115 DEBUG [c.c.c.ClusterServiceServletImpl]
>> (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Executing
>> ClusterServicePdu with service URL: https://10.1.1.1:9090/clusterservice
>> 2025-04-03 00:21:28,122 ERROR [c.c.c.ClusterServiceServletImpl]
>> (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Exception from :
>> https://10.1.1.1:9090/clusterservice, method : null, exception :
>> [javax.net](http://javax.net/).ssl.SSLPeerUnverifiedException:
>> Certificate for <10.1.1.1> doesn't match any of the subject
>> alternative names: [fde0:f:2897:1:1:0:0:1, 172.1.1.1,
>> fe80:0:0:0:e643:4bff:fe81:9660, cs1.myinternaldomain.int,
>> cloudstack.internal]
>>
>>
>> Tata Y.
>
Re: CloudStack 4.20 VXLAN-EVPN - Management VXLAN generates error - Public VXLAN not advertising - Initial VMs running but Grey State for Agent State
Posted by Chi vediamo <ta...@gmail.com>.
I decided to reinstall everything clean again, drop the tables completely and start all over again.
The management IPs I see them routed. even the ones for the s-VM and v-VM. I see them on the transport switches.
The Public VXLAN is created, the Public IP address I am not able to see them anywhere !
FROM THE UI: If I run a diagnostic Ping to the public IP for the secondarystoragevm or for the consoleproxy - success !!!
Why or how is not being routed ? How can I troubleshoot this?
vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 qdisc noqueue master brvx-1000 state UNKNOWN group default qlen 1000
link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
brvx-1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc noqueue state UP group default qlen 1000
link/ether 1a:00:21:de:dc:bc brd ff:ff:ff:ff:ff:ff
Any tip will be appreciated.
Thank you Teams.
Tata Y.
> On Apr 3, 2025, at 12:47 PM, Chi vediamo <ta...@gmail.com> wrote:
>
>
> Using [Network 1] Only for all three [Management], [guest],[public] Tagged as VXLAN and with cloudbr1
>
> - **Should I use a separate Network for each ? ? **
>
> What should I check, in addition to the official manual: I reread again the instructions and saw videos
> https://www.youtube.com/watch?v=9gXEmWbgX2o <<Actually this video shown both VMs are down'
> https://www.youtube.com/watch?v=DlJg3LYvIIs << this shows both on green.
> https://www.youtube.com/watch?v=vZfHfoYEMdY << this one shows the [Agent State] green while the Vms are in starting mode with a bue dot.
> I have Read the https://rohityadav.cloud/blog/cloudstack-kvm/
>
> - Management servers are two physical separated servers. all Ubuntu 22.04
> - DB is mysql running on a separate server. Mysql 8
> - Hypervisor is running on a separate server. Ubuntu22.0 KVM
> - Cloudstack 4.20.0.0
> - Primary Storage CEPH RDB
> - Secondary storage EMC NFS
>
>
> **Issue 1) I can't add a second Management server, No idea why !!**
> Thank you Wei, as soon i shut the second Management server the initial VMs proxy and secondary storage come up
>
> **Issue 2) Management Network is Tagged as VXLAN, the UI shows vlan://untagged - "no typo vlan" every-time i use the UI to change to vxlan://untagged**
> I did try vxlan://untagged parameter Got the message: Vlan parameter : vxlan://untagged is not in valid format
> I did try vxlan:// Got the message: Unable to convert to broadcast URI: vxlan://
> I did try vxlan:// Got the message: Vlan parameter : vxlan://100 is not in valid format'
>
> **Issue 3) s-VM and Proxy come up but Public IP addresses are not being routed Advertised.**
> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x> [State-Running] [Agent State-Gray]
> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y> [State-Running] [Agent State-Gray]
> If both [State] and [Agent State] need to be green, What other debug can I enable ??
>
> **NOTE** The VXLAN parameter for the Public network was taken vxlan://1000, But are not routed.
>
> I have to use a Single management server as everything messes UP when I run a second Management server with the Same Database.
>
> I reiterate The initial VMs are running:
> v-154-VM <public-IP-3> <Private-IP> <control-ip-169.x.x.x>
> s-285-VM <public-IP-2> <Private-IP> <control-ip-169.x.x.y>
>
>
>
> I do not see the Public IP anywhere, not even advertised on vni1000 or any other, the public VNI is there was created by the system, but nothing is there - No IP addresses advertised
>
> I used VLAN/VNI vxlan://1000
>
> vxlan1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9050 master brvx-1000 state forwarding priority
> There are 2 VNETS assigned
>
> virsh domiflist s-285-VM
> Interface Type Source Model MAC
> --------------------------------------------------------------
> vnet3 bridge cloud0 virtio 0e:00:a9:fe:a1:14
> vnet4 bridge cloudbr1 virtio 1e:00:3f:00:03:33
> vnet5 bridge brvx-1000 virtio 1e:00:b8:00:0c:fe
>
> virsh domifaddr s-285-VM
> Name MAC address Protocol Address
> -------------------------------------------------------------------------------
> ----- Nothing here -----
>
>
> the command arp -n, does not show Public IP addresses neither the management IP addresses, just the private IP addresses and the <control-ip-169.x.x.x>
>
> sh bgp evpn route vni 1000
> BGP table version is 1, local router ID is 10.1.1.14
> Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
> Origin codes: i - IGP, e - EGP, ? - incomplete
> EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id]
> EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
> EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
> EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
> EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]
>
> Network Next Hop Metric LocPrf Weight Path
> *> [3]:[0]:[32]:[10.1.1.14]
> 10.1.1.14(hv1)
> 32768 i
> ET:8 RT:5641:1000
>
> Displayed 1 prefixes (1 paths)
>
> At the Agent side only WARNINGS I am not using uefi.
>
> 2025-04-03 00:01:44,744 ERROR [kvm.resource.LibvirtComputingResource] (main:[]) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:[]) (logid:) Process [15539] for command [/bin/bash -c virt-v2v --version ] encountered the error: [127].
> 2025-04-03 00:01:47,921 ERROR [cloud.agent.Agent] (agentRequest-Handler-5:[]) (logid:09929bd6) Unexpected arch null, expected x86_64
> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:[]) (logid:) Execution of process [15539] for command [/bin/bash -c virt-v2v --version ] failed.
> 2025-04-03 00:01:46,815 WARN [utils.script.Script] (Agent-Handler-1:[]) (logid:) Process [15539] for command [/bin/bash -c virt-v2v --version ] encountered the error: [127].
>
>
> AT the management the rror is:
> 2025-04-03 00:00:28,115 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Executing ClusterServicePdu with service URL: https://10.1.1.1:9090/clusterservice
> 2025-04-03 00:21:28,122 ERROR [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:[ctx-c96bca9e]) (logid:3fbff637) Exception from : https://10.1.1.1:9090/clusterservice, method : null, exception : [javax.net](http://javax.net/).ssl.SSLPeerUnverifiedException: Certificate for <10.1.1.1> doesn't match any of the subject alternative names: [fde0:f:2897:1:1:0:0:1, 172.1.1.1, fe80:0:0:0:e643:4bff:fe81:9660, cs1.myinternaldomain.int, cloudstack.internal]
>
>
> Tata Y.