You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by cr...@istream.today on 2021/09/13 12:18:06 UTC
virtual router stuck on starting - cloudstack 4.15.1 with kvm
Hello,
I'm trying to understand this error, I created new network with VPC in
a new zone where I added KVM hypervisor, and I have this issue.
2021-09-13 11:36:18,621 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
169.254.52.234
2021-09-13 11:36:21,684 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
169.254.52.234
Here is the full log : https://pastebin.com/0NwsccUc
root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922
Trying 169.254.156.179...
telnet: Unable to connect to remote host: Connection refused ( before
connection refused, I get no route to host, for sure the nic is not up )
root@usa-nyc-kvm-001:~# virsh list
Id Name State
---------------------------
2 v-4048-VM running
4 s-4050-VM running
7 r-4092-VM running
root@usa-nyc-kvm-001:~#
If I check the log inside the router, I see this (the router is also
responding very slow, waiting a lot to connect on it via virsh console):
r-4092-VM login: root
Password:
Login timed out after 60 seconds.
__?.o/ Apache CloudStack SystemVM 4.15.1
( )# https://cloudstack.apache.org
(___(_) Debian GNU/Linux 10 r-4093-VM ttyS0
r-4093-VM login: root
Password:
Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64
root@r-4093-VM:~# cat /var/log/cloud.log
Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config
Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping systemvm appliance
Mon 13 Sep 2021 11:50:57 AM UTC Starting guest services for kvm
Mon 13 Sep 2021 11:51:04 AM UTC acpiphp and pci_hotplug module already
compiled in
Mon 13 Sep 2021 11:52:59 AM UTC Received a new non-empty cmdline file from
qemu-guest-agent
Mon 13 Sep 2021 11:52:59 AM UTC Booting from cloudstack, remove old
configuration files in /etc/cloudstack/
Mon 13 Sep 2021 11:54:30 AM UTC Configuring systemvm type=vpcrouter
Mon 13 Sep 2021 11:54:30 AM UTC Setting up VPC virtual router system vm
Mon 13 Sep 2021 11:54:31 AM UTC Setting up apache web server for VPC
Mon 13 Sep 2021 11:55:05 AM UTC Processors = 1 Enable service = 0
Mon 13 Sep 2021 11:55:05 AM UTC cloud: disable rp_filter
Mon 13 Sep 2021 11:55:05 AM UTC disable rpfilter
Mon 13 Sep 2021 11:55:05 AM UTC cloud: enable_fwding = 1
Mon 13 Sep 2021 11:55:05 AM UTC enable_fwding = 1
Mon 13 Sep 2021 11:55:05 AM UTC cloud: enabling passive FTP for guest VMs
Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
2021-09-13 11:55:13,823 INFO update_config.py :: Processing incoming
file => cmd_line.json
2021-09-13 11:55:13,856 INFO Processing JSON file cmd_line.json
2021-09-13 11:55:13,858 INFO Continuing with the processing of file
'/var/cache/cloud/cmd_line.json'
2021-09-13 11:55:13,877 INFO Command of type cmdline received
2021-09-13 11:55:13,897 INFO Command of type ips received
2021-09-13 11:55:13,923 INFO Executing: ip addr show dev eth0
2021-09-13 11:55:13,975 INFO Executing: ip addr show dev eth0
2021-09-13 11:55:13,979 INFO Address found in DataBag ==> {u'public_ip':
u'169.254.41.129', u'one_to_one_nat': False, u'nic_dev_id': u'0',
u'network': u'169.254.0.0/16', u'netmask': u'255.255.0.0', u'source_nat':
False, u'broadcast': u'169.254.255.255', u'add': True, u'nw_type':
u'control', u'device': u'eth0', u'cidr': u'169.254.41.129/16', u'gateway':
u'', u'size': u'16'}
2021-09-13 11:55:13,979 INFO Address 169.254.41.129/16 on device eth0
already configured
2021-09-13 11:55:13,984 INFO Executing: ip addr show |grep -v secondary
2021-09-13 11:55:14,010 INFO Wrote edited file /etc/dnsmasq.d/cloud.conf
2021-09-13 11:55:14,010 INFO Updated file in-cache configuration
2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcphosts.txt
file did not change
2021-09-13 11:55:14,010 INFO Nothing to commit. The
/var/lib/misc/dnsmasq.leases file did not change
2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcpopts.txt
file did not change
2021-09-13 11:55:14,010 INFO Attempting to delete entries from
dnsmasq.leases file for VMs which are not on dhcphosts file
2021-09-13 11:55:14,010 ERROR Caught error while trying to delete entries
from dnsmasq.leases file: [Errno 2] No such file or directory:
'/etc/dhcphosts.txt'
2021-09-13 11:55:14,034 INFO Wrote edited file /etc/hosts
2021-09-13 11:55:14,044 INFO Updated file in-cache configuration
2021-09-13 11:55:14,044 INFO Updated hosts file
2021-09-13 11:55:14,046 INFO Executing: systemctl restart dnsmasq
2021-09-13 11:55:15,409 INFO Service dnsmasq restart
2021-09-13 11:55:15,453 INFO Executing: ip addr show |grep -v secondary
2021-09-13 11:55:15,832 INFO Nothing to commit. The
/etc/dnsmasq.d/cloud.conf file did not change
2021-09-13 11:55:15,957 INFO Nothing to commit. The /etc/dhcphosts.txt
file did not change
2021-09-13 11:55:15,963 INFO Nothing to commit. The
/var/lib/misc/dnsmasq.leases file did not change
2021-09-13 11:55:15,963 INFO Nothing to commit. The /etc/dhcpopts.txt
file did not change
2021-09-13 11:55:15,967 INFO Executing: systemctl is-active dnsmasq
2021-09-13 11:55:16,179 INFO Executing: systemctl reload dnsmasq
2021-09-13 11:55:16,665 INFO Service dnsmasq reload
2021-09-13 11:55:16,909 INFO Wrote edited file /etc/cron.d/process
2021-09-13 11:55:17,040 INFO Updated file in-cache configuration
2021-09-13 11:55:17,063 INFO Executing: iptables-save | grep
'^:FW_EGRESS_RULES' || iptables -t filter -N FW_EGRESS_RULES
2021-09-13 11:55:17,323 INFO Executing: iptables-save | grep '^-A
FW_EGRESS_RULES -j ACCEPT$' | sed 's/^-A/iptables -t filter -D/g' | bash
2021-09-13 11:55:17,697 INFO Executing: iptables -F FW_EGRESS_RULES
2021-09-13 11:55:17,990 INFO Executing: ipset -L | grep Name: | awk
{'print $2'} | ipset flush
2021-09-13 11:55:19,272 INFO Executing: ipset -L | grep Name: | awk
{'print $2'} | ipset destroy
2021-09-13 11:55:19,833 INFO Executing: iptables-save
2021-09-13 11:55:20,168 INFO Executing: iptables-save
2021-09-13 11:55:20,505 INFO Executing: ip6tables-save
2021-09-13 11:55:20,800 INFO Executing: systemctl stop conntrackd
2021-09-13 11:55:21,201 INFO Service conntrackd stop
2021-09-13 11:55:21,230 INFO Executing: systemctl stop keepalived
2021-09-13 11:55:21,578 INFO Service keepalived stop
2021-09-13 11:55:21,609 INFO Executing: mount
root@r-4093-VM:~#
Please ignore the router name, I did multiple tests to extract these logs, I
always see the same.
Regards,
Cristian
Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Posted by Wei ZHOU <us...@gmail.com>.
It is strange. The cpu/ram should be good enough for a VR.
anyway, good you have solved the problem.
-Wei
On Fri, 17 Sept 2021 at 11:46, <cr...@istream.today> wrote:
> I didn't test with lower values, but with 256 RAM and 500Mhz CPU is not
> working for me. (KVM ubuntu 20.04 ACS 4.15.1 )
>
> Regards,
> Cristian
>
> -----Original Message-----
> From: Daan Hoogland <da...@gmail.com>
> Sent: Friday, September 17, 2021 11:14 AM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm
>
> thanks for the update Christian,
> sounds strange that the router would require that much to start but glad
> it worked.
>
> On Wed, Sep 15, 2021 at 1:32 PM <cr...@istream.today> wrote:
>
> > Hello,
> >
> > I have found the issue, after I have changed the CPU and memory for
> > this " System Offering For Software Router - Local Storage' system
> > offering the router was deployed in few seconds without any issue.
> >
> > CPU mhz from 500 to 2000 and RAM from 256 to 2048.
> >
> >
> >
> > Regards,
> > Cristian
> >
> > -----Original Message-----
> > From: cristian.c@istream.today <cr...@istream.today>
> > Sent: Wednesday, September 15, 2021 10:18 AM
> > To: users@cloudstack.apache.org
> > Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with
> > kvm
> >
> > Any suggestion on this issue?
> >
> > Thank you.
> >
> > -----Original Message-----
> > From: cristian.c@istream.today <cr...@istream.today>
> > Sent: Monday, September 13, 2021 5:40 PM
> > To: users@cloudstack.apache.org
> > Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with
> > kvm
> >
> > Hi,
> >
> >
> > Yes, please see :
> >
> > root@usa-nyc-kvm-001:/var/lib/libvirt/images# libvirtd -v
> > 2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0,
> > package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri,
> > 16 Jul 2021 05:16:36 +0000)
> > 2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
> > 2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutputs:1654 :
> > outputs=1:stderr
> > 2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutput:1492 :
> > output=1:stderr
> > 2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0,
> > package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri,
> > 16 Jul 2021 05:16:36 +0000)
> > 2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
> > 2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 :
> OBJECT_NEW:
> > obj=0x55935a296b60 classname=virAccessManager
> > 2021-09-13 14:33:40.944+0000: 75403: debug :
> > virAccessManagerNewDriver:111
> > : Initialized with stack
> > 2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 :
> OBJECT_NEW:
> > obj=0x55935a297f90 classname=virAccessManager
> > 2021-09-13 14:33:40.944+0000: 75403: debug :
> > virAccessManagerNewDriver:111
> > : Initialized with none
> > 2021-09-13 14:33:40.944+0000: 75403: info : virObjectRef:386 :
> OBJECT_REF:
> > obj=0x55935a296b60
> > 2021-09-13 14:33:40.944+0000: 75403: info : virObjectUnref:348 :
> > OBJECT_UNREF: obj=0x55935a296b60
> > 2021-09-13 14:33:40.944+0000: 75403: debug : main:1177 : Decided on
> > pid file path '/run/libvirtd.pid'
> > 2021-09-13 14:33:40.944+0000: 75403: debug : main:1187 : Decided on
> > socket paths '/run/libvirt/libvirt-sock',
> > '/run/libvirt/libvirt-sock-ro' and '/run/libvirt/libvirt-admin-sock'
> > 2021-09-13 14:33:40.944+0000: 75403: error : virPidFileAcquirePath:367 :
> > Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily
> > unavailable
> > 2021-09-13 14:33:40.944+0000: 75403: debug : virFileClose:110 : Closed
> > fd 3
> >
> >
> > root@usa-nyc-kvm-001:/var/lib/libvirt/images#
> > /usr/bin/qemu-system-x86_64 --version QEMU emulator version 4.2.1
> > (Debian 1:4.2-3ubuntu6.17) Copyright
> > (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
> >
> >
> > root@usa-nyc-kvm-001:/var/lib/libvirt/images# lsb_release -a No LSB
> > modules are available.
> > Distributor ID: Ubuntu
> > Description: Ubuntu 20.04.2 LTS
> > Release: 20.04
> > Codename: focal
> >
> >
> > CPU :
> > vendor_id : AuthenticAMD
> > cpu family : 23
> > model : 49
> > model name : AMD EPYC 7552 48-Core Processor
> >
> >
> > BTW: the SSVM and Proxy console is running fine on this node..
> >
> >
> >
> > Thank you!
> >
> > -----Original Message-----
> > From: Andrija Panic <an...@gmail.com>
> > Sent: Monday, September 13, 2021 4:40 PM
> > To: users <us...@cloudstack.apache.org>
> > Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with
> > kvm
> >
> > Can you share qemu/libvirt version/flavour?
> >
> > On Mon, 13 Sept 2021 at 14:18, <cr...@istream.today> wrote:
> >
> > > Hello,
> > >
> > >
> > >
> > > I'm trying to understand this error, I created new network
> > > with VPC in a new zone where I added KVM hypervisor, and I have this
> issue.
> > >
> > >
> > >
> > >
> > >
> > > 2021-09-13 11:36:18,621 DEBUG
> > > [resource.virtualnetwork.VirtualRoutingResource]
> > > (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
> > > 169.254.52.234
> > >
> > > 2021-09-13 11:36:21,684 DEBUG
> > > [resource.virtualnetwork.VirtualRoutingResource]
> > > (agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
> > > 169.254.52.234
> > >
> > >
> > >
> > > Here is the full log : https://pastebin.com/0NwsccUc
> > >
> > >
> > >
> > > root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922
> > >
> > > Trying 169.254.156.179...
> > >
> > > telnet: Unable to connect to remote host: Connection refused (
> > > before connection refused, I get no route to host, for sure the nic
> > > is not up
> > > )
> > >
> > > root@usa-nyc-kvm-001:~# virsh list
> > >
> > > Id Name State
> > >
> > > ---------------------------
> > >
> > > 2 v-4048-VM running
> > >
> > > 4 s-4050-VM running
> > >
> > > 7 r-4092-VM running
> > >
> > >
> > >
> > > root@usa-nyc-kvm-001:~#
> > >
> > >
> > >
> > >
> > >
> > > If I check the log inside the router, I see this (the router is also
> > > responding very slow, waiting a lot to connect on it via virsh
> console):
> > >
> > >
> > >
> > > r-4092-VM login: root
> > >
> > > Password:
> > >
> > >
> > >
> > > Login timed out after 60 seconds.
> > >
> > >
> > >
> > >
> > >
> > > __?.o/ Apache CloudStack SystemVM 4.15.1
> > >
> > > ( )# https://cloudstack.apache.org
> > >
> > > (___(_) Debian GNU/Linux 10 r-4093-VM ttyS0
> > >
> > >
> > >
> > > r-4093-VM login: root
> > >
> > > Password:
> > >
> > > Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2
> > > (2021-01-30)
> > > x86_64
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > root@r-4093-VM:~# cat /var/log/cloud.log
> > >
> > > Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config
> > >
> > > Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping systemvm appliance
> > >
> > > Mon 13 Sep 2021 11:50:57 AM UTC Starting guest services for kvm
> > >
> > > Mon 13 Sep 2021 11:51:04 AM UTC acpiphp and pci_hotplug module
> > > already compiled in
> > >
> > > Mon 13 Sep 2021 11:52:59 AM UTC Received a new non-empty cmdline
> > > file from qemu-guest-agent
> > >
> > > Mon 13 Sep 2021 11:52:59 AM UTC Booting from cloudstack, remove old
> > > configuration files in /etc/cloudstack/
> > >
> > > Mon 13 Sep 2021 11:54:30 AM UTC Configuring systemvm type=vpcrouter
> > >
> > > Mon 13 Sep 2021 11:54:30 AM UTC Setting up VPC virtual router system
> > > vm
> > >
> > > Mon 13 Sep 2021 11:54:31 AM UTC Setting up apache web server for VPC
> > >
> > > Mon 13 Sep 2021 11:55:05 AM UTC Processors = 1 Enable service = 0
> > >
> > > Mon 13 Sep 2021 11:55:05 AM UTC cloud: disable rp_filter
> > >
> > > Mon 13 Sep 2021 11:55:05 AM UTC disable rpfilter
> > >
> > > Mon 13 Sep 2021 11:55:05 AM UTC cloud: enable_fwding = 1
> > >
> > > Mon 13 Sep 2021 11:55:05 AM UTC enable_fwding = 1
> > >
> > > Mon 13 Sep 2021 11:55:05 AM UTC cloud: enabling passive FTP for
> > > guest VMs
> > >
> > > Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
> > >
> > > Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
> > >
> > > 2021-09-13 11:55:13,823 INFO update_config.py :: Processing
> incoming
> > > file => cmd_line.json
> > >
> > > 2021-09-13 11:55:13,856 INFO Processing JSON file cmd_line.json
> > >
> > > 2021-09-13 11:55:13,858 INFO Continuing with the processing of file
> > > '/var/cache/cloud/cmd_line.json'
> > >
> > > 2021-09-13 11:55:13,877 INFO Command of type cmdline received
> > >
> > > 2021-09-13 11:55:13,897 INFO Command of type ips received
> > >
> > > 2021-09-13 11:55:13,923 INFO Executing: ip addr show dev eth0
> > >
> > > 2021-09-13 11:55:13,975 INFO Executing: ip addr show dev eth0
> > >
> > > 2021-09-13 11:55:13,979 INFO Address found in DataBag ==>
> > > {u'public_ip':
> > > u'169.254.41.129', u'one_to_one_nat': False, u'nic_dev_id': u'0',
> > > u'network': u'169.254.0.0/16', u'netmask': u'255.255.0.0',
> > u'source_nat':
> > > False, u'broadcast': u'169.254.255.255', u'add': True, u'nw_type':
> > > u'control', u'device': u'eth0', u'cidr': u'169.254.41.129/16',
> > u'gateway':
> > > u'', u'size': u'16'}
> > >
> > > 2021-09-13 11:55:13,979 INFO Address 169.254.41.129/16 on device
> > eth0
> > > already configured
> > >
> > > 2021-09-13 11:55:13,984 INFO Executing: ip addr show |grep -v
> > secondary
> > >
> > > 2021-09-13 11:55:14,010 INFO Wrote edited file
> > > /etc/dnsmasq.d/cloud.conf
> > >
> > > 2021-09-13 11:55:14,010 INFO Updated file in-cache configuration
> > >
> > > 2021-09-13 11:55:14,010 INFO Nothing to commit. The
> > /etc/dhcphosts.txt
> > > file did not change
> > >
> > > 2021-09-13 11:55:14,010 INFO Nothing to commit. The
> > > /var/lib/misc/dnsmasq.leases file did not change
> > >
> > > 2021-09-13 11:55:14,010 INFO Nothing to commit. The
> /etc/dhcpopts.txt
> > > file did not change
> > >
> > > 2021-09-13 11:55:14,010 INFO Attempting to delete entries from
> > > dnsmasq.leases file for VMs which are not on dhcphosts file
> > >
> > > 2021-09-13 11:55:14,010 ERROR Caught error while trying to delete
> > > entries
> > > from dnsmasq.leases file: [Errno 2] No such file or directory:
> > > '/etc/dhcphosts.txt'
> > >
> > > 2021-09-13 11:55:14,034 INFO Wrote edited file /etc/hosts
> > >
> > > 2021-09-13 11:55:14,044 INFO Updated file in-cache configuration
> > >
> > > 2021-09-13 11:55:14,044 INFO Updated hosts file
> > >
> > > 2021-09-13 11:55:14,046 INFO Executing: systemctl restart dnsmasq
> > >
> > > 2021-09-13 11:55:15,409 INFO Service dnsmasq restart
> > >
> > > 2021-09-13 11:55:15,453 INFO Executing: ip addr show |grep -v
> > secondary
> > >
> > > 2021-09-13 11:55:15,832 INFO Nothing to commit. The
> > > /etc/dnsmasq.d/cloud.conf file did not change
> > >
> > > 2021-09-13 11:55:15,957 INFO Nothing to commit. The
> > /etc/dhcphosts.txt
> > > file did not change
> > >
> > > 2021-09-13 11:55:15,963 INFO Nothing to commit. The
> > > /var/lib/misc/dnsmasq.leases file did not change
> > >
> > > 2021-09-13 11:55:15,963 INFO Nothing to commit. The
> /etc/dhcpopts.txt
> > > file did not change
> > >
> > > 2021-09-13 11:55:15,967 INFO Executing: systemctl is-active dnsmasq
> > >
> > > 2021-09-13 11:55:16,179 INFO Executing: systemctl reload dnsmasq
> > >
> > > 2021-09-13 11:55:16,665 INFO Service dnsmasq reload
> > >
> > > 2021-09-13 11:55:16,909 INFO Wrote edited file /etc/cron.d/process
> > >
> > > 2021-09-13 11:55:17,040 INFO Updated file in-cache configuration
> > >
> > > 2021-09-13 11:55:17,063 INFO Executing: iptables-save | grep
> > > '^:FW_EGRESS_RULES' || iptables -t filter -N FW_EGRESS_RULES
> > >
> > > 2021-09-13 11:55:17,323 INFO Executing: iptables-save | grep '^-A
> > > FW_EGRESS_RULES -j ACCEPT$' | sed 's/^-A/iptables -t filter -D/g' |
> > > bash
> > >
> > > 2021-09-13 11:55:17,697 INFO Executing: iptables -F FW_EGRESS_RULES
> > >
> > > 2021-09-13 11:55:17,990 INFO Executing: ipset -L | grep Name: |
> awk
> > > {'print $2'} | ipset flush
> > >
> > > 2021-09-13 11:55:19,272 INFO Executing: ipset -L | grep Name: |
> awk
> > > {'print $2'} | ipset destroy
> > >
> > > 2021-09-13 11:55:19,833 INFO Executing: iptables-save
> > >
> > > 2021-09-13 11:55:20,168 INFO Executing: iptables-save
> > >
> > > 2021-09-13 11:55:20,505 INFO Executing: ip6tables-save
> > >
> > > 2021-09-13 11:55:20,800 INFO Executing: systemctl stop conntrackd
> > >
> > > 2021-09-13 11:55:21,201 INFO Service conntrackd stop
> > >
> > > 2021-09-13 11:55:21,230 INFO Executing: systemctl stop keepalived
> > >
> > > 2021-09-13 11:55:21,578 INFO Service keepalived stop
> > >
> > > 2021-09-13 11:55:21,609 INFO Executing: mount
> > >
> > > root@r-4093-VM:~#
> > >
> > >
> > >
> > > Please ignore the router name, I did multiple tests to extract these
> > > logs, I always see the same.
> > >
> > >
> > >
> > >
> > >
> > > Regards,
> > >
> > > Cristian
> > >
> > >
> > >
> > >
> > >
> > >
> >
> > --
> >
> > Andrija Panić
> >
> >
> >
> >
>
> --
> Daan
>
>
RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Posted by cr...@istream.today.
I didn't test with lower values, but with 256 RAM and 500Mhz CPU is not working for me. (KVM ubuntu 20.04 ACS 4.15.1 )
Regards,
Cristian
-----Original Message-----
From: Daan Hoogland <da...@gmail.com>
Sent: Friday, September 17, 2021 11:14 AM
To: users <us...@cloudstack.apache.org>
Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm
thanks for the update Christian,
sounds strange that the router would require that much to start but glad it worked.
On Wed, Sep 15, 2021 at 1:32 PM <cr...@istream.today> wrote:
> Hello,
>
> I have found the issue, after I have changed the CPU and memory for
> this " System Offering For Software Router - Local Storage' system
> offering the router was deployed in few seconds without any issue.
>
> CPU mhz from 500 to 2000 and RAM from 256 to 2048.
>
>
>
> Regards,
> Cristian
>
> -----Original Message-----
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Wednesday, September 15, 2021 10:18 AM
> To: users@cloudstack.apache.org
> Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with
> kvm
>
> Any suggestion on this issue?
>
> Thank you.
>
> -----Original Message-----
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Monday, September 13, 2021 5:40 PM
> To: users@cloudstack.apache.org
> Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with
> kvm
>
> Hi,
>
>
> Yes, please see :
>
> root@usa-nyc-kvm-001:/var/lib/libvirt/images# libvirtd -v
> 2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0,
> package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri,
> 16 Jul 2021 05:16:36 +0000)
> 2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
> 2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutputs:1654 :
> outputs=1:stderr
> 2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutput:1492 :
> output=1:stderr
> 2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0,
> package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri,
> 16 Jul 2021 05:16:36 +0000)
> 2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
> 2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW:
> obj=0x55935a296b60 classname=virAccessManager
> 2021-09-13 14:33:40.944+0000: 75403: debug :
> virAccessManagerNewDriver:111
> : Initialized with stack
> 2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW:
> obj=0x55935a297f90 classname=virAccessManager
> 2021-09-13 14:33:40.944+0000: 75403: debug :
> virAccessManagerNewDriver:111
> : Initialized with none
> 2021-09-13 14:33:40.944+0000: 75403: info : virObjectRef:386 : OBJECT_REF:
> obj=0x55935a296b60
> 2021-09-13 14:33:40.944+0000: 75403: info : virObjectUnref:348 :
> OBJECT_UNREF: obj=0x55935a296b60
> 2021-09-13 14:33:40.944+0000: 75403: debug : main:1177 : Decided on
> pid file path '/run/libvirtd.pid'
> 2021-09-13 14:33:40.944+0000: 75403: debug : main:1187 : Decided on
> socket paths '/run/libvirt/libvirt-sock',
> '/run/libvirt/libvirt-sock-ro' and '/run/libvirt/libvirt-admin-sock'
> 2021-09-13 14:33:40.944+0000: 75403: error : virPidFileAcquirePath:367 :
> Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily
> unavailable
> 2021-09-13 14:33:40.944+0000: 75403: debug : virFileClose:110 : Closed
> fd 3
>
>
> root@usa-nyc-kvm-001:/var/lib/libvirt/images#
> /usr/bin/qemu-system-x86_64 --version QEMU emulator version 4.2.1
> (Debian 1:4.2-3ubuntu6.17) Copyright
> (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
>
>
> root@usa-nyc-kvm-001:/var/lib/libvirt/images# lsb_release -a No LSB
> modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 20.04.2 LTS
> Release: 20.04
> Codename: focal
>
>
> CPU :
> vendor_id : AuthenticAMD
> cpu family : 23
> model : 49
> model name : AMD EPYC 7552 48-Core Processor
>
>
> BTW: the SSVM and Proxy console is running fine on this node..
>
>
>
> Thank you!
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: Monday, September 13, 2021 4:40 PM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with
> kvm
>
> Can you share qemu/libvirt version/flavour?
>
> On Mon, 13 Sept 2021 at 14:18, <cr...@istream.today> wrote:
>
> > Hello,
> >
> >
> >
> > I'm trying to understand this error, I created new network
> > with VPC in a new zone where I added KVM hypervisor, and I have this issue.
> >
> >
> >
> >
> >
> > 2021-09-13 11:36:18,621 DEBUG
> > [resource.virtualnetwork.VirtualRoutingResource]
> > (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
> > 169.254.52.234
> >
> > 2021-09-13 11:36:21,684 DEBUG
> > [resource.virtualnetwork.VirtualRoutingResource]
> > (agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
> > 169.254.52.234
> >
> >
> >
> > Here is the full log : https://pastebin.com/0NwsccUc
> >
> >
> >
> > root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922
> >
> > Trying 169.254.156.179...
> >
> > telnet: Unable to connect to remote host: Connection refused (
> > before connection refused, I get no route to host, for sure the nic
> > is not up
> > )
> >
> > root@usa-nyc-kvm-001:~# virsh list
> >
> > Id Name State
> >
> > ---------------------------
> >
> > 2 v-4048-VM running
> >
> > 4 s-4050-VM running
> >
> > 7 r-4092-VM running
> >
> >
> >
> > root@usa-nyc-kvm-001:~#
> >
> >
> >
> >
> >
> > If I check the log inside the router, I see this (the router is also
> > responding very slow, waiting a lot to connect on it via virsh console):
> >
> >
> >
> > r-4092-VM login: root
> >
> > Password:
> >
> >
> >
> > Login timed out after 60 seconds.
> >
> >
> >
> >
> >
> > __?.o/ Apache CloudStack SystemVM 4.15.1
> >
> > ( )# https://cloudstack.apache.org
> >
> > (___(_) Debian GNU/Linux 10 r-4093-VM ttyS0
> >
> >
> >
> > r-4093-VM login: root
> >
> > Password:
> >
> > Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2
> > (2021-01-30)
> > x86_64
> >
> >
> >
> >
> >
> >
> >
> > root@r-4093-VM:~# cat /var/log/cloud.log
> >
> > Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config
> >
> > Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping systemvm appliance
> >
> > Mon 13 Sep 2021 11:50:57 AM UTC Starting guest services for kvm
> >
> > Mon 13 Sep 2021 11:51:04 AM UTC acpiphp and pci_hotplug module
> > already compiled in
> >
> > Mon 13 Sep 2021 11:52:59 AM UTC Received a new non-empty cmdline
> > file from qemu-guest-agent
> >
> > Mon 13 Sep 2021 11:52:59 AM UTC Booting from cloudstack, remove old
> > configuration files in /etc/cloudstack/
> >
> > Mon 13 Sep 2021 11:54:30 AM UTC Configuring systemvm type=vpcrouter
> >
> > Mon 13 Sep 2021 11:54:30 AM UTC Setting up VPC virtual router system
> > vm
> >
> > Mon 13 Sep 2021 11:54:31 AM UTC Setting up apache web server for VPC
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC Processors = 1 Enable service = 0
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC cloud: disable rp_filter
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC disable rpfilter
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC cloud: enable_fwding = 1
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC enable_fwding = 1
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC cloud: enabling passive FTP for
> > guest VMs
> >
> > Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
> >
> > Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
> >
> > 2021-09-13 11:55:13,823 INFO update_config.py :: Processing incoming
> > file => cmd_line.json
> >
> > 2021-09-13 11:55:13,856 INFO Processing JSON file cmd_line.json
> >
> > 2021-09-13 11:55:13,858 INFO Continuing with the processing of file
> > '/var/cache/cloud/cmd_line.json'
> >
> > 2021-09-13 11:55:13,877 INFO Command of type cmdline received
> >
> > 2021-09-13 11:55:13,897 INFO Command of type ips received
> >
> > 2021-09-13 11:55:13,923 INFO Executing: ip addr show dev eth0
> >
> > 2021-09-13 11:55:13,975 INFO Executing: ip addr show dev eth0
> >
> > 2021-09-13 11:55:13,979 INFO Address found in DataBag ==>
> > {u'public_ip':
> > u'169.254.41.129', u'one_to_one_nat': False, u'nic_dev_id': u'0',
> > u'network': u'169.254.0.0/16', u'netmask': u'255.255.0.0',
> u'source_nat':
> > False, u'broadcast': u'169.254.255.255', u'add': True, u'nw_type':
> > u'control', u'device': u'eth0', u'cidr': u'169.254.41.129/16',
> u'gateway':
> > u'', u'size': u'16'}
> >
> > 2021-09-13 11:55:13,979 INFO Address 169.254.41.129/16 on device
> eth0
> > already configured
> >
> > 2021-09-13 11:55:13,984 INFO Executing: ip addr show |grep -v
> secondary
> >
> > 2021-09-13 11:55:14,010 INFO Wrote edited file
> > /etc/dnsmasq.d/cloud.conf
> >
> > 2021-09-13 11:55:14,010 INFO Updated file in-cache configuration
> >
> > 2021-09-13 11:55:14,010 INFO Nothing to commit. The
> /etc/dhcphosts.txt
> > file did not change
> >
> > 2021-09-13 11:55:14,010 INFO Nothing to commit. The
> > /var/lib/misc/dnsmasq.leases file did not change
> >
> > 2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcpopts.txt
> > file did not change
> >
> > 2021-09-13 11:55:14,010 INFO Attempting to delete entries from
> > dnsmasq.leases file for VMs which are not on dhcphosts file
> >
> > 2021-09-13 11:55:14,010 ERROR Caught error while trying to delete
> > entries
> > from dnsmasq.leases file: [Errno 2] No such file or directory:
> > '/etc/dhcphosts.txt'
> >
> > 2021-09-13 11:55:14,034 INFO Wrote edited file /etc/hosts
> >
> > 2021-09-13 11:55:14,044 INFO Updated file in-cache configuration
> >
> > 2021-09-13 11:55:14,044 INFO Updated hosts file
> >
> > 2021-09-13 11:55:14,046 INFO Executing: systemctl restart dnsmasq
> >
> > 2021-09-13 11:55:15,409 INFO Service dnsmasq restart
> >
> > 2021-09-13 11:55:15,453 INFO Executing: ip addr show |grep -v
> secondary
> >
> > 2021-09-13 11:55:15,832 INFO Nothing to commit. The
> > /etc/dnsmasq.d/cloud.conf file did not change
> >
> > 2021-09-13 11:55:15,957 INFO Nothing to commit. The
> /etc/dhcphosts.txt
> > file did not change
> >
> > 2021-09-13 11:55:15,963 INFO Nothing to commit. The
> > /var/lib/misc/dnsmasq.leases file did not change
> >
> > 2021-09-13 11:55:15,963 INFO Nothing to commit. The /etc/dhcpopts.txt
> > file did not change
> >
> > 2021-09-13 11:55:15,967 INFO Executing: systemctl is-active dnsmasq
> >
> > 2021-09-13 11:55:16,179 INFO Executing: systemctl reload dnsmasq
> >
> > 2021-09-13 11:55:16,665 INFO Service dnsmasq reload
> >
> > 2021-09-13 11:55:16,909 INFO Wrote edited file /etc/cron.d/process
> >
> > 2021-09-13 11:55:17,040 INFO Updated file in-cache configuration
> >
> > 2021-09-13 11:55:17,063 INFO Executing: iptables-save | grep
> > '^:FW_EGRESS_RULES' || iptables -t filter -N FW_EGRESS_RULES
> >
> > 2021-09-13 11:55:17,323 INFO Executing: iptables-save | grep '^-A
> > FW_EGRESS_RULES -j ACCEPT$' | sed 's/^-A/iptables -t filter -D/g' |
> > bash
> >
> > 2021-09-13 11:55:17,697 INFO Executing: iptables -F FW_EGRESS_RULES
> >
> > 2021-09-13 11:55:17,990 INFO Executing: ipset -L | grep Name: | awk
> > {'print $2'} | ipset flush
> >
> > 2021-09-13 11:55:19,272 INFO Executing: ipset -L | grep Name: | awk
> > {'print $2'} | ipset destroy
> >
> > 2021-09-13 11:55:19,833 INFO Executing: iptables-save
> >
> > 2021-09-13 11:55:20,168 INFO Executing: iptables-save
> >
> > 2021-09-13 11:55:20,505 INFO Executing: ip6tables-save
> >
> > 2021-09-13 11:55:20,800 INFO Executing: systemctl stop conntrackd
> >
> > 2021-09-13 11:55:21,201 INFO Service conntrackd stop
> >
> > 2021-09-13 11:55:21,230 INFO Executing: systemctl stop keepalived
> >
> > 2021-09-13 11:55:21,578 INFO Service keepalived stop
> >
> > 2021-09-13 11:55:21,609 INFO Executing: mount
> >
> > root@r-4093-VM:~#
> >
> >
> >
> > Please ignore the router name, I did multiple tests to extract these
> > logs, I always see the same.
> >
> >
> >
> >
> >
> > Regards,
> >
> > Cristian
> >
> >
> >
> >
> >
> >
>
> --
>
> Andrija Panić
>
>
>
>
--
Daan
Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Posted by Daan Hoogland <da...@gmail.com>.
thanks for the update Christian,
sounds strange that the router would require that much to start but glad it
worked.
On Wed, Sep 15, 2021 at 1:32 PM <cr...@istream.today> wrote:
> Hello,
>
> I have found the issue, after I have changed the CPU and memory for
> this " System Offering For Software Router - Local Storage' system offering
> the router was deployed in few seconds without any issue.
>
> CPU mhz from 500 to 2000 and RAM from 256 to 2048.
>
>
>
> Regards,
> Cristian
>
> -----Original Message-----
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Wednesday, September 15, 2021 10:18 AM
> To: users@cloudstack.apache.org
> Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm
>
> Any suggestion on this issue?
>
> Thank you.
>
> -----Original Message-----
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Monday, September 13, 2021 5:40 PM
> To: users@cloudstack.apache.org
> Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm
>
> Hi,
>
>
> Yes, please see :
>
> root@usa-nyc-kvm-001:/var/lib/libvirt/images# libvirtd -v
> 2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0,
> package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri, 16
> Jul 2021 05:16:36 +0000)
> 2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
> 2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutputs:1654 :
> outputs=1:stderr
> 2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutput:1492 :
> output=1:stderr
> 2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0,
> package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri, 16
> Jul 2021 05:16:36 +0000)
> 2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
> 2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW:
> obj=0x55935a296b60 classname=virAccessManager
> 2021-09-13 14:33:40.944+0000: 75403: debug : virAccessManagerNewDriver:111
> : Initialized with stack
> 2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW:
> obj=0x55935a297f90 classname=virAccessManager
> 2021-09-13 14:33:40.944+0000: 75403: debug : virAccessManagerNewDriver:111
> : Initialized with none
> 2021-09-13 14:33:40.944+0000: 75403: info : virObjectRef:386 : OBJECT_REF:
> obj=0x55935a296b60
> 2021-09-13 14:33:40.944+0000: 75403: info : virObjectUnref:348 :
> OBJECT_UNREF: obj=0x55935a296b60
> 2021-09-13 14:33:40.944+0000: 75403: debug : main:1177 : Decided on pid
> file path '/run/libvirtd.pid'
> 2021-09-13 14:33:40.944+0000: 75403: debug : main:1187 : Decided on socket
> paths '/run/libvirt/libvirt-sock', '/run/libvirt/libvirt-sock-ro' and
> '/run/libvirt/libvirt-admin-sock'
> 2021-09-13 14:33:40.944+0000: 75403: error : virPidFileAcquirePath:367 :
> Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily
> unavailable
> 2021-09-13 14:33:40.944+0000: 75403: debug : virFileClose:110 : Closed fd 3
>
>
> root@usa-nyc-kvm-001:/var/lib/libvirt/images# /usr/bin/qemu-system-x86_64
> --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17) Copyright
> (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
>
>
> root@usa-nyc-kvm-001:/var/lib/libvirt/images# lsb_release -a No LSB
> modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 20.04.2 LTS
> Release: 20.04
> Codename: focal
>
>
> CPU :
> vendor_id : AuthenticAMD
> cpu family : 23
> model : 49
> model name : AMD EPYC 7552 48-Core Processor
>
>
> BTW: the SSVM and Proxy console is running fine on this node..
>
>
>
> Thank you!
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: Monday, September 13, 2021 4:40 PM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm
>
> Can you share qemu/libvirt version/flavour?
>
> On Mon, 13 Sept 2021 at 14:18, <cr...@istream.today> wrote:
>
> > Hello,
> >
> >
> >
> > I'm trying to understand this error, I created new network with
> > VPC in a new zone where I added KVM hypervisor, and I have this issue.
> >
> >
> >
> >
> >
> > 2021-09-13 11:36:18,621 DEBUG
> > [resource.virtualnetwork.VirtualRoutingResource]
> > (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
> > 169.254.52.234
> >
> > 2021-09-13 11:36:21,684 DEBUG
> > [resource.virtualnetwork.VirtualRoutingResource]
> > (agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
> > 169.254.52.234
> >
> >
> >
> > Here is the full log : https://pastebin.com/0NwsccUc
> >
> >
> >
> > root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922
> >
> > Trying 169.254.156.179...
> >
> > telnet: Unable to connect to remote host: Connection refused ( before
> > connection refused, I get no route to host, for sure the nic is not up
> > )
> >
> > root@usa-nyc-kvm-001:~# virsh list
> >
> > Id Name State
> >
> > ---------------------------
> >
> > 2 v-4048-VM running
> >
> > 4 s-4050-VM running
> >
> > 7 r-4092-VM running
> >
> >
> >
> > root@usa-nyc-kvm-001:~#
> >
> >
> >
> >
> >
> > If I check the log inside the router, I see this (the router is also
> > responding very slow, waiting a lot to connect on it via virsh console):
> >
> >
> >
> > r-4092-VM login: root
> >
> > Password:
> >
> >
> >
> > Login timed out after 60 seconds.
> >
> >
> >
> >
> >
> > __?.o/ Apache CloudStack SystemVM 4.15.1
> >
> > ( )# https://cloudstack.apache.org
> >
> > (___(_) Debian GNU/Linux 10 r-4093-VM ttyS0
> >
> >
> >
> > r-4093-VM login: root
> >
> > Password:
> >
> > Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30)
> > x86_64
> >
> >
> >
> >
> >
> >
> >
> > root@r-4093-VM:~# cat /var/log/cloud.log
> >
> > Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config
> >
> > Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping systemvm appliance
> >
> > Mon 13 Sep 2021 11:50:57 AM UTC Starting guest services for kvm
> >
> > Mon 13 Sep 2021 11:51:04 AM UTC acpiphp and pci_hotplug module already
> > compiled in
> >
> > Mon 13 Sep 2021 11:52:59 AM UTC Received a new non-empty cmdline file
> > from qemu-guest-agent
> >
> > Mon 13 Sep 2021 11:52:59 AM UTC Booting from cloudstack, remove old
> > configuration files in /etc/cloudstack/
> >
> > Mon 13 Sep 2021 11:54:30 AM UTC Configuring systemvm type=vpcrouter
> >
> > Mon 13 Sep 2021 11:54:30 AM UTC Setting up VPC virtual router system
> > vm
> >
> > Mon 13 Sep 2021 11:54:31 AM UTC Setting up apache web server for VPC
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC Processors = 1 Enable service = 0
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC cloud: disable rp_filter
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC disable rpfilter
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC cloud: enable_fwding = 1
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC enable_fwding = 1
> >
> > Mon 13 Sep 2021 11:55:05 AM UTC cloud: enabling passive FTP for guest
> > VMs
> >
> > Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
> >
> > Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
> >
> > 2021-09-13 11:55:13,823 INFO update_config.py :: Processing incoming
> > file => cmd_line.json
> >
> > 2021-09-13 11:55:13,856 INFO Processing JSON file cmd_line.json
> >
> > 2021-09-13 11:55:13,858 INFO Continuing with the processing of file
> > '/var/cache/cloud/cmd_line.json'
> >
> > 2021-09-13 11:55:13,877 INFO Command of type cmdline received
> >
> > 2021-09-13 11:55:13,897 INFO Command of type ips received
> >
> > 2021-09-13 11:55:13,923 INFO Executing: ip addr show dev eth0
> >
> > 2021-09-13 11:55:13,975 INFO Executing: ip addr show dev eth0
> >
> > 2021-09-13 11:55:13,979 INFO Address found in DataBag ==>
> > {u'public_ip':
> > u'169.254.41.129', u'one_to_one_nat': False, u'nic_dev_id': u'0',
> > u'network': u'169.254.0.0/16', u'netmask': u'255.255.0.0',
> u'source_nat':
> > False, u'broadcast': u'169.254.255.255', u'add': True, u'nw_type':
> > u'control', u'device': u'eth0', u'cidr': u'169.254.41.129/16',
> u'gateway':
> > u'', u'size': u'16'}
> >
> > 2021-09-13 11:55:13,979 INFO Address 169.254.41.129/16 on device
> eth0
> > already configured
> >
> > 2021-09-13 11:55:13,984 INFO Executing: ip addr show |grep -v
> secondary
> >
> > 2021-09-13 11:55:14,010 INFO Wrote edited file
> > /etc/dnsmasq.d/cloud.conf
> >
> > 2021-09-13 11:55:14,010 INFO Updated file in-cache configuration
> >
> > 2021-09-13 11:55:14,010 INFO Nothing to commit. The
> /etc/dhcphosts.txt
> > file did not change
> >
> > 2021-09-13 11:55:14,010 INFO Nothing to commit. The
> > /var/lib/misc/dnsmasq.leases file did not change
> >
> > 2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcpopts.txt
> > file did not change
> >
> > 2021-09-13 11:55:14,010 INFO Attempting to delete entries from
> > dnsmasq.leases file for VMs which are not on dhcphosts file
> >
> > 2021-09-13 11:55:14,010 ERROR Caught error while trying to delete
> > entries
> > from dnsmasq.leases file: [Errno 2] No such file or directory:
> > '/etc/dhcphosts.txt'
> >
> > 2021-09-13 11:55:14,034 INFO Wrote edited file /etc/hosts
> >
> > 2021-09-13 11:55:14,044 INFO Updated file in-cache configuration
> >
> > 2021-09-13 11:55:14,044 INFO Updated hosts file
> >
> > 2021-09-13 11:55:14,046 INFO Executing: systemctl restart dnsmasq
> >
> > 2021-09-13 11:55:15,409 INFO Service dnsmasq restart
> >
> > 2021-09-13 11:55:15,453 INFO Executing: ip addr show |grep -v
> secondary
> >
> > 2021-09-13 11:55:15,832 INFO Nothing to commit. The
> > /etc/dnsmasq.d/cloud.conf file did not change
> >
> > 2021-09-13 11:55:15,957 INFO Nothing to commit. The
> /etc/dhcphosts.txt
> > file did not change
> >
> > 2021-09-13 11:55:15,963 INFO Nothing to commit. The
> > /var/lib/misc/dnsmasq.leases file did not change
> >
> > 2021-09-13 11:55:15,963 INFO Nothing to commit. The /etc/dhcpopts.txt
> > file did not change
> >
> > 2021-09-13 11:55:15,967 INFO Executing: systemctl is-active dnsmasq
> >
> > 2021-09-13 11:55:16,179 INFO Executing: systemctl reload dnsmasq
> >
> > 2021-09-13 11:55:16,665 INFO Service dnsmasq reload
> >
> > 2021-09-13 11:55:16,909 INFO Wrote edited file /etc/cron.d/process
> >
> > 2021-09-13 11:55:17,040 INFO Updated file in-cache configuration
> >
> > 2021-09-13 11:55:17,063 INFO Executing: iptables-save | grep
> > '^:FW_EGRESS_RULES' || iptables -t filter -N FW_EGRESS_RULES
> >
> > 2021-09-13 11:55:17,323 INFO Executing: iptables-save | grep '^-A
> > FW_EGRESS_RULES -j ACCEPT$' | sed 's/^-A/iptables -t filter -D/g' |
> > bash
> >
> > 2021-09-13 11:55:17,697 INFO Executing: iptables -F FW_EGRESS_RULES
> >
> > 2021-09-13 11:55:17,990 INFO Executing: ipset -L | grep Name: | awk
> > {'print $2'} | ipset flush
> >
> > 2021-09-13 11:55:19,272 INFO Executing: ipset -L | grep Name: | awk
> > {'print $2'} | ipset destroy
> >
> > 2021-09-13 11:55:19,833 INFO Executing: iptables-save
> >
> > 2021-09-13 11:55:20,168 INFO Executing: iptables-save
> >
> > 2021-09-13 11:55:20,505 INFO Executing: ip6tables-save
> >
> > 2021-09-13 11:55:20,800 INFO Executing: systemctl stop conntrackd
> >
> > 2021-09-13 11:55:21,201 INFO Service conntrackd stop
> >
> > 2021-09-13 11:55:21,230 INFO Executing: systemctl stop keepalived
> >
> > 2021-09-13 11:55:21,578 INFO Service keepalived stop
> >
> > 2021-09-13 11:55:21,609 INFO Executing: mount
> >
> > root@r-4093-VM:~#
> >
> >
> >
> > Please ignore the router name, I did multiple tests to extract these
> > logs, I always see the same.
> >
> >
> >
> >
> >
> > Regards,
> >
> > Cristian
> >
> >
> >
> >
> >
> >
>
> --
>
> Andrija Panić
>
>
>
>
--
Daan
RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Posted by cr...@istream.today.
Hello,
I have found the issue, after I have changed the CPU and memory for this " System Offering For Software Router - Local Storage' system offering the router was deployed in few seconds without any issue.
CPU mhz from 500 to 2000 and RAM from 256 to 2048.
Regards,
Cristian
-----Original Message-----
From: cristian.c@istream.today <cr...@istream.today>
Sent: Wednesday, September 15, 2021 10:18 AM
To: users@cloudstack.apache.org
Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Any suggestion on this issue?
Thank you.
-----Original Message-----
From: cristian.c@istream.today <cr...@istream.today>
Sent: Monday, September 13, 2021 5:40 PM
To: users@cloudstack.apache.org
Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Hi,
Yes, please see :
root@usa-nyc-kvm-001:/var/lib/libvirt/images# libvirtd -v
2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0, package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri, 16 Jul 2021 05:16:36 +0000)
2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutputs:1654 : outputs=1:stderr
2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutput:1492 : output=1:stderr
2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0, package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri, 16 Jul 2021 05:16:36 +0000)
2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW: obj=0x55935a296b60 classname=virAccessManager
2021-09-13 14:33:40.944+0000: 75403: debug : virAccessManagerNewDriver:111 : Initialized with stack
2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW: obj=0x55935a297f90 classname=virAccessManager
2021-09-13 14:33:40.944+0000: 75403: debug : virAccessManagerNewDriver:111 : Initialized with none
2021-09-13 14:33:40.944+0000: 75403: info : virObjectRef:386 : OBJECT_REF: obj=0x55935a296b60
2021-09-13 14:33:40.944+0000: 75403: info : virObjectUnref:348 : OBJECT_UNREF: obj=0x55935a296b60
2021-09-13 14:33:40.944+0000: 75403: debug : main:1177 : Decided on pid file path '/run/libvirtd.pid'
2021-09-13 14:33:40.944+0000: 75403: debug : main:1187 : Decided on socket paths '/run/libvirt/libvirt-sock', '/run/libvirt/libvirt-sock-ro' and '/run/libvirt/libvirt-admin-sock'
2021-09-13 14:33:40.944+0000: 75403: error : virPidFileAcquirePath:367 : Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily unavailable
2021-09-13 14:33:40.944+0000: 75403: debug : virFileClose:110 : Closed fd 3
root@usa-nyc-kvm-001:/var/lib/libvirt/images# /usr/bin/qemu-system-x86_64 --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
root@usa-nyc-kvm-001:/var/lib/libvirt/images# lsb_release -a No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
CPU :
vendor_id : AuthenticAMD
cpu family : 23
model : 49
model name : AMD EPYC 7552 48-Core Processor
BTW: the SSVM and Proxy console is running fine on this node..
Thank you!
-----Original Message-----
From: Andrija Panic <an...@gmail.com>
Sent: Monday, September 13, 2021 4:40 PM
To: users <us...@cloudstack.apache.org>
Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Can you share qemu/libvirt version/flavour?
On Mon, 13 Sept 2021 at 14:18, <cr...@istream.today> wrote:
> Hello,
>
>
>
> I'm trying to understand this error, I created new network with
> VPC in a new zone where I added KVM hypervisor, and I have this issue.
>
>
>
>
>
> 2021-09-13 11:36:18,621 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
> 169.254.52.234
>
> 2021-09-13 11:36:21,684 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
> 169.254.52.234
>
>
>
> Here is the full log : https://pastebin.com/0NwsccUc
>
>
>
> root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922
>
> Trying 169.254.156.179...
>
> telnet: Unable to connect to remote host: Connection refused ( before
> connection refused, I get no route to host, for sure the nic is not up
> )
>
> root@usa-nyc-kvm-001:~# virsh list
>
> Id Name State
>
> ---------------------------
>
> 2 v-4048-VM running
>
> 4 s-4050-VM running
>
> 7 r-4092-VM running
>
>
>
> root@usa-nyc-kvm-001:~#
>
>
>
>
>
> If I check the log inside the router, I see this (the router is also
> responding very slow, waiting a lot to connect on it via virsh console):
>
>
>
> r-4092-VM login: root
>
> Password:
>
>
>
> Login timed out after 60 seconds.
>
>
>
>
>
> __?.o/ Apache CloudStack SystemVM 4.15.1
>
> ( )# https://cloudstack.apache.org
>
> (___(_) Debian GNU/Linux 10 r-4093-VM ttyS0
>
>
>
> r-4093-VM login: root
>
> Password:
>
> Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30)
> x86_64
>
>
>
>
>
>
>
> root@r-4093-VM:~# cat /var/log/cloud.log
>
> Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config
>
> Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping systemvm appliance
>
> Mon 13 Sep 2021 11:50:57 AM UTC Starting guest services for kvm
>
> Mon 13 Sep 2021 11:51:04 AM UTC acpiphp and pci_hotplug module already
> compiled in
>
> Mon 13 Sep 2021 11:52:59 AM UTC Received a new non-empty cmdline file
> from qemu-guest-agent
>
> Mon 13 Sep 2021 11:52:59 AM UTC Booting from cloudstack, remove old
> configuration files in /etc/cloudstack/
>
> Mon 13 Sep 2021 11:54:30 AM UTC Configuring systemvm type=vpcrouter
>
> Mon 13 Sep 2021 11:54:30 AM UTC Setting up VPC virtual router system
> vm
>
> Mon 13 Sep 2021 11:54:31 AM UTC Setting up apache web server for VPC
>
> Mon 13 Sep 2021 11:55:05 AM UTC Processors = 1 Enable service = 0
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: disable rp_filter
>
> Mon 13 Sep 2021 11:55:05 AM UTC disable rpfilter
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: enable_fwding = 1
>
> Mon 13 Sep 2021 11:55:05 AM UTC enable_fwding = 1
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: enabling passive FTP for guest
> VMs
>
> Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
>
> Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
>
> 2021-09-13 11:55:13,823 INFO update_config.py :: Processing incoming
> file => cmd_line.json
>
> 2021-09-13 11:55:13,856 INFO Processing JSON file cmd_line.json
>
> 2021-09-13 11:55:13,858 INFO Continuing with the processing of file
> '/var/cache/cloud/cmd_line.json'
>
> 2021-09-13 11:55:13,877 INFO Command of type cmdline received
>
> 2021-09-13 11:55:13,897 INFO Command of type ips received
>
> 2021-09-13 11:55:13,923 INFO Executing: ip addr show dev eth0
>
> 2021-09-13 11:55:13,975 INFO Executing: ip addr show dev eth0
>
> 2021-09-13 11:55:13,979 INFO Address found in DataBag ==>
> {u'public_ip':
> u'169.254.41.129', u'one_to_one_nat': False, u'nic_dev_id': u'0',
> u'network': u'169.254.0.0/16', u'netmask': u'255.255.0.0', u'source_nat':
> False, u'broadcast': u'169.254.255.255', u'add': True, u'nw_type':
> u'control', u'device': u'eth0', u'cidr': u'169.254.41.129/16', u'gateway':
> u'', u'size': u'16'}
>
> 2021-09-13 11:55:13,979 INFO Address 169.254.41.129/16 on device eth0
> already configured
>
> 2021-09-13 11:55:13,984 INFO Executing: ip addr show |grep -v secondary
>
> 2021-09-13 11:55:14,010 INFO Wrote edited file
> /etc/dnsmasq.d/cloud.conf
>
> 2021-09-13 11:55:14,010 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcphosts.txt
> file did not change
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The
> /var/lib/misc/dnsmasq.leases file did not change
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcpopts.txt
> file did not change
>
> 2021-09-13 11:55:14,010 INFO Attempting to delete entries from
> dnsmasq.leases file for VMs which are not on dhcphosts file
>
> 2021-09-13 11:55:14,010 ERROR Caught error while trying to delete
> entries
> from dnsmasq.leases file: [Errno 2] No such file or directory:
> '/etc/dhcphosts.txt'
>
> 2021-09-13 11:55:14,034 INFO Wrote edited file /etc/hosts
>
> 2021-09-13 11:55:14,044 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:14,044 INFO Updated hosts file
>
> 2021-09-13 11:55:14,046 INFO Executing: systemctl restart dnsmasq
>
> 2021-09-13 11:55:15,409 INFO Service dnsmasq restart
>
> 2021-09-13 11:55:15,453 INFO Executing: ip addr show |grep -v secondary
>
> 2021-09-13 11:55:15,832 INFO Nothing to commit. The
> /etc/dnsmasq.d/cloud.conf file did not change
>
> 2021-09-13 11:55:15,957 INFO Nothing to commit. The /etc/dhcphosts.txt
> file did not change
>
> 2021-09-13 11:55:15,963 INFO Nothing to commit. The
> /var/lib/misc/dnsmasq.leases file did not change
>
> 2021-09-13 11:55:15,963 INFO Nothing to commit. The /etc/dhcpopts.txt
> file did not change
>
> 2021-09-13 11:55:15,967 INFO Executing: systemctl is-active dnsmasq
>
> 2021-09-13 11:55:16,179 INFO Executing: systemctl reload dnsmasq
>
> 2021-09-13 11:55:16,665 INFO Service dnsmasq reload
>
> 2021-09-13 11:55:16,909 INFO Wrote edited file /etc/cron.d/process
>
> 2021-09-13 11:55:17,040 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:17,063 INFO Executing: iptables-save | grep
> '^:FW_EGRESS_RULES' || iptables -t filter -N FW_EGRESS_RULES
>
> 2021-09-13 11:55:17,323 INFO Executing: iptables-save | grep '^-A
> FW_EGRESS_RULES -j ACCEPT$' | sed 's/^-A/iptables -t filter -D/g' |
> bash
>
> 2021-09-13 11:55:17,697 INFO Executing: iptables -F FW_EGRESS_RULES
>
> 2021-09-13 11:55:17,990 INFO Executing: ipset -L | grep Name: | awk
> {'print $2'} | ipset flush
>
> 2021-09-13 11:55:19,272 INFO Executing: ipset -L | grep Name: | awk
> {'print $2'} | ipset destroy
>
> 2021-09-13 11:55:19,833 INFO Executing: iptables-save
>
> 2021-09-13 11:55:20,168 INFO Executing: iptables-save
>
> 2021-09-13 11:55:20,505 INFO Executing: ip6tables-save
>
> 2021-09-13 11:55:20,800 INFO Executing: systemctl stop conntrackd
>
> 2021-09-13 11:55:21,201 INFO Service conntrackd stop
>
> 2021-09-13 11:55:21,230 INFO Executing: systemctl stop keepalived
>
> 2021-09-13 11:55:21,578 INFO Service keepalived stop
>
> 2021-09-13 11:55:21,609 INFO Executing: mount
>
> root@r-4093-VM:~#
>
>
>
> Please ignore the router name, I did multiple tests to extract these
> logs, I always see the same.
>
>
>
>
>
> Regards,
>
> Cristian
>
>
>
>
>
>
--
Andrija Panić
RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Posted by cr...@istream.today.
Any suggestion on this issue?
Thank you.
-----Original Message-----
From: cristian.c@istream.today <cr...@istream.today>
Sent: Monday, September 13, 2021 5:40 PM
To: users@cloudstack.apache.org
Subject: RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Hi,
Yes, please see :
root@usa-nyc-kvm-001:/var/lib/libvirt/images# libvirtd -v
2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0, package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri, 16 Jul 2021 05:16:36 +0000)
2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutputs:1654 : outputs=1:stderr
2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutput:1492 : output=1:stderr
2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0, package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri, 16 Jul 2021 05:16:36 +0000)
2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW: obj=0x55935a296b60 classname=virAccessManager
2021-09-13 14:33:40.944+0000: 75403: debug : virAccessManagerNewDriver:111 : Initialized with stack
2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW: obj=0x55935a297f90 classname=virAccessManager
2021-09-13 14:33:40.944+0000: 75403: debug : virAccessManagerNewDriver:111 : Initialized with none
2021-09-13 14:33:40.944+0000: 75403: info : virObjectRef:386 : OBJECT_REF: obj=0x55935a296b60
2021-09-13 14:33:40.944+0000: 75403: info : virObjectUnref:348 : OBJECT_UNREF: obj=0x55935a296b60
2021-09-13 14:33:40.944+0000: 75403: debug : main:1177 : Decided on pid file path '/run/libvirtd.pid'
2021-09-13 14:33:40.944+0000: 75403: debug : main:1187 : Decided on socket paths '/run/libvirt/libvirt-sock', '/run/libvirt/libvirt-sock-ro' and '/run/libvirt/libvirt-admin-sock'
2021-09-13 14:33:40.944+0000: 75403: error : virPidFileAcquirePath:367 : Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily unavailable
2021-09-13 14:33:40.944+0000: 75403: debug : virFileClose:110 : Closed fd 3
root@usa-nyc-kvm-001:/var/lib/libvirt/images# /usr/bin/qemu-system-x86_64 --version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
root@usa-nyc-kvm-001:/var/lib/libvirt/images# lsb_release -a No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
CPU :
vendor_id : AuthenticAMD
cpu family : 23
model : 49
model name : AMD EPYC 7552 48-Core Processor
BTW: the SSVM and Proxy console is running fine on this node..
Thank you!
-----Original Message-----
From: Andrija Panic <an...@gmail.com>
Sent: Monday, September 13, 2021 4:40 PM
To: users <us...@cloudstack.apache.org>
Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Can you share qemu/libvirt version/flavour?
On Mon, 13 Sept 2021 at 14:18, <cr...@istream.today> wrote:
> Hello,
>
>
>
> I'm trying to understand this error, I created new network with
> VPC in a new zone where I added KVM hypervisor, and I have this issue.
>
>
>
>
>
> 2021-09-13 11:36:18,621 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
> 169.254.52.234
>
> 2021-09-13 11:36:21,684 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
> 169.254.52.234
>
>
>
> Here is the full log : https://pastebin.com/0NwsccUc
>
>
>
> root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922
>
> Trying 169.254.156.179...
>
> telnet: Unable to connect to remote host: Connection refused ( before
> connection refused, I get no route to host, for sure the nic is not up
> )
>
> root@usa-nyc-kvm-001:~# virsh list
>
> Id Name State
>
> ---------------------------
>
> 2 v-4048-VM running
>
> 4 s-4050-VM running
>
> 7 r-4092-VM running
>
>
>
> root@usa-nyc-kvm-001:~#
>
>
>
>
>
> If I check the log inside the router, I see this (the router is also
> responding very slow, waiting a lot to connect on it via virsh console):
>
>
>
> r-4092-VM login: root
>
> Password:
>
>
>
> Login timed out after 60 seconds.
>
>
>
>
>
> __?.o/ Apache CloudStack SystemVM 4.15.1
>
> ( )# https://cloudstack.apache.org
>
> (___(_) Debian GNU/Linux 10 r-4093-VM ttyS0
>
>
>
> r-4093-VM login: root
>
> Password:
>
> Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30)
> x86_64
>
>
>
>
>
>
>
> root@r-4093-VM:~# cat /var/log/cloud.log
>
> Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config
>
> Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping systemvm appliance
>
> Mon 13 Sep 2021 11:50:57 AM UTC Starting guest services for kvm
>
> Mon 13 Sep 2021 11:51:04 AM UTC acpiphp and pci_hotplug module already
> compiled in
>
> Mon 13 Sep 2021 11:52:59 AM UTC Received a new non-empty cmdline file
> from qemu-guest-agent
>
> Mon 13 Sep 2021 11:52:59 AM UTC Booting from cloudstack, remove old
> configuration files in /etc/cloudstack/
>
> Mon 13 Sep 2021 11:54:30 AM UTC Configuring systemvm type=vpcrouter
>
> Mon 13 Sep 2021 11:54:30 AM UTC Setting up VPC virtual router system
> vm
>
> Mon 13 Sep 2021 11:54:31 AM UTC Setting up apache web server for VPC
>
> Mon 13 Sep 2021 11:55:05 AM UTC Processors = 1 Enable service = 0
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: disable rp_filter
>
> Mon 13 Sep 2021 11:55:05 AM UTC disable rpfilter
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: enable_fwding = 1
>
> Mon 13 Sep 2021 11:55:05 AM UTC enable_fwding = 1
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: enabling passive FTP for guest
> VMs
>
> Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
>
> Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
>
> 2021-09-13 11:55:13,823 INFO update_config.py :: Processing incoming
> file => cmd_line.json
>
> 2021-09-13 11:55:13,856 INFO Processing JSON file cmd_line.json
>
> 2021-09-13 11:55:13,858 INFO Continuing with the processing of file
> '/var/cache/cloud/cmd_line.json'
>
> 2021-09-13 11:55:13,877 INFO Command of type cmdline received
>
> 2021-09-13 11:55:13,897 INFO Command of type ips received
>
> 2021-09-13 11:55:13,923 INFO Executing: ip addr show dev eth0
>
> 2021-09-13 11:55:13,975 INFO Executing: ip addr show dev eth0
>
> 2021-09-13 11:55:13,979 INFO Address found in DataBag ==>
> {u'public_ip':
> u'169.254.41.129', u'one_to_one_nat': False, u'nic_dev_id': u'0',
> u'network': u'169.254.0.0/16', u'netmask': u'255.255.0.0', u'source_nat':
> False, u'broadcast': u'169.254.255.255', u'add': True, u'nw_type':
> u'control', u'device': u'eth0', u'cidr': u'169.254.41.129/16', u'gateway':
> u'', u'size': u'16'}
>
> 2021-09-13 11:55:13,979 INFO Address 169.254.41.129/16 on device eth0
> already configured
>
> 2021-09-13 11:55:13,984 INFO Executing: ip addr show |grep -v secondary
>
> 2021-09-13 11:55:14,010 INFO Wrote edited file
> /etc/dnsmasq.d/cloud.conf
>
> 2021-09-13 11:55:14,010 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcphosts.txt
> file did not change
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The
> /var/lib/misc/dnsmasq.leases file did not change
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcpopts.txt
> file did not change
>
> 2021-09-13 11:55:14,010 INFO Attempting to delete entries from
> dnsmasq.leases file for VMs which are not on dhcphosts file
>
> 2021-09-13 11:55:14,010 ERROR Caught error while trying to delete
> entries
> from dnsmasq.leases file: [Errno 2] No such file or directory:
> '/etc/dhcphosts.txt'
>
> 2021-09-13 11:55:14,034 INFO Wrote edited file /etc/hosts
>
> 2021-09-13 11:55:14,044 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:14,044 INFO Updated hosts file
>
> 2021-09-13 11:55:14,046 INFO Executing: systemctl restart dnsmasq
>
> 2021-09-13 11:55:15,409 INFO Service dnsmasq restart
>
> 2021-09-13 11:55:15,453 INFO Executing: ip addr show |grep -v secondary
>
> 2021-09-13 11:55:15,832 INFO Nothing to commit. The
> /etc/dnsmasq.d/cloud.conf file did not change
>
> 2021-09-13 11:55:15,957 INFO Nothing to commit. The /etc/dhcphosts.txt
> file did not change
>
> 2021-09-13 11:55:15,963 INFO Nothing to commit. The
> /var/lib/misc/dnsmasq.leases file did not change
>
> 2021-09-13 11:55:15,963 INFO Nothing to commit. The /etc/dhcpopts.txt
> file did not change
>
> 2021-09-13 11:55:15,967 INFO Executing: systemctl is-active dnsmasq
>
> 2021-09-13 11:55:16,179 INFO Executing: systemctl reload dnsmasq
>
> 2021-09-13 11:55:16,665 INFO Service dnsmasq reload
>
> 2021-09-13 11:55:16,909 INFO Wrote edited file /etc/cron.d/process
>
> 2021-09-13 11:55:17,040 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:17,063 INFO Executing: iptables-save | grep
> '^:FW_EGRESS_RULES' || iptables -t filter -N FW_EGRESS_RULES
>
> 2021-09-13 11:55:17,323 INFO Executing: iptables-save | grep '^-A
> FW_EGRESS_RULES -j ACCEPT$' | sed 's/^-A/iptables -t filter -D/g' |
> bash
>
> 2021-09-13 11:55:17,697 INFO Executing: iptables -F FW_EGRESS_RULES
>
> 2021-09-13 11:55:17,990 INFO Executing: ipset -L | grep Name: | awk
> {'print $2'} | ipset flush
>
> 2021-09-13 11:55:19,272 INFO Executing: ipset -L | grep Name: | awk
> {'print $2'} | ipset destroy
>
> 2021-09-13 11:55:19,833 INFO Executing: iptables-save
>
> 2021-09-13 11:55:20,168 INFO Executing: iptables-save
>
> 2021-09-13 11:55:20,505 INFO Executing: ip6tables-save
>
> 2021-09-13 11:55:20,800 INFO Executing: systemctl stop conntrackd
>
> 2021-09-13 11:55:21,201 INFO Service conntrackd stop
>
> 2021-09-13 11:55:21,230 INFO Executing: systemctl stop keepalived
>
> 2021-09-13 11:55:21,578 INFO Service keepalived stop
>
> 2021-09-13 11:55:21,609 INFO Executing: mount
>
> root@r-4093-VM:~#
>
>
>
> Please ignore the router name, I did multiple tests to extract these
> logs, I always see the same.
>
>
>
>
>
> Regards,
>
> Cristian
>
>
>
>
>
>
--
Andrija Panić
RE: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Posted by cr...@istream.today.
Hi,
Yes, please see :
root@usa-nyc-kvm-001:/var/lib/libvirt/images# libvirtd -v
2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0, package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri, 16 Jul 2021 05:16:36 +0000)
2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutputs:1654 : outputs=1:stderr
2021-09-13 14:33:40.944+0000: 75403: debug : virLogParseOutput:1492 : output=1:stderr
2021-09-13 14:33:40.944+0000: 75403: info : libvirt version: 6.0.0, package: 0ubuntu8.13 (Frode Nordahl <fr...@canonical.com> Fri, 16 Jul 2021 05:16:36 +0000)
2021-09-13 14:33:40.944+0000: 75403: info : hostname: usa-nyc-kvm-001
2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW: obj=0x55935a296b60 classname=virAccessManager
2021-09-13 14:33:40.944+0000: 75403: debug : virAccessManagerNewDriver:111 : Initialized with stack
2021-09-13 14:33:40.944+0000: 75403: info : virObjectNew:251 : OBJECT_NEW: obj=0x55935a297f90 classname=virAccessManager
2021-09-13 14:33:40.944+0000: 75403: debug : virAccessManagerNewDriver:111 : Initialized with none
2021-09-13 14:33:40.944+0000: 75403: info : virObjectRef:386 : OBJECT_REF: obj=0x55935a296b60
2021-09-13 14:33:40.944+0000: 75403: info : virObjectUnref:348 : OBJECT_UNREF: obj=0x55935a296b60
2021-09-13 14:33:40.944+0000: 75403: debug : main:1177 : Decided on pid file path '/run/libvirtd.pid'
2021-09-13 14:33:40.944+0000: 75403: debug : main:1187 : Decided on socket paths '/run/libvirt/libvirt-sock', '/run/libvirt/libvirt-sock-ro' and '/run/libvirt/libvirt-admin-sock'
2021-09-13 14:33:40.944+0000: 75403: error : virPidFileAcquirePath:367 : Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily unavailable
2021-09-13 14:33:40.944+0000: 75403: debug : virFileClose:110 : Closed fd 3
root@usa-nyc-kvm-001:/var/lib/libvirt/images# /usr/bin/qemu-system-x86_64 --version
QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.17)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
root@usa-nyc-kvm-001:/var/lib/libvirt/images# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
CPU :
vendor_id : AuthenticAMD
cpu family : 23
model : 49
model name : AMD EPYC 7552 48-Core Processor
BTW: the SSVM and Proxy console is running fine on this node..
Thank you!
-----Original Message-----
From: Andrija Panic <an...@gmail.com>
Sent: Monday, September 13, 2021 4:40 PM
To: users <us...@cloudstack.apache.org>
Subject: Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Can you share qemu/libvirt version/flavour?
On Mon, 13 Sept 2021 at 14:18, <cr...@istream.today> wrote:
> Hello,
>
>
>
> I'm trying to understand this error, I created new network with
> VPC in a new zone where I added KVM hypervisor, and I have this issue.
>
>
>
>
>
> 2021-09-13 11:36:18,621 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
> 169.254.52.234
>
> 2021-09-13 11:36:21,684 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
> 169.254.52.234
>
>
>
> Here is the full log : https://pastebin.com/0NwsccUc
>
>
>
> root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922
>
> Trying 169.254.156.179...
>
> telnet: Unable to connect to remote host: Connection refused ( before
> connection refused, I get no route to host, for sure the nic is not up
> )
>
> root@usa-nyc-kvm-001:~# virsh list
>
> Id Name State
>
> ---------------------------
>
> 2 v-4048-VM running
>
> 4 s-4050-VM running
>
> 7 r-4092-VM running
>
>
>
> root@usa-nyc-kvm-001:~#
>
>
>
>
>
> If I check the log inside the router, I see this (the router is also
> responding very slow, waiting a lot to connect on it via virsh console):
>
>
>
> r-4092-VM login: root
>
> Password:
>
>
>
> Login timed out after 60 seconds.
>
>
>
>
>
> __?.o/ Apache CloudStack SystemVM 4.15.1
>
> ( )# https://cloudstack.apache.org
>
> (___(_) Debian GNU/Linux 10 r-4093-VM ttyS0
>
>
>
> r-4093-VM login: root
>
> Password:
>
> Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30)
> x86_64
>
>
>
>
>
>
>
> root@r-4093-VM:~# cat /var/log/cloud.log
>
> Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config
>
> Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping systemvm appliance
>
> Mon 13 Sep 2021 11:50:57 AM UTC Starting guest services for kvm
>
> Mon 13 Sep 2021 11:51:04 AM UTC acpiphp and pci_hotplug module already
> compiled in
>
> Mon 13 Sep 2021 11:52:59 AM UTC Received a new non-empty cmdline file
> from qemu-guest-agent
>
> Mon 13 Sep 2021 11:52:59 AM UTC Booting from cloudstack, remove old
> configuration files in /etc/cloudstack/
>
> Mon 13 Sep 2021 11:54:30 AM UTC Configuring systemvm type=vpcrouter
>
> Mon 13 Sep 2021 11:54:30 AM UTC Setting up VPC virtual router system
> vm
>
> Mon 13 Sep 2021 11:54:31 AM UTC Setting up apache web server for VPC
>
> Mon 13 Sep 2021 11:55:05 AM UTC Processors = 1 Enable service = 0
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: disable rp_filter
>
> Mon 13 Sep 2021 11:55:05 AM UTC disable rpfilter
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: enable_fwding = 1
>
> Mon 13 Sep 2021 11:55:05 AM UTC enable_fwding = 1
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: enabling passive FTP for guest
> VMs
>
> Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
>
> Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
>
> 2021-09-13 11:55:13,823 INFO update_config.py :: Processing incoming
> file => cmd_line.json
>
> 2021-09-13 11:55:13,856 INFO Processing JSON file cmd_line.json
>
> 2021-09-13 11:55:13,858 INFO Continuing with the processing of file
> '/var/cache/cloud/cmd_line.json'
>
> 2021-09-13 11:55:13,877 INFO Command of type cmdline received
>
> 2021-09-13 11:55:13,897 INFO Command of type ips received
>
> 2021-09-13 11:55:13,923 INFO Executing: ip addr show dev eth0
>
> 2021-09-13 11:55:13,975 INFO Executing: ip addr show dev eth0
>
> 2021-09-13 11:55:13,979 INFO Address found in DataBag ==>
> {u'public_ip':
> u'169.254.41.129', u'one_to_one_nat': False, u'nic_dev_id': u'0',
> u'network': u'169.254.0.0/16', u'netmask': u'255.255.0.0', u'source_nat':
> False, u'broadcast': u'169.254.255.255', u'add': True, u'nw_type':
> u'control', u'device': u'eth0', u'cidr': u'169.254.41.129/16', u'gateway':
> u'', u'size': u'16'}
>
> 2021-09-13 11:55:13,979 INFO Address 169.254.41.129/16 on device eth0
> already configured
>
> 2021-09-13 11:55:13,984 INFO Executing: ip addr show |grep -v secondary
>
> 2021-09-13 11:55:14,010 INFO Wrote edited file
> /etc/dnsmasq.d/cloud.conf
>
> 2021-09-13 11:55:14,010 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcphosts.txt
> file did not change
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The
> /var/lib/misc/dnsmasq.leases file did not change
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcpopts.txt
> file did not change
>
> 2021-09-13 11:55:14,010 INFO Attempting to delete entries from
> dnsmasq.leases file for VMs which are not on dhcphosts file
>
> 2021-09-13 11:55:14,010 ERROR Caught error while trying to delete
> entries
> from dnsmasq.leases file: [Errno 2] No such file or directory:
> '/etc/dhcphosts.txt'
>
> 2021-09-13 11:55:14,034 INFO Wrote edited file /etc/hosts
>
> 2021-09-13 11:55:14,044 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:14,044 INFO Updated hosts file
>
> 2021-09-13 11:55:14,046 INFO Executing: systemctl restart dnsmasq
>
> 2021-09-13 11:55:15,409 INFO Service dnsmasq restart
>
> 2021-09-13 11:55:15,453 INFO Executing: ip addr show |grep -v secondary
>
> 2021-09-13 11:55:15,832 INFO Nothing to commit. The
> /etc/dnsmasq.d/cloud.conf file did not change
>
> 2021-09-13 11:55:15,957 INFO Nothing to commit. The /etc/dhcphosts.txt
> file did not change
>
> 2021-09-13 11:55:15,963 INFO Nothing to commit. The
> /var/lib/misc/dnsmasq.leases file did not change
>
> 2021-09-13 11:55:15,963 INFO Nothing to commit. The /etc/dhcpopts.txt
> file did not change
>
> 2021-09-13 11:55:15,967 INFO Executing: systemctl is-active dnsmasq
>
> 2021-09-13 11:55:16,179 INFO Executing: systemctl reload dnsmasq
>
> 2021-09-13 11:55:16,665 INFO Service dnsmasq reload
>
> 2021-09-13 11:55:16,909 INFO Wrote edited file /etc/cron.d/process
>
> 2021-09-13 11:55:17,040 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:17,063 INFO Executing: iptables-save | grep
> '^:FW_EGRESS_RULES' || iptables -t filter -N FW_EGRESS_RULES
>
> 2021-09-13 11:55:17,323 INFO Executing: iptables-save | grep '^-A
> FW_EGRESS_RULES -j ACCEPT$' | sed 's/^-A/iptables -t filter -D/g' |
> bash
>
> 2021-09-13 11:55:17,697 INFO Executing: iptables -F FW_EGRESS_RULES
>
> 2021-09-13 11:55:17,990 INFO Executing: ipset -L | grep Name: | awk
> {'print $2'} | ipset flush
>
> 2021-09-13 11:55:19,272 INFO Executing: ipset -L | grep Name: | awk
> {'print $2'} | ipset destroy
>
> 2021-09-13 11:55:19,833 INFO Executing: iptables-save
>
> 2021-09-13 11:55:20,168 INFO Executing: iptables-save
>
> 2021-09-13 11:55:20,505 INFO Executing: ip6tables-save
>
> 2021-09-13 11:55:20,800 INFO Executing: systemctl stop conntrackd
>
> 2021-09-13 11:55:21,201 INFO Service conntrackd stop
>
> 2021-09-13 11:55:21,230 INFO Executing: systemctl stop keepalived
>
> 2021-09-13 11:55:21,578 INFO Service keepalived stop
>
> 2021-09-13 11:55:21,609 INFO Executing: mount
>
> root@r-4093-VM:~#
>
>
>
> Please ignore the router name, I did multiple tests to extract these
> logs, I always see the same.
>
>
>
>
>
> Regards,
>
> Cristian
>
>
>
>
>
>
--
Andrija Panić
Re: virtual router stuck on starting - cloudstack 4.15.1 with kvm
Posted by Andrija Panic <an...@gmail.com>.
Can you share qemu/libvirt version/flavour?
On Mon, 13 Sept 2021 at 14:18, <cr...@istream.today> wrote:
> Hello,
>
>
>
> I'm trying to understand this error, I created new network with VPC
> in
> a new zone where I added KVM hypervisor, and I have this issue.
>
>
>
>
>
> 2021-09-13 11:36:18,621 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Trying to connect to
> 169.254.52.234
>
> 2021-09-13 11:36:21,684 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-1:null) (logid:42d715f2) Could not connect to
> 169.254.52.234
>
>
>
> Here is the full log : https://pastebin.com/0NwsccUc
>
>
>
> root@usa-nyc-kvm-001:~# telnet 169.254.156.179 3922
>
> Trying 169.254.156.179...
>
> telnet: Unable to connect to remote host: Connection refused ( before
> connection refused, I get no route to host, for sure the nic is not up )
>
> root@usa-nyc-kvm-001:~# virsh list
>
> Id Name State
>
> ---------------------------
>
> 2 v-4048-VM running
>
> 4 s-4050-VM running
>
> 7 r-4092-VM running
>
>
>
> root@usa-nyc-kvm-001:~#
>
>
>
>
>
> If I check the log inside the router, I see this (the router is also
> responding very slow, waiting a lot to connect on it via virsh console):
>
>
>
> r-4092-VM login: root
>
> Password:
>
>
>
> Login timed out after 60 seconds.
>
>
>
>
>
> __?.o/ Apache CloudStack SystemVM 4.15.1
>
> ( )# https://cloudstack.apache.org
>
> (___(_) Debian GNU/Linux 10 r-4093-VM ttyS0
>
>
>
> r-4093-VM login: root
>
> Password:
>
> Linux r-4093-VM 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30)
> x86_64
>
>
>
>
>
>
>
> root@r-4093-VM:~# cat /var/log/cloud.log
>
> Mon 13 Sep 2021 11:50:55 AM UTC Executing cloud-early-config
>
> Mon 13 Sep 2021 11:50:55 AM UTC Bootstrapping systemvm appliance
>
> Mon 13 Sep 2021 11:50:57 AM UTC Starting guest services for kvm
>
> Mon 13 Sep 2021 11:51:04 AM UTC acpiphp and pci_hotplug module already
> compiled in
>
> Mon 13 Sep 2021 11:52:59 AM UTC Received a new non-empty cmdline file from
> qemu-guest-agent
>
> Mon 13 Sep 2021 11:52:59 AM UTC Booting from cloudstack, remove old
> configuration files in /etc/cloudstack/
>
> Mon 13 Sep 2021 11:54:30 AM UTC Configuring systemvm type=vpcrouter
>
> Mon 13 Sep 2021 11:54:30 AM UTC Setting up VPC virtual router system vm
>
> Mon 13 Sep 2021 11:54:31 AM UTC Setting up apache web server for VPC
>
> Mon 13 Sep 2021 11:55:05 AM UTC Processors = 1 Enable service = 0
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: disable rp_filter
>
> Mon 13 Sep 2021 11:55:05 AM UTC disable rpfilter
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: enable_fwding = 1
>
> Mon 13 Sep 2021 11:55:05 AM UTC enable_fwding = 1
>
> Mon 13 Sep 2021 11:55:05 AM UTC cloud: enabling passive FTP for guest VMs
>
> Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
>
> Mon 13 Sep 2021 11:55:06 AM UTC Finished setting up systemvm
>
> 2021-09-13 11:55:13,823 INFO update_config.py :: Processing incoming
> file => cmd_line.json
>
> 2021-09-13 11:55:13,856 INFO Processing JSON file cmd_line.json
>
> 2021-09-13 11:55:13,858 INFO Continuing with the processing of file
> '/var/cache/cloud/cmd_line.json'
>
> 2021-09-13 11:55:13,877 INFO Command of type cmdline received
>
> 2021-09-13 11:55:13,897 INFO Command of type ips received
>
> 2021-09-13 11:55:13,923 INFO Executing: ip addr show dev eth0
>
> 2021-09-13 11:55:13,975 INFO Executing: ip addr show dev eth0
>
> 2021-09-13 11:55:13,979 INFO Address found in DataBag ==>
> {u'public_ip':
> u'169.254.41.129', u'one_to_one_nat': False, u'nic_dev_id': u'0',
> u'network': u'169.254.0.0/16', u'netmask': u'255.255.0.0', u'source_nat':
> False, u'broadcast': u'169.254.255.255', u'add': True, u'nw_type':
> u'control', u'device': u'eth0', u'cidr': u'169.254.41.129/16', u'gateway':
> u'', u'size': u'16'}
>
> 2021-09-13 11:55:13,979 INFO Address 169.254.41.129/16 on device eth0
> already configured
>
> 2021-09-13 11:55:13,984 INFO Executing: ip addr show |grep -v secondary
>
> 2021-09-13 11:55:14,010 INFO Wrote edited file
> /etc/dnsmasq.d/cloud.conf
>
> 2021-09-13 11:55:14,010 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcphosts.txt
> file did not change
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The
> /var/lib/misc/dnsmasq.leases file did not change
>
> 2021-09-13 11:55:14,010 INFO Nothing to commit. The /etc/dhcpopts.txt
> file did not change
>
> 2021-09-13 11:55:14,010 INFO Attempting to delete entries from
> dnsmasq.leases file for VMs which are not on dhcphosts file
>
> 2021-09-13 11:55:14,010 ERROR Caught error while trying to delete
> entries
> from dnsmasq.leases file: [Errno 2] No such file or directory:
> '/etc/dhcphosts.txt'
>
> 2021-09-13 11:55:14,034 INFO Wrote edited file /etc/hosts
>
> 2021-09-13 11:55:14,044 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:14,044 INFO Updated hosts file
>
> 2021-09-13 11:55:14,046 INFO Executing: systemctl restart dnsmasq
>
> 2021-09-13 11:55:15,409 INFO Service dnsmasq restart
>
> 2021-09-13 11:55:15,453 INFO Executing: ip addr show |grep -v secondary
>
> 2021-09-13 11:55:15,832 INFO Nothing to commit. The
> /etc/dnsmasq.d/cloud.conf file did not change
>
> 2021-09-13 11:55:15,957 INFO Nothing to commit. The /etc/dhcphosts.txt
> file did not change
>
> 2021-09-13 11:55:15,963 INFO Nothing to commit. The
> /var/lib/misc/dnsmasq.leases file did not change
>
> 2021-09-13 11:55:15,963 INFO Nothing to commit. The /etc/dhcpopts.txt
> file did not change
>
> 2021-09-13 11:55:15,967 INFO Executing: systemctl is-active dnsmasq
>
> 2021-09-13 11:55:16,179 INFO Executing: systemctl reload dnsmasq
>
> 2021-09-13 11:55:16,665 INFO Service dnsmasq reload
>
> 2021-09-13 11:55:16,909 INFO Wrote edited file /etc/cron.d/process
>
> 2021-09-13 11:55:17,040 INFO Updated file in-cache configuration
>
> 2021-09-13 11:55:17,063 INFO Executing: iptables-save | grep
> '^:FW_EGRESS_RULES' || iptables -t filter -N FW_EGRESS_RULES
>
> 2021-09-13 11:55:17,323 INFO Executing: iptables-save | grep '^-A
> FW_EGRESS_RULES -j ACCEPT$' | sed 's/^-A/iptables -t filter -D/g' | bash
>
> 2021-09-13 11:55:17,697 INFO Executing: iptables -F FW_EGRESS_RULES
>
> 2021-09-13 11:55:17,990 INFO Executing: ipset -L | grep Name: | awk
> {'print $2'} | ipset flush
>
> 2021-09-13 11:55:19,272 INFO Executing: ipset -L | grep Name: | awk
> {'print $2'} | ipset destroy
>
> 2021-09-13 11:55:19,833 INFO Executing: iptables-save
>
> 2021-09-13 11:55:20,168 INFO Executing: iptables-save
>
> 2021-09-13 11:55:20,505 INFO Executing: ip6tables-save
>
> 2021-09-13 11:55:20,800 INFO Executing: systemctl stop conntrackd
>
> 2021-09-13 11:55:21,201 INFO Service conntrackd stop
>
> 2021-09-13 11:55:21,230 INFO Executing: systemctl stop keepalived
>
> 2021-09-13 11:55:21,578 INFO Service keepalived stop
>
> 2021-09-13 11:55:21,609 INFO Executing: mount
>
> root@r-4093-VM:~#
>
>
>
> Please ignore the router name, I did multiple tests to extract these logs,
> I
> always see the same.
>
>
>
>
>
> Regards,
>
> Cristian
>
>
>
>
>
>
--
Andrija Panić