You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Cloud List <cl...@sg.or.id> on 2020/12/26 04:12:09 UTC

SSVM and CPVM agent unable to start after console proxy SSL certificate update

Hi,

Merry Christmas to all.

We are using Cloudstack with KVM hypervisor. Since our console proxy SSL
certificate has expired, we updated our new SSL certificate using below
method:

http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy

We have done the above method in the past years without any issues, however
this time round, both the SSVM and CPVM agents are not able to start after
the update.

The state for both VMs are up but agents are in "disconnected" state. We
are still able to login to the SSVM, and found out that the cloud service
is not running.

root@s-4200-VM:~# service cloud status
CloudStack cloud service is not running

Tried to start the service:

root@s-4200-VM:~# service cloud start
Starting CloudStack cloud service (type=secstorage) Success

But the service is not started:

root@s-4200-VM:~# service cloud status
CloudStack cloud service is not running

Below is the logs from /var/log/cloud.log:

=====
Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config
Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest
Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit
the loop and proceed with configuration.
Sat Dec 26 03:45:04 UTC 2020 Patching  cloud service
Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml
Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm
Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP
Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip
timer=0
Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP
Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP
Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP
Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with
STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR:
Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to 10.19.16.1
Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server
Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of
volume/template
Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file
/etc/apache2/sites-available/default-ssl
Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file:
/etc/apache2/sites-available/default-ssl
Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter
Sat Dec 26 03:45:21 UTC 2020 disable rpfilter
Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0
Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0
Sat Dec 26 03:45:21 UTC 2020 Processors = 1  Enable service  = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1
=====

Result of /usr/local/cloud/systemvm/ssvm-check.sh:

=====
root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is  8.8.8.8
PING 8.8.8.8 (8.8.8.8): 48 data bytes
56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms
56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms
Good: Can ping DNS server
================================================
Good: DNS resolves download.cloud.com
================================================
ERROR: NFS is not currently mounted
Try manually mounting from inside the VM
NFS server is  X.X.201.1
PING X.X.201.1 (X.X.201.1): 48 data bytes
56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms
56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms
--- X.X.201.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms
Good: Can ping nfs server
================================================
Management server is 10.237.3.8. Checking connectivity.
Good: Can connect to management server port 8250
================================================
ERROR: Java process not running.  Try restarting the SSVM.
root@s-4200-VM:/var/log#
=====

The result is OK except the NFS test, but we checked the IP address is not
correct (X.X.201.1 which is the public IP address of the gateway rather
than the actual NFS server IP). We tested mounting to the actual NFS server
and it works fine.

Have tried stopping and starting back the SSVM and the issue still persists.

Anyone can help to advice how we can resolve the problem?

Looking forward to your reply, thank you.

-ip-

<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
www.avg.com
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: SSVM and CPVM agent unable to start after console proxy SSL certificate update

Posted by Andrija Panic <an...@gmail.com>.
The issue, most probably, is due to different SSL provider or different
names used for the certificates - I've seen this in past.

I would *strongly* suggest, removing all relevant records from the
cloud.keystore table (all records related to the domain you are using -
that probably means indeed ALL records from the table...)
Then upload the SSL and it's intermediate/Root certificates again, i.e.
from scratch.
restart mgmt, and ensure SSVM/CPVM are destroyed

Best,



On Mon, 28 Dec 2020 at 11:43, Rohit Yadav <ro...@shapeblue.com> wrote:

> Hi,
>
> Can you try to manually start the cloud service, for example: "service
> cloud start" and tail/share the logs which may explain why the java process
> is not running.
> If that does not work, you may also try to validate/verify the
> certificates (including any chain/intermediate certificates) you've
> uploaded and destroy the old CPVM/SSVM.
>
> For more information on SSL certificate setup, you may read this
> 4.11-specific blog
> https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/ which
> I think is applicable for 4.9 as well.
>
>
> Regards.
>
> ________________________________
> From: Cloud List <cl...@sg.or.id>
> Sent: Saturday, December 26, 2020 09:42
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>; dev <
> dev@cloudstack.apache.org>
> Subject: SSVM and CPVM agent unable to start after console proxy SSL
> certificate update
>
> Hi,
>
> Merry Christmas to all.
>
> We are using Cloudstack with KVM hypervisor. Since our console proxy SSL
> certificate has expired, we updated our new SSL certificate using below
> method:
>
>
> http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy
>
> We have done the above method in the past years without any issues, however
> this time round, both the SSVM and CPVM agents are not able to start after
> the update.
>
> The state for both VMs are up but agents are in "disconnected" state. We
> are still able to login to the SSVM, and found out that the cloud service
> is not running.
>
> root@s-4200-VM:~# service cloud status
> CloudStack cloud service is not running
>
> Tried to start the service:
>
> root@s-4200-VM:~# service cloud start
> Starting CloudStack cloud service (type=secstorage) Success
>
> But the service is not started:
>
> root@s-4200-VM:~# service cloud status
> CloudStack cloud service is not running
>
> Below is the logs from /var/log/cloud.log:
>
> =====
> Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config
> Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest
> Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit
> the loop and proceed with configuration.
> Sat Dec 26 03:45:04 UTC 2020 Patching  cloud service
> Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml
> Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm
> Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP
> Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip
> timer=0
> Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP
> Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP
> Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP
> Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with
> STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR:
> Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to
> 10.19.16.1
> Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server
> Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of
> volume/template
> Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file
> /etc/apache2/sites-available/default-ssl
> Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file:
> /etc/apache2/sites-available/default-ssl
> Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter
> Sat Dec 26 03:45:21 UTC 2020 disable rpfilter
> Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0
> Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0
> Sat Dec 26 03:45:21 UTC 2020 Processors = 1  Enable service  = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1
> =====
>
> Result of /usr/local/cloud/systemvm/ssvm-check.sh:
>
> =====
> root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh
> ================================================
> First DNS server is  8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms
> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms
> --- 8.8.8.8 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms
> Good: Can ping DNS server
> ================================================
> Good: DNS resolves download.cloud.com
> ================================================
> ERROR: NFS is not currently mounted
> Try manually mounting from inside the VM
> NFS server is  X.X.201.1
> PING X.X.201.1 (X.X.201.1): 48 data bytes
> 56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms
> 56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms
> --- X.X.201.1 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms
> Good: Can ping nfs server
> ================================================
> Management server is 10.237.3.8. Checking connectivity.
> Good: Can connect to management server port 8250
> ================================================
> ERROR: Java process not running.  Try restarting the SSVM.
> root@s-4200-VM:/var/log#
> =====
>
> The result is OK except the NFS test, but we checked the IP address is not
> correct (X.X.201.1 which is the public IP address of the gateway rather
> than the actual NFS server IP). We tested mounting to the actual NFS server
> and it works fine.
>
> Have tried stopping and starting back the SSVM and the issue still
> persists.
>
> Anyone can help to advice how we can resolve the problem?
>
> Looking forward to your reply, thank you.
>
> -ip-
>
> <
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> >
> Virus-free.
> www.avg.com<http://www.avg.com>
> <
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> >
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> @shapeblue
>
>
>
>

-- 

Andrija Panić

Re: SSVM and CPVM agent unable to start after console proxy SSL certificate update

Posted by Andrija Panic <an...@gmail.com>.
The issue, most probably, is due to different SSL provider or different
names used for the certificates - I've seen this in past.

I would *strongly* suggest, removing all relevant records from the
cloud.keystore table (all records related to the domain you are using -
that probably means indeed ALL records from the table...)
Then upload the SSL and it's intermediate/Root certificates again, i.e.
from scratch.
restart mgmt, and ensure SSVM/CPVM are destroyed

Best,



On Mon, 28 Dec 2020 at 11:43, Rohit Yadav <ro...@shapeblue.com> wrote:

> Hi,
>
> Can you try to manually start the cloud service, for example: "service
> cloud start" and tail/share the logs which may explain why the java process
> is not running.
> If that does not work, you may also try to validate/verify the
> certificates (including any chain/intermediate certificates) you've
> uploaded and destroy the old CPVM/SSVM.
>
> For more information on SSL certificate setup, you may read this
> 4.11-specific blog
> https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/ which
> I think is applicable for 4.9 as well.
>
>
> Regards.
>
> ________________________________
> From: Cloud List <cl...@sg.or.id>
> Sent: Saturday, December 26, 2020 09:42
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>; dev <
> dev@cloudstack.apache.org>
> Subject: SSVM and CPVM agent unable to start after console proxy SSL
> certificate update
>
> Hi,
>
> Merry Christmas to all.
>
> We are using Cloudstack with KVM hypervisor. Since our console proxy SSL
> certificate has expired, we updated our new SSL certificate using below
> method:
>
>
> http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy
>
> We have done the above method in the past years without any issues, however
> this time round, both the SSVM and CPVM agents are not able to start after
> the update.
>
> The state for both VMs are up but agents are in "disconnected" state. We
> are still able to login to the SSVM, and found out that the cloud service
> is not running.
>
> root@s-4200-VM:~# service cloud status
> CloudStack cloud service is not running
>
> Tried to start the service:
>
> root@s-4200-VM:~# service cloud start
> Starting CloudStack cloud service (type=secstorage) Success
>
> But the service is not started:
>
> root@s-4200-VM:~# service cloud status
> CloudStack cloud service is not running
>
> Below is the logs from /var/log/cloud.log:
>
> =====
> Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config
> Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest
> Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit
> the loop and proceed with configuration.
> Sat Dec 26 03:45:04 UTC 2020 Patching  cloud service
> Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml
> Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm
> Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP
> Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip
> timer=0
> Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP
> Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP
> Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP
> Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with
> STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR:
> Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to
> 10.19.16.1
> Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server
> Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of
> volume/template
> Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file
> /etc/apache2/sites-available/default-ssl
> Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file:
> /etc/apache2/sites-available/default-ssl
> Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter
> Sat Dec 26 03:45:21 UTC 2020 disable rpfilter
> Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0
> Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0
> Sat Dec 26 03:45:21 UTC 2020 Processors = 1  Enable service  = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1
> =====
>
> Result of /usr/local/cloud/systemvm/ssvm-check.sh:
>
> =====
> root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh
> ================================================
> First DNS server is  8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms
> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms
> --- 8.8.8.8 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms
> Good: Can ping DNS server
> ================================================
> Good: DNS resolves download.cloud.com
> ================================================
> ERROR: NFS is not currently mounted
> Try manually mounting from inside the VM
> NFS server is  X.X.201.1
> PING X.X.201.1 (X.X.201.1): 48 data bytes
> 56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms
> 56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms
> --- X.X.201.1 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms
> Good: Can ping nfs server
> ================================================
> Management server is 10.237.3.8. Checking connectivity.
> Good: Can connect to management server port 8250
> ================================================
> ERROR: Java process not running.  Try restarting the SSVM.
> root@s-4200-VM:/var/log#
> =====
>
> The result is OK except the NFS test, but we checked the IP address is not
> correct (X.X.201.1 which is the public IP address of the gateway rather
> than the actual NFS server IP). We tested mounting to the actual NFS server
> and it works fine.
>
> Have tried stopping and starting back the SSVM and the issue still
> persists.
>
> Anyone can help to advice how we can resolve the problem?
>
> Looking forward to your reply, thank you.
>
> -ip-
>
> <
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> >
> Virus-free.
> www.avg.com<http://www.avg.com>
> <
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> >
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
> @shapeblue
>
>
>
>

-- 

Andrija Panić

Re: SSVM and CPVM agent unable to start after console proxy SSL certificate update

Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi,

Can you try to manually start the cloud service, for example: "service cloud start" and tail/share the logs which may explain why the java process is not running.
If that does not work, you may also try to validate/verify the certificates (including any chain/intermediate certificates) you've uploaded and destroy the old CPVM/SSVM.

For more information on SSL certificate setup, you may read this 4.11-specific blog https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/ which I think is applicable for 4.9 as well.


Regards.

________________________________
From: Cloud List <cl...@sg.or.id>
Sent: Saturday, December 26, 2020 09:42
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>; dev <de...@cloudstack.apache.org>
Subject: SSVM and CPVM agent unable to start after console proxy SSL certificate update

Hi,

Merry Christmas to all.

We are using Cloudstack with KVM hypervisor. Since our console proxy SSL
certificate has expired, we updated our new SSL certificate using below
method:

http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy

We have done the above method in the past years without any issues, however
this time round, both the SSVM and CPVM agents are not able to start after
the update.

The state for both VMs are up but agents are in "disconnected" state. We
are still able to login to the SSVM, and found out that the cloud service
is not running.

root@s-4200-VM:~# service cloud status
CloudStack cloud service is not running

Tried to start the service:

root@s-4200-VM:~# service cloud start
Starting CloudStack cloud service (type=secstorage) Success

But the service is not started:

root@s-4200-VM:~# service cloud status
CloudStack cloud service is not running

Below is the logs from /var/log/cloud.log:

=====
Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config
Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest
Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit
the loop and proceed with configuration.
Sat Dec 26 03:45:04 UTC 2020 Patching  cloud service
Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml
Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm
Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP
Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip
timer=0
Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP
Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP
Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP
Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with
STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR:
Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to 10.19.16.1
Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server
Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of
volume/template
Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file
/etc/apache2/sites-available/default-ssl
Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file:
/etc/apache2/sites-available/default-ssl
Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter
Sat Dec 26 03:45:21 UTC 2020 disable rpfilter
Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0
Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0
Sat Dec 26 03:45:21 UTC 2020 Processors = 1  Enable service  = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1
=====

Result of /usr/local/cloud/systemvm/ssvm-check.sh:

=====
root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is  8.8.8.8
PING 8.8.8.8 (8.8.8.8): 48 data bytes
56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms
56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms
Good: Can ping DNS server
================================================
Good: DNS resolves download.cloud.com
================================================
ERROR: NFS is not currently mounted
Try manually mounting from inside the VM
NFS server is  X.X.201.1
PING X.X.201.1 (X.X.201.1): 48 data bytes
56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms
56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms
--- X.X.201.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms
Good: Can ping nfs server
================================================
Management server is 10.237.3.8. Checking connectivity.
Good: Can connect to management server port 8250
================================================
ERROR: Java process not running.  Try restarting the SSVM.
root@s-4200-VM:/var/log#
=====

The result is OK except the NFS test, but we checked the IP address is not
correct (X.X.201.1 which is the public IP address of the gateway rather
than the actual NFS server IP). We tested mounting to the actual NFS server
and it works fine.

Have tried stopping and starting back the SSVM and the issue still persists.

Anyone can help to advice how we can resolve the problem?

Looking forward to your reply, thank you.

-ip-

<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
www.avg.com<http://www.avg.com>
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

rohit.yadav@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue
  
 


Re: SSVM and CPVM agent unable to start after console proxy SSL certificate update

Posted by Wei ZHOU <us...@gmail.com>.
Hi,

Can you create an issue on github ?
https://github.com/apache/cloudstack/issues

-Wei

On Sat, 26 Dec 2020 at 05:12, Cloud List <cl...@sg.or.id> wrote:

> Hi,
>
> Merry Christmas to all.
>
> We are using Cloudstack with KVM hypervisor. Since our console proxy SSL
> certificate has expired, we updated our new SSL certificate using below
> method:
>
>
> http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy
>
> We have done the above method in the past years without any issues, however
> this time round, both the SSVM and CPVM agents are not able to start after
> the update.
>
> The state for both VMs are up but agents are in "disconnected" state. We
> are still able to login to the SSVM, and found out that the cloud service
> is not running.
>
> root@s-4200-VM:~# service cloud status
> CloudStack cloud service is not running
>
> Tried to start the service:
>
> root@s-4200-VM:~# service cloud start
> Starting CloudStack cloud service (type=secstorage) Success
>
> But the service is not started:
>
> root@s-4200-VM:~# service cloud status
> CloudStack cloud service is not running
>
> Below is the logs from /var/log/cloud.log:
>
> =====
> Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config
> Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest
> Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit
> the loop and proceed with configuration.
> Sat Dec 26 03:45:04 UTC 2020 Patching  cloud service
> Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml
> Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm
> Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP
> Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip
> timer=0
> Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP
> Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP
> Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP
> Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with
> STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR:
> Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to
> 10.19.16.1
> Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server
> Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of
> volume/template
> Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file
> /etc/apache2/sites-available/default-ssl
> Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file:
> /etc/apache2/sites-available/default-ssl
> Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter
> Sat Dec 26 03:45:21 UTC 2020 disable rpfilter
> Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0
> Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0
> Sat Dec 26 03:45:21 UTC 2020 Processors = 1  Enable service  = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1
> =====
>
> Result of /usr/local/cloud/systemvm/ssvm-check.sh:
>
> =====
> root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh
> ================================================
> First DNS server is  8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms
> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms
> --- 8.8.8.8 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms
> Good: Can ping DNS server
> ================================================
> Good: DNS resolves download.cloud.com
> ================================================
> ERROR: NFS is not currently mounted
> Try manually mounting from inside the VM
> NFS server is  X.X.201.1
> PING X.X.201.1 (X.X.201.1): 48 data bytes
> 56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms
> 56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms
> --- X.X.201.1 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms
> Good: Can ping nfs server
> ================================================
> Management server is 10.237.3.8. Checking connectivity.
> Good: Can connect to management server port 8250
> ================================================
> ERROR: Java process not running.  Try restarting the SSVM.
> root@s-4200-VM:/var/log#
> =====
>
> The result is OK except the NFS test, but we checked the IP address is not
> correct (X.X.201.1 which is the public IP address of the gateway rather
> than the actual NFS server IP). We tested mounting to the actual NFS server
> and it works fine.
>
> Have tried stopping and starting back the SSVM and the issue still
> persists.
>
> Anyone can help to advice how we can resolve the problem?
>
> Looking forward to your reply, thank you.
>
> -ip-
>
> <
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> >
> Virus-free.
> www.avg.com
> <
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail
> >
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>

Re: SSVM and CPVM agent unable to start after console proxy SSL certificate update

Posted by Rakesh v <ww...@gmail.com>.
Probably try destroying them once? Any warn or error message in mgt logs or ssvm /var/log/messages?

Sent from my iPhone

> On 26-Dec-2020, at 5:12 AM, Cloud List <cl...@sg.or.id> wrote:
> 
> Hi,
> 
> Merry Christmas to all.
> 
> We are using Cloudstack with KVM hypervisor. Since our console proxy SSL
> certificate has expired, we updated our new SSL certificate using below
> method:
> 
> http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy
> 
> We have done the above method in the past years without any issues, however
> this time round, both the SSVM and CPVM agents are not able to start after
> the update.
> 
> The state for both VMs are up but agents are in "disconnected" state. We
> are still able to login to the SSVM, and found out that the cloud service
> is not running.
> 
> root@s-4200-VM:~# service cloud status
> CloudStack cloud service is not running
> 
> Tried to start the service:
> 
> root@s-4200-VM:~# service cloud start
> Starting CloudStack cloud service (type=secstorage) Success
> 
> But the service is not started:
> 
> root@s-4200-VM:~# service cloud status
> CloudStack cloud service is not running
> 
> Below is the logs from /var/log/cloud.log:
> 
> =====
> Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config
> Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest
> Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit
> the loop and proceed with configuration.
> Sat Dec 26 03:45:04 UTC 2020 Patching  cloud service
> Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml
> Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm
> Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP
> Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip
> timer=0
> Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP
> Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP
> Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP
> Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with
> STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR:
> Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to 10.19.16.1
> Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server
> Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of
> volume/template
> Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file
> /etc/apache2/sites-available/default-ssl
> Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file:
> /etc/apache2/sites-available/default-ssl
> Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter
> Sat Dec 26 03:45:21 UTC 2020 disable rpfilter
> Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0
> Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0
> Sat Dec 26 03:45:21 UTC 2020 Processors = 1  Enable service  = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1
> =====
> 
> Result of /usr/local/cloud/systemvm/ssvm-check.sh:
> 
> =====
> root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh
> ================================================
> First DNS server is  8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms
> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms
> --- 8.8.8.8 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms
> Good: Can ping DNS server
> ================================================
> Good: DNS resolves download.cloud.com
> ================================================
> ERROR: NFS is not currently mounted
> Try manually mounting from inside the VM
> NFS server is  X.X.201.1
> PING X.X.201.1 (X.X.201.1): 48 data bytes
> 56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms
> 56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms
> --- X.X.201.1 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms
> Good: Can ping nfs server
> ================================================
> Management server is 10.237.3.8. Checking connectivity.
> Good: Can connect to management server port 8250
> ================================================
> ERROR: Java process not running.  Try restarting the SSVM.
> root@s-4200-VM:/var/log#
> =====
> 
> The result is OK except the NFS test, but we checked the IP address is not
> correct (X.X.201.1 which is the public IP address of the gateway rather
> than the actual NFS server IP). We tested mounting to the actual NFS server
> and it works fine.
> 
> Have tried stopping and starting back the SSVM and the issue still persists.
> 
> Anyone can help to advice how we can resolve the problem?
> 
> Looking forward to your reply, thank you.
> 
> -ip-
> 
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: SSVM and CPVM agent unable to start after console proxy SSL certificate update

Posted by Rakesh v <ww...@gmail.com>.
Probably try destroying them once? Any warn or error message in mgt logs or ssvm /var/log/messages?

Sent from my iPhone

> On 26-Dec-2020, at 5:12 AM, Cloud List <cl...@sg.or.id> wrote:
> 
> Hi,
> 
> Merry Christmas to all.
> 
> We are using Cloudstack with KVM hypervisor. Since our console proxy SSL
> certificate has expired, we updated our new SSL certificate using below
> method:
> 
> http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy
> 
> We have done the above method in the past years without any issues, however
> this time round, both the SSVM and CPVM agents are not able to start after
> the update.
> 
> The state for both VMs are up but agents are in "disconnected" state. We
> are still able to login to the SSVM, and found out that the cloud service
> is not running.
> 
> root@s-4200-VM:~# service cloud status
> CloudStack cloud service is not running
> 
> Tried to start the service:
> 
> root@s-4200-VM:~# service cloud start
> Starting CloudStack cloud service (type=secstorage) Success
> 
> But the service is not started:
> 
> root@s-4200-VM:~# service cloud status
> CloudStack cloud service is not running
> 
> Below is the logs from /var/log/cloud.log:
> 
> =====
> Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config
> Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest
> Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit
> the loop and proceed with configuration.
> Sat Dec 26 03:45:04 UTC 2020 Patching  cloud service
> Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml
> Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm
> Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP
> Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip
> timer=0
> Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP
> Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP
> Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP
> Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with
> STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR:
> Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to 10.19.16.1
> Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server
> Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of
> volume/template
> Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file
> /etc/apache2/sites-available/default-ssl
> Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file:
> /etc/apache2/sites-available/default-ssl
> Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter
> Sat Dec 26 03:45:21 UTC 2020 disable rpfilter
> Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0
> Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0
> Sat Dec 26 03:45:21 UTC 2020 Processors = 1  Enable service  = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0
> Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1
> =====
> 
> Result of /usr/local/cloud/systemvm/ssvm-check.sh:
> 
> =====
> root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh
> ================================================
> First DNS server is  8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 48 data bytes
> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms
> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms
> --- 8.8.8.8 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms
> Good: Can ping DNS server
> ================================================
> Good: DNS resolves download.cloud.com
> ================================================
> ERROR: NFS is not currently mounted
> Try manually mounting from inside the VM
> NFS server is  X.X.201.1
> PING X.X.201.1 (X.X.201.1): 48 data bytes
> 56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms
> 56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms
> --- X.X.201.1 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms
> Good: Can ping nfs server
> ================================================
> Management server is 10.237.3.8. Checking connectivity.
> Good: Can connect to management server port 8250
> ================================================
> ERROR: Java process not running.  Try restarting the SSVM.
> root@s-4200-VM:/var/log#
> =====
> 
> The result is OK except the NFS test, but we checked the IP address is not
> correct (X.X.201.1 which is the public IP address of the gateway rather
> than the actual NFS server IP). We tested mounting to the actual NFS server
> and it works fine.
> 
> Have tried stopping and starting back the SSVM and the issue still persists.
> 
> Anyone can help to advice how we can resolve the problem?
> 
> Looking forward to your reply, thank you.
> 
> -ip-
> 
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> Virus-free.
> www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: SSVM and CPVM agent unable to start after console proxy SSL certificate update

Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi,

Can you try to manually start the cloud service, for example: "service cloud start" and tail/share the logs which may explain why the java process is not running.
If that does not work, you may also try to validate/verify the certificates (including any chain/intermediate certificates) you've uploaded and destroy the old CPVM/SSVM.

For more information on SSL certificate setup, you may read this 4.11-specific blog https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/ which I think is applicable for 4.9 as well.


Regards.

________________________________
From: Cloud List <cl...@sg.or.id>
Sent: Saturday, December 26, 2020 09:42
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>; dev <de...@cloudstack.apache.org>
Subject: SSVM and CPVM agent unable to start after console proxy SSL certificate update

Hi,

Merry Christmas to all.

We are using Cloudstack with KVM hypervisor. Since our console proxy SSL
certificate has expired, we updated our new SSL certificate using below
method:

http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/systemvm.html#using-a-ssl-certificate-for-the-console-proxy

We have done the above method in the past years without any issues, however
this time round, both the SSVM and CPVM agents are not able to start after
the update.

The state for both VMs are up but agents are in "disconnected" state. We
are still able to login to the SSVM, and found out that the cloud service
is not running.

root@s-4200-VM:~# service cloud status
CloudStack cloud service is not running

Tried to start the service:

root@s-4200-VM:~# service cloud start
Starting CloudStack cloud service (type=secstorage) Success

But the service is not started:

root@s-4200-VM:~# service cloud status
CloudStack cloud service is not running

Below is the logs from /var/log/cloud.log:

=====
Sat Dec 26 03:45:04 UTC 2020 Executing cloud-early-config
Sat Dec 26 03:45:04 UTC 2020 Detected that we are running inside kvm guest
Sat Dec 26 03:45:04 UTC 2020 Found a non empty cmdline file. Will now exit
the loop and proceed with configuration.
Sat Dec 26 03:45:04 UTC 2020 Patching  cloud service
Sat Dec 26 03:45:10 UTC 2020 Updating log4j-cloud.xml
Sat Dec 26 03:45:10 UTC 2020 Setting up secondary storage system vm
Sat Dec 26 03:45:10 UTC 2020 checking that eth0 has IP
Sat Dec 26 03:45:11 UTC 2020 waiting for eth0 interface setup with ip
timer=0
Sat Dec 26 03:45:11 UTC 2020 checking that eth1 has IP
Sat Dec 26 03:45:11 UTC 2020 checking that eth2 has IP
Sat Dec 26 03:45:20 UTC 2020 checking that eth3 has IP
Sat Dec 26 03:45:20 UTC 2020 Successfully setup storage network with
STORAGE_IP:10.19.22.67, STORAGE_NETMASK:255.255.240.0, STORAGE_CIDR:
Sat Dec 26 03:45:20 UTC 2020 Setting up route of RFC1918 space to 10.19.16.1
Sat Dec 26 03:45:20 UTC 2020 Setting up apache web server
Sat Dec 26 03:45:20 UTC 2020 setting up apache2 for post upload of
volume/template
Sat Dec 26 03:45:20 UTC 2020 rewrite rules already exist in file
/etc/apache2/sites-available/default-ssl
Sat Dec 26 03:45:20 UTC 2020 adding cors rules to file:
/etc/apache2/sites-available/default-ssl
Sat Dec 26 03:45:21 UTC 2020 cloud: disable rp_filter
Sat Dec 26 03:45:21 UTC 2020 disable rpfilter
Sat Dec 26 03:45:21 UTC 2020 cloud: enable_fwding = 0
Sat Dec 26 03:45:21 UTC 2020 enable_fwding = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service haproxy = 0
Sat Dec 26 03:45:21 UTC 2020 Processors = 1  Enable service  = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service dnsmasq = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service cloud-passwd-srvr = 0
Sat Dec 26 03:45:21 UTC 2020 Enable service cloud = 1
=====

Result of /usr/local/cloud/systemvm/ssvm-check.sh:

=====
root@s-4200-VM:/var/log# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is  8.8.8.8
PING 8.8.8.8 (8.8.8.8): 48 data bytes
56 bytes from 8.8.8.8: icmp_seq=0 ttl=122 time=0.531 ms
56 bytes from 8.8.8.8: icmp_seq=1 ttl=122 time=0.676 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.531/0.604/0.676/0.073 ms
Good: Can ping DNS server
================================================
Good: DNS resolves download.cloud.com
================================================
ERROR: NFS is not currently mounted
Try manually mounting from inside the VM
NFS server is  X.X.201.1
PING X.X.201.1 (X.X.201.1): 48 data bytes
56 bytes from X.X.201.1: icmp_seq=0 ttl=255 time=0.463 ms
56 bytes from X.X.201.1: icmp_seq=1 ttl=255 time=0.482 ms
--- X.X.201.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.463/0.473/0.482/0.000 ms
Good: Can ping nfs server
================================================
Management server is 10.237.3.8. Checking connectivity.
Good: Can connect to management server port 8250
================================================
ERROR: Java process not running.  Try restarting the SSVM.
root@s-4200-VM:/var/log#
=====

The result is OK except the NFS test, but we checked the IP address is not
correct (X.X.201.1 which is the public IP address of the gateway rather
than the actual NFS server IP). We tested mounting to the actual NFS server
and it works fine.

Have tried stopping and starting back the SSVM and the issue still persists.

Anyone can help to advice how we can resolve the problem?

Looking forward to your reply, thank you.

-ip-

<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
www.avg.com<http://www.avg.com>
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

rohit.yadav@shapeblue.com 
www.shapeblue.com
3 London Bridge Street,  3rd floor, News Building, London  SE1 9SGUK
@shapeblue